Step 4 : Deploy your app on dev cluster with kubernetes

Note

We consider that you have tested your application with docker compose according to instructions described in Step 3

Step 4.1: Define a deploy token (Optionnal)

Only if your GitLab project’s visibility is private or internal, you need to create deploy token.

As your goal is to deploy your app in kubernetes, you need to let kubernetes access to the private gitlab docker registry of your project. You achieve that by creating a deploy token, and providing it to the continuous integration (CI). A deploy token is a pair of values:

• username: username in the HTTP authentication framework. The default username format is gitlab+deploy-token-{n}. You can specify a custom username when you create the deploy token.

• token: password in the HTTP authentication framework.

Note

More details on gitlab documentation

4.1.1 Check the visibility of your GitLab project

After creation, you can view/modify the visibility of your gitlab project in Settings –> General –> Visibility, project features, permissions section in gitlab page.

4.1.2 Create a token that can read the registry

Here, we need to define a token with only read_registry scope which correspond to Read-only access to the images in the project’s container registry.

To create such token:

• From your Gitlab project page, go to Settings - Repository –> https://gitlab.pasteur.fr/ada-team/my-project/-/settings/repository

• Expand “Deploy tokens” section.

• Set a Name for your token. For example, name it read_registry, easier to keep a track on what it can do

• Add a Username or let default valule

• Chose the scope read_registry

• Apply your chose by clinking on Create deploy token button

Here an example with default values:

Immediately after clicking on the Create deploy token button, your token is created with a password.

See an example below :

Warning

Save the token password you just created !! You need it for add it on CI/CD variables

To see created tokens, you can go again to the “Deploy tokens” section of your Settings/Repository. You can find the list of active deploy token at the end of the section, like below :

4.1.3 Provide the token to the CI

The deploy token is stored in two distinct variables :

• the username will be contained in the dedicated variable DEPLOY_USER

• the password will be store in the dedicated variable DEPLOY_TOKEN

To create the CI variables :

• From your Gitlab project page, go to Settings - CI/CD –> https://gitlab.pasteur.fr/ada-team/my-project/-/settings/ci_cd

• Expand “Variables” section

• Add two variables named DEPLOY_USER and DEPLOY_TOKEN, keep the flags, type, … to their default values

• Apply your chose by clinking on Add variable button

With the previous values defined during the creation of deploy token, the two variables will be like following :

• for DEPLOY_USER :

• for DEPLOY_TOKEN :

You can retrieve (also modify by cliking on the pen icon) the token variables at the end of Settings - CI/CD - Variables section :

Tip

This section follows the IT departement documentation available here “Requesting a namespace”.

Step 4.2: Ask a namespace in the kubernetes dev cluster

Now you have you project hosted in GitLab (see Step 1), your shiny app is containerized in the Docker image (see Step 2) and you have tested it using the given example locally on your computer (see Step 3). The next step is to deploy your shiny app in the kubernetes dev cluster, also named dev cluster.

In the dev cluster, you can test your shiny app at each step of development as it will later be deployed in production. The urls to access to applications are shaped as https://<anything>.dev.pasteur.cloud. These urls, and the shiny app, are only accessible within the Institut Pasteur internet network or outsite Pasteur with the VPN.

Like compute cluster, when you use Kubernetes cluster, you need to have a space with specific ressources dedicated to your app.

In the later part of this page we talk about pod, which can be approximated here to a container, a running docker image.

4.2.1 : Estimate memory (CPU, RAM, storage) resources you need

Ressources should be estimated carefully. While asking and using for too few CPU will only slow down your application, asking and configuring too few RAM will make your application unusable.

Tip

During development, you can request more resources than what was requested at the beginning.

How to observe my memory usage locally

To observe the memory consumption of your application, build it, start it, monitor it, and then use your application as you expect it will be used. Use it multiple time in a row as shiny will keep some variable cached in memory.

# Clean up all docker running on your computer
docker stop $(docker ps -aq)

# Start you application
docker-compose up --build -d

# Start watching
while true; do
    docker stats --no-stream | grep -v "<CONTAINER ID OR NAME>" | tee -a shinylogs/stats.txt; sleep 1;
done

In the following example, we used the application, reloaded the page and re used it. In an actual use case, you should use it more than twice in order to have an acurate estimation of the memory consumption.

fca3d08757e1   rshiny-k8s-example-r_shiny-server_1   0.00%     35.47MiB / 31.09GiB   ...
fca3d08757e1   rshiny-k8s-example-r_shiny-server_1   17.30%    430.63MiB / 31.09GiB   ...
fca3d08757e1   rshiny-k8s-example-r_shiny-server_1   13.87%    920.30MiB / 31.09GiB   ...
fca3d08757e1   rshiny-k8s-example-r_shiny-server_1   44.62%    1.03GiB / 31.09GiB    ...
fca3d08757e1   rshiny-k8s-example-r_shiny-server_1   0.24%     230.99MiB / 31.09GiB   ...
fca3d08757e1   rshiny-k8s-example-r_shiny-server_1   22.60%    850.08MiB / 31.09GiB   ...
fca3d08757e1   rshiny-k8s-example-r_shiny-server_1   51.20%    1.31GiB / 31.09GiB    ...

The memory usage peaked up to 1.31Gi, maybe the memory went higher between two measures, maybe using it more than two times would have use more memory. Defining the limit to 2Gi seems thus resonable.

Limits and requests for memory are measured in bytes. You can express memory as a plain integer or as a fixed-point number suffixed by G, M, k, respectivly GigaBytes, MegaBytes and KilaBytes. You can also use the power-of-two equivalents: Gi, Mi, Ki.

Tip

For R developer, you can also use the information displayed in R Studio IDE.

How to estimate quota needs from your app ressources

In the following we will estimate a multiplication factor that have to be applied to your request and limits of memory of your application.

In the previous example, the memory peaked to 1.31Gi, and we defined that its RAM needs is 2Gi. We thus need at least this memory times one instance.

–> Quota memory limit >= \(2\text{Gi} * 1\)

During development stage - “in dev”-, you might want to have multiple version running in parallel, one for the main branch, one when developping a new feature, one stagging anther feature, … We thus introduce \(qte_{branch}\).

–> Quota memory limit >= \(2\text{Gi} * qte_{branch}\)

For example, with 3 branches, your multiplier is 3, you would need in the worst case scenario 6Gi.

4.2.2 : Estimate CPU resources you need

Limits and requests for CPU resources are measured in cpu units. In Kubernetes, 1 CPU unit is equivalent to 1 physical CPU core, or 1 virtual core. Fractional requests are allowed. When you define a container with 0.5, you are requesting half as much CPU time compared to if you asked for 1.0 CPU. For CPU resource units, the quantity expression 0.1 is equivalent to the expression 100m, which can be read as “one hundred millicpu”. To avoid accidentally using an invalid CPU quantity, it’s useful to specify CPU units using the milliCPU.

As we did with memory, keep in mind that the reasoning for estimating CPU resources is the same for the CPU.

–> Quota CPU limit >= \(0.5\text{m} * qte_{branch}\)

Adapted from kubernetes documentation

4.2.3 : Estimate Storage resources you need

Regarding the storage, Pasteur offer two storage class for persistent storage : ceph-block and ceph-fs.

• ceph-block is the default storage class. Such storage cannot be mounted by multiple pod at a time.

• ceph-fs is slower, and should be used when you need to have a shared storage accross multiple pods.

4.2.4 : List the users that will have access to the namespaces

You need to list the users that will have access to the namespaces, but also the one who should be notified when the application goes down.

It is possible to grant read only and read write access to the namespaces.

4.2.5 Ask it!

Now you have all informations needed for ask your namespace. You can ask it on Rocket Chat - #Ask-DevOps #Support-Kubernetes or send an email to IT Support (informatique@pasteur.fr), but ask for this issue to be routed to OPSI (and not cluster team).

Here is a summary of the information to transmit :

Name of the application : My application
Project ID of Gitlab project (find it on the main page of your Gitlab project) : ######
Project in GitLab: https://gitlab.pasteur.fr/ada-team/my-project
Namespace in dev: my-project-dev
CPU quota in dev: <...> cpu
RAM quota in dev: <...> GiB
Disk quota in dev: <...> GiB
Name(s) of person(s) who will administrate your application on Kubernetes : Project Manager Name, ...
Name(s) of person(s) who will get read only access on Kubernetes : Project Manager Name, ... (optional)
Name(s) of person(s) who would get notified (For alerting purpose) : Project Manager Name, ...

The IT support - OPSI Team - will configure your project so it can deploy to the Kubernetes clusters, and will indicate you the names of the namespaces.

Step 4.3: Adapte the GitLab CI file to your project

In your GItLab project, from shiny-k8s-example zipped sources, you have to update .gitlab-ci.yml file with your value of the development NAMESPACE variable.

Replace each rshiny-dev occurences, use the *-dev namespace name the IT support provided you.

Example:

 ...
 deploy-example-in-dev:
     extends: .deploy
     stage: 🚀 🚧 deploy-in-dev
     variables:
-       NAMESPACE: "rshiny-dev"
+       NAMESPACE: "my-project-dev"
 ...

At this stage, you don’t need to uncomment the commented parts present in the .gitlab-ci.yml file.

Note

A full documentation on the CI is available at Gitlab CI of shiny-k8s-example.

Step 4.4: Configuring your application in Kubernetes (values.yaml)

Even though the application is preconfigured to be up and running, you can adapt its behavior by overriding some variables. To do so, set the variables you wish to change in chart/values.yaml.

Here are all the settings you may want to change

shiny-k8s-toolkit-helm:
    registry:
        private: false
    nodowntime:
        enabled: false
    autoscaling:
        enabled: false
        minReplicas: 1
        maxReplicas: 2
        targetCPUUtilizationPercentage: 80
        targetMemoryUtilizationPercentage: 80
    resources:
        requests:
            memory: "256Mi"
            cpu: "250m"
        limits:
            memory: "512Mi"
            cpu: "1000m"
    ingress:
        annotations:
            nginx.ingress.kubernetes.io/affinity: cookie
            nginx.ingress.kubernetes.io/proxy-body-size: 256m
    storage:
        enabled: false
        size: "1Gi"
        className: "ceph-fs" # could be ceph-block, incompatible with nodowntime
        keepOnDelete: true # should be false in dev, true in prod

4.4.1 Private registry and deploy token

If the GitLab visibility’s project is private or internal (see Step 1), you need to set the project as private. Therefore, the registry containing the Docker image build from your sources.

• private/internal project
• public project

shiny-k8s-toolkit-helm:
    registry:
        private: true

shiny-k8s-toolkit-helm:
    registry:
        private: false

By setting it to true, the helm package and associated CI scripts are expecting a DEPLOY_TOKEN and a DEPLOY_USER. See Step 4.1: Define a deploy token (Optionnal) to do so.

4.4.2 Adjusting resources (memory and CPU)

Here you need to update the default values by the values of resources limit and requests, previously defined at Step 4.2

shiny-k8s-toolkit-helm:
    resources:
        requests:
            memory: "256Mi"
            cpu: "250m"
        limits:
            memory: "2Gi"
            cpu: "500m"

How Kubernetes applies resource requests and limits

When Kubernetes starts a container as part of a Pod, container’s requests and limits for memory and CPU are passed to the container runtime. On Linux, the container runtime typically configures kernel cgroups that apply and enforce the limits you defined.

• The memory request is mainly used during (Kubernetes) Pod scheduling.

• The CPU request typically defines a weighting.

• The CPU limit defines a hard ceiling on how much CPU time the container can use. During each scheduling interval (time slice), the Linux kernel checks to see if this limit is exceeded; if so, the kernel waits before allowing to resume execution.

• The memory limit defines a memory limit for that cgroup. If the container tries to allocate more memory than this limit, the Linux kernel out-of-memory subsystem activates and, typically, intervenes by stopping one of the processes in the container that tried to allocate memory.

Tip

You can later change the resources allocated to a pod/container. Keep in mind that these resources must be within your quota (See quota and logs).

You can request more resources than what was requested at the beginning by asking the news values on Rocket Chat - #Ask-DevOps #Support-Kubernetes

Follow How to observe my memory usage locally to evaluate how much memory you need, and also the quota you need.

4.4.3 Adjusting storage configuration

Note

This feature was released in v0.7, to update your dependencies, see How to upgrade my helm dependency / I can’t enable a feature

By default, everything you write on the storage of your application is lost when your application restart or have an upgrade. You can have folders with a persistent storage :

• /srv/shiny-server/www/persistent here you can write data that are served to user such as images, csv, …

• /srv/shiny-server/data the content of this folder is only visible by the application, not end users.

shiny-k8s-toolkit-helm:
    storage:
        enabled: true
        size: "1Gi"
        className: "ceph-fs" # could be ceph-block, incompatible with nodowntime
        keepOnDelete: true # should be false in dev, true in prod

• The size indicated is for the total of the two folders (i.e: it is not per folder). It can be increased later, but cannot be reduced.

• The className allows to choose between disk storage (ceph-fs) or flash storage (ceph-block) which is faster but incompatible with nodowntime: Storage using ceph-block cannot be mounted by multiple pod (i.e: multiple version of your application) which is precisly what does the nodowntime option.

• The keepOnDelete indicate whetherthe storage and its content should be kept after removing the application. In production you should set it to true as user might want to be able to re-consult analysis results later without re-running their analysis. In dev you should set it to false as a new storage will be created for each new branch and you don’t want to let them eat up your quota when you will have merged the branch.

To upload data in the data dir, see FAQ Uploading to data dir

Step 4.5: Access to logs and quota

To access to the logs, you can use the one-click CI job we defined for help you, (or use Kubernetes utilities).

Go to Pipelines tab of your project at https://gitlab.pasteur.fr/ada-team/my-project/-/pipelines In the last pipelines, clic on the second round circle (1 in image), note that it can be green, grey, red. Once openned the pipelines popup, clic on the text fetch-log-dev (2 in the image).

• Pipeline starting from Dec. 2025
• Pipeline starting from Mai. 2025
• Pipeline prior to Mai. 2025

You can then either directly clic on Run job, you will see the last lines (200 by default as of july 2023). You can also specify a custom amount of line, to do so specify the value of the variable LAST_ROWS_COUNT. If you want all logs since startup set the value to -1.

Which produce the following:

The logs will be at the bottom of the page (1 in the image). Here we see that the Shiny server has started well.

The quota will be at the beginning of the logs (2 in the image). Quota should be read as:

Kind	Requests	Limits	Quota
CPU	0.75	1.5	10
Ram	768Mi	1536Gi	10Gi

The logs can also be downloaded, the file(s) can be found in the “Job artifacts” section (3 in the image).

If you want to restart the task with a different number of rows, first you have to delete the log with the trash button at the top right of the page (4 in the image).