Step 5 : Deploy your app on prod cluster with kubernetes

Note

We consider that you have tested your application with docker compose according to instructions described in Step 3, and succeeded to deploy it in the dev (cf Step 4)

Step 5.1: Ask Kubernetes namespace on prod cluster

At Pasteur, we seperate production environmenent from developpement environment (see Step 4) by using different kubernetes clusters (see Kubernetes infrastructure documentation).

In the production environement, only the main branch will be deployed and accessible to users. This allows you to easily develop new features or fix bugs using development environement and dev namespace without interupting the production environement.

Your two environments can coexists in parallel. In this case, you will need two namespaces. At Pasteur, the naming convention is like -dev and -prod.

On the prod cluster, you can have urls https://<anything>.pasteur.cloud, and be opened to the world, or kept accessible only to the Institut Pasteur with the VPN.

5.1.1 Do I want a public URL accessible without the VPN?

If you want to have an URL accessible by anyone in the word without the VPN, you have to deploy to the prod cluster, but you also have to declare this URL to the IT support. To do so, ask either on Rocket Chat - #Ask-DevOps #Support-Kubernetes or send an email to informatique@pasteur.fr to add the url in the “public DNS”, this will allow incoming trafic on this URL.

5.1.2 Estimate (CPU, RAM, storage) resources you need

Like in development environment, ressources should be estimated carefully.

You can use the same resoning for dev namespace (see Estimate memory resources)

In you production environement you will have only one branch, the main branch. But in production using nodowntime is arguable, and autoscaling could also be a good idea if you may face peek of usage by numerous users. Also dedicating more CPU and RAM to your app can make sens.

How to estimate quota needs from your app ressources in prod

In the following we will estimate a multiplication factor that have to be applied to your request and limits of memory and cpu of your application.

We will work on memory, but keep mind that the resoning is the same for memory and cpu, request and limits.

In the previous example, the memory peaked to 1.31Gi. We can assume that with multiple users (3 at the same time?) and the RAM needs is 4Gi. We thus need at least this memory times one instance.

–> Quota memory limit >= \(4\text{Gi} * 1\)

Assuming you want to use Nodowntime, two instances will shortly consume resources at the same time, so you have to consider a second one is running.

–> Quota memory limit >= \(4\text{Gi} * (1 + (1 \text{ if }Nodowntime\text{ else } 0))\)

Assuming you want to use Autoscaling, with a maxReplicas: ... it is not one but \(qte_{replicat}\) that will run in parallel.

–> Quota memory limit >= \(4\text{Gi} * (qte_{replicat} + (1 \text{ if }Nodowntime\text{ else } 0))\)

For example, with a maxReplicas to 3 and Nodowntime enabled, your multiplier is 4, you would need in the worst case scenario 16Gi:

Quota memory limit >= \(4\text{Gi} * (3 + (1 ))\) = 4 * 4 = 16

Dev vs Prod quota

As a reminder, in your dev environement multiple branches will coexiste, but autoscaling and nodowntime might not be that much relevant as you may accept to have an instance going down with a wrong commit, or a slowed down as you started multiple computation on it.

By re-using previous example,(see Step 4) but after adapting settings after the previous discussion, we then have a quota needs of:

Container memory limit

Autoscaling

No downtime

Branches

Multiplier

Quota memory limit

Dev

2Gi

1

no

3

=3

6Gi

Prod

4Gi

3

yes

1

=4

16Gi

5.1.3 Ask the second namespace

Now you have all informations needed for ask your prod namespace. You can ask it on Rocket Chat - #Ask-DevOps #Support-Kubernetes or send an email to IT Support (informatique@pasteur.fr), but ask for this issue to be routed to OPSI (and not cluster team).

Here is a summary of the information to transmit :

Name of the application : My application
Project ID of Gitlab project (find it on the main page of your Gitlab project) : ######
Project in GitLab: https://gitlab.pasteur.fr/ada-team/my-project
Namespace in dev: my-project-dev
CPU quota in dev: <...> cpu
RAM quota in dev: <...> GiB
Disk quota in dev (if needed): <...> GiB
Namespace in prod: my-project-prod
CPU quota in prod: <...> cpu
RAM quota in prod: <...> GiB
Disk quota in prod (if needed): <...> GiB
Public URL in prod (if needed, the complete URL): https://my-project.pasteur.cloud.
Name(s) of person(s) who will administrate your application on Kubernetes : Project Manager Name, ...
Name(s) of person(s) who will get read only access on Kubernetes : Project Manager Name, ... (optional)
Name(s) of person(s) who would get notified (For alerting purpose) : Project Manager Name, ...

The IT support - OPSI Team - will configure your project so it can deploy to the Kubernetes clusters, and will indicate you the names of the namespaces.

Step 5.2: Adapte the CI to your project

In your GItLab project, from shiny-k8s-example zipped sources, you have to update .gitlab-ci.yml file with the value of the production NAMESPACE variables and uncomment the jobs dedicated to be used in prod ( deploy-prod, fetch-log-prod, send-fex-to-prod)

  • Replace each rshiny-prod occurrences, use the *-prod namespace name the IT support provided you.

  • Change the value of PUBLIC_URL variable with your URL as my-project.pasteur.cloud.

If you asked an public url, keep the INGRESS_CLASS variable to external (line 10), otherwise set to in internal

  • Before
  • After
 1# Deployment job for production
 2#deploy-prod:
 3#  extends: .deploy
 4#  stage: "🚀 🌐 deploy-in-prod"
 5#  only:
 6#    - main
 7#  variables:
 8#    NAMESPACE: "rshiny-prod"
 9#    PUBLIC_URL: "${CI_PROJECT_NAME}.pasteur.cloud"
10#    INGRESS_CLASS: "external"
11#    CHART_LOCATION: "chart"
12#    IMAGE: "${CI_REGISTRY_IMAGE}/${CI_COMMIT_REF_SLUG}:${CI_COMMIT_SHORT_SHA}"
13#    VALUES_OVERRIDE_FILENAME: "values.prod.yaml"
14#  environment:
15#    name: "k8sprod-02/${NAMESPACE}/${CI_COMMIT_REF_SLUG}"
16#    url: "https://${CI_PROJECT_NAME}.pasteur.cloud"

 1# Deployment job for production
 2deploy-prod:
 3  extends: .deploy
 4  stage: "🚀 🌐 deploy-in-prod"
 5  only:
 6    - main
 7  variables:
 8    NAMESPACE: "rshiny-prod"
 9    PUBLIC_URL: "${CI_PROJECT_NAME}.pasteur.cloud"
10    INGRESS_CLASS: "external"
11    CHART_LOCATION: "chart"
12    IMAGE: "${CI_REGISTRY_IMAGE}/${CI_COMMIT_REF_SLUG}:${CI_COMMIT_SHORT_SHA}"
13    VALUES_OVERRIDE_FILENAME: "values.prod.yaml"
14  environment:
15    name: "k8sprod-02/${NAMESPACE}/${CI_COMMIT_REF_SLUG}"
16    url: "https://${CI_PROJECT_NAME}.pasteur.cloud"

Step 5.3: Configuring your prod environement in Kubernetes (values.prod.yaml)

Even though the application is preconfigured to be up and running, you can adapt its behavior by overriding some variables. To do so set the variables you wish to change in chart/values.prod.yaml.

Here are all the settings you may want to change

 1shiny-k8s-toolkit-helm:
 2    nodowntime:
 3        enabled: true
 4    autoscaling:
 5        enabled: true
 6    resources:
 7        requests:
 8            memory: "256Mi"
 9            cpu: "250m"
10        limits:
11            memory: "4Gi"
12            cpu: "2000m"
13    storage:
14        size: "10Gi"
15        keepOnDelete: true # should be false in dev, true in prod

Nodowntime

When a new version of your application is released, the old version is stopped, and the new one is then started. Between these two events, your application is not reachable. You can enable the nodowntime. If so, the new version will be started, and only when ready, the old version will be stopped down. Another advantage of nodowntime is that if the new version fails to start, the outdated version is still available.

The drawback of the nodowntime is that temporarily two version of the application are running, which both consume resources. You thus have to adjust the resources so the limits is at most half your quota (cf See quota and logs).

shiny-k8s-toolkit-helm:
    nodowntime:
        enabled: tue

Autoscaling

If you plan to have spike of activity for your application, you may want to enable autoscaling. It will start new instance(s) of you application and will dispatch new users to the less used instance. Note that each instance started will consume resources, so keep an eye on your quota and your resources.

shiny-k8s-toolkit-helm:
    autoscaling:
        enabled: true
        minReplicas: 1
        maxReplicas: 2
        targetCPUUtilizationPercentage: 80
        targetMemoryUtilizationPercentage: 80

Adjusting resources

You can ask for more resources, keep in mind that these resources must be within your quota (See quota and logs). Follow How to observe my memory usage locally to evaluate how much memory you need, and also the quota you need.

shiny-k8s-toolkit-helm:
    resources:
        requests:
            memory: "256Mi"
            cpu: "250m"
        limits:
            memory: "4Gi"
            cpu: "2000m"

Storage

shiny-k8s-toolkit-helm:
    storage:
        size: "10Gi"
        keepOnDelete: true # should be false in dev, true in prod

The keepOnDelete indicate whetherthe storage and its content should be kept after removing the application. In production you should set it to true as user might want to be able to re-consult analysis results later without re-running their analysis.

The disk size should also be higher in production, as more analysis will be done. You have to implement mecanisme in your app to keep the disk usage under control.

Step 5.4 Access to logs and quota

Like for dev environement, to access to the logs, you can either use the one-click CI job.

Go to Pipelines tab of your project at https://gitlab.pasteur.fr/ada-team/my-project/-/pipelines In the last pipelines, clic on the second round circle (1 in image), note that it can be green, grey, red. Once openned the pipelines popup, clic on the text fetch-log-prod (2 in the image).

To be adapted

  • Gitlab UI starting from 16.1
  • Gitlab UI prior to 16.1
Find task log-fetcher
Find task log-fetcher

See quota and logs for details on how to interpret it.

More ?

You can add a task to be able to delete your instance from production (see FAQ here). If your app log usage internally and store it in data dir, you can retrieve them (see FAQ here).