instruction: How do you implement GitLab CI/CD with Kubernetes deployment?
output: Kubernetes executor | GitLab Docs Skip to main content Go to GitLab Docs homepage What's new? What's new? Get free trial Toggle menu Use GitLab GitLab Duo Extend Install Administer Subscribe Contribute Solutions Help us learn about your current experience with the documentation. Take the survey . Kubernetes executor Tier : Free, Premium, Ultimate Offering : GitLab.com, GitLab Self-Managed, GitLab Dedicated Use the Kubernetes executor to use Kubernetes clusters for your builds. The executor calls the Kubernetes cluster API and creates a pod for each GitLab CI job. The Kubernetes executor divides the build into multiple steps: Prepare : Create the Pod against the Kubernetes Cluster. This creates the containers required for the build and services to run. Pre-build : Clone, restore cache, and download artifacts from previous stages. This step runs on a special container as part of the pod. Build : User build. Post-build : Create cache, upload artifacts to GitLab. This step also uses the special container as part of the pod. How the runner creates Kubernetes pods The following diagram shows the interaction between a GitLab instance and a runner hosted on a Kubernetes cluster. The runner calls the Kubernetes API to create pods on the cluster. The pod consists of the following containers for each service defined in the .gitlab-ci.yml or config.toml files: A build container defined as build . A helper container defined as helper . A services containers defined as svc-X , where X is [0-9]+ . Services and containers run in the same Kubernetes pod and share the same localhost address. The following restrictions apply: The services are accessible through their DNS names. If you use an older version, you must use localhost . You cannot use several services that use the same port. For example, you cannot have two mysql services at the same time. sequenceDiagram participant G as GitLab instance participant R as Runner on Kubernetes cluster participant Kube as Kubernetes API participant P as POD R-&gt;&gt;&#43;G: Get a CI job. loop G--&gt;R: ; end Note over R,G: POST /api/v4/jobs/request G-&gt;&gt;&#43;R: CI job data. R--&gt;&gt;-Kube: Create a POD to run the CI job. Note over R,Kube: POST to Kube API P-&gt;&gt;&#43;P: Execute job. Note over P: CI build job = Prepare &#43; Pre-build &#43; Build &#43; Post-build P-&gt;&gt;&#43;G: Job logs The interaction in the diagram is valid for any Kubernetes cluster. For example, turnkey solutions hosted on the major public cloud providers, or self-managed Kubernetes installations. Connect to the Kubernetes API Use the following options to connect to the Kubernetes API. The user account provided must have permission to create, list, and attach to Pods in the specified namespace. Option Description host Optional Kubernetes API server host URL (auto-discovery attempted if not specified). cert_file Optional Kubernetes API server user auth certificate. key_file Optional Kubernetes API server user auth private key. ca_file Optional Kubernetes API server ca certificate. If you&rsquo;re running GitLab Runner in the Kubernetes cluster, omit these fields so that the GitLab Runner auto-discovers the Kubernetes API. If you&rsquo;re running GitLab Runner externally to the Cluster, these settings ensure that GitLab Runner has access to the Kubernetes API on the cluster. Set the bearer token for Kubernetes API calls To set the bearer token for API calls to create pods, use the KUBERNETES_BEARER_TOKEN variable. This allows project owners to use project secret variables to specify a bearer token. When specifying the bearer token, you must set the Host configuration setting. variables : KUBERNETES_BEARER_TOKEN : thebearertokenfromanothernamespace Configure runner API permissions To configure permissions for the core API group, update the values.yml file for GitLab Runner Helm charts. You can either: Set rbac.create to true . Specify a service account serviceAccount.name: &lt;service_account_name> with the following permissions in the values.yml file. Resource Verb (Optional Feature/Config Flags) events list ( print_pod_warning_events=true ), watch ( FF_PRINT_POD_EVENTS=true ) namespaces create ( kubernetes.NamespacePerJob=true ), delete ( kubernetes.NamespacePerJob=true ) pods create, delete, get, list ( using Informers ), watch ( using Informers , FF_KUBERNETES_HONOR_ENTRYPOINT=true , FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=false ) pods/attach create ( FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=false ), delete ( FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=false ), get ( FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=false ), patch ( FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=false ) pods/exec create, delete, get, patch pods/log get ( FF_KUBERNETES_HONOR_ENTRYPOINT=true , FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=false , FF_WAIT_FOR_POD_TO_BE_REACHABLE=true ), list ( FF_KUBERNETES_HONOR_ENTRYPOINT=true , FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=false ) secrets create, delete, get, update serviceaccounts get services create, get You can use the following YAML role definition to create a role with the required permissions. apiVersion : rbac.authorization.k8s.io/v1 kind : Role metadata : name : gitlab-runner namespace : default rules : - apiGroups : [ &#34;&#34; ] resources : [ &#34;events&#34; ] verbs : - &#34;list&#34; # Required when `print_pod_warning_events=true` - &#34;watch&#34; # Required when `FF_PRINT_POD_EVENTS=true` - apiGroups : [ &#34;&#34; ] resources : [ &#34;namespaces&#34; ] verbs : - &#34;create&#34; # Required when `kubernetes.NamespacePerJob=true` - &#34;delete&#34; # Required when `kubernetes.NamespacePerJob=true` - apiGroups : [ &#34;&#34; ] resources : [ &#34;pods&#34; ] verbs : - &#34;create&#34; - &#34;delete&#34; - &#34;get&#34; - &#34;list&#34; # Required when using Informers (https://docs.gitlab.com/runner/executors/kubernetes/#informers) - &#34;watch&#34; # Required when `FF_KUBERNETES_HONOR_ENTRYPOINT=true`, `FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=false`, using Informers (https://docs.gitlab.com/runner/executors/kubernetes/#informers) - apiGroups : [ &#34;&#34; ] resources : [ &#34;pods/attach&#34; ] verbs : - &#34;create&#34; # Required when `FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=false` - &#34;delete&#34; # Required when `FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=false` - &#34;get&#34; # Required when `FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=false` - &#34;patch&#34; # Required when `FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=false` - apiGroups : [ &#34;&#34; ] resources : [ &#34;pods/exec&#34; ] verbs : - &#34;create&#34; - &#34;delete&#34; - &#34;get&#34; - &#34;patch&#34; - apiGroups : [ &#34;&#34; ] resources : [ &#34;pods/log&#34; ] verbs : - &#34;get&#34; # Required when `FF_KUBERNETES_HONOR_ENTRYPOINT=true`, `FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=false`, `FF_WAIT_FOR_POD_TO_BE_REACHABLE=true` - &#34;list&#34; # Required when `FF_KUBERNETES_HONOR_ENTRYPOINT=true`, `FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=false` - apiGroups : [ &#34;&#34; ] resources : [ &#34;secrets&#34; ] verbs : - &#34;create&#34; - &#34;delete&#34; - &#34;get&#34; - &#34;update&#34; - apiGroups : [ &#34;&#34; ] resources : [ &#34;serviceaccounts&#34; ] verbs : - &#34;get&#34; - apiGroups : [ &#34;&#34; ] resources : [ &#34;services&#34; ] verbs : - &#34;create&#34; - &#34;get&#34; Additional details: The event permission is needed only for GitLab 16.2.1 and later. The namespace permission is needed only when enabling namespace isolation by using namespace_per_job . The pods/log permission is only needed when one of the following scenarios are true: The FF_KUBERNETES_HONOR_ENTRYPOINT feature flag is enabled. The FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY feature flag is disabled when the CI_DEBUG_SERVICES variable is set to true . The FF_WAIT_FOR_POD_TO_BE_REACHABLE feature flag is enabled. Informers In GitLab Runner 17.9.0 and later, a Kubernetes informer tracks build pod changes. This helps the executor detect the changes more quickly. The informer requires list and watch permissions for pods . When the executor starts the build, it checks the Kubernetes API for the permissions. If all permissions are granted, the executor uses an informer. If any permission is missing, GitLab Runner logs a warning. The build continues and uses the previous mechanism to track the build pod&rsquo;s status and changes. Configuration settings Use the following settings in the config.toml file to configure the Kubernetes executor. CPU requests and limits Setting Description cpu_limit The CPU allocation given to build containers. cpu_limit_overwrite_max_allowed The maximum amount that the CPU allocation can be written to for build containers. When empty, it disables the CPU limit overwrite feature. cpu_request The CPU allocation requested for build containers. cpu_request_overwrite_max_allowed The maximum amount that the CPU allocation request can be written to for build containers. When empty, it disables the CPU request overwrite feature. helper_cpu_limit The CPU allocation given to build helper containers. helper_cpu_limit_overwrite_max_allowed The maximum amount that the CPU allocation can be written to for helper containers. When empty, it disables the CPU limit overwrite feature. helper_cpu_request The CPU allocation requested for build helper containers. helper_cpu_request_overwrite_max_allowed The maximum amount that the CPU allocation request can be written to for helper containers. When empty, it disables the CPU request overwrite feature. service_cpu_limit The CPU allocation given to build service containers. service_cpu_limit_overwrite_max_allowed The maximum amount that the CPU allocation can be written to for service containers. When empty, it disables the CPU limit overwrite feature. service_cpu_request The CPU allocation requested for build service containers. service_cpu_request_overwrite_max_allowed The maximum amount that the CPU allocation request can be written to for service containers. When empty, it disables the CPU request overwrite feature. Memory requests and limits Setting Description memory_limit The amount of memory allocated to build containers. memory_limit_overwrite_max_allowed The maximum amount that the memory allocation can be written to for build containers. When empty, it disables the memory limit overwrite feature. memory_request The amount of memory requested from build containers. memory_request_overwrite_max_allowed The maximum amount that the memory allocation request can be written to for build containers. When empty, it disables the memory request overwrite feature. helper_memory_limit The amount of memory allocated to build helper containers. helper_memory_limit_overwrite_max_allowed The maximum amount that the memory allocation can be written to for helper containers. When empty, it disables the memory limit overwrite feature. helper_memory_request The amount of memory requested for build helper containers. helper_memory_request_overwrite_max_allowed The maximum amount that the memory allocation request can be written to for helper containers. When empty, it disables the memory request overwrite feature. service_memory_limit The amount of memory allocated to build service containers. service_memory_limit_overwrite_max_allowed The maximum amount that the memory allocation can be written to for service containers. When empty, it disables the memory limit overwrite feature. service_memory_request The amount of memory requested for build service containers. service_memory_request_overwrite_max_allowed The maximum amount that the memory allocation request can be written to for service containers. When empty, it disables the memory request overwrite feature. Helper container memory sizing recommendations For optimal performance, set helper container memory limits based on your workload requirements: Workloads with caching and artifact generation : Minimum 250 MiB Basic workloads without cache/artifacts : Might work with lower limits (128-200 MiB) Basic configuration example: [[ runners ]] executor = &#34;kubernetes&#34; [ runners . kubernetes ] helper_memory_limit = &#34;250Mi&#34; helper_memory_request = &#34;250Mi&#34; helper_memory_limit_overwrite_max_allowed = &#34;1Gi&#34; Job-specific memory overrides: Use the KUBERNETES_HELPER_MEMORY_LIMIT variable to adjust memory for specific jobs without requiring administrator changes: job_with_higher_helper_memory_limit : variables : KUBERNETES_HELPER_MEMORY_LIMIT : &#34;512Mi&#34; script: This approach allows developers to optimize resource usage per job while maintaining cluster-wide limits through helper_memory_limit_overwrite_max_allowed . Storage requests and limits Setting Description ephemeral_storage_limit The ephemeral storage limit for build containers. ephemeral_storage_limit_overwrite_max_allowed The maximum amount that the ephemeral storage limit for build containers can be overwritten. When empty, it disables the ephemeral storage limit overwrite feature. ephemeral_storage_request The ephemeral storage request given to build containers. ephemeral_storage_request_overwrite_max_allowed The maximum amount that the ephemeral storage request can be overwritten by for build containers. When empty, it disables the ephemeral storage request overwrite feature. helper_ephemeral_storage_limit The ephemeral storage limit given to helper containers. helper_ephemeral_storage_limit_overwrite_max_allowed The maximum amount that the ephemeral storage limit can be overwritten by for helper containers. When empty, it disables the ephemeral storage request overwrite feature. helper_ephemeral_storage_request The ephemeral storage request given to helper containers. helper_ephemeral_storage_request_overwrite_max_allowed The maximum amount that the ephemeral storage request can be overwritten by for helper containers. When empty, it disables the ephemeral storage request overwrite feature. service_ephemeral_storage_limit The ephemeral storage limit given to service containers. service_ephemeral_storage_limit_overwrite_max_allowed The maximum amount that the ephemeral storage limit can be overwritten by for service containers. When empty, it disables the ephemeral storage request overwrite feature. service_ephemeral_storage_request The ephemeral storage request given to service containers. service_ephemeral_storage_request_overwrite_max_allowed The maximum amount that the ephemeral storage request can be overwritten by for service containers. When empty, it disables the ephemeral storage request overwrite feature. Other config.toml settings Setting Description affinity Specify affinity rules that determine which node runs the build. Read more about using affinity . allow_privilege_escalation Run all containers with the allowPrivilegeEscalation flag enabled. When empty, it does not define the allowPrivilegeEscalation flag in the container SecurityContext and allows Kubernetes to use the default privilege escalation behavior. allowed_images Wildcard list of images that can be specified in .gitlab-ci.yml . If not present all images are allowed (equivalent to ["*/*:*"] ). View details . allowed_pull_policies List of pull policies that can be specified in the .gitlab-ci.yml file or the config.toml file. allowed_services Wildcard list of services that can be specified in .gitlab-ci.yml . If not present all images are allowed (equivalent to ["*/*:*"] ). View details . automount_service_account_token Boolean to control whether the service account token automatically mounts in the build pod. bearer_token Default bearer token used to launch build pods. bearer_token_overwrite_allowed Boolean to allow projects to specify a bearer token used to create the build pod. build_container_security_context Sets a container security context for the build container. Read more about security context . cap_add Specify Linux capabilities that should be added to the job pod containers. Read more about capabilities configuration in Kubernetes executor . cap_drop Specify Linux capabilities that should be dropped from the job pod containers. Read more about capabilities configuration in Kubernetes executor . cleanup_grace_period_seconds When a job completes, the duration in seconds that the pod has to terminate gracefully. After this period, the processes are forcibly halted with a kill signal. Ignored if terminationGracePeriodSeconds is specified. dns_policy Specify the DNS policy that should be used when constructing the pod: none , default , cluster-first , cluster-first-with-host-net . The Kubernetes default ( cluster-first ) is used if not set. dns_config Specify the DNS configuration that should be used when constructing the pod. Read more about using pod&rsquo;s DNS config . helper_container_security_context Sets a container security context for the helper container. Read more about security context . helper_image (Advanced) Override the default helper image used to clone repositories and upload artifacts. helper_image_flavor Sets the helper image flavor ( alpine , alpine3.19 , alpine3.21 , or ubuntu ). Defaults to alpine . Using alpine is the same as alpine3.19 . host_aliases List of additional host name aliases that are added to all containers. Read more about using extra host aliases . image_pull_secrets An array of items containing the Kubernetes docker-registry secret names used to authenticate Docker image pulling from private registries. init_permissions_container_security_context Sets a container security context for the init-permissions container. Read more about security context . namespace Namespace in which to run Kubernetes Pods. namespace_per_job Isolate jobs in separate namespaces. If enabled, namespace and namespace_overwrite_allowed are ignored. namespace_overwrite_allowed Regular expression to validate the contents of the namespace overwrite environment variable (documented below). When empty, it disables the namespace overwrite feature. node_selector A table of key=value pairs in the format of string=string ( string:string in the case of environment variables). Setting this limits the creation of pods to Kubernetes nodes matching all the key=value pairs. Read more about using node selectors . node_tolerations A table of "key=value" = "Effect" pairs in the format of string=string:string . Setting this allows pods to schedule to nodes with all or a subset of tolerated taints. Only one toleration can be supplied through environment variable configuration. The key , value , and effect match with the corresponding field names in Kubernetes pod toleration configuration. pod_annotations A table of key=value pairs in the format of string=string . The table contains a list of annotations to be added to each build pod created by the runner. The value of these can include environment variables for expansion. Pod annotations can be overwritten in each build. pod_annotations_overwrite_allowed Regular expression to validate the contents of the pod annotations overwrite environment variable. When empty, it disables the pod annotations overwrite feature. pod_labels A table of key=value pairs in the format of string=string . The table contains a list of labels to be added to each build pod created by the runner. The value of these can include environment variables for expansion. Pod labels can be overwritten in each build by using pod_labels_overwrite_allowed . pod_labels_overwrite_allowed Regular expression to validate the contents of the pod labels overwrite environment variable. When empty, it disables the pod labels overwrite feature. Note that pod labels in the runner.gitlab.com label namespace cannot be overwritten. pod_security_context Configured through the configuration file, this sets a pod security context for the build pod. Read more about security context . pod_termination_grace_period_seconds Pod-level setting which determines the duration in seconds which the pod has to terminate gracefully. After this, the processes are forcibly halted with a kill signal. Ignored if terminationGracePeriodSeconds is specified. poll_interval How frequently, in seconds, the runner polls the Kubernetes pod it has just created to check its status (default = 3). poll_timeout The amount of time, in seconds, that needs to pass before the runner times out attempting to connect to the container it has just created. Use this setting for queueing more builds than the cluster can handle at a time (default = 180). cleanup_resources_timeout The total amount of time for Kubernetes resources to be cleaned up after the job completes. Supported syntax: 1h30m , 300s , 10m . Default is 5 minutes ( 5m ). priority_class_name Specify the Priority Class to be set to the pod. The default one is used if not set. privileged Run containers with the privileged flag. pull_policy Specify the image pull policy: never , if-not-present , always . If not set, the cluster&rsquo;s image default pull policy is used. For more information and instructions on how to set multiple pull policies, see using pull policies . See also if-not-present , never security considerations . You can also restrict pull policies . resource_availability_check_max_attempts The maximum number of attempts to check if a resource (service account and/or pull secret) set is available before giving up. There is 5 seconds interval between each attempt. Read more about resources check during prepare step . runtime_class_name A Runtime class to use for all created pods. If the feature is unsupported by the cluster, jobs exit or fail. service_container_security_context Sets a container security context for the service containers. Read more about security context . scheduler_name Scheduler to use for scheduling build pods. service_account Default service account job/executor pods use to talk to Kubernetes API. service_account_overwrite_allowed Regular expression to validate the contents of the service account overwrite environment variable. When empty, it disables the service account overwrite feature. services List of services attached to the build container using the sidecar pattern . Read more about using services . use_service_account_image_pull_secrets When enabled, the pod created by the executor lacks imagePullSecrets . This causes the pod to be created using the imagePullSecrets from the service account , if set. terminationGracePeriodSeconds Duration after the processes running in the pod are sent a termination signal and the time when the processes are forcibly halted with a kill signal. Deprecated in favour of cleanup_grace_period_seconds and pod_termination_grace_period_seconds . volumes Configured through the configuration file, the list of volumes that is mounted in the build container. Read more about using volumes . pod_spec This setting is an experiment. Overwrites the pod specification generated by the runner manager with a list of configurations set on the pod used to run the CI Job. All the properties listed Kubernetes Pod Specification can be set. For more information, see Overwrite generated pod specifications (experiment) . retry_limit The maximum number of attempts to communicate with Kubernetes API. The retry interval between each attempt is based on a backoff algorithm starting at 500 ms. retry_backoff_max Custom maximum backoff value in milliseconds for the retry interval to reach for each attempt. The default value is 2000 ms and it can not be lower than 500 ms. The default maximum retry interval to reach for each attempt is 2 seconds and can be customized with retry_backoff_max . retry_limits How many times each request error is to be retried. logs_base_dir Base directory to be prepended to the generated path to store build logs. For more information, see Change the base directory for build logs and scripts . scripts_base_dir Base directory to be prepended to the generated path to store build scripts. For more information, see Change the base directory for build logs and scripts . print_pod_warning_events When enabled, this feature retrieves all warning events associated with the pod when jobs fail. This functionality is enabled by default and requires a service account with at least events: list permissions . Configuration example The following sample shows an example configuration of the config.toml file for the Kubernetes executor. concurrent = 4 [[ runners ]] name = &#34;myRunner&#34; url = &#34;https://gitlab.com/ci&#34; token = &#34;......&#34; executor = &#34;kubernetes&#34; [ runners . kubernetes ] host = &#34;https://45.67.34.123:4892&#34; cert_file = &#34;/etc/ssl/kubernetes/api.crt&#34; key_file = &#34;/etc/ssl/kubernetes/api.key&#34; ca_file = &#34;/etc/ssl/kubernetes/ca.crt&#34; namespace = &#34;gitlab&#34; namespace_overwrite_allowed = &#34;ci-.*&#34; bearer_token_overwrite_allowed = true privileged = true cpu_limit = &#34;1&#34; memory_limit = &#34;1Gi&#34; service_cpu_limit = &#34;1&#34; service_memory_limit = &#34;1Gi&#34; helper_cpu_limit = &#34;500m&#34; helper_memory_limit = &#34;100Mi&#34; poll_interval = 5 poll_timeout = 3600 dns_policy = &#34;cluster-first&#34; priority_class_name = &#34;priority-1&#34; logs_base_dir = &#34;/tmp&#34; scripts_base_dir = &#34;/tmp&#34; [ runners . kubernetes . node_selector ] gitlab = &#34;true&#34; [ runners . kubernetes . node_tolerations ] &#34;node-role.kubernetes.io/master&#34; = &#34;NoSchedule&#34; &#34;custom.toleration=value&#34; = &#34;NoSchedule&#34; &#34;empty.value=&#34; = &#34;PreferNoSchedule&#34; &#34;onlyKey&#34; = &#34;&#34; Configure the executor service account To configure the executor service account, you can set the KUBERNETES_SERVICE_ACCOUNT environment variable or use the --kubernetes-service-account flag. Pods and containers You can configure pods and containers to control how jobs are executed. Default labels for job pods You cannot override these labels through runner configuration or .gitlab-ci.yml files. Any attempts to set or modify labels in the runner.gitlab.com namespace are ignored and logged as debug messages. Key Description project.runner.gitlab.com/id The ID of the project, unique across projects in the GitLab instance. project.runner.gitlab.com/name The name of the project. project.runner.gitlab.com/namespace-id The ID of the project&rsquo;s namespace. project.runner.gitlab.com/namespace The name of the project&rsquo;s namespace. project.runner.gitlab.com/root-namespace The ID of the project&rsquo;s root namespace. For example, /gitlab-org/group-a/subgroup-a/project , where the root namespace is gitlab-org manager.runner.gitlab.com/name The name of the runner configuration that launched this job. manager.runner.gitlab.com/id-short The ID of the runner configuration that launched the job. job.runner.gitlab.com/pod Internal label used by the Kubernetes executor. Default annotations for job pods The following annotations are added by default on the Pod running the jobs: Key Description job.runner.gitlab.com/id The ID of the job, unique across all jobs in the GitLab instance. job.runner.gitlab.com/url The URL for the job details. job.runner.gitlab.com/sha The commit revision the project is built for. job.runner.gitlab.com/before_sha The previous latest commit present on a branch or tag. job.runner.gitlab.com/ref The branch or tag name for which the project is built. job.runner.gitlab.com/name The name of the job. job.runner.gitlab.com/timeout The job execution timeout in the time duration format. For example, 2h3m0.5s . project.runner.gitlab.com/id The project ID of the job. To overwrite default annotations, use the pod_annotations in the GitLab Runner configuration. You can also overwrite annotations for each CI/CD job in the .gitlab-ci.yml file . Pod lifecycle A pod&rsquo;s lifecycle can be affected by: Setting the pod_termination_grace_period_seconds property in the TOML configuration file. The process running on the pod can run for the given duration after the TERM signal is sent. A kill signal is sent if the Pod is not successfully terminated after this period of time. Enabling the FF_USE_POD_ACTIVE_DEADLINE_SECONDS feature flag . When enabled and the job times out, the pod running the CI/CD job is marked as failed and all associated containers are killed. To have the job time out on GitLab first, activeDeadlineSeconds is set to configured timeout + 1 second . If you enable the FF_USE_POD_ACTIVE_DEADLINE_SECONDS feature flag and set pod_termination_grace_period_seconds to a non-zero value, the CI/CD job pod is not terminated immediately. The pod terminationGracePeriods ensures the pod is terminated only when it expired. Overwrite pod tolerations To overwrite Kubernetes pod tolerations: In the config.toml or Helm values.yaml file, to enable the overwrite of CI job pod tolerations, define a regular expression for node_tolerations_overwrite_allowed . This regular expression validates the values of CI variable names that start with KUBERNETES_NODE_TOLERATIONS_ . runners : ... config : | [[ runners ]] [ runners . kubernetes ] node_tolerations_overwrite_allowed = &#34;.*&#34; In the .gitlab-ci.yml file, define one or more CI variables to overwrite CI job pod tolerations. variables : KUBERNETES_NODE_TOLERATIONS_1 : &#39;node-role.kubernetes.io/master:NoSchedule&#39; KUBERNETES_NODE_TOLERATIONS_2 : &#39;custom.toleration=value:NoSchedule&#39; KUBERNETES_NODE_TOLERATIONS_3 : &#39;empty.value=:PreferNoSchedule&#39; KUBERNETES_NODE_TOLERATIONS_4 : &#39;onlyKey&#39; KUBERNETES_NODE_TOLERATIONS_5 : &#39;&#39; # tolerate all taints Overwrite pod labels To overwrite Kubernetes pod labels for each CI/CD job: In the .config.yaml file, define a regular expression for pod_labels_overwrite_allowed . In the .gitlab-ci.yml file, set the KUBERNETES_POD_LABELS_* variables with values of key=value . The pod labels are overwritten to the key=value . You can apply multiple values: variables : KUBERNETES_POD_LABELS_1 : &#34;Key1=Val1&#34; KUBERNETES_POD_LABELS_2 : &#34;Key2=Val2&#34; KUBERNETES_POD_LABELS_3 : &#34;Key3=Val3&#34; Labels in the runner.gitlab.com namespace are read-only. GitLab ignores any attempts to add, modify, or remove these GitLab-internal labels. Overwrite pod annotations To overwrite Kubernetes pod annotations for each CI/CD job: In the .config.yaml file, define a regular expression for pod_annotations_overwrite_allowed . In the .gitlab-ci.yml file, set the KUBERNETES_POD_ANNOTATIONS_* variables and use key=value for the value. Pod annotations are overwritten to the key=value . You can specify multiple annotations: variables : KUBERNETES_POD_ANNOTATIONS_1 : &#34;Key1=Val1&#34; KUBERNETES_POD_ANNOTATIONS_2 : &#34;Key2=Val2&#34; KUBERNETES_POD_ANNOTATIONS_3 : &#34;Key3=Val3&#34; In the example below, the pod_annotations and the pod_annotations_overwrite_allowed are set. This configuration allows overwrite of any of the pod_annotations configured in the config.toml . [[ runners ]] # usual configuration executor = &#34;kubernetes&#34; [ runners . kubernetes ] image = &#34;alpine&#34; pod_annotations_overwrite_allowed = &#34;.*&#34; [ runners . kubernetes . pod_annotations ] &#34;Key1&#34; = &#34;Val1&#34; &#34;Key2&#34; = &#34;Val2&#34; &#34;Key3&#34; = &#34;Val3&#34; &#34;Key4&#34; = &#34;Val4&#34; Overwrite generated pod specifications Status : Beta This feature is in beta . We strongly recommend that you use this feature on a test Kubernetes cluster before you use it on a production cluster. To use this feature, you must enable the FF_USE_ADVANCED_POD_SPEC_CONFIGURATION feature flag . To add feedback before the feature is made generally available, leave a comment on issue 556286 . To modify the PodSpec generated by the runner manager, use the pod_spec setting in the config.toml file. For runner operator-specific configuration, see patch structure . The pod_spec setting: Overwrites and completes fields for the generated pod specification. Overwrites configuration values that might have been set in your config.toml under [runners.kubernetes] . You can configure multiple pod_spec settings. Setting Description name Name given to the custom pod_spec . patch_path Path to the file that defines the changes to apply to the final PodSpec object before it is generated. The file must be a JSON or YAML file. patch A JSON or YAML format string that describes the changes which must be applied to the final PodSpec object before it is generated. patch_type The strategy the runner uses to apply the specified changes to the PodSpec object generated by GitLab Runner. The accepted values are merge , json , and strategic . You cannot set the patch_path and patch in the same pod_spec configuration, otherwise an error occurs. Example of multiple pod_spec configurations in the config.toml : [[ runners ]] [ runners . kubernetes ] [[ runners . kubernetes . pod_spec ]] name = &#34;hostname&#34; patch = &#39;&#39;&#39; hostname: &#34;custom-pod-hostname&#34; &#39;&#39;&#39; patch_type = &#34;merge&#34; [[ runners . kubernetes . pod_spec ]] name = &#34;subdomain&#34; patch = &#39;&#39;&#39; subdomain: &#34;subdomain&#34; &#39;&#39;&#39; patch_type = &#34;strategic&#34; [[ runners . kubernetes . pod_spec ]] name = &#34;terminationGracePeriodSeconds&#34; patch = &#39;&#39;&#39; [{&#34;op&#34;: &#34;replace&#34;, &#34;path&#34;: &#34;/terminationGracePeriodSeconds&#34;, &#34;value&#34;: 60}] &#39;&#39;&#39; patch_type = &#34;json&#34; Merge patch strategy The merge patch strategy applies a key-value replacement on the existing PodSpec . If you use this strategy, the pod_spec configuration in the config.toml overwrites the values in the final PodSpec object before it is generated. Because the values are completely overwritten, you should use this patch strategy with caution. Example of a pod_spec configuration with the merge patch strategy: concurrent = 1 check_interval = 1 log_level = &#34;debug&#34; shutdown_timeout = 0 [ session_server ] session_timeout = 1800 [[ runners ]] name = &#34;&#34; url = &#34;https://gitlab.example.com&#34; id = 0 token = &#34;__REDACTED__&#34; token_obtained_at = 0001-01-01T00:00:00Z token_expires_at = 0001-01-01T00:00:00Z executor = &#34;kubernetes&#34; shell = &#34;bash&#34; environment = [ &#34;FF_USE_ADVANCED_POD_SPEC_CONFIGURATION=true&#34; , &#34;CUSTOM_VAR=value&#34; ] [ runners . kubernetes ] image = &#34;alpine&#34; ... [[ runners . kubernetes . pod_spec ]] name = &#34;build envvars&#34; patch = &#39;&#39;&#39; containers: - env: - name: env1 value: &#34;value1&#34; - name: env2 value: &#34;value2&#34; name: build &#39;&#39;&#39; patch_type = &#34;merge&#34; With this configuration, the final PodSpec has only one container called build with two environment variables env1 and env2 . The example above make the related CI Job failed as: The helper container specification is removed. The build container specification lost all necessary configuration set by GitLab Runner. To prevent the job from failing, in this example, the pod_spec must contain the untouched properties generated by GitLab Runner. JSON patch strategy The json patch strategy uses the JSON Patch specification to give control over the PodSpec objects and arrays to update. You cannot use this strategy on array properties. Example of a pod_spec configuration with the json patch strategy. In this configuration, a new key: value pair is added to the existing nodeSelector . The existing values are not overwritten. concurrent = 1 check_interval = 1 log_level = &#34;debug&#34; shutdown_timeout = 0 [ session_server ] session_timeout = 1800 [[ runners ]] name = &#34;&#34; url = &#34;https://gitlab.example.com&#34; id = 0 token = &#34;__REDACTED__&#34; token_obtained_at = 0001-01-01T00:00:00Z token_expires_at = 0001-01-01T00:00:00Z executor = &#34;kubernetes&#34; shell = &#34;bash&#34; environment = [ &#34;FF_USE_ADVANCED_POD_SPEC_CONFIGURATION=true&#34; , &#34;CUSTOM_VAR=value&#34; ] [ runners . kubernetes ] image = &#34;alpine&#34; ... [[ runners . kubernetes . pod_spec ]] name = &#34;val1 node&#34; patch = &#39;&#39;&#39; [{ &#34;op&#34;: &#34;add&#34;, &#34;path&#34;: &#34;/nodeSelector&#34;, &#34;value&#34;: { key1: &#34;val1&#34; } }] &#39;&#39;&#39; patch_type = &#34;json&#34; Strategic patch strategy This strategic patch strategy uses the existing patchStrategy applied to each field of the PodSpec object. Example of a pod_spec configuration with the strategic patch strategy. In this configuration, a resource request is set to on the build container. concurrent = 1 check_interval = 1 log_level = &#34;debug&#34; shutdown_timeout = 0 [ session_server ] session_timeout = 1800 [[ runners ]] name = &#34;&#34; url = &#34;https://gitlab.example.com&#34; id = 0 token = &#34;__REDACTED__&#34; token_obtained_at = 0001-01-01T00:00:00Z token_expires_at = 0001-01-01T00:00:00Z executor = &#34;kubernetes&#34; shell = &#34;bash&#34; environment = [ &#34;FF_USE_ADVANCED_POD_SPEC_CONFIGURATION=true&#34; , &#34;CUSTOM_VAR=value&#34; ] [ runners . kubernetes ] image = &#34;alpine&#34; ... [[ runners . kubernetes . pod_spec ]] name = &#34;cpu request 500m&#34; patch = &#39;&#39;&#39; containers: - name: build resources: requests: cpu: &#34;500m&#34; &#39;&#39;&#39; patch_type = &#34;strategic&#34; With this configuration, a resource request is set to on the build container. Best practices Test the added pod_spec in a test environment before deployment in a production environment. Make sure that the pod_spec configuration does not negatively impact the GitLab Runner generated specification. Do not use the merge patch strategy for complex pod specification updates. Where possible, use the config.toml when the configuration is available. For example, the following configuration replaces the first environment variables set by GitLab Runner by the one set in the custom pod_spec instead of adding the environment variable set to the existing list. concurrent = 1 check_interval = 1 log_level = &#34;debug&#34; shutdown_timeout = 0 [ session_server ] session_timeout = 1800 [[ runners ]] name = &#34;&#34; url = &#34;https://gitlab.example.com&#34; id = 0 token = &#34;__REDACTED__&#34; token_obtained_at = 0001-01-01T00:00:00Z token_expires_at = 0001-01-01T00:00:00Z executor = &#34;kubernetes&#34; shell = &#34;bash&#34; environment = [ &#34;FF_USE_ADVANCED_POD_SPEC_CONFIGURATION=true&#34; , &#34;CUSTOM_VAR=value&#34; ] [ runners . kubernetes ] image = &#34;alpine&#34; ... [[ runners . kubernetes . pod_spec ]] name = &#34;build envvars&#34; patch = &#39;&#39;&#39; containers: - env: - name: env1 value: &#34;value1&#34; name: build &#39;&#39;&#39; patch_type = &#34;strategic&#34; Create a PVC for each build job by modifying the Pod Spec To create a PersistentVolumeClaim for each build job make sure to check out how to enable the Pod Spec functionality . Kubernetes allows you to create an ephemeral PersistentVolumeClaim attached to a pod&rsquo;s lifecycle. This approach works if dynamic provisioning is enabled on your Kubernetes cluster. Each PVC can request a new Volume . The volume is also tied to the pod&rsquo;s lifecycle. After dynamic provisioning is enabled, the config.toml can be modified as follows to create an ephemeral PVC : [[ runners . kubernetes . pod_spec ]] name = &#34;ephemeral-pvc&#34; patch = &#39;&#39;&#39; containers: - name: build volumeMounts: - name: builds mountPath: /builds - name: helper volumeMounts: - name: builds mountPath: /builds volumes: - name: builds ephemeral: volumeClaimTemplate: spec: storageClassName: &lt;The Storage Class that will dynamically provision a Volume&gt; accessModes: [ ReadWriteOnce ] resources: requests: storage: 1Gi &#39;&#39;&#39; Set a security policy for the pod Configure the security context in the config.toml to set a security policy for the build pod. Use the following options: Option Type Required Description fs_group int No A special supplemental group that applies to all containers in a pod. run_as_group int No The GID to run the entry point of the container process. run_as_non_root boolean No Indicates that the container must run as a non-root user. run_as_user int No The UID to run the entry point of the container process. supplemental_groups int list No A list of groups applied to the first process run in each container, in addition to the container&rsquo;s primary GID. selinux_type string No The SELinux type label that applies to all containers in a pod. Example of a pod security context in the config.toml : concurrent = %( concurrent ) s check_interval = 30 [[ runners ]] name = &#34;myRunner&#34; url = &#34;gitlab.example.com&#34; executor = &#34;kubernetes&#34; [ runners . kubernetes ] helper_image = &#34;gitlab-registry.example.com/helper:latest&#34; [ runners . kubernetes . pod_security_context ] run_as_non_root = true run_as_user = 59417 run_as_group = 59417 fs_group = 59417 Remove old runner pods Sometimes old runner pods are not cleared. This can happen when the runner manager is incorrectly shut down. To handle this situation, you can use the GitLab Runner Pod Cleanup application to schedule cleanup of old pods. For more information, see: The GitLab Runner Pod Cleanup project README . GitLab Runner Pod Cleanup documentation . Set a security policy for the container Configure the container security context in the config.toml executor to set a container security policy for the build, helper, or service pods. Use the following options: Option Type Required Description run_as_group int No The GID to run the entry point of the container process. run_as_non_root boolean No Indicates that the container must run as a non-root user. run_as_user int No The UID to run the entry point of the container process. capabilities.add string list No The capabilities to add when running the container. capabilities.drop string list No The capabilities to drop when running the container. selinux_type string No The SELinux type label that is associated with the container process. In the following example in the config.toml , the security context configuration: Sets a pod security context. Overrides run_as_user and run_as_group for the build and helper containers. Specifies that all service containers inherit run_as_user and run_as_group from the pod security context. concurrent = 4 check_interval = 30 [[ runners ]] name = &#34;myRunner&#34; url = &#34;gitlab.example.com&#34; executor = &#34;kubernetes&#34; [ runners . kubernetes ] helper_image = &#34;gitlab-registry.example.com/helper:latest&#34; [ runners . kubernetes . pod_security_context ] run_as_non_root = true run_as_user = 59417 run_as_group = 59417 fs_group = 59417 [ runners . kubernetes . init_permissions_container_security_context ] run_as_user = 1000 run_as_group = 1000 [ runners . kubernetes . build_container_security_context ] run_as_user = 65534 run_as_group = 65534 [ runners . kubernetes . build_container_security_context . capabilities ] add = [ &#34;NET_ADMIN&#34; ] [ runners . kubernetes . helper_container_security_context ] run_as_user = 1000 run_as_group = 1000 [ runners . kubernetes . service_container_security_context ] run_as_user = 1000 run_as_group = 1000 Set a pull policy Use the pull_policy parameter in the config.toml file to specify a single or multiple pull policies. The policy controls how an image is fetched and updated, and applies to the build image, helper image, and any services. To determine which policy to use, see the Kubernetes documentation about pull policies . For a single pull policy: [ runners . kubernetes ] pull_policy = &#34;never&#34; For multiple pull policies: [ runners . kubernetes ] # use multiple pull policies pull_policy = [ &#34;always&#34; , &#34;if-not-present&#34; ] When you define multiple policies, each policy is attempted until the image is obtained successfully. For example, when you use [ always, if-not-present ] , the policy if-not-present is used if the always policy fails due to a temporary registry problem. To retry a failed pull: [ runners . kubernetes ] pull_policy = [ &#34;always&#34; , &#34;always&#34; ] The GitLab naming convention is different to the Kubernetes one. Runner pull policy Kubernetes pull policy Description none none Uses the default policy, as specified by Kubernetes. if-not-present IfNotPresent The image is pulled only if it is not already present on the node that executes the job. Review the security considerations before you use this pull policy. always Always The image is pulled every time the job is executed. never Never The image is never pulled and requires the node to already have it. Specify container capabilities You can specify the Kubernetes capabilities to use in the container. To specify the container capabilities, use the cap_add and cap_drop options in the config.toml . Container runtimes can also define a default list of capabilities, like those in Docker or the container . There is a list of capabilities that the runner drops by default. Capabilities that you list in cap_add option are excluded from being dropped. Example configuration in the config.toml file: concurrent = 1 check_interval = 30 [[ runners ]] name = &#34;myRunner&#34; url = &#34;gitlab.example.com&#34; executor = &#34;kubernetes&#34; [ runners . kubernetes ] # ... cap_add = [ &#34;SYS_TIME&#34; , &#34;IPC_LOCK&#34; ] cap_drop = [ &#34;SYS_ADMIN&#34; ] # ... When you specify the capabilities: User-defined cap_drop has priority over user-defined cap_add . If you define the same capability in both settings, only the capability from cap_drop is passed to the container. Remove the CAP_ prefix from capability identifiers passed to the container configuration. For example, if you want to add or drop the CAP_SYS_TIME capability, in the configuration file, enter the string, SYS_TIME . The owner of the Kubernetes cluster can define a PodSecurityPolicy , where specific capabilities are allowed, restricted, or added by default. These rules take precedence over any user-defined configuration. Overwrite container resources You can overwrite Kubernetes CPU and memory allocations for each CI/CD job. You can apply settings for requests and limits for the build, helper, and service containers. To overwrite container resources, use the following variables in the .gitlab-ci.yml file. The values for the variables are restricted to the maximum overwrite setting for that resource. If the maximum overwrite has not been set for a resource, the variable is not used. variables : KUBERNETES_CPU_REQUEST : &#34;3&#34; KUBERNETES_CPU_LIMIT : &#34;5&#34; KUBERNETES_MEMORY_REQUEST : &#34;2Gi&#34; KUBERNETES_MEMORY_LIMIT : &#34;4Gi&#34; KUBERNETES_EPHEMERAL_STORAGE_REQUEST : &#34;512Mi&#34; KUBERNETES_EPHEMERAL_STORAGE_LIMIT : &#34;1Gi&#34; KUBERNETES_HELPER_CPU_REQUEST : &#34;3&#34; KUBERNETES_HELPER_CPU_LIMIT : &#34;5&#34; KUBERNETES_HELPER_MEMORY_REQUEST : &#34;2Gi&#34; KUBERNETES_HELPER_MEMORY_LIMIT : &#34;4Gi&#34; KUBERNETES_HELPER_EPHEMERAL_STORAGE_REQUEST : &#34;512Mi&#34; KUBERNETES_HELPER_EPHEMERAL_STORAGE_LIMIT : &#34;1Gi&#34; KUBERNETES_SERVICE_CPU_REQUEST : &#34;3&#34; KUBERNETES_SERVICE_CPU_LIMIT : &#34;5&#34; KUBERNETES_SERVICE_MEMORY_REQUEST : &#34;2Gi&#34; KUBERNETES_SERVICE_MEMORY_LIMIT : &#34;4Gi&#34; KUBERNETES_SERVICE_EPHEMERAL_STORAGE_REQUEST : &#34;512Mi&#34; KUBERNETES_SERVICE_EPHEMERAL_STORAGE_LIMIT : &#34;1Gi&#34; Define a list of services History Introduced support for HEALTCHECK_TCP_SERVICES in GitLab Runner 16.9. Define a list of services in the config.toml . concurrent = 1 check_interval = 30 [[ runners ]] name = &#34;myRunner&#34; url = &#34;gitlab.example.com&#34; executor = &#34;kubernetes&#34; [ runners . kubernetes ] helper_image = &#34;gitlab-registy.example.com/helper:latest&#34; [[ runners . kubernetes . services ]] name = &#34;postgres:12-alpine&#34; alias = &#34;db1&#34; [[ runners . kubernetes . services ]] name = &#34;registry.example.com/svc1&#34; alias = &#34;svc1&#34; entrypoint = [ &#34;entrypoint.sh&#34; ] command = [ &#34;executable&#34; , &#34;param1&#34; , &#34;param2&#34; ] environment = [ &#34;ENV=value1&#34; , &#34;ENV2=value2&#34; ] If the service environment includes HEALTHCHECK_TCP_PORT , GitLab Runner waits until the service responds on that port before starting user CI scripts. You can also configure the HEALTHCHECK_TCP_PORT environment variable in a services section of .gitlab-ci.yml . Overwrite service containers resources If a job has multiple service containers, you can set explicit resource requests and limits to each service container. Use the variables attribute in each service to overwrite container resources specified in .gitlab-ci.yml . services : - name : redis:5 alias : redis5 variables : KUBERNETES_SERVICE_CPU_REQUEST : &#34;3&#34; KUBERNETES_SERVICE_CPU_LIMIT : &#34;6&#34; KUBERNETES_SERVICE_MEMORY_REQUEST : &#34;3Gi&#34; KUBERNETES_SERVICE_MEMORY_LIMIT : &#34;6Gi&#34; KUBERNETES_EPHEMERAL_STORAGE_REQUEST : &#34;2Gi&#34; KUBERNETES_EPHEMERAL_STORAGE_LIMIT : &#34;3Gi&#34; - name : postgres:12 alias : MY_relational-database.12 variables : KUBERNETES_CPU_REQUEST : &#34;2&#34; KUBERNETES_CPU_LIMIT : &#34;4&#34; KUBERNETES_MEMORY_REQUEST : &#34;1Gi&#34; KUBERNETES_MEMORY_LIMIT : &#34;2Gi&#34; KUBERNETES_EPHEMERAL_STORAGE_REQUEST : &#34;1Gi&#34; KUBERNETES_EPHEMERAL_STORAGE_LIMIT : &#34;2Gi&#34; These specific settings take precedence over the general settings for the job. The values are still restricted to the maximum overwrite setting for that resource. Overwrite the Kubernetes default service account To overwrite the Kubernetes service account for each CI/CD job in the .gitlab-ci.yml file, set the variable KUBERNETES_SERVICE_ACCOUNT_OVERWRITE . You can use this variable to specify a service account attached to the namespace, which you may need for complex RBAC configurations. variables : KUBERNETES_SERVICE_ACCOUNT_OVERWRITE : ci-service-account To ensure only designated service accounts are used during CI runs, define a regular expression for either: The service_account_overwrite_allowed setting. The KUBERNETES_SERVICE_ACCOUNT_OVERWRITE_ALLOWED environment variable. If you don&rsquo;t set either, the overwrite is disabled. Set the RuntimeClass Use runtime_class_name to set the RuntimeClass for each job container. If you specify a RuntimeClass name but did not configure it in the cluster, or the feature is not supported, the executor fails to create jobs. concurrent = 1 check_interval = 30 [[ runners ]] name = &#34;myRunner&#34; url = &#34;gitlab.example.com&#34; executor = &#34;kubernetes&#34; [ runners . kubernetes ] runtime_class_name = &#34;myclass&#34; Change the base directory for build logs and scripts History Introduced in GitLab Runner 17.2. You can change the directory where emptyDir volumes are mounted to the pod for build logs and scripts. You can use the directory to: Run job pods with a modified image. Run as an unprivileged user. Customize SecurityContext settings. To change the directory: For build logs, set logs_base_dir . For build scripts, set scripts_base_dir . The expected value is a string that represents a base directory without the trailing slash (for example, /tmp or /mydir/example ). The directory must already exist . This value is prepended to the generated path for build logs and scripts. For example: [[ runners ]] name = &#34;myRunner&#34; url = &#34;gitlab.example.com&#34; executor = &#34;kubernetes&#34; [ runners . kubernetes ] logs_base_dir = &#34;/tmp&#34; scripts_base_dir = &#34;/tmp&#34; This configuration would result in an emptyDir volume mounted in: /tmp/logs-${CI_PROJECT_ID}-${CI_JOB_ID} for build logs instead of the default /logs-${CI_PROJECT_ID}-${CI_JOB_ID} . /tmp/scripts-${CI_PROJECT_ID}-${CI_JOB_ID} for build scripts. User namespaces In Kubernetes 1.30 and later, you can isolate the user running in the container from the one on the host with user namespaces . A process running as root in the container can run as a different unprivileged user on the host. With user namespaces, you can have more control over which images are used to run your CI/CD jobs. Operations that require additional settings (such as running as root) can also function without opening up additional attack surface on the host. To use this feature, ensure your cluster has been properly configured . The following example adds pod_spec for the hostUsers key and disables both privileged pods and privilege escalation: [[ runners ]] environment = [ &#34;FF_USE_ADVANCED_POD_SPEC_CONFIGURATION=true&#34; ] builds_dir = &#34;/tmp/builds&#34; [ runners . kubernetes ] logs_base_dir = &#34;/tmp&#34; scripts_base_dir = &#34;/tmp&#34; privileged = false allowPrivilegeEscalation = false [[ runners . kubernetes . pod_spec ]] name = &#34;hostUsers&#34; patch = &#39;&#39;&#39; [{&#34;op&#34;: &#34;add&#34;, &#34;path&#34;: &#34;/hostUsers&#34;, &#34;value&#34;: false}] &#39;&#39;&#39; patch_type = &#34;json&#34; With user namespaces, you cannot use the default path for the build directory ( builds_dir ), build logs ( logs_base_dir ), or build scripts ( scripts_base_dir ). Even the container&rsquo;s root user does not have the permission to mount volumes. They also cannot create directories in the root of the container&rsquo;s file system. Instead, you can change the base directory for build logs and scripts . You can also change the build directory by setting [[runners]].builds_dir . Operating system, architecture, and Windows kernel version GitLab Runner with the Kubernetes executor can run builds on different operating systems if the configured cluster has nodes running those operating systems. The system determines the helper image&rsquo;s operating system, architecture, and Windows kernel version (if applicable). It then uses those parameters for other aspects of the build, for example the containers or images to use. The following diagram explains how the system detects these details: %%|fig-align: center flowchart TB init[&lt;b&gt;Initial defaults&lt;/b&gt;:&lt;br/&gt;OS: linux&lt;/br&gt;Arch: amd64] hasAutoset{Configuration&lt;br/&gt;&lt;tt&gt;&lt;a href=&#34;https://docs.gitlab.com/runner/configuration/advanced-configuration/&#34;&gt;helper_image_autoset_arch_and_os&lt;/a&gt; == true&lt;/tt&gt;?} setArch[&lt;b&gt;Update&lt;/b&gt;:&lt;br/&gt;Arch: &lt;i&gt;same as runner&lt;/i&gt;] isWin{GitLab Runner runs on Windows?} setWin[&lt;b&gt;Update&lt;/b&gt;:&lt;br/&gt;OS: windows&lt;br/&gt;KernelVersion: &lt;i&gt;same as runner&lt;/i&gt;] hasNodeSel{&lt;a href=&#34;https://docs.gitlab.com/runner/configuration/advanced-configuration/&#34;&gt;&lt;tt&gt;node_selector&lt;/tt&gt;&lt;/a&gt; configured&lt;br/&gt;in &lt;tt&gt;runners.kubernetes&lt;/tt&gt; section?} hasNodeSelOverride{&lt;tt&gt;node_selector&lt;/tt&gt; configured&lt;br/&gt;&lt;a href=&#34;https://docs.gitlab.com/runner/executors/kubernetes/#overwrite-the-node-selector&#34;&gt;as overwrite&lt;/a&gt;?} updateNodeSel[&lt;b&gt;Update from &lt;tt&gt;node_selector&lt;/tt&gt;&lt;/b&gt; if set:&lt;br/&gt;OS: from &lt;tt&gt;kubernetes.io/os&lt;/tt&gt;&lt;br/&gt;Arch: from &lt;tt&gt;kubernetes.io/arch&lt;/tt&gt;&lt;br/&gt;KernelVersion: from &lt;tt&gt;node.kubernetes.io/windows-build&lt;/tt&gt;] updateNodeSelOverride[&lt;b&gt;Update from &lt;tt&gt;node_selector&lt;/tt&gt; overwrites&lt;/b&gt; if set:&lt;/br&gt;OS: from &lt;tt&gt;kubernetes.io/os&lt;/tt&gt;&lt;br/&gt;Arch: from &lt;tt&gt;kubernetes.io/arch&lt;/tt&gt;&lt;br/&gt;KernelVersion: from &lt;tt&gt;node.kubernetes.io/windows-build&lt;/tt&gt;] result[final &lt;b&gt;OS&lt;/b&gt;, &lt;b&gt;Arch&lt;/b&gt;, &lt;b&gt;kernelVersion&lt;/b&gt;] init --&gt; hasAutoset hasAutoset --&gt;|false | hasNodeSel hasAutoset --&gt;|true | setArch setArch --&gt; isWin isWin --&gt;|false | hasNodeSel isWin --&gt;|true | setWin setWin --&gt; hasNodeSel hasNodeSel --&gt;|false | hasNodeSelOverride hasNodeSel --&gt;|true | updateNodeSel updateNodeSel --&gt; hasNodeSelOverride hasNodeSelOverride --&gt;|false | result hasNodeSelOverride --&gt;|true | updateNodeSelOverride updateNodeSelOverride --&gt; result The following are the only parameters that influence the operating system, architecture, and Windows kernel version selection of the build. The helper_image_autoset_arch_and_os configuration The kubernetes.io/os , kubernetes.io/arch , and node.kubernetes.io/windows-build label selectors from: node_selector configuration node_selector overwrites Other parameters don&rsquo;t influence the selection process described above. However, you can use parameters like affinity to further limit the nodes on which builds are scheduled. Nodes Specify the node to execute builds Use the node_selector option to specify which node in a Kubernetes cluster can be used to execute the builds. It is a key=value pair in string=string format ( string:string in the case of environment variables). Runner uses the information provided to determine the operating system and architecture for the build. This ensures that the correct helper image is used. The default operating system and architecture is linux/amd64 . You can use specific labels to schedule nodes with different operating systems and architectures. Example for linux/arm64 [[ runners ]] name = &#34;myRunner&#34; url = &#34;gitlab.example.com&#34; executor = &#34;kubernetes&#34; [ runners . kubernetes . node_selector ] &#34;kubernetes.io/arch&#34; = &#34;arm64&#34; &#34;kubernetes.io/os&#34; = &#34;linux&#34; Example for windows/amd64 Kubernetes for Windows has certain limitations . If you are using process isolation, you must also provide the specific Windows build version with the node.kubernetes.io/windows-build label. [[ runners ]] name = &#34;myRunner&#34; url = &#34;gitlab.example.com&#34; executor = &#34;kubernetes&#34; # The FF_USE_POWERSHELL_PATH_RESOLVER feature flag has to be enabled for PowerShell # to resolve paths for Windows correctly when Runner is operating in a Linux environment # but targeting Windows nodes. environment = [ &#34;FF_USE_POWERSHELL_PATH_RESOLVER=true&#34; ] [ runners . kubernetes . node_selector ] &#34;kubernetes.io/arch&#34; = &#34;amd64&#34; &#34;kubernetes.io/os&#34; = &#34;windows&#34; &#34;node.kubernetes.io/windows-build&#34; = &#34;10.0.20348&#34; Overwrite the node selector To overwrite the node selector: In the config.toml or Helm values.yaml file, enable overwriting of the node selector: runners : ... config : | [[ runners ]] [ runners . kubernetes ] node_selector_overwrite_allowed = &#34;.*&#34; In the .gitlab-ci.yml file, define the variable to overwrite the node selector: variables : KUBERNETES_NODE_SELECTOR_* = &#39;&#39; In the following example, to overwrite the Kubernetes node architecture, the settings are configured in the config.toml and .gitlab-ci.yml file: `config.toml` concurrent = 1 check_interval = 1 log_level = &#34;debug&#34; shutdown_timeout = 0 listen_address = &#39;:9252&#39; [ session_server ] session_timeout = 1800 [[ runners ]] name = &#34;&#34; url = &#34;https://gitlab.com/&#34; id = 0 token = &#34;__REDACTED__&#34; token_obtained_at = &#34;0001-01-01T00:00:00Z&#34; token_expires_at = &#34;0001-01-01T00:00:00Z&#34; executor = &#34;kubernetes&#34; shell = &#34;bash&#34; [ runners . kubernetes ] host = &#34;&#34; bearer_token_overwrite_allowed = false image = &#34;alpine&#34; namespace = &#34;&#34; namespace_overwrite_allowed = &#34;&#34; pod_labels_overwrite_allowed = &#34;&#34; service_account_overwrite_allowed = &#34;&#34; pod_annotations_overwrite_allowed = &#34;&#34; node_selector_overwrite_allowed = &#34;kubernetes.io/arch=.*&#34; # &lt;--- allows overwrite of the architecture `.gitlab-ci.yml` job : image : IMAGE_NAME variables : KUBERNETES_NODE_SELECTOR_ARCH : &#39;kubernetes.io/arch=amd64&#39; # &lt;--- select the architecture Define a list of node affinities Define a list of node affinities to add to a pod specification at build time. node_affinities does not determine which operating system a build should run with, only node_selectors . For more information, see Operating system, architecture, and Windows kernel version . Example configuration in the config.toml : concurrent = 1 [[ runners ]] name = &#34;myRunner&#34; url = &#34;gitlab.example.com&#34; executor = &#34;kubernetes&#34; [ runners . kubernetes ] [ runners . kubernetes . affinity ] [ runners . kubernetes . affinity . node_affinity ] [[ runners . kubernetes . affinity . node_affinity . preferred_during_scheduling_ignored_during_execution ]] weight = 100 [ runners . kubernetes . affinity . node_affinity . preferred_during_scheduling_ignored_during_execution . preference ] [[ runners . kubernetes . affinity . node_affinity . preferred_during_scheduling_ignored_during_execution . preference . match_expressions ]] key = &#34;cpu_speed&#34; operator = &#34;In&#34; values = [ &#34;fast&#34; ] [[ runners . kubernetes . affinity . node_affinity . preferred_during_scheduling_ignored_during_execution . preference . match_expressions ]] key = &#34;mem_speed&#34; operator = &#34;In&#34; values = [ &#34;fast&#34; ] [[ runners . kubernetes . affinity . node_affinity . preferred_during_scheduling_ignored_during_execution ]] weight = 50 [ runners . kubernetes . affinity . node_affinity . preferred_during_scheduling_ignored_during_execution . preference ] [[ runners . kubernetes . affinity . node_affinity . preferred_during_scheduling_ignored_during_execution . preference . match_expressions ]] key = &#34;core_count&#34; operator = &#34;In&#34; values = [ &#34;high&#34; , &#34;32&#34; ] [[ runners . kubernetes . affinity . node_affinity . preferred_during_scheduling_ignored_during_execution . preference . match_fields ]] key = &#34;cpu_type&#34; operator = &#34;In&#34; values = [ &#34;arm64&#34; ] [ runners . kubernetes . affinity . node_affinity . required_during_scheduling_ignored_during_execution ] [[ runners . kubernetes . affinity . node_affinity . required_during_scheduling_ignored_during_execution . node_selector_terms ]] [[ runners . kubernetes . affinity . node_affinity . required_during_scheduling_ignored_during_execution . node_selector_terms . match_expressions ]] key = &#34;kubernetes.io/e2e-az-name&#34; operator = &#34;In&#34; values = [ &#34;e2e-az1&#34; , &#34;e2e-az2&#34; ] Define nodes where pods are scheduled Use pod affinity and anti-affinity to constrain the nodes your pod is eligible to be scheduled on, based on labels on other pods. Example configuration in the config.toml : concurrent = 1 [[ runners ]] name = &#34;myRunner&#34; url = &#34;gitlab.example.com&#34; executor = &#34;kubernetes&#34; [ runners . kubernetes ] [ runners . kubernetes . affinity ] [ runners . kubernetes . affinity . pod_affinity ] [[ runners . kubernetes . affinity . pod_affinity . required_during_scheduling_ignored_during_execution ]] topology_key = &#34;failure-domain.beta.kubernetes.io/zone&#34; namespaces = [ &#34;namespace_1&#34; , &#34;namespace_2&#34; ] [ runners . kubernetes . affinity . pod_affinity . required_during_scheduling_ignored_during_execution . label_selector ] [[ runners . kubernetes . affinity . pod_affinity . required_during_scheduling_ignored_during_execution . label_selector . match_expressions ]] key = &#34;security&#34; operator = &#34;In&#34; values = [ &#34;S1&#34; ] [[ runners . kubernetes . affinity . pod_affinity . preferred_during_scheduling_ignored_during_execution ]] weight = 100 [ runners . kubernetes . affinity . pod_affinity . preferred_during_scheduling_ignored_during_execution . pod_affinity_term ] topology_key = &#34;failure-domain.beta.kubernetes.io/zone&#34; [ runners . kubernetes . affinity . pod_affinity . preferred_during_scheduling_ignored_during_execution . pod_affinity_term . label_selector ] [[ runners . kubernetes . affinity . pod_affinity . preferred_during_scheduling_ignored_during_execution . pod_affinity_term . label_selector . match_expressions ]] key = &#34;security_2&#34; operator = &#34;In&#34; values = [ &#34;S2&#34; ] [ runners . kubernetes . affinity . pod_anti_affinity ] [[ runners . kubernetes . affinity . pod_anti_affinity . required_during_scheduling_ignored_during_execution ]] topology_key = &#34;failure-domain.beta.kubernetes.io/zone&#34; namespaces = [ &#34;namespace_1&#34; , &#34;namespace_2&#34; ] [ runners . kubernetes . affinity . pod_anti_affinity . required_during_scheduling_ignored_during_execution . label_selector ] [[ runners . kubernetes . affinity . pod_anti_affinity . required_during_scheduling_ignored_during_execution . label_selector . match_expressions ]] key = &#34;security&#34; operator = &#34;In&#34; values = [ &#34;S1&#34; ] [ runners . kubernetes . affinity . pod_anti_affinity . required_during_scheduling_ignored_during_execution . namespace_selector ] [[ runners . kubernetes . affinity . pod_anti_affinity . required_during_scheduling_ignored_during_execution . namespace_selector . match_expressions ]] key = &#34;security&#34; operator = &#34;In&#34; values = [ &#34;S1&#34; ] [[ runners . kubernetes . affinity . pod_anti_affinity . preferred_during_scheduling_ignored_during_execution ]] weight = 100 [ runners . kubernetes . affinity . pod_anti_affinity . preferred_during_scheduling_ignored_during_execution . pod_affinity_term ] topology_key = &#34;failure-domain.beta.kubernetes.io/zone&#34; [ runners . kubernetes . affinity . pod_anti_affinity . preferred_during_scheduling_ignored_during_execution . pod_affinity_term . label_selector ] [[ runners . kubernetes . affinity . pod_anti_affinity . preferred_during_scheduling_ignored_during_execution . pod_affinity_term . label_selector . match_expressions ]] key = &#34;security_2&#34; operator = &#34;In&#34; values = [ &#34;S2&#34; ] [ runners . kubernetes . affinity . pod_anti_affinity . preferred_during_scheduling_ignored_during_execution . pod_affinity_term . namespace_selector ] [[ runners . kubernetes . affinity . pod_anti_affinity . preferred_during_scheduling_ignored_during_execution . pod_affinity_term . namespace_selector . match_expressions ]] key = &#34;security_2&#34; operator = &#34;In&#34; values = [ &#34;S2&#34; ] Networking Configure a container lifecycle hook Use container lifecycle hooks to run code configured for a handler when the corresponding lifecycle hook is executed. You can configure two types of hooks: PreStop and PostStart . Each of them allows only one type of handler to be set. Example configuration in the config.toml file: [[ runners ]] name = &#34;kubernetes&#34; url = &#34;https://gitlab.example.com/&#34; executor = &#34;kubernetes&#34; token = &#34;yrnZW46BrtBFqM7xDzE7dddd&#34; [ runners . kubernetes ] image = &#34;alpine:3.11&#34; privileged = true namespace = &#34;default&#34; [ runners . kubernetes . container_lifecycle . post_start . exec ] command = [ &#34;touch&#34; , &#34;/builds/postStart.txt&#34; ] [ runners . kubernetes . container_lifecycle . pre_stop . http_get ] port = 8080 host = &#34;localhost&#34; path = &#34;/test&#34; [[ runners . kubernetes . container_lifecycle . pre_stop . http_get . http_headers ]] name = &#34;header_name_1&#34; value = &#34;header_value_1&#34; [[ runners . kubernetes . container_lifecycle . pre_stop . http_get . http_headers ]] name = &#34;header_name_2&#34; value = &#34;header_value_2&#34; Use the following settings to configure each lifecycle hook: Option Type Required Description exec KubernetesLifecycleExecAction No Exec specifies the action to take. http_get KubernetesLifecycleHTTPGet No HTTPGet specifies the http request to perform. tcp_socket KubernetesLifecycleTcpSocket No TCPsocket specifies an action involving a TCP port. KubernetesLifecycleExecAction Option Type Required Description command string list Yes The command line to execute inside the container. KubernetesLifecycleHTTPGet Option Type Required Description port int Yes The number of the port to access on the container. host string No The host name to connect to, defaults to the pod IP (optional). path string No The path to access on the HTTP server (optional). scheme string No The scheme used for connecting to the host. Defaults to HTTP (optional). http_headers KubernetesLifecycleHTTPGetHeader list No Custom headers to set in the request (optional). KubernetesLifecycleHTTPGetHeader Option Type Required Description name string Yes HTTP header name. value string Yes HTTP header value. KubernetesLifecycleTcpSocket Option Type Required Description port int Yes The number of the port to access on the container. host string No The host name to connect to, defaults to the pod IP (optional). Configure pod DNS settings Use the following options to configure the DNS settings of the pods. Option Type Required Description nameservers string list No A list of IP addresses that are used as DNS servers for the pod. options KubernetesDNSConfigOption No A optional list of objects where each object may have a name property (required) and a value property (optional). searches string lists No A list of DNS search domains for hostname lookup in the pod. Example configuration in the config.toml file: concurrent = 1 check_interval = 30 [[ runners ]] name = &#34;myRunner&#34; url = &#34;https://gitlab.example.com&#34; token = &#34;__REDACTED__&#34; executor = &#34;kubernetes&#34; [ runners . kubernetes ] image = &#34;alpine:latest&#34; [ runners . kubernetes . dns_config ] nameservers = [ &#34;1.2.3.4&#34; , ] searches = [ &#34;ns1.svc.cluster-domain.example&#34; , &#34;my.dns.search.suffix&#34; , ] [[ runners . kubernetes . dns_config . options ]] name = &#34;ndots&#34; value = &#34;2&#34; [[ runners . kubernetes . dns_config . options ]] name = &#34;edns0&#34; KubernetesDNSConfigOption Option Type Required Description name string Yes Configuration option name. value *string No Configuration option value. Default list of dropped capabilities GitLab Runner drops the following capabilities by default. User-defined cap_add has priority over the default list of dropped capabilities. If you want to add the capability that is dropped by default, add it to cap_add . NET_RAW Add extra host aliases This feature is available in Kubernetes 1.7 and higher. Configure a host aliases to instruct Kubernetes to add entries to /etc/hosts file in the container. Use the following options: Option Type Required Description IP string Yes The IP address you want to attach hosts to. Hostnames string list Yes A list of host name aliases that are attached to the IP. Example configuration in the config.toml file: concurrent = 4 [[ runners ]] # usual configuration executor = &#34;kubernetes&#34; [ runners . kubernetes ] [[ runners . kubernetes . host_aliases ]] ip = &#34;127.0.0.1&#34; hostnames = [ &#34;web1&#34; , &#34;web2&#34; ] [[ runners . kubernetes . host_aliases ]] ip = &#34;192.168.1.1&#34; hostnames = [ &#34;web14&#34; , &#34;web15&#34; ] You can also configure host aliases by using the command-line parameter --kubernetes-host_aliases with JSON input. For example: gitlab-runner register --kubernetes-host_aliases &#39;[{&#34;ip&#34;:&#34;192.168.1.100&#34;,&#34;hostnames&#34;:[&#34;myservice.local&#34;]},{&#34;ip&#34;:&#34;192.168.1.101&#34;,&#34;hostnames&#34;:[&#34;otherservice.local&#34;]}]&#39; Volumes Using the cache with the Kubernetes executor When the cache is used with the Kubernetes executor, a volume called /cache is mounted on the pod. During job execution, if cached data is needed, the runner checks if cached data is available. Cached data is available if a compressed file is available on the cache volume. To set the cache volume, use the cache_dir setting in the config.toml file. If available, the compressed file is extracted into the build folder and can then be used in the job. If not available, the cached data is downloaded from the configured storage and saved into the cache dir as a compressed file. The compressed file is then extracted into the build folder. Configure volume types You can mount the following volume types: hostPath persistentVolumeClaim configMap secret emptyDir csi Example of a configuration with multiple volume types: concurrent = 4 [[ runners ]] # usual configuration executor = &#34;kubernetes&#34; [ runners . kubernetes ] [[ runners . kubernetes . volumes . host_path ]] name = &#34;hostpath-1&#34; mount_path = &#34;/path/to/mount/point&#34; read_only = true host_path = &#34;/path/on/host&#34; [[ runners . kubernetes . volumes . host_path ]] name = &#34;hostpath-2&#34; mount_path = &#34;/path/to/mount/point_2&#34; read_only = true [[ runners . kubernetes . volumes . pvc ]] name = &#34;pvc-1&#34; mount_path = &#34;/path/to/mount/point1&#34; [[ runners . kubernetes . volumes . config_map ]] name = &#34;config-map-1&#34; mount_path = &#34;/path/to/directory&#34; [ runners . kubernetes . volumes . config_map . items ] &#34;key_1&#34; = &#34;relative/path/to/key_1_file&#34; &#34;key_2&#34; = &#34;key_2&#34; [[ runners . kubernetes . volumes . secret ]] name = &#34;secrets&#34; mount_path = &#34;/path/to/directory1&#34; read_only = true [ runners . kubernetes . volumes . secret . items ] &#34;secret_1&#34; = &#34;relative/path/to/secret_1_file&#34; [[ runners . kubernetes . volumes . empty_dir ]] name = &#34;empty-dir&#34; mount_path = &#34;/path/to/empty_dir&#34; medium = &#34;Memory&#34; [[ runners . kubernetes . volumes . csi ]] name = &#34;csi-volume&#34; mount_path = &#34;/path/to/csi/volume&#34; driver = &#34;my-csi-driver&#34; [ runners . kubernetes . volumes . csi . volume_attributes ] size = &#34;2Gi&#34; hostPath volume Configure the hostPath volume to instruct Kubernetes to mount a specified host path in the container. Use the following options in the config.toml file: Option Type Required Description name string Yes The name of the volume. mount_path string Yes The path where the volume is mounted in the container. sub_path string No The sub-path inside the mounted volume instead of its root. host_path string No The path on the host mounted as a volume. If you don&rsquo;t specify a value, it defaults to the same path as mount_path . read_only boolean No Sets the volume in read-only mode. Defaults to false . mount_propagation string No Share mounted volumes between containers. For more information, see Mount Propagation . persistentVolumeClaim volume Configure the persistentVolumeClaim volume to instruct Kubernetes to use a persistentVolumeClaim defined in a Kubernetes cluster and mount it in the container. Use the following options in the config.toml file: Option Type Required Description name string Yes The name of the volume and at the same time the name of PersistentVolumeClaim that should be used. Supports variables. For more information, see Persistent per-concurrency build volumes . mount_path string Yes Path in the container where the volume is mounted. read_only boolean No Sets the volume to read-only mode (defaults to false). sub_path string No Mount a sub-path in the volume instead of the root. mount_propagation string No Set the mount propagation mode for the volume. For more details, see Kubernetes mount propagation . configMap volume Configure the configMap volume to instruct Kubernetes to use a configMap defined in a Kubernetes cluster and mount it in the container. Use the following options in the config.toml : Option Type Required Description name string Yes The name of the volume and at the same time the name of configMap that should be used. mount_path string Yes Path in the container where the volume is mounted. read_only boolean No Sets the volume to read-only mode (defaults to false). sub_path string No Mount a sub-path in the volume instead of the root. items map[string]string no Key-to-path mapping for keys from the configMap that should be used. Each key from the configMap is changed into a file and stored in the mount path. By default: All keys are included. The configMap key is used as the filename. The value is stored in the file contents. To change the default key and value storage, use the items option. If you use the items option, only specified keys are added to the volumes and all other keys are skipped. If you use a key that doesn&rsquo;t exist, the job fails on the pod creation stage. secret volume Configure a secret volume to instruct Kubernetes to use a secret defined in a Kubernetes cluster and mount it in the container. Use the following options in the config.toml file: Option Type Required Description name string Yes The name of the volume and at the same time the name of secret that should be used. mount_path string Yes Path inside of container where the volume should be mounted. read_only boolean No Sets the volume in read-only mode (defaults to false). sub_path string No Mount a sub-path in the volume instead of the root. items map[string]string No Key-to-path mapping for keys from the configMap that should be used. Each key from selected secret is changed into a file stored in the selected mount path. By default: All keys are included. The configMap key is used as the filename. The value is stored in the file contents. To change default key and value storage, use the items option. If you use the items option, only specified keys are added to the volumes and all other keys are skipped. If you use a key that doesn&rsquo;t exist, the job fails on the pod creation stage. emptyDir volume Configure an emptyDir volume to instruct Kubernetes to mount an empty directory in the container. Use the following options in the config.toml file: Option Type Required Description name string Yes The name of the volume. mount_path string Yes Path inside of container where the volume should be mounted. sub_path string No Mount a sub-path in the volume instead of the root. medium string No &ldquo;Memory&rdquo; provides a tmpfs , otherwise it defaults to the node disk storage (defaults to &ldquo;&rdquo;). size_limit string No The total amount of local storage required for the emptyDir volume. csi volume Configure a Container Storage Interface ( csi ) volume to instruct Kubernetes to use a custom csi driver to mount an arbitrary storage system in the container. Use the following options in the config.toml : Option Type Required Description name string Yes The name of the volume. mount_path string Yes Path inside of container where the volume should be mounted. driver string Yes A string value that specifies the name of the volume driver to use. fs_type string No A string value that specifies the name of the file system type (for example, ext4 , xfs , ntfs ). volume_attributes map[string]string No Key-value pair mapping for attributes of the csi volume. sub_path string No Mount a sub-path in the volume instead of the root. read_only boolean No Sets the volume in read-only mode (defaults to false). Mount volumes on service containers Volumes defined for the build container are also automatically mounted for all services containers. You can use this functionality as an alternative to services_tmpfs (available only to Docker executor), to mount database storage in RAM to speed up tests. Example configuration in the config.toml file: [[ runners ]] # usual configuration executor = &#34;kubernetes&#34; [ runners . kubernetes ] [[ runners . kubernetes . volumes . empty_dir ]] name = &#34;mysql-tmpfs&#34; mount_path = &#34;/var/lib/mysql&#34; medium = &#34;Memory&#34; Custom volume mount To store the builds directory for the job, define custom volume mounts to the configured builds_dir ( /builds by default). If you use pvc volumes , based on the access mode , you might be limited to running jobs on one node. Example configuration in the config.toml file: concurrent = 4 [[ runners ]] # usual configuration executor = &#34;kubernetes&#34; builds_dir = &#34;/builds&#34; [ runners . kubernetes ] [[ runners . kubernetes . volumes . empty_dir ]] name = &#34;repo&#34; mount_path = &#34;/builds&#34; medium = &#34;Memory&#34; Persistent per-concurrency build volumes History Support for variable injection to pvc.name introduced in GitLab 16.3. The build directories in Kubernetes CI jobs are ephemeral by default. If you want to persist your Git clone across jobs (to make GIT_STRATEGY=fetch work), you must mount a persistent volume claim for your build folder. Because multiple jobs can run concurrently, you must either use a ReadWriteMany volume, or have one volume for each potential concurrent job on the same runner. The latter is likely to be more performant. Here is an example of such a configuration: concurrent = 4 [[ runners ]] executor = &#34;kubernetes&#34; builds_dir = &#34;/mnt/builds&#34; [ runners . kubernetes ] [[ runners . kubernetes . volumes . pvc ]] # CI_CONCURRENT_ID identifies parallel jobs of the same runner. name = &#34;build-pvc-$CI_CONCURRENT_ID&#34; mount_path = &#34;/mnt/builds&#34; In this example, create the persistent volume claims named build-pvc-0 to build-pvc-3 yourself. Create as many as the runner&rsquo;s concurrent setting dictates. Use a helper image After you set the security policy, the helper image must conform to the policy. The image does not receive privileges from the root group, so you must ensure that the user ID is part of the root group. If you only need the nonroot environment, you can use the GitLab Runner UBI OpenShift Container Platform images instead of a helper image. You can also use the GitLab Runner Helper UBI OpenShift Container Platform images. The following example creates a user and group called nonroot and sets the helper image to run as that user. ARG tag FROM registry.gitlab.com/gitlab-org/ci-cd/gitlab-runner-ubi-images/gitlab-runner-helper-ocp:${tag} USER root RUN groupadd -g 59417 nonroot &amp;&amp; \ useradd -u 59417 nonroot -g nonroot WORKDIR /home/nonroot USER 59417:59417 Using Docker in builds When you use Docker in your builds, there are several considerations you should be aware of. Exposed /var/run/docker.sock There is risk involved if you use the runners.kubernetes.volumes.host_path option to expose /var/run/docker.sock of your host into your build container. Be careful when you run builds in the same cluster as your production containers. The node&rsquo;s containers are accessible from the build container. Using docker:dind If you run the docker:dind , also called the docker-in-docker image, containers must run in privileged mode. This may have potential risks and cause additional issues. The Docker daemon runs as a separate container in the pod because it is started as a service , typically in the .gitlab-ci.yml . Containers in pods only share volumes assigned to them and an IP address, that they use to communicate with each other with localhost . The docker:dind container does not share /var/run/docker.sock and the docker binary tries to use it by default. To configure the client use TCP to contact the Docker daemon, in the other container, include the environment variables of the build container: DOCKER_HOST=tcp://docker:2375 for no TLS connection. DOCKER_HOST=tcp://docker:2376 for TLS connection. In Docker 19.03 and later, TLS is enabled by default but you must map certificates to your client. You can enable non-TLS connection for Docker-in-Docker or mount certificates. For more information, see Use the Docker executor with Docker-in-Docker . Prevent host kernel exposure If you use docker:dind or /var/run/docker.sock , the Docker daemon has access to the underlying kernel of the host machine. This means that any limits set in the pod do not work when Docker images are built. The Docker daemon reports the full capacity of the node, regardless of limits imposed on the Docker build containers spawned by Kubernetes. If you run build containers in privileged mode, or if /var/run/docker.sock is exposed, the host kernel may become exposed to build containers. To minimize exposure, specify a label in the node_selector option. This ensures that the node matches the labels before any containers can be deployed to the node. For example, if you specify the label role=ci , the build containers only run on nodes labeled role=ci , and all other production services run on other nodes. To further separate build containers, you can use node taints . Taints prevent other pods from scheduling on the same nodes as the build pods, without extra configuration for the other pods. Restrict Docker images and services You can restrict the Docker images that are used to run your jobs. To do this, you specify wildcard patterns. For example, to allow images from your private Docker registry only: [[ runners ]] ( ... ) executor = &#34;kubernetes&#34; [ runners . kubernetes ] ( ... ) allowed_images = [ &#34;my.registry.tld:5000/*:*&#34; ] allowed_services = [ &#34;my.registry.tld:5000/*:*&#34; ] Or, to restrict to a specific list of images from this registry: [[ runners ]] ( ... ) executor = &#34;kubernetes&#34; [ runners . kubernetes ] ( ... ) allowed_images = [ &#34;my.registry.tld:5000/ruby:*&#34; , &#34;my.registry.tld:5000/node:*&#34; ] allowed_services = [ &#34;postgres:9.4&#34; , &#34;postgres:latest&#34; ] Restrict Docker pull policies In the .gitlab-ci.yml file, you can specify a pull policy. This policy determines how a CI/CD job should fetch images. To restrict which pull policies can be used from those specified in the .gitlab-ci.yml file, use allowed_pull_policies . For example, to allow only the always and if-not-present pull policies: [[ runners ]] ( ... ) executor = &#34;kubernetes&#34; [ runners . kubernetes ] ( ... ) allowed_pull_policies = [ &#34;always&#34; , &#34;if-not-present&#34; ] If you don&rsquo;t specify allowed_pull_policies , the default is the value in the pull_policy keyword. If you don&rsquo;t specify pull_policy , the cluster&rsquo;s image default pull policy is used. The job uses only the pull policies that are listed in both pull_policy and allowed_pull_policies . The effective pull policy is determined by comparing the policies in pull_policy keyword and allowed_pull_policies . GitLab uses the intersection of these two policy lists. For example, if pull_policy is ["always", "if-not-present"] and allowed_pull_policies is ["if-not-present"] , then the job uses only if-not-present because it&rsquo;s the only pull policy defined in both lists. The existing pull_policy keyword must include at least one pull policy specified in allowed_pull_policies . The job fails if none of the pull_policy values match allowed_pull_policies . Job execution GitLab Runner uses kube attach instead of kube exec by default. This should avoid problems like when a job is marked successful midway in environments with an unstable network. Follow issue #27976 for progress on legacy execution strategy removal. Configure the number of request attempts to the Kubernetes API By default, the Kubernetes executor retries specific requests to the Kubernetes API after five failed attempts. The delay is controlled by a backoff algorithm with a 500 millisecond floor and a customizable ceiling with default value of two seconds. To configure the number of retries, use the retry_limit option in the config.toml file. Similarly, for backoff ceiling, use the retry_backoff_max option. The following failures are automatically retried: error dialing backend TLS handshake timeout read: connection timed out connect: connection timed out Timeout occurred http2: client connection lost connection refused tls: internal error io.unexpected EOF syscall.ECONNRESET syscall.ECONNREFUSED syscall.ECONNABORTED syscall.EPIPE To control the amount of retries for each error, use the retry_limits option. The rety_limits specifies the amount of retries for each error separately, and is a map of error messages to the amount of retries. The error message can be a substring of the error message returned by the Kubernetes API. The retry_limits option has precedence over the retry_limit option. For example, configure the retry_limits option to retry the TLS related errors in your environment 10 times instead of the default five times: [[ runners ]] name = &#34;myRunner&#34; url = &#34;https://gitlab.example.com/&#34; executor = &#34;kubernetes&#34; [ runners . kubernetes ] retry_limit = 5 [ runners . kubernetes . retry_limits ] &#34;TLS handshake timeout&#34; = 10 &#34;tls: internal error&#34; = 10 To retry an entirely different error, such as exceeded quota 20 times: [[ runners ]] name = &#34;myRunner&#34; url = &#34;https://gitlab.example.com/&#34; executor = &#34;kubernetes&#34; [ runners . kubernetes ] retry_limit = 5 [ runners . kubernetes . retry_limits ] &#34;exceeded quota&#34; = 20 Container entrypoint known issues In GitLab 15.1 and later, the entrypoint defined in a Docker image is used with the Kubernetes executor when FF_KUBERNETES_HONOR_ENTRYPOINT is set. The container entry point has the following known issues: If an entrypoint is defined in the Dockerfile for an image, it must open a valid shell. Otherwise, the job hangs. To open a shell, the system passes the command as args for the build container. File type CI/CD variables are not written to disk when the entrypoint is executed. The file is only accessible in the job during script execution. The following CI/CD variables are not accessible in the entrypoint. You can use before_script to make any setup changes before running script commands: CI/CD variables defined in the settings . Masked CI/CD variables . Before GitLab Runner 17.4: The entrypoint logs were not forwarded to the build&rsquo;s log. With the Kubernetes executor with kube exec , GitLab Runner did not wait for the entrypoint to open a shell (see above ). Starting with GitLab Runner 17.4, the entrypoint logs are now forwarded. The system waits for the entrypoint to run and spawn the shell. This has the following implications: If FF_KUBERNETES_HONOR_ENTRYPOINT is set, and the image&rsquo;s entrypoint takes longer than poll_timeout (default: 180 s), the build fails. The poll_timeout value (and potentially poll_interval ) must be adapted if the entrypoint is expected to run longer. When FF_KUBERNETES_HONOR_ENTRYPOINT and FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY are set, the system adds a startup probe to the build container, so that it knows when the entrypoint is spawning the shell. If a custom entrypoint uses the provided args to spawn the expected shell, then the startup probe is resolved automatically. However, if the container image is spawning the shell without using the command passed in through args , the entrypoint must resolve the startup probe itself by creating a file named .gitlab-startup-marker inside the root of the build directory. The startup probe checks every poll_interval for the .gitlab-startup-marker file. If the file is not present in poll_timeout , the pod is considered unhealthy, and the system abort the build. Restrict access to job variables When using Kubernetes executor, users with access to the Kubernetes cluster can read variables used in the job. By default, job variables are stored in: Pod&rsquo;s environment section To restrict access to job variable data, you should use role-based access control (RBAC). When you use RBAC, only GitLab administrators have access to the namespace used by the GitLab Runner. If you need other users to access the GitLab Runner namespace, set the following verbs to restrict the user access in the GitLab Runner namespace: For pods and configmaps : get watch list For pods/exec and pods/attach , use create . Example RBAC definition for authorized users: kind : Role apiVersion : rbac.authorization.k8s.io/v1 metadata : name : gitlab-runner-authorized-users rules : - apiGroups : [ &#34;&#34; ] resources : [ &#34;configmaps&#34; , &#34;pods&#34; ] verbs : [ &#34;get&#34; , &#34;watch&#34; , &#34;list&#34; ] - apiGroups : [ &#34;&#34; ] resources : [ &#34;pods/exec&#34; , &#34;pods/attach&#34; ] verbs : [ &#34;create&#34; ] Resources check during prepare step Prerequisites: image_pull_secrets or service_account is set. resource_availability_check_max_attempts is set to a number greater than zero. Kubernetes serviceAccount used with the get and list permissions. GitLab Runner checks if the new service accounts or secrets are available with a 5-second interval between each try. This feature is disabled by default. To enable this feature, set resource_availability_check_max_attempts to any value other than 0 . The value you set defines the amount of times the runner checks for service accounts or secrets. Overwrite the Kubernetes namespace Prerequisites: In the values.yml file for GitLab Runner Helm charts, rbac.clusterWideAccess is set to true . The runner has permissions configured in the core API group. You can overwrite Kubernetes namespaces to designate a namespace for CI purposes, and deploy a custom set of pods to it. The pods spawned by the runner are in the overwritten namespace to enable access between containers during the CI stages. To overwrite the Kubernetes namespace for each CI/CD job, set the KUBERNETES_NAMESPACE_OVERWRITE variable in the .gitlab-ci.yml file. variables : KUBERNETES_NAMESPACE_OVERWRITE : ci-${CI_COMMIT_REF_SLUG} This variable does not create a namespace on your cluster. Ensure that the namespace exists before you run the job. To use only designated namespaces during CI runs, in the config.toml file, define a regular expression for namespace_overwrite_allowed : [ runners . kubernetes ] ... namespace_overwrite_allowed = &#34;ci-.*&#34; Facebook LinkedIn Twitter YouTube Company About GitLab View pricing Try GitLab for free Feedback View page source Edit in web IDE Contribute to GitLab Suggest updates Help & Community Get certified Get support Post on the GitLab forum Resources Terms Privacy statement Use of generative AI Acceptable use of user licenses