instruction: How do you implement GitOps workflows with ArgoCD and Kustomize?
output: Helm - Argo CD - Declarative GitOps CD for Kubernetes Skip to content Argo CD - Declarative GitOps CD for Kubernetes Helm Initializing search GitHub Argo CD - Declarative GitOps CD for Kubernetes GitHub Overview Understand The Basics Core Concepts Getting Started Operator Manual Operator Manual Overview Architectural Overview Installation Feature Maturity Argo CD Core Declarative Setup Applications in any namespace Ingress Configuration High Availability High Availability Overview Dynamic Cluster Distribution User Management User Management Overview Auth0 Microsoft Okta OneLogin Keycloak OpenUnison Google Zitadel Identity Center (AWS SSO) RBAC Configuration Security Security Overview Snyk Scans Verification of Argo CD Artifacts TLS configuration Cluster Management Cluster Bootstrapping Secret Management Disaster Recovery Reconcile Optimization Git Webhook Configuration Resource Health Resource Actions Custom Tooling Custom Styles UI Customization Metrics Web-based Terminal Config Management Plugins Deep Links Notifications Notifications Overview Triggers Templates Functions Triggers and Templates Catalog Monitoring Subscriptions Troubleshooting Troubleshooting commands Troubleshooting errors Examples Notification Services Notification Services Alertmanager AWS SQS Email GitHub Google Chat Grafana Mattermost NewRelic Opsgenie Overview PagerDuty PagerDuty V2 Pushover Rocket.Chat Slack Teams Telegram Webex Teams Webhook Troubleshooting Tools ApplicationSet ApplicationSet Introduction Installations Use Cases Security How ApplicationSet controller interacts with Argo CD Generators Generators Generators List Generator Cluster Generator Git Generator Matrix Generator Merge Generator SCM Provider Generator Cluster Decision Resource Generator Pull Request Generator Post Selector all generators Plugin Generator Template fields Template fields Templates Go Template Controlling Resource Modification Application Pruning & Resource Deletion Progressive Syncs Git File Generator Globbing ApplicationSet Specification Reference ApplicationSet in any namespace Server Configuration Parameters Server Configuration Parameters argocd-server Command Reference argocd-application-controller Command Reference argocd-repo-server Command Reference argocd-dex Command Reference Additional configuration method Upgrading Upgrading Overview v2.14 to 3.0 v2.13 to 2.14 v2.12 to 2.13 v2.11 to 2.12 v2.10 to 2.11 v2.9 to 2.10 v2.8 to 2.9 v2.7 to 2.8 v2.6 to 2.7 v2.5 to 2.6 v2.4 to 2.5 v2.3 to 2.4 v2.2 to 2.3 v2.1 to 2.2 v2.0 to 2.1 v1.8 to 2.0 v1.7 to 1.8 v1.6 to 1.7 v1.5 to 1.6 v1.4 to 1.5 v1.3 to 1.4 v1.2 to 1.3 v1.1 to 1.2 v1.0 to 1.1 Project Specification Reference User Guide User Guide Overview Tools Kustomize Helm Helm Table of contents Declarative Values Files Values Helm Parameters Helm Value Precedence Helm --set-file support Helm Release Name Helm Hooks Hook Tips Random Data Build Environment Helm plugins Modifying the ArgoCD container image Using initContainers Helm Version Helm --pass-credentials Helm --skip-crds Helm --skip-schema-validation Helm --skip-tests OCI Importing Argo CD go packages Jsonnet Directory Tool Detection Projects Private Repositories Plugins Multiple Sources for an Application GnuPG verification Automated Sync Policy Diffing Diffing Diff Strategies Diff Customization Orphaned Resources Monitoring Compare Options Sync Options Parameter Overrides Environment Variables Build Environment Tracking and Deployment Strategies Resource Tracking Resource hooks Selective Sync Sync Phases and Waves Sync Windows Sync Applications with Kubectl Skip Application Reconcile Generating Applications with ApplicationSet Automation from CI Pipelines App Deletion Best Practices Status Badge Add external URL Add extra Application info Notification subscriptions Annotations and Labels used by Argo CD Command Reference Application Specification Reference Developer Guide Developer Guide Overview Development Environment Development Cycle Submit Your PR Architecture Architecture Authentication and Authorization Component Architecture Code Contribution Guide Toolchain Guide MacOS users Release Process And Cadence Running Argo CD locally Debugging a local Argo CD instance Debugging a Remote ArgoCD Environment API Docs E2E Tests Managing Dependencies Continuous Integration (CI) Releasing Documentation Site Static Code Analysis Extensions Extensions UI Extensions Proxy Extensions Contribution FAQ Tilt Development FAQ Security Considerations Support Roadmap Releases ⧉ Blog ⧉ Table of contents Declarative Values Files Values Helm Parameters Helm Value Precedence Helm --set-file support Helm Release Name Helm Hooks Hook Tips Random Data Build Environment Helm plugins Modifying the ArgoCD container image Using initContainers Helm Version Helm --pass-credentials Helm --skip-crds Helm --skip-schema-validation Helm --skip-tests Helm # Helm Declarative &para; You can install Helm charts through the UI, or in the declarative GitOps way. Helm is only used to inflate charts with helm template . The lifecycle of the application is handled by Argo CD instead of Helm. Here is an example: apiVersion : argoproj.io/v1alpha1 kind : Application metadata : name : sealed-secrets namespace : argocd spec : project : default source : chart : sealed-secrets repoURL : https://bitnami-labs.github.io/sealed-secrets targetRevision : 1.16.1 helm : releaseName : sealed-secrets destination : server : &quot;https://kubernetes.default.svc&quot; namespace : kubeseal Another example using a public OCI helm chart: apiVersion : argoproj.io/v1alpha1 kind : Application metadata : name : nginx spec : project : default source : chart : nginx repoURL : registry-1.docker.io/bitnamicharts # note: the oci:// syntax is not included. targetRevision : 15.9.0 destination : name : &quot;in-cluster&quot; namespace : nginx When using Helm there are multiple ways to provide values Order of precedence is parameters &gt; valuesObject &gt; values &gt; valueFiles &gt; helm repository values.yaml (see Here for a more detailed example) See here for more info about how to configure private Helm repositories and private OCI registries. Values Files &para; Helm has the ability to use a different, or even multiple "values.yaml" files to derive its parameters from. Alternate or multiple values file(s), can be specified using the --values flag. The flag can be repeated to support multiple values files: argocd app set helm-guestbook --values values-production.yaml Note Before v2.6 of Argo CD, Values files must be in the same git repository as the Helm chart. The files can be in a different location in which case it can be accessed using a relative path relative to the root directory of the Helm chart. As of v2.6 , values files can be sourced from a separate repository than the Helm chart by taking advantage of multiple sources for Applications . In the declarative syntax: source : helm : valueFiles : - values-production.yaml If Helm is passed a non-existing value file during template expansion, it will error out. Missing values files can be ignored (meaning, not passed to Helm) using the --ignore-missing-value-files . This can be particularly helpful to implement a default/override pattern with Application Sets . In the declarative syntax: source : helm : valueFiles : - values-common.yaml - values-optional-override.yaml ignoreMissingValueFiles : true Values &para; Argo CD supports the equivalent of a values file directly in the Application manifest using the source.helm.valuesObject key. source : helm : valuesObject : ingress : enabled : true path : / hosts : - mydomain.example.com annotations : kubernetes.io/ingress.class : nginx kubernetes.io/tls-acme : &quot;true&quot; labels : {} tls : - secretName : mydomain-tls hosts : - mydomain.example.com Alternatively, values can be passed in as a string using the source.helm.values key. source : helm : values : | ingress: enabled: true path: / hosts: - mydomain.example.com annotations: kubernetes.io/ingress.class: nginx kubernetes.io/tls-acme: &quot;true&quot; labels: {} tls: - secretName: mydomain-tls hosts: - mydomain.example.com Helm Parameters &para; Helm has the ability to set parameter values, which override any values in a values.yaml . For example, service.type is a common parameter which is exposed in a Helm chart: helm template . --set service.type = LoadBalancer Similarly, Argo CD can override values in the values.yaml parameters using argocd app set command, in the form of -p PARAM=VALUE . For example: argocd app set helm-guestbook -p service.type = LoadBalancer In the declarative syntax: source : helm : parameters : - name : &quot;service.type&quot; value : LoadBalancer Helm Value Precedence &para; Values injections have the following order of precedence parameters &gt; valuesObject &gt; values &gt; valueFiles &gt; helm repository values.yaml Or rather lowest -&gt; valueFiles -&gt; values -&gt; valuesObject highest -&gt; parameters So valuesObject trumps values - therefore values will be ignored, and both valuesObject and values trump valueFiles. Parameters trump all of them. Precedence of multiple valueFiles: When multiple valueFiles are specified, the last file listed has the highest precedence: valueFiles: - values-file-2.yaml - values-file-1.yaml In this case, values-file-1.yaml will override values from values-file-2.yaml. When multiple of the same key are found the last one wins i.e e.g. if we only have values-file-1.yaml and it contains param1: value1 param1: value3000 we get param1=value3000 parameters: - name: &quot;param1&quot; value: value2 - name: &quot;param1&quot; value: value1 the result will be param1=value1 values: | param1: value2 param1: value5 the result will be param1=value5 When valueFiles or values is used The list of parameters seen in the ui is not what is used for resources, rather it is the values/valuesObject merged with parameters (see this issue incase it has been resolved) As a workaround using parameters instead of values/valuesObject will provide a better overview of what will be used for resources Helm --set-file support &para; The --set-file argument to helm can be used with the following syntax on the cli: argocd app set helm-guestbook --helm-set-file some.key = path/to/file.ext or using the fileParameters for yaml: source : helm : fileParameters : - name : some.key path : path/to/file.ext Helm Release Name &para; By default, the Helm release name is equal to the Application name to which it belongs. Sometimes, especially on a centralised Argo CD, you may want to override that name, and it is possible with the release-name flag on the cli: argocd app set helm-guestbook --release-name myRelease or using the releaseName for yaml: source : helm : releaseName : myRelease Important notice on overriding the release name Please note that overriding the Helm release name might cause problems when the chart you are deploying is using the app.kubernetes.io/instance label. Argo CD injects this label with the value of the Application name for tracking purposes. So when overriding the release name, the Application name will stop being equal to the release name. Because Argo CD will overwrite the label with the Application name it might cause some selectors on the resources to stop working. In order to avoid this we can configure Argo CD to use another label for tracking in the ArgoCD configmap argocd-cm.yaml - check the lines describing application.instanceLabelKey . Helm Hooks &para; Helm hooks are similar to Argo CD hooks . In Helm, a hook is any normal Kubernetes resource annotated with the helm.sh/hook annotation. Argo CD supports many (most?) Helm hooks by mapping the Helm annotations onto Argo CD's own hook annotations: Helm Annotation Notes helm.sh/hook: crd-install Supported as equivalent to argocd.argoproj.io/hook: PreSync . helm.sh/hook: pre-delete Not supported. In Helm stable there are 3 cases used to clean up CRDs and 3 to clean-up jobs. helm.sh/hook: pre-rollback Not supported. Never used in Helm stable. helm.sh/hook: pre-install Supported as equivalent to argocd.argoproj.io/hook: PreSync . helm.sh/hook: pre-upgrade Supported as equivalent to argocd.argoproj.io/hook: PreSync . helm.sh/hook: post-upgrade Supported as equivalent to argocd.argoproj.io/hook: PostSync . helm.sh/hook: post-install Supported as equivalent to argocd.argoproj.io/hook: PostSync . helm.sh/hook: post-delete Supported as equivalent to argocd.argoproj.io/hook: PostDelete . helm.sh/hook: post-rollback Not supported. Never used in Helm stable. helm.sh/hook: test-success Not supported. No equivalent in Argo CD. helm.sh/hook: test-failure Not supported. No equivalent in Argo CD. helm.sh/hook-delete-policy Supported. See also argocd.argoproj.io/hook-delete-policy ). helm.sh/hook-delete-timeout Not supported. Never used in Helm stable helm.sh/hook-weight Supported as equivalent to argocd.argoproj.io/sync-wave . helm.sh/resource-policy: keep Supported as equivalent to argocd.argoproj.io/sync-options: Delete=false . Unsupported hooks are ignored. In Argo CD, hooks are created by using kubectl apply , rather than kubectl create . This means that if the hook is named and already exists, it will not change unless you have annotated it with before-hook-creation . Helm hooks + ArgoCD hooks If you define any Argo CD hooks, all Helm hooks will be ignored. 'install' vs 'upgrade' vs 'sync' Argo CD cannot know if it is running a first-time "install" or an "upgrade" - every operation is a "sync'. This means that, by default, apps that have pre-install and pre-upgrade will have those hooks run at the same time. Hook Tips &para; Make your hook idempotent. Annotate crd-install with hook-weight: "-2" to make sure it runs to success before any install or upgrade hooks. Annotate pre-install and post-install with hook-weight: "-1" . This will make sure it runs to success before any upgrade hooks. Annotate pre-upgrade and post-upgrade with hook-delete-policy: before-hook-creation to make sure it runs on every sync. Read more about Argo hooks and Helm hooks . Random Data &para; Helm templating has the ability to generate random data during chart rendering via the randAlphaNum function. Many helm charts from the charts repository make use of this feature. For example, the following is the secret for the redis helm chart : data : {{ - if .Values.password }} redis-password : {{ .Values.password | b64enc | quote }} {{ - else }} redis-password : {{ randAlphaNum 10 | b64enc | quote }} {{ - end }} The Argo CD application controller periodically compares Git state against the live state, running the helm template &lt;CHART&gt; command to generate the helm manifests. Because the random value is regenerated every time the comparison is made, any application which makes use of the randAlphaNum function will always be in an OutOfSync state. This can be mitigated by explicitly setting a value in the values.yaml or using argocd app set command to override the value such that the value is stable between each comparison. For example: argocd app set redis -p password = abc123 Build Environment &para; Helm apps have access to the standard build environment via substitution as parameters. E.g. via the CLI: argocd app create APPNAME \ --helm-set-string &#39;app=${ARGOCD_APP_NAME}&#39; Or via declarative syntax: spec : source : helm : parameters : - name : app value : $ARGOCD_APP_NAME It's also possible to use build environment variables for the Helm values file path: spec : source : helm : valueFiles : - values.yaml - myprotocol://somepath/$ARGOCD_APP_NAME/$ARGOCD_APP_REVISION Helm plugins &para; Argo CD is un-opinionated on what cloud provider you use and what kind of Helm plugins you are using, that's why there are no plugins delivered with the ArgoCD image. But sometimes you want to use a custom plugin. Perhaps you would like to use Google Cloud Storage or Amazon S3 storage to save the Helm charts, for example: https://github.com/hayorov/helm-gcs where you can use gs:// protocol for Helm chart repository access. There are two ways to install custom plugins; you can modify the ArgoCD container image, or you can use a Kubernetes initContainer . Modifying the ArgoCD container image &para; One way to use this plugin is to prepare your own ArgoCD image where it is included. Example Dockerfile : FROM argoproj/argocd:v1.5.7 USER root RUN apt-get update &amp;&amp; \ apt-get install -y \ curl &amp;&amp; \ apt-get clean &amp;&amp; \ rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* USER argocd ARG GCS_PLUGIN_VERSION = &quot;0.3.5&quot; ARG GCS_PLUGIN_REPO = &quot;https://github.com/hayorov/helm-gcs.git&quot; RUN helm plugin install ${ GCS_PLUGIN_REPO } --version ${ GCS_PLUGIN_VERSION } ENV HELM_PLUGINS = &quot;/home/argocd/.local/share/helm/plugins/&quot; The HELM_PLUGINS environment property required for ArgoCD to locale plugins correctly. Once built, use the custom image for ArgoCD installation. Using initContainers &para; Another option is to install Helm plugins via Kubernetes initContainers . Some users find this pattern preferable to maintaining their own version of the ArgoCD container image. Below is an example of how to add Helm plugins when installing ArgoCD with the official ArgoCD helm chart : repoServer : volumes : - name : gcp-credentials secret : secretName : my-gcp-credentials volumeMounts : - name : gcp-credentials mountPath : /gcp env : - name : HELM_CACHE_HOME value : /helm-working-dir - name : HELM_CONFIG_HOME value : /helm-working-dir - name : HELM_DATA_HOME value : /helm-working-dir initContainers : - name : helm-gcp-authentication image : alpine/helm:3.16.1 volumeMounts : - name : helm-working-dir mountPath : /helm-working-dir - name : gcp-credentials mountPath : /gcp env : - name : HELM_CACHE_HOME value : /helm-working-dir - name : HELM_CONFIG_HOME value : /helm-working-dir - name : HELM_DATA_HOME value : /helm-working-dir command : [ &quot;/bin/sh&quot; , &quot;-c&quot; ] args : - apk --no-cache add curl; helm plugin install https://github.com/hayorov/helm-gcs.git; helm repo add my-gcs-repo gs://my-private-helm-gcs-repository; chmod -R 777 $HELM_DATA_HOME; Helm Version &para; Argo CD will assume that the Helm chart is v3 (even if the apiVersion field in the chart is Helm v2), unless v2 is explicitly specified within the Argo CD Application (see below). If needed, it is possible to specifically set the Helm version to template with by setting the helm-version flag on the cli (either v2 or v3): argocd app set helm-guestbook --helm-version v3 Or using declarative syntax: spec : source : helm : version : v3 Helm --pass-credentials &para; Helm, starting with v3.6.1 , prevents sending repository credentials to download charts that are being served from a different domain than the repository. If needed, it is possible to opt into passing credentials for all domains by setting the helm-pass-credentials flag on the cli: argocd app set helm-guestbook --helm-pass-credentials Or using declarative syntax: spec : source : helm : passCredentials : true Helm --skip-crds &para; Helm installs custom resource definitions in the crds folder by default if they are not existing. See the CRD best practices for details. If needed, it is possible to skip the CRD installation step with the helm-skip-crds flag on the cli: argocd app set helm-guestbook --helm-skip-crds Or using declarative syntax: spec : source : helm : skipCrds : true Helm --skip-schema-validation &para; Helm validates the values.yaml file using a values.schema.json file. See Schema files for details. If needed, it is possible to skip the schema validation step with the helm-skip-schema-validation flag on the cli: argocd app set helm-guestbook --helm-skip-schema-validation Or using declarative syntax: spec : source : helm : skipSchemaValidation : true Helm --skip-tests &para; By default, Helm includes test manifests when rendering templates. Argo CD currently skips manifests that include hooks not supported by Argo CD, including Helm test hooks . While this feature covers many testing use cases, it is not totally congruent with --skip-tests, so the --skip-tests option can be used. If needed, it is possible to skip the test manifests installation step with the helm-skip-tests flag on the cli: argocd app set helm-guestbook --helm-skip-tests Or using declarative syntax: spec : source : helm : skipTests : true # or false Previous Kustomize Next OCI Made with Material for MkDocs