GitOps with Argo CD at scale: the layout, the roles and when not to use it

GitOps is easy with one cluster and five apps: put YAML in git, point Argo CD at it, go home. It gets interesting around the hundredth service, when the question stops being “how do I deploy this?” and becomes “where does each kind of YAML live, who can change it, and how far can a mistake travel?”

This is the layout I work with: well over a hundred services, ten environments, two or three clusters in each, all driven by Argo CD. Names are changed and numbers are rounded. It is a long post because the reasons matter more than the YAML, so every choice comes with its “why”, and some sections tell you when to skip the whole approach.

In this post

• The problem it solves
• Is this for you?
• The one idea: two repos, two questions
• Who owns what
• The gitops repo, folder by folder
• A spec file and an ApplicationSet nobody edits
• Two sources: chart from a registry, values from git
• The platform repo: bases, overlays and tenants
• Building it, in an order that works
• Adding a cluster
• How it scales
• Security
• What happens when things break
• One change, end to end
• What I would do differently
• If you take only a few things

The problem it solves

Every team that outgrows a single cluster ends up picking one of a few approaches. Here is what each one costs:

Approach	Why people pick it	Where it hurts
CI pushes to the cluster (helm upgrade in the pipeline)	Simple, no extra tools	Every pipeline holds cluster credentials. Nothing notices drift. The audit trail is a pipeline log. Adding a cluster means editing every pipeline.
A repo or branch per environment	Familiar from code branching	Promotion becomes a merge or cherry-pick. Environments drift apart. Thirty services times four environments is a lot of branches.
One hand-written Argo CD Application per service per environment	Explicit, easy to read	Hundreds of near-identical files. Copy-paste drift. A rename touches everything.
Two repos plus generated Applications (this post)	One shape for everything, promotion is a one-line diff, the cluster pulls and holds no inbound credentials	Real up-front complexity, and it needs someone to own the platform

The last row is not free, which is why the next section exists.

Is this for you?

I would not recommend this layout to everyone. It pays off when most of these are true:

• Many services, several teams. Roughly twenty or more services, with more than one team shipping into shared clusters.
• Several clusters or environments, and more coming. Three or more, and growing: regions, tenants, customers.
• You need an audit trail. Regulated industry, enterprise customers, or just a “who changed that, and who approved it?” question you cannot answer today.
• Someone owns the platform. A platform team, or at least one person with time set aside. Argo CD, the bases and the policies need an owner or they rot.
• Private clusters. You would rather the cluster pull its config than have CI hold credentials to reach in.
• Kubernetes-native workloads. Helm and Kustomize are already how you package things.
• Teams are happy to work through pull requests. The review is the control, so it has to be a normal part of the day.

It is probably too much when:

• One team, one cluster, under about ten services. One repo, Kustomize overlays and a single Argo CD is plenty. Come back when it hurts.
• Nobody can own it. An unowned GitOps repo becomes a museum of stale YAML that everyone is afraid to touch.
• Your workloads mostly are not on Kubernetes. Serverless and managed platforms have their own deploy tooling, and Argo CD adds little.
• You need a deploy in seconds, with no review. GitOps adds a pull request to every change. That is the point, and also the cost.
• Releases are really orchestrations. Database migrations with strict ordering, multi-step cutovers and manual gates fit a workflow engine better. Argo CD has sync waves and hooks, but it is not one.
• Hundreds of short-lived preview environments per day. Possible with ApplicationSet’s pull request generator, but this layout, with its static spec files, is not built for that.

A rough guide to where you sit:

Your situation	What I would build
1 team, 1 cluster	One repo, Kustomize overlays, one Argo CD. Skip ApplicationSets.
2 to 5 teams, 2 to 4 clusters	Two repos, one ApplicationSet per service, one AppProject per team. This post, minus the extras.
10+ teams, 10+ clusters	All of it, plus cluster labels and a cluster generator, controller sharding and one Argo CD per tier.

The one idea: two repos, two questions

Repo	Answers	Holds
platform-configs	What is this thing?	Helm charts, Kustomize bases, per-cluster overlays
gitops	What runs where, at which version?	Argo CD projects, cluster registrations, one ApplicationSet per service, per-environment values

Why split them instead of one big repo:

• Different speeds. A chart changes when its template changes. The gitops repo changes every time anyone ships anything, many times a day. In one repo, the rare chart change drowns in the daily noise.
• Different owners. The platform team owns charts and bases. Service teams own their own spec and values files. A CODEOWNERS file routes the reviews, so nobody has to be a gatekeeper for everything.
• Artifacts, not branches. Merging to platform-configs publishes only the charts that changed, as versioned OCI artifacts. The gitops repo points at a version number, never at “whatever main is today”. A chart cannot change under a running service by accident.
• Small blast radius. A rollout is a one-line change in a repo whose only job is to say what runs. A rollback is a revert of that line.

There is a third repo in the picture, the infrastructure repo (Terraform) that creates the clusters. It matters when we get to adding one.

Who owns what

Most GitOps pain is people pain: who is allowed to change this, and who has to approve it? Think in layers, and give each layer exactly one owner.

These are roles, not departments. In a small company one person wears all three hats, and the layers still help, because they tell you which hat you are wearing when you open a pull request.

• Developers own their service: the spec.yaml and the values-env files. They never need cluster credentials, and they never edit a base.
• DevOps (delivery or release engineering) owns the road from commit to artifact: the CI pipelines, the chart templates, the ApplicationSet template, the promotion process. They also review the sizing in values-infra.
• Platform owns everything the services stand on: the clusters, Argo CD itself, the bases and overlays, the projects, the tenant namespaces and RBAC, the policies, the secrets plumbing, the upgrades.

A change table makes it concrete. This is the split I recommend; the real enforcement is just CODEOWNERS paths, so you can encode whatever you decide:

Change	Author	Approver
New build in dev	Developer, or a CI bot	Developer lead
Promote to test	Developer	Developer lead and DevOps
Promote to prod	Developer or DevOps	DevOps, plus whatever release gate you have
Change replicas, requests, limits	Developer	DevOps, and Platform for big asks
New service	Developer	DevOps (template), Platform (project and namespace)
New tenant, namespace or quota	Developer asks	Platform
Change a chart template	DevOps	Platform
New cluster, base change, policy, Argo upgrade	Platform	Platform, with DevOps informed

Four mechanisms make those rules real. None of them depends on people remembering:

1. CODEOWNERS routes pull request reviews by path.

*                                                    @org/platform
/projects/  /clusters/  /repo-creds/                 @org/platform
/deployments/applications/**/applicationset.yaml     @org/devops
/deployments/applications/payments/**/spec.yaml     @org/payments-devs  @org/devops
/deployments/applications/payments/**/values-env/    @org/payments-devs

2. Argo CD RBAC says who may sync what. Sign in through your identity provider, map groups to roles, and make the default role read-only:

p, role:payments-dev, applications, get,  payments/*, allow
p, role:payments-dev, applications, sync, payments/*, allow
p, role:payments-dev, logs,         get,  payments/*, allow
g, payments-devs@example.com, role:payments-dev
g, platform@example.com,      role:admin
# policy.default: role:readonly

3. AppProjects bound what an application can touch. A project lists the repos it may read, the clusters and namespaces it may deploy to, and the kinds of object it may create. We come back to this under security.

4. Kubernetes RBAC bounds what people can do directly. Every tenant namespace gets three roles bound to groups: viewer (read), editor (create and update, but not delete) and admin. In an overlay:

# environments/test/application-cluster/tenants/payments/editor_rolebinding.yaml
subjects:
  - kind: Group
    name: payments-devs@example.com
    apiGroup: rbac.authorization.k8s.io

Three rules keep this stable. Every change is a pull request. Each layer has one owner. And lower layers never reach up: a developer never edits a base, and the platform team never edits a service’s values.

The gitops repo, folder by folder

gitops/
├── clusters/                     # one Secret per cluster: how Argo reaches it
│   └── <env>/{application,data}.yaml
├── projects/                     # AppProjects: who may deploy what, from where, to where
├── repo-creds/                   # how Argo authenticates to git and to the chart registry
├── platform-common-configs/      # things every cluster gets
│   ├── projects/platform.yaml
│   └── deployments/{base-apps,external-secrets,kyverno,...}/
└── deployments/
    ├── applications/<team>/<service>/
    │   ├── applicationset.yaml
    │   ├── spec.yaml
    │   ├── values-env/<env>.yaml
    │   └── values-infra/<env>.yaml
    ├── data/<service>/           # Kafka, Flink, schema registry, NATS
    └── tools/<service>/          # Grafana, Keda, Argo Workflows

The top half (clusters, projects, repo-creds) is the bootstrap layer. It rarely changes and only the pipeline applies it. The deployments half is the day-to-day layer.

Under deployments there are three buckets: applications (what teams build), data (stateful platform pieces) and tools (shared operations). They are separate because they differ in who owns them and in how aggressively Argo may touch them. More on that under failure modes.

A spec file and an ApplicationSet nobody edits

Every service has the same two files. spec.yaml is the only thing people edit:

environments:                       # where each environment lives
  dev:  { server: https://k8s-dev.example.internal,  cluster: dev-application }
  test: { server: https://k8s-test.example.internal, cluster: test-application }
common:
  project: payments                 # the AppProject that limits what this may do
services:
  - name: orders-api
    helmValuesRepoURL: git@git.example.com:org/gitops.git
    helmValuesTargetRevision: main
    namespace: payments
    environments:
      - name: dev
        helmChartURL: registry.example.com/helm-nonprod
        helmChartVersion: 1.4.0-482.a3f9c21
      - name: test
        helmChartURL: registry.example.com/helm-nonprod
        helmChartVersion: 1.3.9-471.9be04d7

The applicationset.yaml next to it reads that file and fans it out. It is the same template for every service:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata: { name: orders-api, namespace: argocd }
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]      # a typo fails loudly, not silently
  generators:
    - matrix:
        generators:
          - git:                               # 1. read the spec file...
              repoURL: git@git.example.com:org/gitops.git
              revision: main
              files:
                - path: deployments/applications/payments/orders-api/spec.yaml
          - list:                              # 2. ...and fan out: service x environment
              elementsYaml: |-
                {{- range .services }}
                {{- $svc := . }}
                {{- range .environments }}
                - environment: {{ .name }}
                  server: {{ index $.environments .name "server" }}
                  helmChartURL: {{ .helmChartURL }}
                  helmChartVersion: {{ .helmChartVersion }}
                  name: {{ $svc.name }}-{{ .name }}
                {{- end }}
                {{- end }}
  template:
    # ...one Application per element, shown in the next section

Why this shape, and not a hand-written Application per environment:

• Promotion is a one-line pull request. Moving a build from dev to test is changing one version string:

        - name: test
          helmChartURL: registry.example.com/helm-nonprod
  -       helmChartVersion: 1.3.9-471.9be04d7
  +       helmChartVersion: 1.4.0-482.a3f9c21

  The diff is the audit log. The version is semver, then a build number, then the commit it came from, so you can trace any running pod back to source.

• A new environment is a new list item. No new files, no copied YAML.

• missingkey=error. Without it, a misspelled key quietly renders as an empty string and Argo happily deploys nonsense. With it, the ApplicationSet refuses to render. I would rather have a loud failure in review than a silent one in a cluster.

• One ApplicationSet per service. A bad template change breaks one service, not a hundred.

Two sources: chart from a registry, values from git

The template half of that ApplicationSet gives each Application two sources:

  template:
    metadata:
      name: '{{.name}}'
      finalizers: [resources-finalizer.argocd.argoproj.io]   # deleting the app cleans up what it made
    spec:
      project: payments
      sources:
        - repoURL: '{{.helmChartURL}}'                       # 1. the chart: versioned, immutable
          chart: orders-api
          targetRevision: '{{.helmChartVersion}}'
          helm:
            valueFiles:
              - $values/deployments/applications/payments/orders-api/values-env/{{.environment}}.yaml
              - $values/deployments/applications/payments/orders-api/values-infra/{{.environment}}.yaml
        - repoURL: git@git.example.com:org/gitops.git        # 2. the values: just a git ref
          targetRevision: main
          ref: values
      destination: { server: '{{.server}}', namespace: payments }
      syncPolicy:
        syncOptions: [CreateNamespace=true]

The chart is built by the service’s own pipeline and never edited by hand. The values live in git. That split buys two things:

• Config changes do not need a rebuild. Changing a URL or a replica count is a values pull request. No new chart, no new image.
• Two kinds of values, two reviewers. values-env is application config (image tag, URLs, feature flags). values-infra is sizing (replicas, requests, limits). Developers review the first. DevOps and Platform keep an eye on the second, because that is where the cluster bill and the noisy-neighbour problems come from.

The platform repo: bases, overlays and tenants

Charts are only half of platform-configs. The rest is plain Kustomize, because most platform components are “an upstream manifest plus a few patches”, and Kustomize says exactly that:

platform-configs/
├── charts/                          # ~40 charts, published when changed
├── kustomize/
│   ├── platform-base-config/        # cert-manager, external-dns, gateway, policies, Argo CD itself
│   ├── tenant-cluster-config/       # namespace + roles + quota template, per tenant
│   └── dataapps-base-config/        # Flink, Elastic, and friends
└── environments/
    └── <env>/<cluster>/
        ├── kustomization.yaml
        ├── common/  tenants/  gateway/  certmanager/  externaldns/ ...

Every cluster’s folder is a set of small overlays on the shared bases. Here is the one that defines Argo CD itself:

# environments/mgmt/nonprod-cluster/argocd/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: argocd
resources:
  - ../../../../kustomize/platform-base-config/argocd   # the shared base
  - sso-external-secret.yaml                            # things only this cluster has
patches:
  - path: application-controller-sts.yaml               # sizing for this cluster
    target: { kind: StatefulSet, name: argocd-application-controller }

And one small ApplicationSet, base-apps, creates one Application per cluster that simply points at that cluster’s folder:

source:
  repoURL: git@git.example.com:org/platform-configs.git
  targetRevision: main
  path: environments/{{.env}}/{{.cluster}}
syncPolicy:
  automated: { selfHeal: true, prune: false }
  syncOptions: [CreateNamespace=true]

Why base plus overlay:

• A fix reaches every cluster in one pull request. Patch the base once and every cluster picks it up on its next sync. (That is also a risk. See “What happens when things break”.)
• The overlay is the only place a cluster differs. “How is prod different from test?” becomes a diff -r between two folders.
• Tenants are boring on purpose. A new team’s namespace comes from tenant-cluster-config: the namespace, the viewer, editor and admin roles with bindings, and a quota template you can switch on per environment. Each cluster lists its workload namespaces in one patch file, labelled tenant: <name>. Same shape everywhere, so the RBAC review is the same review everywhere.

I use Helm for things I version and release, and Kustomize for things I configure per cluster. Trying to make one tool do both is where most of the pain I have seen comes from.

Building it, in an order that works

Do not try to build all of this in one go. This order gives you something working at each step:

1. Install Argo CD once, by hand. Somebody has to bootstrap the thing that manages everything else. I keep the install as an overlay in platform-configs and apply it with kubectl apply -k. After that, nothing else is applied by hand.

   kubectl apply -k environments/mgmt/nonprod-cluster/argocd

2. Turn on SSO and RBAC before anyone else logs in. Map groups to roles, make the default read-only (see the RBAC snippet above).

3. Register your clusters. One Secret each. Use an exec plugin so Argo gets short-lived tokens from your cloud’s IAM instead of holding a kubeconfig. The next section shows the Secret.

4. Create the repos with guard rails. Protected main, no direct pushes, required reviews, CODEOWNERS from the start, and a CI check on every pull request.

5. Build the chart pipeline. On pull request, helm lint the changed charts. On merge, package and push only the changed charts to the OCI registry, with the version read from Chart.yaml. Decide the version scheme now (semver plus build plus commit) because you cannot easily change it later.

6. Move one service. Pick a boring, low-risk one. Write its spec.yaml, applicationset.yaml and values, let CI apply it, and watch it sync. Fix the rough edges of the template here, while it is cheap.

7. Add the platform layer. Create the base-apps ApplicationSet and bring cert-manager, external-dns, the gateway and friends under per-cluster overlays.

8. Take secrets out of git. Run External Secrets with a store that reads from your cloud’s secret manager. Applications reference a secret by name. The value never enters a repo.

   apiVersion: external-secrets.io/v1beta1
   kind: ClusterSecretStore
   metadata: { name: cloud-store }
   spec:
     provider:
       gcpsm:
         projectID: example-secrets-project
         auth:
           workloadIdentity:
             clusterLocation: us-central1
             clusterName: example-cluster
             serviceAccountRef: { name: external-secrets, namespace: external-secrets }
   ---
   apiVersion: external-secrets.io/v1beta1
   kind: ExternalSecret
   metadata: { name: orders-api-db, namespace: payments }
   spec:
     secretStoreRef: { kind: ClusterSecretStore, name: cloud-store }
     target: { name: orders-api-db }
     dataFrom:
       - extract: { key: payments-orders-api }     # one secret in the cloud secret manager

9. Add guard rails and eyes. Policies in audit mode, notifications into chat, a dashboard of “which version is running where”.

10. Migrate the rest, a few at a time. Each migration is the same two files, which is the whole point.

Adding a cluster

This is the test of the design. Adding a cluster touches three repos, each with one job:

1. Infra repo: make the cluster exist. Terraform (I use Terragrunt) creates the cluster, node pools, firewall rules and a bastion for private access, and outputs the identity Argo CD will use to reach it. The infra repo has the same pull-request loop as the others: plan on the pull request, apply on merge.

2. gitops repo: let Argo reach it. Add a cluster Secret and add the new environment to the environments: map in the spec files that should run there:

apiVersion: v1
kind: Secret
metadata:
  name: eu-application-secret
  namespace: argocd
  labels: { argocd.argoproj.io/secret-type: cluster }
stringData:
  name: eu-application
  server: https://k8s-eu.example.internal
  config: |
    { "execProviderConfig": { "command": "argocd-k8s-auth", "args": ["gcp"],
        "apiVersion": "client.authentication.k8s.io/v1beta1" },
      "tlsClientConfig": { "insecure": false, "caData": "<cluster-ca>" } }

3. platform-configs: describe what the cluster is. Copy an existing overlay folder to environments/<env>/<cluster>/, set the workload namespaces and tenants, and adjust the few things that differ (domain, sizing).

4. Let Argo do the rest. Add the cluster to the base-apps spec:

  environments:
    test-application: { server: https://k8s-test.example.internal, cluster: application-cluster, env: test }
+   eu-application:   { server: https://k8s-eu.example.internal,   cluster: application-cluster, env: eu }
  services:
    - name: platform
      environments:
        - { name: test-application, targetRevision: main }
+       - { name: eu-application,   targetRevision: main }

Argo creates the platform Application. Certificates, DNS, the gateway and the policies come up on their own. Then each service adds an eu entry to its own spec when it is ready to run there.

That last step is the part that does not scale gracefully. Ten services means ten spec edits, and a hundred means a hundred. The fix is to stop listing clusters by hand. Label your cluster Secrets and let a cluster generator find them:

generators:
  - clusters:
      selector:
        matchLabels: { tier: nonprod, role: application }

With labels, registering a cluster (step 2) is enough to give it the platform layer. Services still opt in explicitly, which I think is the right call: a new region should not silently receive a payment service.

How it scales

Different things grow with different numbers, so they have different knobs:

What grows	Grows with	What to do about it
Applications	services times environments	One ApplicationSet per service, so a bad template change hits one service
Application controller load	resources watched in the destination clusters (more than application count)	Run several replicas (sharding), give them memory, and keep the reconcile interval sane
Repo server load	Helm and Kustomize renders	More replicas. Add the argocd.argoproj.io/manifest-generate-paths annotation so a commit to one folder does not re-render everything
Git traffic	polling every service	Webhooks from your git host, and a longer poll interval as a backstop
Human attention	number of apps in the UI	Projects, labels and per-team views
Review load	pull requests per day	CODEOWNERS by path, so the platform team is not on every pull request
Cluster count	environments times regions	Labels, a cluster generator, and an overlay you copy instead of invent

For scale, my Argo CD runs the application controller as a StatefulSet of three replicas (4 CPU and 12 GiB each), plus three repo servers, three ApplicationSet controllers and a three-replica Redis, with a five-minute reconcile interval. That serves hundreds of Applications across a handful of clusters. Treat those as a data point, not a recipe: measure your own.

Two honest caveats:

• Sharding is per destination cluster. Argo spreads load by assigning clusters to controller replicas. If most of your workload lives in one enormous cluster, extra replicas will not share it. More, smaller clusters shard better.
• Copies drift. I have well over a hundred nearly identical ApplicationSet files. They are generated from one template, but they are still copies. That is the first thing I would change.

A small trick that pays off: label each Application with its chart version and let the controller export it as a metric label (--metrics-application-labels). One dashboard then answers “which version is running in which environment?” without opening a single Argo page.

Security

The design tries to hold as few secrets as possible:

• No cloud keys in CI. The pipeline authenticates with workload identity federation, exchanging a short-lived token from the CI system for cloud access.
• No kubeconfigs in Argo. The cluster Secret above uses an exec plugin, so Argo asks the cloud for a short-lived token each time.
• No secrets in git. External Secrets reads from the cloud’s secret manager through workload identity. The repository credentials in repo-creds/ are templates, and the pipeline fills in the values from masked CI variables when it applies them.
• Private clusters. The pipeline reaches them through a bastion and an identity-aware tunnel, so there is no public API endpoint to defend.
• People come through SSO. Groups map to Argo roles, and the default role is read-only.

On top of that, AppProjects are the main permission boundary. A project says which repos an application may pull from, which clusters and namespaces it may push to, and which kinds of object it may create. This is the shape I aim for:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata: { name: payments, namespace: argocd }
spec:
  sourceRepos:                          # where it may read from
    - git@git.example.com:org/gitops.git
    - registry.example.com/helm-nonprod
  destinations:                         # where it may deploy
    - { server: https://k8s-test.example.internal, namespace: payments }
  clusterResourceWhitelist: []          # no cluster-scoped objects: no CRDs, no RBAC

Start tight and widen when something genuinely needs it. A project with wildcards everywhere is just a label.

Policy lives in git too. A set of Kyverno policies ships through the same pipeline: resource requests and limits, liveness and readiness probes, PodDisruptionBudgets, more than one replica, allowed image registries. I start new policies in audit mode. You learn what would break before you break it, then switch to enforce one policy at a time.

One Argo CD per tier. Nonprod and prod each get their own instance, defined as an overlay like the one above. An experiment on the nonprod controller cannot take down prod’s. Each Argo sits in a management cluster and pushes outward to the workload clusters.

What happens when things break

A design is only as good as its bad days. Here is what I expect, and what I do:

What breaks	What happens	Recovery
Argo CD goes down	Workloads keep running. Nothing syncs and nothing notices drift.	Argo is not in the data path. Re-apply the bootstrap overlay, let the pipeline re-apply projects, clusters and ApplicationSets, and the Applications regenerate from git.
The git host is down	Argo works from cached manifests. No new commits are seen.	Wait. Nothing deploys and nothing breaks. For a true emergency there is a break-glass kubectl path, followed by a pull request to make git match.
A bad chart version is merged	The Application shows OutOfSync. Someone syncs, pods fail.	Revert the pull request and sync again. Manual sync is why this stops at one environment.
A bad base change	Platform Applications self-heal, so it reaches every cluster that tracks main.	See below.
A cluster is lost	Its workloads are gone. Git is untouched.	Create a new cluster, register it, and the platform and services re-sync. Data is a separate problem: GitOps does not back it up.
Someone edits a live object by hand	Platform pieces self-heal back. Service workloads show as OutOfSync.	The diff makes the drift visible. Put the real change in git.
The chart registry is down	New syncs fail. Running pods are unaffected.	Mirror or cache the registry that production pulls from.

Sync policy follows the kind of thing it is.

Bucket	Policy	Reason
Platform pieces	selfHeal: true, prune: false	Undo hand-edits, but never let Argo delete on its own initiative
Service workloads	No automation	A merge is not a rollout. A person presses sync, and sees the diff first

prune: false on shared infrastructure is cheap insurance. One bad rename in git should not be able to delete a gateway.

About “a bad base change”: the property that lets one fix reach every cluster also lets one mistake reach every cluster. The brake is the per-environment targetRevision in the base-apps spec. Lower environments track main. Production tracks a tag that you move on purpose, after the change has soaked below. If you take only one operational idea from this section, take that one.

CI is the only door to the bootstrap layer. Cluster registrations, projects and ApplicationSets are applied by the pipeline, not by hand:

• On a pull request, it runs kubectl apply --dry-run=server against only the files that changed. The API server validates them, so schema mistakes show up before merge.
• On merge, it applies those same files.

One change, end to end

1. The service’s pipeline builds an image and a chart, and pushes the chart to the registry as 1.4.0-482.a3f9c21.
2. Someone (or a bot) opens a pull request in gitops changing that one version string in spec.yaml.
3. CODEOWNERS pulls in the right reviewers. The diff is one line.
4. Merge. Argo shows the Application as OutOfSync, with the exact resource diff.
5. A person syncs it. If it goes wrong, they revert the line and sync again.

Nothing in that list needs kubectl, and nobody needs cluster credentials.

What I would do differently

• Collapse the ApplicationSets. Well over a hundred of them are near-identical copies of the same template, and copies drift. One ApplicationSet with a git files generator over deployments/**/spec.yaml would do the same job with a single template to maintain.
• Use cluster labels from day one. Register clusters with labels and a cluster generator, and the manual “add the environment to every spec” step mostly disappears.
• Let Argo manage the bootstrap layer too. Today the pipeline applies projects, clusters and ApplicationSets. An app-of-apps root Application would make that layer self-healing and visible in the UI like everything else.
• Pin production to a tag early. Do not wait for the bad base change to teach you.
• Stop bumping versions by hand. A small bot that opens the “new version” pull request keeps the human review and removes the typing.
• Do not leave policies in audit forever. Audit mode is a great on-ramp and a terrible place to live.

If you take only a few things

• Check the criteria first. Under about twenty services, one team or one cluster, this is more machinery than you need.
• Split what a thing is from where it runs at which version.
• Give every layer one owner, and let CODEOWNERS, Argo RBAC, projects and Kubernetes RBAC enforce it.
• Reference immutable, versioned artifacts, never a moving branch.
• Make promotion a one-line diff that someone has to approve.
• Use base plus overlay so a cluster’s differences fit on one screen.
• Adding a cluster should be three small pull requests, not a project.
• Keep secrets and credentials out of git and out of CI, and put the guard rails in git instead.