Scaling Argo CD GitOps Across 34 Kubernetes Clusters

Managing 34 Kubernetes clusters with one Argo CD instance shifts the hard problem from deploying applications to controlling change across the fleet. At Lucid Motors, a GitOps control plane spanned 22 AWS accounts, an Oracle Cloud tenancy, five regions and three continents. One repository contained 174 Helm charts that rendered 1,214 Argo CD Applications. At that size, a small configuration mistake can travel far; safe defaults and repeatable validation matter as much as automation.

The timing sharpened another operational concern. On Oct. 6, 2026, Red Hat published a walkthrough of using Argo CD ApplicationSets to provision hosted control planes from Git. The next day, GitHub reported an outage affecting Actions, Git operations and pull requests, following smaller incidents on Oct. 5 and 6. GitOps can now manage entire clusters, but its workflow still depends on services that may become unavailable. The architecture has to keep an interruption in the control plane from turning into an interruption for running workloads.

One hub, with authority kept in each spoke

The central hub ran the only Argo CD instance responsible for platform components. It held cluster registrations, the identity used by the Cluster API controller and three ApplicationSets: one distributed a baseline to each cluster through an app-of-apps chart, another installed shared platform services where they belonged, and a third managed the Cluster API definitions for the clusters.

That central view did not mean the hub owned every permission. Each spoke decided whether to trust the hub’s role through its own authentication configuration. The spoke also controlled the role Cluster API used inside it, while workloads received cloud roles through the spoke’s identity provider.

AWS made it possible to avoid keeping long-lived cross-account secrets in the hub. The hub obtained short-lived credentials, and each grant was declared on the spoke side and reviewed through Git. Oracle Cloud did not support the same arrangement in this setup: registration still read a stored credential from a vault. That difference is operationally important. A multi-cloud design should document where its security properties vary, rather than assuming one provider’s identity model carries over to another.

A small cluster contract drives the fleet

The key design choice was to represent each cluster with a configuration file containing about 20 values. Those described identity, account boundary, region, environment, API access, delegated DNS and certificates, plus any opt-outs. The system derived the rest: which Applications to install, IAM role names, Cluster API identity, DNS delegation and cost tags.

That contract made fleet changes easier to reason about because downstream settings did not need separate edits. Adding a cluster to a fleet meant changing a label on its registration; adding a component meant adding one line to a list. Matrix generators handled the resulting expansion across clusters.

The same pattern also had to work across infrastructure boundaries. A Middle East expansion brought an Oracle Cloud tenancy into the fleet. It joined through the existing cluster contract instead of receiving a one-off configuration. That provides a useful test for any platform abstraction: a new region or provider should fit the model without quietly creating a second operating process.

Render the graph before merging

With 174 charts in play, the main risk was not Argo CD itself. It was a values change that rendered successfully for one cluster and produced a bad manifest for many others. Continuous integration therefore rendered the full configuration graph for every merge request.

Validation covered charts on their own, app-of-apps charts against every cluster contract, cluster definitions, and each chart referenced by an Application using the exact values that Application would receive. This approach catches failures at the point where templates meet real fleet configuration, instead of relying on a successful render in a single representative environment.

CI also rendered the target branch and attached the resulting manifest diff to the merge request. Reviewers could inspect the resources that would change on clusters, rather than trying to infer deployment effects from template edits alone. For a large fleet, that distinction makes review more concrete: the proposed output becomes part of the change under discussion.

Drift measurements need comparable methods

Reported out-of-band configuration drift fell from 24.3 incidents per month before cutover to 0.9 afterward, using two 90-day windows. Those figures are useful context, but they do not support a clean percentage comparison. Before cutover, the count came from incident tickets tagged as drift, and tagging was incomplete. Afterward, the count came from controller diff events confirmed to have an out-of-band cause.

The more defensible takeaway is that instrumented drift was uncommon after the change, and the controller corrected each confirmed case within its reconciliation interval without manual intervention. Teams measuring a migration should use the same detection and classification method on both sides where possible. Otherwise, better instrumentation can change the count even when the underlying system improves.

Make destructive changes deliberate

At this scale, a bad commit can affect a large portion of the fleet. The team reduced that risk by changing Argo CD’s default deletion behavior in two places. ApplicationSets preserved deployed resources when an entry was removed, preventing that removal from cascading into resource deletion. Cluster objects carried a no-prune setting, so reverting a commit could not delete a running cluster.

These safeguards match the consequences of the actions they govern: reconciliation should be easier to reverse when the potential blast radius is large. The same principle applies during an outage. If Git or the hub becomes unavailable, workloads already running in the spokes can continue, even while GitOps deployments and configuration updates pause. A control-plane outage should leave the data plane serving traffic wherever the workload design allows it.

The hub also deserves treatment as a highly privileged system. Once a change merges, a compromised repository or CI pipeline can reach the resources the hub manages. Keeping credentials and permissions scoped, and making changes visible in review, are part of operating the control plane safely.

What onboarding time says—and doesn’t

For a new cluster, the process had four steps: create the matching IAM roles once in Terraform, write the cluster values file, open a merge request so CI renders the graph and posts its diff, then merge and label the registration. Older clusters originally built by hand could be adopted in place through an explicit import path.

Mean onboarding time was 4.0 days across eight earlier clusters, reconstructed from change tickets. After cutover, it was 1.7 hours across eleven clusters, measured from merge to first successful sync. The samples were small, the measurement windows differed, and the cutover removed manual approval steps, so the figures aren’t a direct apples-to-apples comparison. Still, the new workflow shows how a consistent contract and automated rendering can reduce the work required to bring a cluster under management.

The operating principle is familiar: declare the intended state, review it, and let controllers reconcile. Scale changes the cost of shortcuts. A credential in the hub, an unrendered values file or a default prune behavior may pass unnoticed in a small environment; across dozens of clusters, each can become a fleet-wide incident. The practical test for a GitOps control plane is whether it makes ordinary changes predictable and failure safe by design.

About the author: Nitish Mane is a Software Engineer at Philo, where he builds infrastructure for AI-assisted production operations. He previously worked on platform infrastructure at Lucid Motors, including a large-scale GitOps control plane.