Running one OpenShift cluster with Argo CD is easy. Running thirty — three environments, four regions, two tenancy models — is a different problem. At that scale, the bottleneck is never the sync engine: it is the number of Application objects a human has to author, review and keep consistent. Thirty clusters times fifteen platform components is 450 Applications. Nobody maintains that by hand.

This article describes a working topology: a single root Application (app-of-apps) bootstrapping a set of ApplicationSet resources, cluster secrets carrying environment metadata as labels, and Kustomize overlays resolving the per-environment delta. Everything is declarative, everything is in Git, and adding a cluster is a one-line label change.

Context: why one Application per cluster breaks down

The naive pattern is a directory of Applications:

apps/ prd-eu-01-logging.yaml prd-eu-01-monitoring.yaml prd-eu-02-logging.yaml ...

This fails for three reasons.

Drift by copy-paste. Each file duplicates the same 25 lines. A fix to syncPolicy or to ignoreDifferences lands in 12 files and misses 18. Reviewers cannot diff 450 near-identical manifests meaningfully.

No lifecycle for clusters. Decommissioning a cluster means finding and deleting every Application referencing it. Forgotten Applications stay Unknown forever and pollute health dashboards and alerting.

No rollout control. You want dev first, then stg, then prd in batches of two. Static Applications with automated.selfHeal sync everything simultaneously the moment a commit lands on main.

ApplicationSet solves all three: it is a controller that generates and reconciles Applications from a generator (clusters, git, matrix, merge), owns their full lifecycle, and supports progressive rollout strategies.

Foundation: OpenShift GitOps and cluster registration

Start from the operator-managed instance. Verify what you are running — generator and strategy support varies significantly across versions:

oc get csv -n openshift-gitops -o custom-columns=NAME:.metadata.name,VERSION:.spec.version oc -n openshift-gitops exec deploy/openshift-gitops-server -- argocd version --short

Cluster secrets are your inventory

Argo CD stores remote clusters as Secret objects labeled argocd.argoproj.io/secret-type: cluster. The labels on that secret are the metadata the cluster generator will expose to templates. Treat them as your source of truth.

ARGOCD_HOST=$(oc get route openshift-gitops-server -n openshift-gitops -o jsonpath='{.spec.host}') argocd login "$ARGOCD_HOST" --sso --grpc-web # kubeconfig context must already exist locally argocd cluster add prd-eu-01-admin \ --name prd-eu-01 \ --label env=prd --label region=eu --label tier=platinum --label wave=1 \ --service-account argocd-manager \ --system-namespace openshift-gitops-agent \ --grpc-web --yes

For reproducibility, prefer declaring the secret in Git with the token injected by External Secrets Operator or Sealed Secrets:

apiVersion: v1 kind: Secret metadata: name: cluster-prd-eu-01 namespace: openshift-gitops labels: argocd.argoproj.io/secret-type: cluster env: prd region: eu tier: platinum type: Opaque stringData: name: prd-eu-01 server: https://api.prd-eu-01.example.com:6443 config: | { "bearerToken": "<injected-by-eso>", "tlsClientConfig": { "insecure": false, "caData": "<base64-cluster-ca>" } }

Verify the inventory is complete before building anything on top of it:

oc get secrets -n openshift-gitops -l argocd.argoproj.io/secret-type=cluster \ -o custom-columns=CLUSTER:.metadata.name,ENV:.metadata.labels.env,REGION:.metadata.labels.region argocd cluster list

Repository layout

Two repositories: one for the GitOps control plane, one (or several) for workloads. The control-plane repo:

gitops-platform/ ├── bootstrap/ │ ├── root-application.yaml # applied once, by hand or by the operator │ └── appsets/ │ ├── platform-components.yaml │ └── tenant-namespaces.yaml ├── projects/ │ ├── platform.yaml │ └── tenants.yaml └── components/ ├── cluster-logging/ │ ├── base/ │ │ ├── kustomization.yaml │ │ ├── namespace.yaml │ │ ├── operatorgroup.yaml │ │ ├── subscription.yaml │ │ └── clusterlogging.yaml │ └── overlays/ │ ├── dev/kustomization.yaml │ ├── stg/kustomization.yaml │ └── prd/kustomization.yaml ├── compliance-operator/ └── network-policies/

The contract is strict: every component exposes overlays/<env> for every env label used on cluster secrets. A missing overlay must fail loudly, not silently fall back to base.

The root Application (app-of-apps)

One object bootstraps everything. It is the only manifest you ever apply imperatively.

apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: root namespace: openshift-gitops annotations: argocd.argoproj.io/sync-wave: "-10" finalizers: - resources-finalizer.argocd.argoproj.io spec: project: default source: repoURL: https://github.com/acme/gitops-platform.git targetRevision: main path: bootstrap directory: recurse: true include: '{appsets/*.yaml,../projects/*.yaml}' destination: server: https://kubernetes.default.svc namespace: openshift-gitops syncPolicy: automated: prune: true selfHeal: true syncOptions: - ApplyOutOfSyncOnly=true - ServerSideApply=true 
oc apply -f bootstrap/root-application.yaml argocd app get root --grpc-web

From here, adding an ApplicationSet is a pull request against bootstrap/appsets/.

ApplicationSet with a matrix generator

The cluster generator alone gives you one Application per cluster. Combined with a git directory generator in a matrix, you get one Application per (cluster, component) pair — the 450 objects, generated from 40 lines.

apiVersion: argoproj.io/v1alpha1 kind: ApplicationSet metadata: name: platform-components namespace: openshift-gitops spec: goTemplate: true goTemplateOptions: ["missingkey=error"] generators: - matrix: generators: - clusters: selector: matchExpressions: - key: env operator: In values: [dev, stg, prd] - git: repoURL: https://github.com/acme/gitops-platform.git revision: main directories: - path: components/* - path: components/_experimental exclude: true template: metadata: name: '{{.nameNormalized}}-{{.path.basename}}' namespace: openshift-gitops labels: env: '{{index .metadata.labels "env"}}' region: '{{index .metadata.labels "region"}}' component: '{{.path.basename}}' finalizers: - resources-finalizer.argocd.argoproj.io spec: project: platform source: repoURL: https://github.com/acme/gitops-platform.git targetRevision: main path: '{{.path.path}}/overlays/{{index .metadata.labels "env"}}' destination: server: '{{.server}}' syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true - ServerSideApply=true - RespectIgnoreDifferences=true retry: limit: 5 backoff: duration: 20s factor: 2 maxDuration: 5m

goTemplateOptions: ["missingkey=error"] is not cosmetic: without it, a cluster secret missing the env label silently renders components/cluster-logging/overlays/ and syncs the wrong tree. With it, the generator errors and the ApplicationSet reports the failure.

Dry-run the generation before merging:

argocd appset generate bootstrap/appsets/platform-components.yaml -o yaml | \ yq '.[].metadata.name'

Then inspect what the controller actually produced:

argocd appset get platform-components argocd app list -l component=cluster-logging -o wide oc get applications.argoproj.io -n openshift-gitops -l env=prd --no-headers | wc -l

Kustomize overlays carry the environment delta

Keep base cluster-agnostic. The overlay encodes retention, replicas, storage class, and nothing else.

# components/cluster-logging/overlays/prd/kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization namespace: openshift-logging resources: - ../../base patches: - target: group: logging.openshift.io version: v1 kind: ClusterLogging name: instance patch: |- - op: replace path: /spec/logStore/elasticsearch/nodeCount value: 5 - op: replace path: /spec/logStore/retentionPolicy/application/maxAge value: 30d - op: add path: /spec/logStore/elasticsearch/storage/storageClassName value: gp3-csi labels: - pairs: app.kubernetes.io/managed-by: openshift-gitops env: prd includeSelectors: false

Validate locally — this is the cheapest CI gate you can add:

for env in dev stg prd; do for c in components/*/; do kustomize build "${c}overlays/${env}" >/dev/null \ || echo "BROKEN: ${c}overlays/${env}" done done

Ordering: sync waves inside an Application

Within one Application, argocd.argoproj.io/sync-wave sequences resources. The canonical operator pattern is namespace → OperatorGroup → Subscription → custom resource, with a guard so Argo CD does not dry-run a CR whose CRD does not exist yet.

apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: cluster-logging namespace: openshift-logging annotations: argocd.argoproj.io/sync-wave: "0" spec: channel: stable-6.1 name: cluster-logging source: redhat-operators sourceNamespace: openshift-marketplace installPlanApproval: Automatic --- apiVersion: logging.openshift.io/v1 kind: ClusterLogging metadata: name: instance namespace: openshift-logging annotations: argocd.argoproj.io/sync-wave: "10" argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true spec: managementState: Managed

Waves are per Application, not global. To order across Applications, put the sync-wave annotation on the generated Application itself (the root app treats them as children) or rely on RollingSync.

Progressive rollout across 30 clusters

Enable progressive syncs on the operator-managed instance, then declare steps matching the labels you set in the ApplicationSet template.

apiVersion: argoproj.io/v1beta1 kind: ArgoCD metadata: name: openshift-gitops namespace: openshift-gitops spec: applicationSet: extraCommandArgs: - --enable-progressive-syncs controller: sharding: enabled: true replicas: 3 processors: operation: 25 status: 50 resources: requests: {cpu: "1", memory: 2Gi} limits: {memory: 6Gi} repo: replicas: 3 
# inside spec of the ApplicationSet strategy: type: RollingSync rollingSync: steps: - matchExpressions: - key: env operator: In values: [dev] - matchExpressions: - key: env operator: In values: [stg] maxUpdate: 50% - matchExpressions: - key: env operator: In values: [prd] maxUpdate: 2

Each step waits for the previous batch to reach Healthy. A failing dev cluster stops the promotion before it reaches production. Drive and observe it from the CLI:

argocd app wait -l env=stg --health --timeout 900 --grpc-web argocd app list -l env=prd --output wide | awk '$3!="Synced" || $4!="Healthy"' argocd app diff prd-eu-01-cluster-logging --local components/cluster-logging/overlays/prd

Three application-controller shards comfortably handle 30 clusters; the default sharding algorithm distributes clusters by hash. If one cluster is much larger than the others, switch to round-robin via spec.controller.env with ARGOCD_CONTROLLER_SHARDING_ALGORITHM.

Security best practices

Never grant cluster-admin to the Argo CD service account on remote clusters. argocd cluster add --service-account lets you bind a scoped ClusterRole. Build it from the component inventory: operator subscriptions, namespaces, network policies, RBAC — not */*. Audit periodically with oc auth can-i --list --as=system:serviceaccount:openshift-gitops-agent:argocd-manager.

Make AppProject the real boundary. ApplicationSet can generate any Application; the project is what constrains it. Deny cluster-scoped resources you never intend to manage:

apiVersion: argoproj.io/v1alpha1 kind: AppProject metadata: name: platform namespace: openshift-gitops spec: sourceRepos: - https://github.com/acme/gitops-platform.git destinations: - server: '*' namespace: 'openshift-*' - server: '*' namespace: 'platform-*' clusterResourceWhitelist: - group: '' kind: Namespace - group: operators.coreos.com kind: '*' - group: rbac.authorization.k8s.io kind: ClusterRole namespaceResourceBlacklist: - group: '' kind: ResourceQuota - group: security.openshift.io kind: SecurityContextConstraints signatureKeys: - keyID: 4AEE18F83AFDEB23 orphanedResources: warn: true

Require signed commits. signatureKeys on the project plus GPG keys loaded into Argo CD means an unsigned commit on main cannot sync. This is the single most effective control against a compromised CI token pushing to the GitOps repo.

No plaintext secrets in Git, ever. Use External Secrets Operator or Sealed Secrets, and add spec.extraConfig resource exclusions so Argo CD does not cache sensitive objects it has no business tracking. Cluster bearer tokens in openshift-gitops are the crown jewels: restrict get secrets in that namespace to the operator's own service accounts.

Lock down the ApplicationSet controller. It can create and delete Applications cluster-wide. Run it with --policy create-update during onboarding so a bad generator cannot mass-delete Applications, and disable SCM providers (--enable-scm-providers=false) if you only use cluster and git generators — SCM generators can be abused to reach arbitrary endpoints with the controller's token.

Use OpenShift OAuth, disable the local admin. Set spec.sso.provider: dex with the OpenShift connector (or your OIDC IdP), map groups to Argo CD RBAC roles, and set spec.disableAdmin: true once SSO works. Give tenants sync on their project only, never applications, delete, */*.

Pin everything. targetRevision: main is convenient for the control plane but dangerous for workloads: pin tenant Applications to tags or commit SHAs, and use installPlanApproval: Manual for operators on production clusters so an operator upgrade is a deliberate Git change.

Conclusion

The pattern that scales to 30 clusters has four moving parts, and none of them is a per-cluster manifest:

  • Cluster secrets with labels are the inventory. env, region, tier drive every downstream decision. Onboarding a cluster is one labeled secret; offboarding is deleting it, and the ApplicationSet garbage-collects the Applications.
  • A single root Application bootstraps the ApplicationSets and AppProjects. Exactly one oc apply in the entire lifecycle of the platform.
  • Matrix generators (clusters × git directories) turn 40 lines of YAML into hundreds of Applications, with goTemplateOptions: ["missingkey=error"] to fail fast on incomplete metadata.
  • Kustomize overlays plus sync waves plus RollingSync handle, respectively, the environment delta, intra-application ordering, and inter-cluster promotion.

Add kustomize build validation and argocd appset generate to CI, scope your remote service accounts, enforce signed commits, and the thirty-cluster fleet becomes less operational work than the three-cluster one you started with.