Quotas, LimitRange and ResourceQuota on OpenShift: Industrializing Multi-Tenancy per Project

A shared OpenShift cluster behaves like a shared bank account: it works perfectly until someone forgets to tell the others they spent everything. One team ships a Deployment with replicas: 30 and no resource requests, the scheduler happily packs it onto every worker node, and by Monday morning three other teams are staring at 0/12 nodes are available: insufficient memory.

Resource governance is not about saying "no" to developers. It is about making the cluster's capacity explicit, bounded and auditable per tenant, so that one namespace's bad day never becomes everyone's incident. This article walks through the three layers OpenShift gives you — LimitRange, ResourceQuota, and the OpenShift-specific ClusterResourceQuota — and then wires them into a self-applying project template so that every new project is born governed, with no ticket and no manual step.

The problem: why namespaces alone are not isolation

A namespace (or OpenShift project) is an RBAC and naming boundary. It is not a capacity boundary. By default:

• A pod without resources.requests is scheduled as BestEffort. It competes for CPU with everything else and is the first candidate for eviction under memory pressure — including pods belonging to other teams on the same node.
• A pod without resources.limits can consume all the CPU on a node, and OOM-kill its neighbours by exhausting node memory.
• Nothing stops a team from creating 5,000 ConfigMaps, 200 Services of type LoadBalancer, or 40 TiB of PVCs.

The consequences are concrete: degraded latency for unrelated workloads, etcd bloat, surprise cloud bills for load balancers, and a cluster autoscaler that scales to the moon because a broken CronJob keeps spawning pods.

The fix is a layered model:

Each layer is necessary. LimitRange without quota caps nothing globally. Quota without LimitRange breaks every pod that forgets to set requests.

Layer 1: LimitRange, the per-object guardrail

LimitRange does two jobs: it injects defaults into pods that do not declare resources, and it rejects objects outside a min/max envelope at admission time.

apiVersion: v1
kind: LimitRange
metadata:
  name: tenant-defaults
  namespace: team-payments-dev
spec:
  limits:
    - type: Container
      default:                 # becomes spec.containers[].resources.limits
        cpu: 500m
        memory: 512Mi
      defaultRequest:          # becomes spec.containers[].resources.requests
        cpu: 50m
        memory: 128Mi
      min:
        cpu: 10m
        memory: 32Mi
      max:
        cpu: "4"
        memory: 8Gi
      maxLimitRequestRatio:    # blocks 10m request / 4 CPU limit overcommit abuse
        cpu: "20"
        memory: "4"
    - type: Pod
      max:
        cpu: "8"
        memory: 16Gi
    - type: PersistentVolumeClaim
      min:
        storage: 1Gi
      max:
        storage: 200Gi

Two OpenShift-specific types are worth adding on clusters that use the internal registry heavily:

    - type: openshift.io/Image
      max:
        storage: 2Gi           # rejects pushing an image layer set above 2Gi
    - type: openshift.io/ImageStream
      max:
        openshift.io/image-tags: "30"
        openshift.io/images: "60"

Verify what a tenant actually gets:

oc describe limitrange tenant-defaults -n team-payments-dev

# Prove the defaulting works
oc run nginx --image=registry.access.redhat.com/ubi9/nginx-124 -n team-payments-dev
oc get pod nginx -n team-payments-dev -o jsonpath='{.spec.containers[0].resources}{"\n"}'
# {"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"50m","memory":"128Mi"}}

maxLimitRequestRatio is the under-used knob. It is what prevents a team from requesting 10m CPU to dodge the quota while setting a 4-core limit and then saturating a node.

Layer 2: ResourceQuota, the namespace budget

ResourceQuota caps aggregate consumption. Split it into two objects — compute and objects — so that a developer hitting the Service limit gets a readable error instead of a wall of text.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-quota
  namespace: team-payments-dev
spec:
  hard:
    requests.cpu: "8"
    requests.memory: 16Gi
    limits.cpu: "16"
    limits.memory: 32Gi
    pods: "60"
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: object-quota
  namespace: team-payments-dev
spec:
  hard:
    persistentvolumeclaims: "20"
    requests.storage: 500Gi
    gold.storageclass.storage.k8s.io/requests.storage: 100Gi
    gold.storageclass.storage.k8s.io/persistentvolumeclaims: "5"
    services: "30"
    services.loadbalancers: "0"
    services.nodeports: "0"
    secrets: "60"
    configmaps: "60"
    count/deployments.apps: "40"
    count/cronjobs.batch: "10"
    openshift.io/imagestreams: "25"

Setting services.loadbalancers: "0" and services.nodeports: "0" is a cheap way to force every tenant through Routes/Ingress instead of opening cloud load balancers at will.

Scopes: treat batch and interactive workloads differently

Quota scopes let you carve the budget. A classic pattern is to reserve headroom for long-running services by bounding what short-lived jobs can take:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: batch-quota
  namespace: team-payments-dev
spec:
  scopes:
    - Terminating            # pods with activeDeadlineSeconds set
  hard:
    pods: "20"
    requests.cpu: "4"
    requests.memory: 8Gi

And with scopeSelector, you can stop tenants from grabbing high-priority scheduling slots:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: no-high-priority
  namespace: team-payments-dev
spec:
  scopeSelector:
    matchExpressions:
      - operator: In
        scopeName: PriorityClass
        values: ["system-cluster-critical", "system-node-critical"]
  hard:
    pods: "0"

The gotchas that will page you

• Quota on requests.* makes requests mandatory. Any pod without them is rejected with must specify requests.cpu. This is exactly why the LimitRange must be created before or with the quota.
• Quota rejections surface on the controller, not the pod. A blocked Deployment shows nothing in oc get pods. Look at the ReplicaSet: oc describe rs -n team-payments-dev | grep -A5 Events.
• Lowering a quota never evicts running pods. It only blocks new creations until usage drops below the new ceiling.
• count/<resource>.<group> works for any resource, including CRDs — useful to cap count/virtualmachines.kubevirt.io or count/routes.route.openshift.io.

Layer 3: ClusterResourceQuota, the team budget across projects

Real teams own several projects: team-payments-dev, -staging, -sandbox. Per-namespace quotas let a team multiply its footprint simply by creating more projects. OpenShift's ClusterResourceQuota (CRQ) is a cluster-scoped object that applies a single shared budget across all namespaces matching a selector.

apiVersion: quota.openshift.io/v1
kind: ClusterResourceQuota
metadata:
  name: crq-team-payments
spec:
  selector:
    labels:
      matchLabels:
        tenant: payments
  quota:
    hard:
      requests.cpu: "40"
      requests.memory: 80Gi
      limits.cpu: "80"
      limits.memory: 160Gi
      pods: "300"
      persistentvolumeclaims: "60"
      requests.storage: 2Ti

The selector can also match project annotations, which is how you bind a quota to everything a given user self-provisions:

oc create clusterresourcequota crq-user-alice \
  --project-annotation-selector=openshift.io/requester=alice \
  --hard=pods=50 \
  --hard=requests.cpu=10 \
  --hard=requests.memory=20Gi

If you specify both labels and annotations in spec.selector, a project must match both to be included. Start with labels — they are easier to drive from GitOps.

AppliedClusterResourceQuota: what the tenant sees

A developer with admin on their project cannot read cluster-scoped objects. OpenShift exposes a namespaced, read-only projection for exactly this: AppliedClusterResourceQuota.

# As the tenant, inside one of their projects
oc get appliedclusterresourcequota -n team-payments-dev

oc describe appliedclusterresourcequota crq-team-payments -n team-payments-dev
# Name:       crq-team-payments
# Namespace:  team-payments-dev
# Resource            Used    Hard
# --------            ----    ----
# pods                187     300
# requests.cpu        26200m  40
# requests.memory     52Gi    80Gi

Put that command in your onboarding docs. "Why is my deployment stuck?" becomes self-service.

Industrializing it: the self-applying project template

Creating these objects by hand for each new project does not scale. OpenShift lets you replace the template used by oc new-project / "Create Project" in the console, so every project is provisioned with its guardrails already in place.

Start from the built-in template:

oc adm create-bootstrap-project-template -o yaml > project-request-template.yaml

Then extend it. The key parameters injected by the API server are PROJECT_NAME, PROJECT_DISPLAYNAME, PROJECT_DESCRIPTION, PROJECT_ADMIN_USER and PROJECT_REQUESTING_USER.

apiVersion: template.openshift.io/v1
kind: Template
metadata:
  name: project-request
  namespace: openshift-config
objects:
  - apiVersion: project.openshift.io/v1
    kind: Project
    metadata:
      name: ${PROJECT_NAME}
      annotations:
        openshift.io/description: ${PROJECT_DESCRIPTION}
        openshift.io/display-name: ${PROJECT_DISPLAYNAME}
        openshift.io/requester: ${PROJECT_REQUESTING_USER}
      labels:
        tenant: unassigned          # platform team relabels to attach the CRQ
  - apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: admin
      namespace: ${PROJECT_NAME}
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: admin
    subjects:
      - apiGroup: rbac.authorization.k8s.io
        kind: User
        name: ${PROJECT_ADMIN_USER}
  - apiVersion: v1
    kind: LimitRange
    metadata:
      name: tenant-defaults
      namespace: ${PROJECT_NAME}
    spec:
      limits:
        - type: Container
          default:
            cpu: 500m
            memory: 512Mi
          defaultRequest:
            cpu: 50m
            memory: 128Mi
          max:
            cpu: "4"
            memory: 8Gi
          min:
            cpu: 10m
            memory: 32Mi
          maxLimitRequestRatio:
            cpu: "20"
  - apiVersion: v1
    kind: ResourceQuota
    metadata:
      name: compute-quota
      namespace: ${PROJECT_NAME}
    spec:
      hard:
        requests.cpu: ${{QUOTA_REQUESTS_CPU}}
        requests.memory: ${QUOTA_REQUESTS_MEMORY}
        limits.cpu: ${{QUOTA_LIMITS_CPU}}
        limits.memory: ${QUOTA_LIMITS_MEMORY}
        pods: ${{QUOTA_PODS}}
        services.loadbalancers: "0"
        services.nodeports: "0"
  - apiVersion: networking.k8s.io/v1
    kind: NetworkPolicy
    metadata:
      name: default-deny-ingress
      namespace: ${PROJECT_NAME}
    spec:
      podSelector: {}
      policyTypes:
        - Ingress
  - apiVersion: networking.k8s.io/v1
    kind: NetworkPolicy
    metadata:
      name: allow-from-openshift-ingress
      namespace: ${PROJECT_NAME}
    spec:
      podSelector: {}
      ingress:
        - from:
            - namespaceSelector:
                matchLabels:
                  policy-group.network.openshift.io/ingress: ""
      policyTypes:
        - Ingress
parameters:
  - name: PROJECT_NAME
  - name: PROJECT_DISPLAYNAME
  - name: PROJECT_DESCRIPTION
  - name: PROJECT_ADMIN_USER
  - name: PROJECT_REQUESTING_USER
  - name: QUOTA_REQUESTS_CPU
    value: "4"
  - name: QUOTA_REQUESTS_MEMORY
    value: 8Gi
  - name: QUOTA_LIMITS_CPU
    value: "8"
  - name: QUOTA_LIMITS_MEMORY
    value: 16Gi
  - name: QUOTA_PODS
    value: "30"

Note the ${{...}} syntax for numeric values: single-brace substitution always yields a string, which the quota schema accepts for pods but not cleanly for everything — using ${{ }} emits a proper JSON value. Keep memory values as plain strings.

Install and activate it:

oc create -f project-request-template.yaml -n openshift-config

oc patch project.config.openshift.io/cluster --type=merge \
  -p '{"spec":{"projectRequestTemplate":{"name":"project-request"}}}'

# The openshift-apiserver rolls out; wait for it
oc rollout status deploy/apiserver -n openshift-apiserver --timeout=5m

Test end to end, as a normal user, never as kubeadmin:

oc login -u alice
oc new-project demo-governance
oc get limitrange,resourcequota,networkpolicy -n demo-governance

Store this template in Git and apply it with Argo CD or oc apply. A project template is cluster policy: it deserves a pull request and a reviewer.

Auditing consumption

Governance without measurement is wishful thinking. Three levels of audit:

Ad hoc, across the cluster:

# Namespaces closest to saturation on memory requests
oc get resourcequota --all-namespaces \
  -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,USED:.status.used.requests\.memory,HARD:.status.hard.requests\.memory'

# Team-level view
oc get clusterresourcequota -o custom-columns=\
'NAME:.metadata.name,CPU_USED:.status.total.used.requests\.cpu,CPU_HARD:.status.total.hard.requests\.cpu,NS:.status.namespaces[*].namespace'

# Find ungoverned namespaces — the real risk
for ns in $(oc get ns -o name | cut -d/ -f2 | grep -v '^openshift\|^kube\|^default'); do
  [ "$(oc get resourcequota -n "$ns" --no-headers 2>/dev/null | wc -l)" -eq 0 ] && echo "NO QUOTA: $ns"
done

Continuous, with Prometheus. kube-state-metrics exposes kube_resourcequota with a type label of hard or used:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: tenant-quota-alerts
  namespace: openshift-monitoring
spec:
  groups:
    - name: quotas
      rules:
        - alert: NamespaceQuotaNearlyExhausted
          expr: |
            kube_resourcequota{type="used", resource=~"requests.cpu|requests.memory|pods"}
              / ignoring(type)
            kube_resourcequota{type="hard", resource=~"requests.cpu|requests.memory|pods"}
              > 0.9
          for: 15m
          labels:
            severity: warning
          annotations:
            summary: "{{ $labels.namespace }} is at >90% of {{ $labels.resource }}"

(Deploy user-workload rules in the tenant's own namespace; cluster-wide rules belong with the platform monitoring stack.)

Capacity reconciliation. Periodically compare the sum of all requests.cpu hard values against real allocatable capacity. Overcommitting quotas 2–3× is normal and healthy; overcommitting 10× means your quotas are decorative.

Security best practices

• Tenants must not be able to edit their own quotas. The built-in admin ClusterRole grants get/list/watch on resourcequotas but not create/update/delete. Verify it after any RBAC customization:

  oc auth can-i delete resourcequota -n team-payments-dev --as alice   # must print "no"
  oc auth can-i patch limitrange -n team-payments-dev --as alice       # must print "no"

• Control self-provisioning. If anyone can create unlimited projects, per-namespace quotas are trivially bypassed. Either remove the self-provisioner binding and provision projects via GitOps, or always pair self-service with an annotation-selector ClusterResourceQuota per user/team.
• Quota the control plane too. Caps on secrets, configmaps and count/* protect etcd from a runaway operator or CI loop — an availability control for the whole cluster, not just a billing one.
• Block implicit network exposure with services.loadbalancers: "0" and services.nodeports: "0", and ship a default-deny NetworkPolicy in the same template. Resource isolation and network isolation should be provisioned together.
• Deny high-priority classes by quota scope so tenants cannot set priorityClassName: system-cluster-critical to jump the queue and preempt platform workloads.
• Keep SCCs in the picture. restricted-v2 should remain the default; a tenant able to run privileged pods can mount host paths and read other tenants' data regardless of any quota.
• Treat the project template as audited policy. Changes to it propagate to every future project. Git, review, signed commits, and an alert on project.config.openshift.io/cluster drift.
• Label-driven CRQ membership is a privilege. Whoever can set tenant=payments on a namespace can move it into (or out of) a budget. Restrict namespace labelling to the platform team.

Key takeaways

Multi-tenancy on OpenShift is an engineering problem with a well-defined answer, and it is only three objects deep:

1. LimitRange makes every pod well-formed — defaults for requests and limits, a sane min/max envelope, and maxLimitRequestRatio to stop overcommit games. Always deploy it before a quota that constrains requests.*.
2. ResourceQuota caps a namespace on compute, storage per StorageClass, and object counts. Use scopes to separate batch from long-running workloads and to forbid critical priority classes.
3. ClusterResourceQuota caps a team across all its projects, matched by namespace labels or the openshift.io/requester annotation, and AppliedClusterResourceQuota makes that budget visible to developers without any cluster-scoped RBAC.
4. The project template is what makes it industrial. oc adm create-bootstrap-project-template, extend it with LimitRange, ResourceQuota, RoleBinding and NetworkPolicy, store it in Git, and point project.config.openshift.io/cluster at it. Governance becomes a property of project creation instead of a recurring ticket.
5. Audit continuously. Alert at 90% saturation, hunt for namespaces with no quota at all, and reconcile the sum of quotas against real allocatable capacity every quarter.

Get these in place and the noisy-neighbour incident stops being a postmortem — it becomes a 403 exceeded quota that the responsible team sees immediately, in their own namespace, with their own name on it.