Kubernetes HorizontalPodAutoscaler has the ability to dynamically adjust Deployment replicas depending on resource utilization.

Me and my friends found this weird behavior with HorizontalPodAutoscaler and Deployment when we tried to apply HPA and remove replicas from Deployment manifest.

Steps to Reproduce

  1. 1.

    Apply Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: app-one
  name: app-one
spec:
  replicas: 2
  selector:
    matchLabels:
      app: app-one
  template:
    metadata:
      labels:
        app: app-one
    spec:
      containers:
        - name: busybox
          image: busybox
          command: ["/bin/sh", "-c", "sleep infinity"]
  1. 2.

    Check status

21:47:18 ❯ kubectl get deploy,replicasets,pods,hpa
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/app-one   2/2     2            2           104s

NAME                                DESIRED   CURRENT   READY   AGE
replicaset.apps/app-one-557b57844   2         2         2       104s

NAME                          READY   STATUS    RESTARTS   AGE
pod/app-one-557b57844-4gn8w   1/1     Running   0          104s
pod/app-one-557b57844-wtj8v   1/1     Running   0          104s
  1. 3.

    Apply HPA

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: app-one
spec:
  maxReplicas: 3
  minReplicas: 2
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: app-one
  1. 4.

    Check status. Pods are unaffected.

21:48:58 ❯ kubectl get deploy,replicasets,pods,hpa
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/app-one   2/2     2            2           3m24s

NAME                                DESIRED   CURRENT   READY   AGE
replicaset.apps/app-one-557b57844   2         2         2       3m24s

NAME                          READY   STATUS    RESTARTS   AGE
pod/app-one-557b57844-4gn8w   1/1     Running   0          3m24s
pod/app-one-557b57844-wtj8v   1/1     Running   0          3m24s

NAME                                          REFERENCE            TARGETS              MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/app-one   Deployment/app-one   cpu: <unknown>/80%   2         3         2          30s
  1. 5.

    Remove replicas from Deployment manifest as HPA will adjust/manage Deployment replicas

@@ -5,7 +5,6 @@
     app: app-one
   name: app-one
 spec:
-  replicas: 2
   selector:
     matchLabels:
       app: app-one
  1. 6.

    Reapply Deployment and then check status

21:53:54 ❯ kubectl get deploy,replicasets,pods,hpa
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/app-one   2/2     2            2           8m20s

NAME                                DESIRED   CURRENT   READY   AGE
replicaset.apps/app-one-557b57844   2         2         2       8m20s

NAME                          READY   STATUS        RESTARTS   AGE
pod/app-one-557b57844-4gn8w   1/1     Running       0          8m20s
pod/app-one-557b57844-wtj8v   1/1     Terminating   0          8m20s
pod/app-one-557b57844-zsfhr   1/1     Running       0          11s

NAME                                          REFERENCE            TARGETS              MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/app-one   Deployment/app-one   cpu: <unknown>/80%   2         3         1          5m26s

Although HPA minReplicas has the same value with previously running replicas, somehow one pod was terminated and replaced with a new pod. The replicas value is still same at 2 and the ReplicaSet is also the same.

AFAIK this behavior is not officially documented or reported.

Argo CD Workaround

Argo CD has workaround to ignore Deployment replicas drift.

spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas

Argo will ignore Deployment replicas field in case Deployment manifest still have replicas defined. So if HPA is also applied and have adjusted/managed Deployment replicas, Argo will not detect any drift between Deployment replicas in resource and manifest.