With multiple triggers (CPU and HTTP) and minReplicaCount of 0, KEDA erroneously scales to 0.

[mengland-noaa]

Report

With CPU and the http-external-scaler together as triggers in the same scaled object, the http scaler is superseding the CPU scaler.
With CPU under heavy load and with http request(s) it scales up successfully, but KEDA subsequently intervenes and scales to 0 ignoring CPU.

Expected Behavior

Under heavy CPU load even with no http requests KEDA should not scale down to 0.

Actual Behavior

The HTTP add on appears to be overriding the CPU scaler.

Steps to Reproduce the Problem

1. Create an nginx or other deployment paired with a CPU load test side car or init container. The memory scaler behaves similarly.
2. Send an http request and watch as it initially scales up then scales back down to 0.

apiVersion: v1
kind: Service
metadata:
  name: my-service
  namespace: my-namespace
spec:
  selector:
    app: my-app
  type: ClusterIP
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
      name: http
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
  namespace: my-namespace
spec:
  replicas: 0
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: my-container
          image: nginx
          ports:
            - containerPort: 80
          resources:
            requests:
              cpu: "100m"
              memory: "100Mi"
            limits:
              cpu: "500m"
              memory: "100Mi"
        - name: stress-ng
          image: polinux/stress-ng:latest
          command: ["/bin/sh", "-c"]
          args:
            - "echo 'Running stress-ng'; stress-ng --cpu 1 --vm 1 --vm-bytes 64M --timeout 300s; echo 'stress-ng finished'; sleep 3600"
          resources:
            requests:
              cpu: "100m"
              memory: "100Mi"
            limits:
              cpu: "1000m"
              memory: "1000Mi"
---
kind: ScaledObject
apiVersion: keda.sh/v1alpha1
metadata:
  name: my-scaled-object
  namespace: my-namespace
spec:
  initialCooldownPeriod: 120
  cooldownPeriod: 30
  minReplicaCount: 0
  maxReplicaCount: 4
  pollingInterval: 5
  fallback:
    failureThreshold: 5
    replicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-deployment
  advanced:
    horizontalPodAutoscalerConfig:
      name: custom-hpa-name
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 300
  triggers:
    - type: cpu
      name: cpu_trig
      metricType: Utilization
      metadata:
        value: "10"
    - type: external
      name: http_trig
      metadata:
        httpScaledObject: my-scaled-object
        hosts: "myhost"
        scalerAddress: keda-add-ons-http-external-scaler.keda:9090
---
kind: HTTPScaledObject
apiVersion: http.keda.sh/v1alpha1
metadata:
  name: my-scaled-object
  namespace: my-namespace
  annotations:
      httpscaledobject.keda.sh/skip-scaledobject-creation: "true"
spec:
  hosts:
  - "myhost"
  scalingMetric:
    requestRate:
      granularity: 1s
      targetValue: 2
      window: 1m
  scaledownPeriod: 300
  scaleTargetRef:
      name: my-deployment
      service: my-service
      port: 80
  replicas:
      min: 0
      max: 4
  targetPendingRequests: 1
---
kind: Service
apiVersion: v1
metadata:
  name: keda-add-ons-http-interceptor-proxy
  namespace: my-namespace
spec:
  type: ExternalName
  externalName: keda-add-ons-http-interceptor-proxy.keda.svc.cluster.local

Logs from KEDA HTTP operator

No response

HTTP Add-on Version

0.10.0

Kubernetes Version

None

Platform

Any

Anything else?

No response

[rd-zahari-aleksiev]

I have the same problem with HTTP scaler + Cron.
The cron declares 1 replica for some interval, but if there aren't http request it seems the HTTP scaler is pushing 'deactivate' to KEDA and KEDA tries to scale to zero, and few milliseconds later is activating again due the cron. So the replica is constantly starting and terminating.

I'm not GO dev, but from

func (e *impl) IsActive(
	ctx context.Context,
	sor *externalscaler.ScaledObjectRef,
) (*externalscaler.IsActiveResponse, error) {
	lggr := e.lggr.WithName("IsActive")

	gmr, err := e.GetMetrics(ctx, &externalscaler.GetMetricsRequest{
		ScaledObjectRef: sor,
	})
	if err != nil {
		lggr.Error(err, "GetMetrics failed", "scaledObjectRef", sor.String())
		return nil, err
	}

	metricValues := gmr.GetMetricValues()
	if err := errors.New("len(metricValues) != 1"); len(metricValues) != 1 {
		lggr.Error(err, "invalid GetMetricsResponse", "scaledObjectRef", sor.String(), "getMetricsResponse", gmr.String())
		return nil, err
	}
	metricValue := metricValues[0].GetMetricValue()

	active := metricValue > 0
	res := &externalscaler.IsActiveResponse{
		Result: active,
	}
	return res, nil
}

and for the push

func (e *impl) StreamIsActive(
	scaledObject *externalscaler.ScaledObjectRef,
	server externalscaler.ExternalScaler_StreamIsActiveServer,
) error {
	// this function communicates with KEDA via the 'server' parameter.
	// we call server.Send (below) every streamInterval, which tells it to immediately
	// ping our IsActive RPC
	ticker := time.NewTicker(streamInterval)
	defer ticker.Stop()
	for {
		select {
		case <-server.Context().Done():
			return nil
		case <-ticker.C:
			active, err := e.IsActive(server.Context(), scaledObject)
			if err != nil {
				e.lggr.Error(
					err,
					"error getting active status in stream",
				)
				return err
			}
			err = server.Send(&externalscaler.IsActiveResponse{
				Result: active.Result,
			})
			if err != nil {
				e.lggr.Error(
					err,
					"error sending the active result in stream",
				)
				return err
			}
		}
	}
}

I get the feeling the http scaler will push the deactivation to KEDA, no matter what else active scalers there are in the ScaledObject, and this makes KEDA deactivating the workload briefly? Is this something to be fixed in KEDA itself, or in the http-add-on?

Given the example from Implementing StreamIsActive the external push scaler should not push active=false ever ?

StreamIsActive is calling IsActive, and the stream(push) should not return false, IsActive is probably fine to return false when called during polling, just to be clear :-)

[rd-zahari-aleksiev]

I think is the same issue -> #1147

[rd-zahari-aleksiev]

@JorTurFer , what do you think, is my analysis makes sense? :-)

[leorniduv]

Hi, just wanted to +1 this issue, I'm having the same difficulties making a cron trigger work with a http scaler. Version 0.10.0.

    - kind: HTTPScaledObject
      apiVersion: http.keda.sh/v1alpha1
      metadata:
        name: my-app
        annotations:
          httpscaledobject.keda.sh/skip-scaledobject-creation: "true"
      spec:
        hosts:
          - my-app.hello.world
        scaleTargetRef:
          name: my-app
          kind: Deployment
          apiVersion: apps/v1
          service: my-app
          port: 11434
        replicas:
          min: 0
          max: 3
        scaledownPeriod: 30
        scalingMetric:
          concurrency:
            targetValue: 10
    - kind: ScaledObject
      apiVersion: keda.sh/v1alpha1
      metadata:
        name: my-app
      spec:
        scaleTargetRef:
          apiVersion: apps/v1
          kind: Deployment
          name: my-app
        pollingInterval: 10
        cooldownPeriod: 30
        initialCooldownPeriod: 0
        minReplicaCount: 0
        maxReplicaCount: 3
        triggers:
          - type: cron
            metadata:
              timezone: Europe/Paris
              start: 0 9 * * 1-5
              end: 0 19 * * 1-5
              desiredReplicas: "1"
          - type: external-push
            metadata:
              httpScaledObject: my-app
              scalerAddress: keda-add-ons-http-external-scaler.keda:9090

A pod spawns and then is immediately shut down

[leorniduv]

If anyone needs this I've found a very stupid way of making this setup work. You basically create another cron ScaledObject that kills the Http Add-on at the same time 🤦 I tested it and it seems to be doing the job: what needs to be killed outside the cron gets killed, and during the cron, my app is up and scales based on http traffic.

    - kind: HTTPScaledObject
      apiVersion: http.keda.sh/v1alpha1
      metadata:
        name: my-app-http
      spec:
        hosts:
          - bla.bla.foo
        scaleTargetRef:
          name: my-app
          kind: Deployment
          apiVersion: apps/v1
          service: my-app
          port: 3601
        replicas:
          min: 1 # changed from 0 to 1
          max: 3
        scaledownPeriod: 30
        scalingMetric:
          concurrency:
            targetValue: 10
    - kind: ScaledObject
      apiVersion: keda.sh/v1alpha1
      metadata:
        name: my-app-cron
      spec:
        scaleTargetRef:
          apiVersion: apps/v1
          kind: Deployment
          name: my-app
        pollingInterval: 10
        cooldownPeriod: 30
        initialCooldownPeriod: 0
        minReplicaCount: 0
        maxReplicaCount: 3
        triggers:
          - type: cron
            metadata:
              timezone: Europe/Paris
              start: 0 9 * * 1-5
              end: 0 19 * * 1-5
              desiredReplicas: "1"
  # New ScaledObject that kills the external-scaler on the same cron interval
    - kind: ScaledObject
      apiVersion: keda.sh/v1alpha1
      metadata:
        name: keda-external-scaler-cron
        namespace: keda
      spec:
        scaleTargetRef:
          apiVersion: apps/v1
          kind: Deployment
          name: keda-add-ons-http-external-scaler
        pollingInterval: 10
        cooldownPeriod: 30
        initialCooldownPeriod: 0
        minReplicaCount: 0
        maxReplicaCount: 3
        triggers:
          - type: cron
            metadata:
              timezone: Europe/Paris
              start: 0 9 * * 1-5
              end: 0 19 * * 1-5
              desiredReplicas: "1"

Again, this is very stupid and only works if you are using the Http add-on for a specific app (should probably deploy in namespace-mode and not cluster-wide now that I think of it ; to be a bit cleaner). + I'm not familiar with Go so I don't think I can pull off a PR to fix the real issue, this is just a hack to get it working :/