SPARK: Secure Predictive Autoscaling for Robust Kubernetes

SPARK demonstrates a proactive autoscaling framework built on KEDA that uses machine learning to predict incoming load and scale Kubernetes workloads before demand spikes occur — reducing latency and error rates that reactive HPA-based scaling cannot avoid.

Submitted as an academic conference demonstration of leveraging KEDA's predictkube scaler for security-aware, predictive autoscaling.

---

System Architecture

┌──────────────┐  100 req/s   ┌──────────────────────┐
│   ATTACKER   │ ──────────>  │       VICTIM         │
│ (Vegeta load │              │  (scalingtest, :8081) │
│  generator)  │              │  Prometheus metrics  │
└──────────────┘              └──────────┬───────────┘
                                         │ scrape (:8081/metrics)
                                         ▼
                              ┌──────────────────────┐
                              │      PROMETHEUS       │
                              │  (time-series store)  │
                              └──────────┬────────────┘
                                         │ PromQL (3m history)
                                         ▼
                         ┌───────────────────────────────┐
                         │     KEDA ScaledObject          │
                         │  (predictkube trigger, :3s)    │
                         │                                │
                         │  ──> PredictKube ML Service    │
                         │      (gRPC, TLS, JWT auth)     │
                         │  <── 2-minute forecast          │
                         └───────────────┬───────────────┘
                                         │ manages
                                         ▼
                              ┌──────────────────────┐
                              │         HPA           │
                              │  min: 1 / max: 3      │
                              │  replicas of victim   │
                              └──────────────────────┘

Key insight: Traditional HPA reacts after a spike is measured. SPARK queries 3 minutes of historical Prometheus data, sends it to the PredictKube ML engine, receives a 2-minute-ahead forecast, and scales up before the load arrives.

---

Repository Structure

SPARK/
├── cmd/main.go                  # Victim service: HTTP server + Prometheus metrics
├── Dockerfile                   # Multi-stage Alpine build for victim image
├── Makefile                     # docker build & publish targets
├── go.mod / go.sum              # Go 1.21.6, module: nyit.com/m
│
├── victim.yaml                  # Kubernetes Deployment for victim (namespace: keda)
├── keda.yaml                    # ScaledObject + TriggerAuthentication (predictkube)
├── serviceMonitor.yaml          # Prometheus ServiceMonitor for victim metrics
├── prometheus_queries.md        # PromQL reference queries
│
└── modifiedKEDA/                # Modified KEDA source with predictkube scaler
    ├── pkg/scalers/
    │   └── predictkube_scaler.go    # Core ML-based predictive scaler (478 lines)
    ├── controllers/keda/
    │   └── scaledobject_controller.go  # ScaledObject reconciler → HPA management
    ├── apis/keda/v1alpha1/          # CRD type definitions
    ├── cmd/operator / adapter / webhooks/  # KEDA operator entrypoints
    └── tests/john/
        ├── services/
        │   ├── attacker.yaml        # Attacker deployment (Vegeta load generator)
        │   └── victim.yaml          # Test victim deployment
        ├── vegeta.go / vegeta1.go   # Load test implementation (100 req/s, 10s)
        ├── scaleObject.yaml         # Test ScaledObject config
        ├── predictKube.yaml         # PredictKube trigger examples
        ├── 1022.md                  # ScaledObject field reference & debugging
        └── 1113.md                  # Troubleshooting: gRPC errors, PromQL, rate() vs irate()

---

Components

Victim Service (cmd/main.go)

The victim is a Go HTTP server that exposes Prometheus metrics, simulates realistic load behaviour, and acts as the scaling target.

Prometheus metrics exposed on :8081/metrics:

Metric	Type	Labels	Description
testservice_http_requests_total	Counter	method, path, status	Total HTTP requests received
memory_usage_bytes	Gauge	—	Current pod memory usage
http_request_duration_seconds	Histogram	—	Request latency distribution

Behaviour under load:

• 1s simulated processing per request
• Returns 304 if memory exceeds MEMORY_THRESHOLD
• Returns 503 on timeout
• Logs all requests to /tmp/service.log

Kubernetes deployment (victim.yaml):

• Namespace: keda, Deployment: scalingtest
• Image: nyitkeda/scale-victim:latest
• Resources: 32Mi request / 64Mi limit
• Service: ClusterIP on port 8081, label app: scalingapp

Attacker (modifiedKEDA/tests/john/services/attacker.yaml)

A Vegeta-based load generator that continuously fires requests at the victim to trigger autoscaling.

image: nyitkeda/attackservice:latest
env:
  VICTIM_URL: sampletest:8080/ping
  FREQ: 100          # requests per second
  DURATION: 10       # seconds per burst

The Go implementation (vegeta1.go) uses the Vegeta library to fire at a fixed rate, reporting P50/P95/P99 latencies and success rates after each burst.

KEDA ScaledObject (keda.yaml)

Defines the predictive scaling trigger connecting Prometheus → PredictKube ML → HPA:

trigger: predictkube
metadata:
  predictHorizon: 2m          # forecast 2 minutes ahead
  historyTimeWindow: 3m       # use last 3 minutes of data
  prometheusAddress: http://localhost:9090
  query: sum(irate(testservice_http_requests_total[5m]))
  threshold: "8.50"           # predicted value to trigger scale-up
  activationThreshold: "3.5"
pollingInterval: 3
cooldownPeriod: 3
minReplicaCount: 1
maxReplicaCount: 3

Authentication uses a JWT API key stored in a Kubernetes secret (predictkube-secrets).

PredictKube Scaler (modifiedKEDA/pkg/scalers/predictkube_scaler.go)

The core of SPARK's predictive capability. On each polling interval:

1. Executes the PromQL range query against Prometheus (from now - historyTimeWindow to now)
2. Sends historical observations to the PredictKube ML engine via gRPC (api.predictkube.com:443)
3. Receives a predicted metric value for now + predictHorizon
4. Returns max(predicted, last_actual) to KEDA for threshold comparison

Security properties:

• JWT validation on API key
• TLS enforced on all gRPC connections
• Prometheus auth round-tripper support
• Non-root container user in Dockerfile

Prometheus ServiceMonitor (serviceMonitor.yaml)

namespace: monitoring      # Prometheus Operator target
scrape: keda/scalingapp   # victim service
port: http-metrics (8081)
interval: 10s
path: /metrics

---

Getting Started

Prerequisites

• Kubernetes cluster with KEDA installed
• Prometheus + kube-prometheus-stack (for ServiceMonitor support)
• PredictKube API key (predictkube.com)

Build & Push Victim Image

make build    # docker build
make push     # docker push nyitkeda/scale-victim:latest

Deploy

# Store PredictKube API key
kubectl create secret generic predictkube-secrets \
  --from-literal=apiKey=<base64-jwt> -n keda

# Deploy victim and KEDA trigger
kubectl apply -f victim.yaml
kubectl apply -f keda.yaml
kubectl apply -f serviceMonitor.yaml

# Deploy attacker
kubectl apply -f modifiedKEDA/tests/john/services/attacker.yaml

Observe Scaling

# Watch replicas change
kubectl get hpa -n keda -w

# Check KEDA scaler logs
kubectl logs -n keda deployment/keda-operator -f

# View live metrics
kubectl port-forward svc/prometheus-stack-kube-prom-prometheus 9090:9090 -n monitoring
# Open http://localhost:9090 and run:
# sum(irate(testservice_http_requests_total[5m]))

---

Metrics Reference

Access victim metrics directly:

curl http://localhost:8081/metrics

Example output:

# HELP testservice_http_requests_total Total number of HTTP requests
# TYPE testservice_http_requests_total counter
testservice_http_requests_total{method="GET",path="/",status="200"} 5

# HELP memory_usage_bytes Current memory usage in bytes
# TYPE memory_usage_bytes gauge
memory_usage_bytes 123456789

# HELP http_request_duration_seconds Duration of HTTP requests in seconds
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{le="0.005"} 0
http_request_duration_seconds_bucket{le="0.01"} 0
...

See prometheus_queries.md for full PromQL reference including success rates, error rates, and per-status-code aggregations.

---

Extending the Metrics

// Additional gauge
cpuUsage := promauto.NewGauge(prometheus.GaugeOpts{
    Name: "cpu_usage_percent",
    Help: "Current CPU usage percentage",
})

// Counter with labels
errorCounter = promauto.NewCounterVec(prometheus.CounterOpts{
    Name: "errors_total",
    Help: "Total number of errors",
}, []string{"type", "service"})

// Custom registry (advanced)
registry := prometheus.NewRegistry()
registry.MustRegister(customMetric)

// Push Gateway (batch jobs)
push.New("push-gateway:9091", "my_job").Collector(requestsCounter)

---

Notes

• Use irate() (not rate()) in the PromQL query for faster reaction to short spikes — see modifiedKEDA/tests/john/1113.md for detailed analysis
• Label cardinality directly affects Prometheus performance; keep label sets small
• Use Summary instead of Histogram when pre-calculated quantiles are needed
• NewConstMetric is available for metrics that cannot be directly instrumented