Operations

• Observability → Pod stack trace dumps for live thread-stack diagnostics via SIGUSR2.
• Database configuration for DB pool sizing and timeouts.
• LLM options → Azure Vision for OCR provider timeouts, retries, and failover.

​

v1.12.6 to v1.13 streaq migration

env:
  WORKER_PROVIDER: STREAQ_LOCAL

streaqWorkers:
  io:
    enabled: true
    workerName: io
    replicaCount: 1
    kedaScaler: false
  cpu:
    enabled: true
    workerName: cpu
    useFullImage: true
    workerCount: 1
    replicaCount: 1
    kedaScaler: false

{
  "input": "https://example.com/document.pdf",
  "settings": {
    "alpha": {
      "use_streaq_parse": true,
      "compute_target_override": "STREAQ_LOCAL"
    }
  }
}

1. Upgrade to v1.13 with worker.enabled: true. The pre-upgrade check requires an installed release marker from v1.12.6 or a later v1.12 patch and repeats the live Redis worker round trips before changing resources.
2. Scale the streaq pools for production load, then set both the trainable and non-trainable *_STREAQ_*_ROLLOUT_RATIO values to 1.0 for parse, extract, split, edit, and pipeline. Keep the legacy worker enabled while existing PostgreSQL-queue work drains. At full parse cutover, asynchronous requests with use_reducto_lite or reducto_lite.enabled are rejected because those modes require the legacy worker; the synchronous variants remain available because they run inline or route their base arm through streaq.
3. Poll GET /drain_status until drained is true. Then set worker.enabled: false, worker.priorityCount: 0, and gpuWorker.enabled: false. The Helm pre-upgrade check blocks this final apply unless the currently running HTTP deployment reports every effective rollout ratio as 1.0 and the live drain check passes. Ratios supplied via env, secretEnv, or envFrom are all read from that running process, so deploy the ratio change before the separate worker-disable apply.

​

Worker liveness probe

​

How it works

1. An asyncio WorkerWatchdog task runs alongside the worker’s job loops.
2. Every 5 seconds it writes a heartbeat file /tmp/worker-state containing <unix_timestamp> <oldest_in_flight_task_age_seconds>. Idle workers emit -1 for the age.
3. The kubelet runs bin/worker-liveness.sh as an exec probe. The script fails (exit non-zero) and triggers a restart when either:
   • the heartbeat file itself is older than WORKER_LIVENESS_WATCHDOG_STALE_SEC (event loop is wedged so the watchdog can’t tick); or
   • the oldest in-flight task age exceeds WORKER_LIVENESS_STUCK_THRESHOLD_SEC (a real job has hung beyond the threshold).
4. Idle workers always pass the probe. The watchdog ticks even with no work, and the age sentinel -1 is always treated as healthy.

​

Helm configuration

worker:
  livenessProbe:
    enabled: true                # set to false to disable the probe entirely
    stuckTaskThresholdSec: 1800  # restart if any in-flight task runs longer than this
    watchdogStaleSec: 60         # restart if the watchdog heartbeat hasn't ticked in this long
    periodSeconds: 30            # how often kubelet runs the probe
    timeoutSeconds: 5            # exec probe timeout
    initialDelaySeconds: 120     # grace period after pod start before probing begins
    failureThreshold: 2          # consecutive probe failures before pod restart

​

Defaults

​

Environment variables

​

Disabling the probe

worker:
  livenessProbe:
    enabled: false

​

Verifying it’s installed

kubectl describe pod -n reducto -l app=reducto-worker | grep -A 4 Liveness
# Liveness: exec [bin/worker-liveness.sh] delay=120s timeout=5s period=30s ...

kubectl exec -n reducto <pod-name> -- cat /tmp/worker-state
# 1778631873 -1        <- idle (age sentinel -1 is healthy)
# 1778631878 42        <- busy with a 42-second-old task