Observability & Monitoring

Overview

On-premise Reducto deployments include a built-in observability stack called ClickStack, which provides:

HyperDX: Unified observability UI for logs, traces, and metrics
ClickHouse: High-performance analytics database for telemetry storage
OTEL Collector: OpenTelemetry collector for ingesting and routing telemetry data

ClickStack is enabled by setting clickstack.enabled: true in your Helm values. Everything else is automatic. No additional setup required. Telemetry is part of your on-premise security boundary. Reducto emits logs to stdout and can route traces, metrics, and logs through OpenTelemetry, but you control where telemetry is stored, who can access it, and how long it is retained. See the on-prem security model for the shared responsibility model. Reducto telemetry is designed for operational metadata. It should not contain document content, OCR text, extracted values, prompts, model outputs, API tokens, access keys, secrets, or other customer content. File names and URLs can reveal customer identity and should be redacted or avoided before telemetry leaves the deployment.

Accessing HyperDX

Default Credentials

When ClickStack is enabled, a seed admin user is automatically created on first install with default credentials. Contact the Reducto team for the default login details, or configure your own credentials in your Helm values (see Configuration below).

Change the default password immediately after first login.

Accessing the UI

HyperDX can be exposed via:

Ingress: Set clickstack.hyperdx.ingress.enabled: true with your domain
Tailscale: Set clickstack.hyperdx.exposure.tailscale.enabled: true for private access
Cloudflare Tunnel: Set clickstack.hyperdx.exposure.cloudflareTunnel.enabled: true
Port-forward (for testing): kubectl port-forward svc/<release>-clickstack-app 3000:3000

Prometheus Scraping

Prometheus endpoints are intended for internal scraping by monitoring systems. Do not expose /metrics, /prometheus, ClickStack, HyperDX, ClickHouse, or the OTEL collector to the public internet. To collect metrics from services that expose Prometheus endpoints (like NGINX ingress controllers), enable the Prometheus receiver on the OTEL collector together with the Target Allocator (TA) subchart. TA shards scrape targets across collector replicas via consistent-hashing so each target is scraped exactly once. See Scaling the OTEL Collector for the multi-replica rationale.

prometheusScrape:
  enabled: true

targetAllocator:
  enabled: true
  targetAllocator:
    config:
      collector_selector:
        matchlabels:
          app.kubernetes.io/instance: <release-name>
          app.kubernetes.io/name: otel-collector
      config:
        scrape_configs:
          - job_name: nginx-ingress
            scrape_interval: 30s
            static_configs:
              - targets:
                  - ingress-nginx-controller-metrics.ingress-nginx.svc.cluster.local:10254
          - job_name: kube-state-metrics
            scrape_interval: 30s
            static_configs:
              - targets:
                  - prometheus-stack-kube-state-metrics.monitoring.svc.cluster.local:8080

scrape_configs follows the standard Prometheus scrape config schema — any static_configs, kubernetes_sd_configs, relabel_configs, etc. are supported.

When targetAllocator.enabled: true, the OTEL collector pod must expose POD_NAME via the downward API so each replica gets a distinct collector_id for consistent-hashing. Render fails fast otherwise.

otelCollector:
  extraEnvs:
    - name: POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name

Scraped metrics route to ClickHouse by default. See Routing to fan out to other sinks.

Per-user page throughput and throttling

The PostgreSQL page throttle is opt-in. After applying database migrations, set USER_PAGE_THROTTLE_ENABLED=true on both the API and legacy worker pods and restart them. It defaults to false. Requests are limited only when they carry a user identity with an explicit row in user_page_throttles. Identity comes from user-id. Deployments configured with an alternate client-app header use that header first and fall back to user-id; configure limits for the ID that is sent. These identities are supplied by the deployment’s trusted gateway, not bound to Reducto API keys. The gateway must strip caller-supplied identity headers and set the verified user/app ID. Missing identities and IDs without configured limits intentionally bypass throttling. This is an opt-in scheduling control, not an authentication boundary. When disabled, application code does not read or write user_page_throttles or page_throttle_events; schema migrations still create the tables. Processing, scrapes, and periodic logs skip throttle queries. The throttle admin routes and /metrics/page_throttles return 503 to authenticated operators without querying Postgres. Enable the feature before configuring limits through those routes. /admin/page_throttles and /metrics/page_throttles require the bearer token configured in PAGE_THROTTLE_ADMIN_TOKEN on the API pods. Provision a unique, random token per deployment through your secret manager, and restart the API pods after configuring or rotating it. Unset or empty denies access; ordinary document-processing API keys and the legacy shared admin credential are rejected. The existing /metrics and /prometheus scrape endpoints remain private to the monitoring network, as described above. When enabled, each /metrics or /prometheus scrape reads the current throttle snapshot and logs page_throttle.metrics_snapshot with the same values. JSON /metrics includes a page_throttle object. With METRICS_FORMAT=prometheus, /metrics exports Prometheus gauges; /prometheus always exports Prometheus. Throttle gauges do not depend on PROMETHEUS_MULTIPROC_DIR. With the feature disabled, these snapshot reads, logs, and extra response fields are omitted.

Gauge	Meaning
`reducto_pages_per_second`	Overall observed pages/sec
`reducto_page_throttle_observed_pages_per_second{user_id}`	Observed pages/sec for the user
`reducto_page_throttle_observed_pages{user_id}`	Pages charged in the window
`reducto_page_throttle_observed_jobs{user_id}`	Distinct charged jobs in the window
`reducto_page_throttle_limit_pages_per_second{user_id}`	Configured pages/sec limit
`reducto_page_throttle_capacity_pages{user_id}`	Burst capacity
`reducto_page_throttle_tokens{user_id}`	Current credit; negative values represent debt
`reducto_page_throttle_rejected_requests{user_id}`	Throttle rejections in the window

Overall and unattributed page, job, and rejection counts are also exported. Observed throughput is admitted/debited pages divided by PAGE_THROTTLE_METRICS_WINDOW_SECONDS (default 60, capped at the one-hour event retention). It is not a completed-page rate or an instantaneous rate. Counts and rates are sliding-window gauges, so do not apply Prometheus rate() to them. Configured users with no recent traffic report zero throughput; users without configured limits still contribute observed throughput. These are deployment-wide PostgreSQL values: do not sum duplicate scrapes across API replicas. Legacy workers additionally emit worker.queue_metrics every QUEUE_METRICS_LOG_INTERVAL_SECONDS when explicitly configured (for example, 60). The default is 0, which disables periodic worker logs and their queue queries; it does not disable scrape logs or metrics export. Before enabling throttling, verify the BYO integration handles HTTP 429 and Retry-After for page-rate rejections. These responses include status.success=false and error_code="RATE_LIMITED"; other BYO failures retain their existing HTTP 200 envelope. This integration check is required before rollout; automated route tests do not verify the customer’s client behavior.

Scaling the OTEL Collector

The OTEL collector deploys as a Deployment with otelCollector.replicaCount replicas (HPA optional). Two receivers need explicit coordination once replicaCount > 1, otherwise every replica emits the same data and ClickHouse storage grows linearly with the replica count.

Receiver	Coordination mechanism	Values key
`prometheus` (scrape)	Target Allocator subchart shards targets via consistent-hashing	`targetAllocator.enabled`
`k8s_cluster` + `k8sobjects`	`k8s_leader_elector` extension — only the lease-holder emits	`k8sMetrics.leaderElector.enabled`

Leader Elector for Kubernetes Metrics & Events

k8s_cluster (cluster-level pod/node/container metrics) and k8sobjects (K8s events watch) are cluster-singletons — every replica running them independently produces an identical stream. Enable the leader-elector extension so only the lease-holder emits:

k8sMetrics:
  enabled: true
  leaderElector:
    enabled: true

The chart renders a namespace-scoped Role + RoleBinding for the coordination.k8s.io lease, gated on leaderElector.enabled. Lease hand-off on pod rollover is automatic. Both targetAllocator.enabled and k8sMetrics.leaderElector.enabled default to false — single-replica deployments work out of the box with no extra RBAC or subchart.

Configuration

Seed User

Configure the admin user credentials in your Helm values:

clickstack:
  hyperdx:
    seedUser:
      email: "admin@yourcompany.com"
      password: "your-secure-password"
      teamName: "Your Team"

For production deployments, use a Kubernetes secret instead of a plaintext password:

clickstack:
  hyperdx:
    seedUser:
      email: "admin@yourcompany.com"
      existingSecret: "my-hyperdx-secret"
      secretKey: "HYPERDX_ADMIN_PASSWORD"
      teamName: "Your Team"

ClickHouse Storage

clickstack:
  clickhouse:
    persistence:
      dataSize: 50Gi   # Adjust based on expected telemetry volume
      logSize: 10Gi

Data Retention

Telemetry data retention is controlled by the OTEL exporter TTL:

otelConfig:
  exporters:
    clickhouse:
      ttl: 360h  # 15 days (default: 72h)

Telemetry Pipeline

The OTEL collector receives telemetry from multiple sources and routes it to configured sinks:

Source	What it collects	Default sink
Application traces/metrics	OTLP from Reducto services	All enabled sinks
Kubernetes metrics	Cluster-level pod, node, container metrics via k8s_cluster receiver; node-level kubelet stats via DaemonSet	All enabled sinks
Kubernetes events	K8s events via k8sobjects receiver	All enabled sinks
Prometheus scrape	Metrics from any Prometheus endpoint	ClickHouse
Application logs	OTLP logs from Reducto services	ClickHouse

Routing

Each source can be independently routed to any combination of sinks:

otelConfig:
  routing:
    traces: [tinybird, datadog, clickhouse, iceberg]         # default
    metrics: [tinybird, datadog, clickhouse, iceberg]        # default
    k8sMetrics: [tinybird, datadog, clickhouse, iceberg]     # default
    k8sEvents: [tinybird, datadog, clickhouse, iceberg]      # default
    prometheusScrape: [clickhouse]                           # default
    logs: [clickhouse]                                       # default

Available sinks: clickhouse, tinybird, datadog, iceberg. Each sink must also be enabled in otelConfig.exporters. The defaults list all sinks, but only sinks that are both listed and enabled will actually receive data, so the defaults are safe for any exporter combination.

Pod stack trace dumps (SIGUSR2)

Every Reducto worker and HTTP pod installs a SIGUSR2 handler that dumps a per-thread stack trace to stderr when signalled. Use this when a pod is unresponsive (stuck event loop, hung downstream call, contended thread pool) and kubectl logs alone doesn’t explain why. Coverage:

Pod	Process labelled as
`reducto-http` (gunicorn)	`http-worker-<pid>`
`reducto-streaq-worker`	`streaq-<worker-name>-worker` (e.g. `streaq-io-worker`, `streaq-cpu-worker`)
`reducto-worker`, `reducto-priority-worker`, `reducto-gpu-worker` (DB-queue)	`k8s-worker` (or the value of `LOGFIRE_SERVICE_NAME` when set)

The handler is always installed, with no Helm flag to disable it. Output is written directly to stderr (not through the structured logger), so the trace appears even if the application logging pipeline is itself wedged.

Triggering a dump

# Pick a pod that's misbehaving
kubectl get pods -n reducto -l app=reducto-worker

# Find the worker PID (gunicorn / python process)
kubectl exec -n reducto <pod-name> -- ps -eo pid,cmd | grep -E 'gunicorn|streaq|python'

# Send SIGUSR2 to that PID
kubectl exec -n reducto <pod-name> -- kill -s USR2 <pid>

# Read the dump from the pod log
kubectl logs -n reducto <pod-name> --tail=500

You’ll see a single-line banner followed by one frame block per thread:

USR2 signal received [http-worker-42]; dumping thread stacks
USR2 triggered stack trace:

Thread "MainThread" (most recent call first):
  File "/app/.venv/bin/gunicorn", line 8, in <module>
    sys.exit(run())
  ...

Thread "asyncio-loop-0" (most recent call first):
  ...

Notes

Forked gunicorn and streaq child processes each register their own handler, so signalling the main PID alone won’t dump child stacks. Signal each child PID individually if you need full process-tree coverage.
SIGUSR2 is not used by any other component in the worker/HTTP processes, so triggering a dump is safe in production. The signal handler is async-safe and only enqueues work onto a dedicated daemon thread.
For wider diagnostics (CPU profile, off-CPU sampling), consider py-spy dump --pid <pid> from a debug container. SIGUSR2 is the lowest-friction option and works without an extra binary.

Telemetry controls

Reducto telemetry is designed to avoid customer content. Logs, traces, and metrics are for debugging and performance analysis, not for storing customer data or business records. Recommended controls:

Keep observability UIs and scrape endpoints on private networks.
Use SSO, VPN, Zero Trust access, or equivalent controls for operator access.
Send telemetry only to approved sinks.
Set retention periods that match your security and compliance requirements.
Review telemetry exports for customer content, file names, URLs, prompts, model outputs, and secrets before sharing them outside your organization.

Get Started

Developer Tools

Core Functions

Workflows and Pipelines

Configurations

Reference

Components

Enterprise Resources

Security and privacy

On-premise Resources

Observability & Monitoring

Overview

Accessing HyperDX

Default Credentials

Accessing the UI

Prometheus Scraping

Per-user page throughput and throttling

Scaling the OTEL Collector

Leader Elector for Kubernetes Metrics & Events

Configuration

Seed User

ClickHouse Storage

Data Retention

Telemetry Pipeline

Routing

Pod stack trace dumps (SIGUSR2)

Triggering a dump

Notes

Telemetry controls

​Overview

​Accessing HyperDX

​Default Credentials

​Accessing the UI

​Prometheus Scraping

​Per-user page throughput and throttling

​Scaling the OTEL Collector

​Leader Elector for Kubernetes Metrics & Events

​Configuration

​Seed User

​ClickHouse Storage

​Data Retention

​Telemetry Pipeline

​Routing

​Pod stack trace dumps (SIGUSR2)

​Triggering a dump

​Notes

​Telemetry controls

Overview

Accessing HyperDX

Default Credentials

Accessing the UI

Prometheus Scraping

Per-user page throughput and throttling

Scaling the OTEL Collector

Leader Elector for Kubernetes Metrics & Events

Configuration

Seed User

ClickHouse Storage

Data Retention

Telemetry Pipeline

Routing

Pod stack trace dumps (SIGUSR2)

Triggering a dump

Notes

Telemetry controls