Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.reducto.ai/llms.txt

Use this file to discover all available pages before exploring further.

​
Overview

On-premise Reducto deployments include a built-in observability stack called ClickStack, which provides:
  • HyperDX: Unified observability UI for logs, traces, and metrics
  • ClickHouse: High-performance analytics database for telemetry storage
  • OTEL Collector: OpenTelemetry collector for ingesting and routing telemetry data
ClickStack is enabled by setting clickstack.enabled: true in your Helm values. Everything else is automatic. No additional setup required. Telemetry is part of your on-premise security boundary. Reducto emits logs to stdout and can route traces, metrics, and logs through OpenTelemetry, but you control where telemetry is stored, who can access it, and how long it is retained. See the on-prem security model for the shared responsibility model. Reducto telemetry is designed for operational metadata. It should not contain document content, OCR text, extracted values, prompts, model outputs, API tokens, access keys, secrets, or other customer content. File names and URLs can reveal customer identity and should be redacted or avoided before telemetry leaves the deployment.

​
Accessing HyperDX

​
Default Credentials

When ClickStack is enabled, a seed admin user is automatically created on first install with default credentials. Contact the Reducto team for the default login details, or configure your own credentials in your Helm values (see Configuration below).
Change the default password immediately after first login.

​
Accessing the UI

HyperDX can be exposed via:
  • Ingress: Set clickstack.hyperdx.ingress.enabled: true with your domain
  • Tailscale: Set clickstack.hyperdx.exposure.tailscale.enabled: true for private access
  • Cloudflare Tunnel: Set clickstack.hyperdx.exposure.cloudflareTunnel.enabled: true
  • Port-forward (for testing): kubectl port-forward svc/<release>-clickstack-app 3000:3000

​
Prometheus Scraping

Prometheus endpoints are intended for internal scraping by monitoring systems. Do not expose /metrics, /prometheus, ClickStack, HyperDX, ClickHouse, or the OTEL collector to the public internet. To collect metrics from services that expose Prometheus endpoints (like NGINX ingress controllers), enable the Prometheus scrape config: 
prometheusScrape:
  enabled: true
  scrapeJobs:
    - jobName: nginx-ingress
      scrapeInterval: 30s
      targets:
        - ingress-nginx-controller-metrics.ingress-nginx.svc.cluster.local:10254
Each entry in scrapeJobs becomes a scrape target. You can add multiple jobs for different services: 
prometheusScrape:
  enabled: true
  scrapeJobs:
    - jobName: nginx-ingress
      scrapeInterval: 30s
      targets:
        - ingress-nginx-controller-metrics.ingress-nginx.svc.cluster.local:10254
    - jobName: my-app
      scrapeInterval: 60s
      targets:
        - my-app-metrics.default.svc.cluster.local:9090
Scraped metrics are routed to ClickHouse by default. To also send them to other sinks: 
otelConfig:
  routing:
    prometheusScrape: [clickhouse, datadog]  # default: [clickhouse]

​
Configuration

​
Seed User

Configure the admin user credentials in your Helm values: 
clickstack:
  hyperdx:
    seedUser:
      email: "admin@yourcompany.com"
      password: "your-secure-password"
      teamName: "Your Team"
For production deployments, use a Kubernetes secret instead of a plaintext password: 
clickstack:
  hyperdx:
    seedUser:
      email: "admin@yourcompany.com"
      existingSecret: "my-hyperdx-secret"
      secretKey: "HYPERDX_ADMIN_PASSWORD"
      teamName: "Your Team"

​
ClickHouse Storage

clickstack:
  clickhouse:
    persistence:
      dataSize: 50Gi   # Adjust based on expected telemetry volume
      logSize: 10Gi

​
Data Retention

Telemetry data retention is controlled by the OTEL exporter TTL: 
otelConfig:
  exporters:
    clickhouse:
      ttl: 360h  # 15 days (default: 72h)

​
Telemetry Pipeline

The OTEL collector receives telemetry from multiple sources and routes it to configured sinks:
SourceWhat it collectsDefault sink
Application traces/metricsOTLP from Reducto servicesAll enabled sinks
Kubernetes metricsCluster-level pod, node, container metrics via k8s_cluster receiver; node-level kubelet stats via DaemonSetAll enabled sinks
Kubernetes eventsK8s events via k8sobjects receiverAll enabled sinks
Prometheus scrapeMetrics from any Prometheus endpointClickHouse
Application logsOTLP logs from Reducto servicesClickHouse

​
Routing

Each source can be independently routed to any combination of sinks: 
otelConfig:
  routing:
    traces: [tinybird, datadog, logfire, clickhouse]         # default
    metrics: [tinybird, datadog, logfire, clickhouse]        # default
    k8sMetrics: [tinybird, datadog, logfire, clickhouse]     # default
    k8sEvents: [tinybird, datadog, logfire, clickhouse]      # default
    prometheusScrape: [clickhouse]                           # default
    logs: [clickhouse]                                       # default
Available sinks: clickhouse, tinybird, datadog, logfire. Each sink must also be enabled in otelConfig.exporters. The defaults list all sinks, but only sinks that are both listed and enabled will actually receive data, so the defaults are safe for any exporter combination.

​
Pod stack trace dumps (SIGUSR2)

Every Reducto worker and HTTP pod installs a SIGUSR2 handler that dumps a per-thread stack trace to stderr when signalled. Use this when a pod is unresponsive (stuck event loop, hung downstream call, contended thread pool) and kubectl logs alone doesn’t explain why. Coverage:
PodProcess labelled as
reducto-http (gunicorn)http-worker-<pid>
reducto-streaq-workerstreaq-<worker-name>-worker (e.g. streaq-io-worker, streaq-cpu-worker)
reducto-worker, reducto-priority-worker, reducto-gpu-worker (DB-queue)k8s-worker (or the value of LOGFIRE_SERVICE_NAME when set)
The handler is always installed, with no Helm flag to disable it. Output is written directly to stderr (not through the structured logger), so the trace appears even if the application logging pipeline is itself wedged.

​
Triggering a dump

# Pick a pod that's misbehaving
kubectl get pods -n reducto -l app=reducto-worker

# Find the worker PID (gunicorn / python process)
kubectl exec -n reducto <pod-name> -- ps -eo pid,cmd | grep -E 'gunicorn|streaq|python'

# Send SIGUSR2 to that PID
kubectl exec -n reducto <pod-name> -- kill -s USR2 <pid>

# Read the dump from the pod log
kubectl logs -n reducto <pod-name> --tail=500
You’ll see a single-line banner followed by one frame block per thread: 
USR2 signal received [http-worker-42]; dumping thread stacks
USR2 triggered stack trace:

Thread "MainThread" (most recent call first):
  File "/app/.venv/bin/gunicorn", line 8, in <module>
    sys.exit(run())
  ...

Thread "asyncio-loop-0" (most recent call first):
  ...

​
Notes

  • Forked gunicorn and streaq child processes each register their own handler, so signalling the main PID alone won’t dump child stacks. Signal each child PID individually if you need full process-tree coverage.
  • SIGUSR2 is not used by any other component in the worker/HTTP processes, so triggering a dump is safe in production. The signal handler is async-safe and only enqueues work onto a dedicated daemon thread.
  • For wider diagnostics (CPU profile, off-CPU sampling), consider py-spy dump --pid <pid> from a debug container. SIGUSR2 is the lowest-friction option and works without an extra binary.

​
Telemetry controls

Reducto telemetry is designed to avoid customer content. Logs, traces, and metrics are for debugging and performance analysis, not for storing customer data or business records. Recommended controls:
  • Keep observability UIs and scrape endpoints on private networks.
  • Use SSO, VPN, Zero Trust access, or equivalent controls for operator access.
  • Send telemetry only to approved sinks.
  • Set retention periods that match your security and compliance requirements.
  • Review telemetry exports for customer content, file names, URLs, prompts, model outputs, and secrets before sharing them outside your organization.