> ## Documentation Index
> Fetch the complete documentation index at: https://docs.reducto.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability & Monitoring

> Built-in observability stack for on-premise Reducto deployments

## Overview

On-premise Reducto deployments include a built-in observability stack called **ClickStack**, which provides:

* **HyperDX**: Unified observability UI for logs, traces, and metrics
* **ClickHouse**: High-performance analytics database for telemetry storage
* **OTEL Collector**: OpenTelemetry collector for ingesting and routing telemetry data

ClickStack is enabled by setting `clickstack.enabled: true` in your Helm values. Everything else is automatic. No additional setup required.

## Accessing HyperDX

### Default Credentials

When ClickStack is enabled, a seed admin user is automatically created on first install with default credentials. Contact the Reducto team for the default login details, or configure your own credentials in your Helm values (see [Configuration](#configuration) below).

<Warning>
  Change the default password immediately after first login.
</Warning>

### Accessing the UI

HyperDX can be exposed via:

* **Ingress**: Set `clickstack.hyperdx.ingress.enabled: true` with your domain
* **Tailscale**: Set `clickstack.hyperdx.exposure.tailscale.enabled: true` for private access
* **Cloudflare Tunnel**: Set `clickstack.hyperdx.exposure.cloudflareTunnel.enabled: true`
* **Port-forward** (for testing): `kubectl port-forward svc/<release>-clickstack-app 3000:3000`

## Prometheus Scraping

To collect metrics from services that expose Prometheus endpoints (like NGINX ingress controllers), enable the Prometheus scrape config:

```yaml theme={null}
prometheusScrape:
  enabled: true
  scrapeJobs:
    - jobName: nginx-ingress
      scrapeInterval: 30s
      targets:
        - ingress-nginx-controller-metrics.ingress-nginx.svc.cluster.local:10254
```

Each entry in `scrapeJobs` becomes a scrape target. You can add multiple jobs for different services:

```yaml theme={null}
prometheusScrape:
  enabled: true
  scrapeJobs:
    - jobName: nginx-ingress
      scrapeInterval: 30s
      targets:
        - ingress-nginx-controller-metrics.ingress-nginx.svc.cluster.local:10254
    - jobName: my-app
      scrapeInterval: 60s
      targets:
        - my-app-metrics.default.svc.cluster.local:9090
```

Scraped metrics are routed to ClickHouse by default. To also send them to other sinks:

```yaml theme={null}
otelConfig:
  routing:
    prometheusScrape: [clickhouse, datadog]  # default: [clickhouse]
```

## Configuration

### Seed User

Configure the admin user credentials in your Helm values:

```yaml theme={null}
clickstack:
  hyperdx:
    seedUser:
      email: "admin@yourcompany.com"
      password: "your-secure-password"
      teamName: "Your Team"
```

For production deployments, use a Kubernetes secret instead of a plaintext password:

```yaml theme={null}
clickstack:
  hyperdx:
    seedUser:
      email: "admin@yourcompany.com"
      existingSecret: "my-hyperdx-secret"
      secretKey: "HYPERDX_ADMIN_PASSWORD"
      teamName: "Your Team"
```

### ClickHouse Storage

```yaml theme={null}
clickstack:
  clickhouse:
    persistence:
      dataSize: 50Gi   # Adjust based on expected telemetry volume
      logSize: 10Gi
```

### Data Retention

Telemetry data retention is controlled by the OTEL exporter TTL:

```yaml theme={null}
otelConfig:
  exporters:
    clickhouse:
      ttl: 360h  # 15 days (default: 72h)
```

## Telemetry Pipeline

The OTEL collector receives telemetry from multiple sources and routes it to configured sinks:

| Source                     | What it collects                                                                                             | Default sink      |
| -------------------------- | ------------------------------------------------------------------------------------------------------------ | ----------------- |
| Application traces/metrics | OTLP from Reducto services                                                                                   | All enabled sinks |
| Kubernetes metrics         | Cluster-level pod, node, container metrics via k8s\_cluster receiver; node-level kubelet stats via DaemonSet | All enabled sinks |
| Kubernetes events          | K8s events via k8sobjects receiver                                                                           | All enabled sinks |
| Prometheus scrape          | Metrics from any Prometheus endpoint                                                                         | ClickHouse        |
| Application logs           | OTLP logs from Reducto services                                                                              | ClickHouse        |

### Routing

Each source can be independently routed to any combination of sinks:

```yaml theme={null}
otelConfig:
  routing:
    traces: [tinybird, datadog, logfire, clickhouse]         # default
    metrics: [tinybird, datadog, logfire, clickhouse]        # default
    k8sMetrics: [tinybird, datadog, logfire, clickhouse]     # default
    k8sEvents: [tinybird, datadog, logfire, clickhouse]      # default
    prometheusScrape: [clickhouse]                           # default
    logs: [clickhouse]                                       # default
```

Available sinks: `clickhouse`, `tinybird`, `datadog`, `logfire`. Each sink must also be enabled in `otelConfig.exporters`. The defaults list all sinks, but only sinks that are both listed **and** enabled will actually receive data, so the defaults are safe for any exporter combination.