Skip to content
Examples
Docker Compose
observability

observability

What it demonstrates

basic with telemetry on, and a place to look at it:

  • puddle exports every trace (OTEL_TRACES_SAMPLER=always_on) over OTLP to an OpenTelemetry collector, which passes them on to Jaeger (in memory);
  • Prometheus scrapes /metrics on puddle’s management listener;
  • Grafana has Prometheus as a data source and a starter puddle dashboard: requests, latency and errors by operation, commits by outcome, conflicts and uncertain commits, metastore latency, authorization decisions, Go memory.

Grafana’s Jaeger data source uses a query API that Jaeger 2.21 removed, so traces are in Jaeger’s own UI rather than in Grafana.

Run it 

cd examples/compose/observability
docker compose up
Catalog RESThttp://localhost:8089/api/catalog, warehouse=default
puddle’s /metricshttp://localhost:9090/metrics
Jaegerhttp://localhost:16686
Prometheushttp://localhost:9091
Grafanahttp://localhost:3000 (anonymous admin: demo only)

Make a commit:

B=localhost:8089/api/catalog/v1/default
curl -X POST $B/namespaces -H 'Content-Type: application/json' \
  -d '{"namespace":["demo"]}'
curl -X POST $B/namespaces/demo/tables -H 'Content-Type: application/json' \
  -d '{"name":"t","schema":{"type":"struct","fields":[{"id":1,"name":"id","type":"long","required":false}]}}'
curl -X POST $B/namespaces/demo/tables/t -H 'Content-Type: application/json' \
  -d '{"updates":[{"action":"set-properties","updates":{"owner":"me"}}]}'

In Jaeger, search service puddle, operation POST updateTable: the trace shows the commit’s steps (see Traces). In Prometheus or Grafana, sum by (kind, outcome) (puddle_commit_duration_seconds_count) counts it.

docker-compose.yml

Source on GitHub.

docker-compose.yml
# Self-contained docker-compose for puddle with OpenTelemetry on:
# traces over OTLP to a collector and on to Jaeger, metrics scraped by
# Prometheus from puddle's management listener, and Grafana with
# Prometheus as its data source and a starter dashboard. (Grafana's
# Jaeger data source speaks the query API Jaeger 2.21 dropped, so
# traces are in Jaeger's own UI.) SQLite metastore,
# local-filesystem warehouse, no auth.
#
# Run:
#   docker compose up
# Catalog REST: http://localhost:8089/api/catalog (warehouse=default)
# Metrics:      http://localhost:9090/metrics (puddle's management listener)
# Jaeger UI:    http://localhost:16686
# Prometheus:   http://localhost:9091
# Grafana:      http://localhost:3000 (dashboard "puddle"; anonymous admin)
#
# Make a commit, then find its trace in Jaeger (service puddle,
# operation "POST updateTable"):
#   curl -X POST localhost:8089/api/catalog/v1/default/namespaces \
#     -H 'Content-Type: application/json' -d '{"namespace":["demo"]}'
#   curl -X POST localhost:8089/api/catalog/v1/default/namespaces/demo/tables \
#     -H 'Content-Type: application/json' \
#     -d '{"name":"t","schema":{"type":"struct","fields":[{"id":1,"name":"id","type":"long","required":false}]}}'
#   curl -X POST localhost:8089/api/catalog/v1/default/namespaces/demo/tables/t \
#     -H 'Content-Type: application/json' \
#     -d '{"updates":[{"action":"set-properties","updates":{"owner":"me"}}]}'
#
# Every request is traced here (OTEL_TRACES_SAMPLER=always_on); see the
# observability docs for sampling in production.

configs:
  puddle-config:
    content: |
      logging:
        level: info
        format: json

      metastore:
        type: sqlite
        sqlite:
          path: /var/lib/puddle/puddle.db

      warehouses:
        default:
          location: file:///var/lib/puddle/warehouse

      # The management listener: /metrics, /healthz, /readyz. The
      # exporter, sampler and resource come from OTEL_* below.
      telemetry:
        enabled: true
        addr: ":9090"

  collector-config:
    content: |
      receivers:
        otlp:
          protocols:
            grpc:
              endpoint: 0.0.0.0:4317
            http:
              endpoint: 0.0.0.0:4318
      processors:
        batch:
      exporters:
        otlp_grpc/jaeger:
          endpoint: jaeger:4317
          tls:
            insecure: true
      service:
        pipelines:
          traces:
            receivers: [otlp]
            processors: [batch]
            exporters: [otlp_grpc/jaeger]

  prometheus-config:
    content: |
      global:
        scrape_interval: 15s
      scrape_configs:
        - job_name: puddle
          static_configs:
            - targets: ["puddle:9090"]

  grafana-datasources:
    content: |
      apiVersion: 1
      datasources:
        - name: Prometheus
          uid: prometheus
          type: prometheus
          access: proxy
          url: http://prometheus:9090
          isDefault: true

  grafana-dashboards:
    content: |
      apiVersion: 1
      providers:
        - name: puddle
          type: file
          options:
            path: /var/lib/grafana/dashboards

  grafana-dashboard-puddle:
    content: |
      {
        "uid": "puddle",
        "title": "puddle",
        "schemaVersion": 39,
        "time": {"from": "now-1h", "to": "now"},
        "refresh": "30s",
        "panels": [
          {"type": "row", "title": "Requests", "gridPos": {"x": 0, "y": 0, "w": 24, "h": 1}},
          {"type": "timeseries", "title": "Requests/s by operation",
           "datasource": {"type": "prometheus", "uid": "prometheus"},
           "gridPos": {"x": 0, "y": 1, "w": 8, "h": 8},
           "fieldConfig": {"defaults": {"unit": "reqps"}},
           "targets": [{"refId": "A", "legendFormat": "{{http_route}}",
             "expr": "sum by (http_route) (rate(http_server_request_duration_seconds_count[$$__rate_interval]))"}]},
          {"type": "timeseries", "title": "p95 latency by operation",
           "datasource": {"type": "prometheus", "uid": "prometheus"},
           "gridPos": {"x": 8, "y": 1, "w": 8, "h": 8},
           "fieldConfig": {"defaults": {"unit": "s"}},
           "targets": [{"refId": "A", "legendFormat": "{{http_route}}",
             "expr": "histogram_quantile(0.95, sum by (le, http_route) (rate(http_server_request_duration_seconds_bucket[$$__rate_interval])))"}]},
          {"type": "timeseries", "title": "Errors/s by type",
           "datasource": {"type": "prometheus", "uid": "prometheus"},
           "gridPos": {"x": 16, "y": 1, "w": 8, "h": 8},
           "fieldConfig": {"defaults": {"unit": "reqps"}},
           "targets": [{"refId": "A", "legendFormat": "{{http_response_status_code}} {{error_type}}",
             "expr": "sum by (http_response_status_code, error_type) (rate(http_server_request_duration_seconds_count{error_type!=\"\"}[$$__rate_interval]))"}]},
          {"type": "row", "title": "Commits", "gridPos": {"x": 0, "y": 9, "w": 24, "h": 1}},
          {"type": "timeseries", "title": "Commits/s by outcome",
           "datasource": {"type": "prometheus", "uid": "prometheus"},
           "gridPos": {"x": 0, "y": 10, "w": 8, "h": 8},
           "fieldConfig": {"defaults": {"unit": "ops"}},
           "targets": [{"refId": "A", "legendFormat": "{{kind}} {{outcome}}",
             "expr": "sum by (kind, outcome) (rate(puddle_commit_duration_seconds_count[$$__rate_interval]))"}]},
          {"type": "timeseries", "title": "p95 commit duration",
           "datasource": {"type": "prometheus", "uid": "prometheus"},
           "gridPos": {"x": 8, "y": 10, "w": 8, "h": 8},
           "fieldConfig": {"defaults": {"unit": "s"}},
           "targets": [{"refId": "A", "legendFormat": "{{kind}}",
             "expr": "histogram_quantile(0.95, sum by (le, kind) (rate(puddle_commit_duration_seconds_bucket{outcome=\"committed\"}[$$__rate_interval])))"}]},
          {"type": "timeseries", "title": "Conflicts, requirement failures, uncertain commits",
           "datasource": {"type": "prometheus", "uid": "prometheus"},
           "gridPos": {"x": 16, "y": 10, "w": 8, "h": 8},
           "fieldConfig": {"defaults": {"unit": "short"}},
           "targets": [
             {"refId": "A", "legendFormat": "conflicts {{kind}}",
              "expr": "sum by (kind) (increase(puddle_commit_conflicts_total[$$__rate_interval]))"},
             {"refId": "B", "legendFormat": "requirement failures {{kind}}",
              "expr": "sum by (kind) (increase(puddle_commit_requirement_failures_total[$$__rate_interval]))"},
             {"refId": "C", "legendFormat": "uncertain {{resolution}}",
              "expr": "sum by (resolution) (increase(puddle_commit_uncertain_total[$$__rate_interval]))"}]},
          {"type": "row", "title": "Dependencies", "gridPos": {"x": 0, "y": 18, "w": 24, "h": 1}},
          {"type": "timeseries", "title": "p95 metastore operation",
           "datasource": {"type": "prometheus", "uid": "prometheus"},
           "gridPos": {"x": 0, "y": 19, "w": 8, "h": 8},
           "fieldConfig": {"defaults": {"unit": "s"}},
           "targets": [{"refId": "A", "legendFormat": "{{db_operation_name}}",
             "expr": "histogram_quantile(0.95, sum by (le, db_operation_name) (rate(db_client_operation_duration_seconds_bucket[$$__rate_interval])))"}]},
          {"type": "timeseries", "title": "Authorization decisions/s",
           "datasource": {"type": "prometheus", "uid": "prometheus"},
           "gridPos": {"x": 8, "y": 19, "w": 8, "h": 8},
           "fieldConfig": {"defaults": {"unit": "ops"}},
           "targets": [{"refId": "A", "legendFormat": "{{decision}}",
             "expr": "sum by (decision) (rate(puddle_authz_decisions_total{probe=\"false\"}[$$__rate_interval]))"}]},
          {"type": "timeseries", "title": "Go memory and goroutines",
           "datasource": {"type": "prometheus", "uid": "prometheus"},
           "gridPos": {"x": 16, "y": 19, "w": 8, "h": 8},
           "fieldConfig": {"defaults": {"unit": "bytes"},
             "overrides": [{"matcher": {"id": "byName", "options": "goroutines"},
               "properties": [{"id": "unit", "value": "short"}, {"id": "custom.axisPlacement", "value": "right"}]}]},
           "targets": [
             {"refId": "A", "legendFormat": "memory used", "expr": "sum(go_memory_used_bytes)"},
             {"refId": "B", "legendFormat": "goroutines", "expr": "go_goroutine_count"}]}
        ]
      }

services:
  puddle:
    image: ghcr.io/ragnard/puddle:latest
    ports: ["8089:8089", "9090:9090"]
    environment:
      OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4318
      OTEL_TRACES_SAMPLER: always_on
      OTEL_RESOURCE_ATTRIBUTES: deployment.environment.name=example
    volumes:
      - puddle-data:/var/lib/puddle
    configs:
      - source: puddle-config
        target: /etc/puddle/puddle.yaml
    command: ["-c", "/etc/puddle/puddle.yaml"]
    restart: unless-stopped

  otel-collector:
    image: otel/opentelemetry-collector:0.161.0@sha256:b6d2b9a85b1029d05b5ad913150c1f014eed4ae99be81a1813ca5ade4a191913
    configs:
      - source: collector-config
        target: /etc/otelcol/config.yaml
    command: ["--config=/etc/otelcol/config.yaml"]
    restart: unless-stopped

  # All-in-one Jaeger v2: OTLP in, in-memory storage, the UI.
  jaeger:
    image: jaegertracing/jaeger:2.21.0@sha256:3d0ac795ff98aa04d1be04311d2dac6c25b4bfc8322dc02e53bc5b170c5018c3
    ports: ["16686:16686"]
    restart: unless-stopped

  prometheus:
    image: prom/prometheus:v3.15.0@sha256:efd719c99d83b060d9daefdcf00360461adf279f45ef5391f8d111892118753e
    ports: ["9091:9090"]
    configs:
      - source: prometheus-config
        target: /etc/prometheus/prometheus.yml
    volumes:
      - prometheus-data:/prometheus
    restart: unless-stopped

  grafana:
    image: grafana/grafana:13.2.2@sha256:ac461fb352abc50da10a51c7d02462e9c05488f11f53f14b3ad79a8145f638a0
    ports: ["3000:3000"]
    environment:
      # Demo only: anyone reaching port 3000 is an admin.
      GF_AUTH_ANONYMOUS_ENABLED: "true"
      GF_AUTH_ANONYMOUS_ORG_ROLE: Admin
      GF_AUTH_DISABLE_LOGIN_FORM: "true"
    configs:
      - source: grafana-datasources
        target: /etc/grafana/provisioning/datasources/puddle.yaml
      - source: grafana-dashboards
        target: /etc/grafana/provisioning/dashboards/puddle.yaml
      - source: grafana-dashboard-puddle
        target: /var/lib/grafana/dashboards/puddle.json
    restart: unless-stopped

volumes:
  puddle-data:
  prometheus-data:
trino-rustfs-opa-vending