Skip to content

Observability

Sluice writes logs, serves Prometheus metrics and sends traces. The record of every query is the audit log.

Example

Goal: find out why a query failed.

curl -s http://localhost:8080/v1/query \
  -H "X-Api-Key: sl_demo_analyst.supersecret" \
  -H "Content-Type: application/json" \
  -d '{"sql": "SELECT CAST(name AS INTEGER) FROM shop.main.customers WHERE id = 1"}'

{
  "code": "ERR_QUERY_FAILED",
  "message": "the database rejected the query",
  "query_id": "<query-id>"
}

Search the log for this error:

grep "query rejected by duckdb" data/sluice.log

time=<time> level=WARN msg="query rejected by duckdb" query_id=<id> error="executor: query execution failed: executor: statement rejected by duckdb: Conversion Error: Could not convert string 'Alice' to INT32 when casting from source column name\n\nLINE 1: SELECT CAST(name AS INTEGER) FROM shop.main.customers WHERE id = 1\n               ^"

What happened: the client gets an error code and a query_id. The log line with this query_id holds the cause from DuckDB.

Logs

Sluice writes one line per event to stderr.

logging:
  level: "info"
  # The default is "json". The text format is easier to read in a terminal.
  format: "text"
  • logging.level is debug, info, warn or error. logging.format is json or text.
  • Sluice accepts an unknown value. It then uses info or json.
  • ERR_QUERY_FAILED logs a WARN line. ERR_INTERNAL logs an ERROR line with the message internal error.
  • These lines are not redacted. They can hold SQL and cell values, like 'Alice' above.
  • logging.maxErrorBytes cuts a long error text. The default is 4096, and 0 turns the cut off.

Metrics

The admin listener serves GET /metrics. It needs admin.enabled: true and the admin token.

curl -s http://localhost:9091/metrics \
  -H "Authorization: Bearer demo-admin-token-change-me" | grep "^sluice_audit"

sluice_audit_enqueued_total{event_type="query"} 2
sluice_audit_enqueued_total{event_type="query-result"} 1
sluice_audit_queue_depth{sink="file"} 0

Metrics lists every metric. Two alerts are useful:

  • sluice_audit_dropped_total above 0: Sluice lost audit records. The reason label is queue_full, context, buffer_full or write_error.
  • sluice_audit_queue_depth close to 10000: the audit queue is almost full.

Tracing

Add a tracing block to the server config. Sluice then sends spans to an OpenTelemetry (OTLP) collector. A span is one timed step of a request.

tracing:
  enabled: true
  endpoint: "otel-collector.internal:4317"
  protocol: "grpc"
  sampleRatio: 0.1
  • protocol is grpc or http. insecure: true sends without TLS.
  • Sluice continues the trace of a W3C traceparent request header.
  • Every REST request and every MCP request over HTTP gets a span.
  • A query adds the span query with the children query.parse, query.policy, query.rewrite and query.execute. A post-query column mask adds query.mask.
  • The query span holds the query id, origin, decision, SQL fingerprint, catalogs and error code, never the SQL text. It ends before Sluice sends the rows.

See also