Skip to content

Data sources

A DataSource object attaches one database or file store as a catalog. The files live in the policy directory.

Example

Goal: attach a SQLite file and a DuckDB file, then join them in one query.

# Transactional data in a SQLite file.
apiVersion: sluice.bino.bi/v1alpha1
kind: DataSource
metadata:
  name: shop
spec:
  type: sqlite
  path: ./data/shop.db

# Reference data in a DuckDB file.
apiVersion: sluice.bino.bi/v1alpha1
kind: DataSource
metadata:
  name: ref
spec:
  type: duckdb_file
  path: ./data/ref.duckdb

curl -s http://localhost:8080/v1/query \
  -H "X-Api-Key: sl_demo_analyst.supersecret" \
  -H "Content-Type: application/json" \
  -d '{"sql": "SELECT o.id, o.amount_cents, c.name AS country, c.region FROM shop.main.orders o JOIN ref.main.countries c ON c.code = o.country_code ORDER BY o.id"}'

{
  "query_id": "<query-id>",
  "columns": ["id", "amount_cents", "country", "region"],
  "rows": [
    [1, 1999, "United States", "Americas"],
    [2, 4599, "Germany", "EMEA"],
    [3, 7800, "France", "EMEA"],
    [4, 250, "United States", "Americas"],
    [5, 32000, "Japan", "APAC"]
  ],
  "row_count": 5,
  "truncated": false
}

What happened: metadata.name became the catalog name, so the tables are shop.main.orders and ref.main.countries. Sluice attached both files at start.

limits.disableCrossCatalog: true rejects a query on several catalogs with ACL_REJECTED. See Cross-source joins.

Types

type Reads Fields Credentials
sqlite A SQLite file path None
duckdb_file A DuckDB file path None
postgres A PostgreSQL database connection credentialsRef: the password
mysql A MySQL or MariaDB database connection credentialsRef: the password
s3 Files in an S3 bucket bucket, region, allowedPaths, endpoint, format credentialsRef: JSON with key_id, secret, session_token
http Files behind HTTP URLs urls, format headersRef: JSON object of headers
motherduck A MotherDuck database database tokenRef: the token
synthetic Generated rows from the profile of another data source synthetic synthetic.seed: the seed. See Synthetic data

Fields that end in Ref take a secret reference. The files below are validated, but not tested against a real service.

apiVersion: sluice.bino.bi/v1alpha1
kind: DataSource
metadata:
  name: pg
spec:
  type: postgres
  # The URL holds the user and the database. The password comes from credentialsRef.
  connection: "postgres://sluice@db.internal:5432/orders?sslmode=require"
  credentialsRef: "secret://env/SLUICE_PG_PASSWORD"

apiVersion: sluice.bino.bi/v1alpha1
kind: DataSource
metadata:
  name: crm
spec:
  type: mysql
  connection: "mysql://sluice@mysql.internal:3306/crm"
  credentialsRef: "secret://env/SLUICE_MYSQL_PASSWORD"

apiVersion: sluice.bino.bi/v1alpha1
kind: DataSource
metadata:
  name: lake
spec:
  type: s3
  bucket: "ref-data"
  region: "us-east-1"
  # One view per path: lake.main.countries_csv and lake.main.facts___parquet.
  allowedPaths:
    - "s3://ref-data/countries.csv"
    - "s3://ref-data/facts/*.parquet"
  credentialsRef: "secret://env/SLUICE_S3_CREDS"

apiVersion: sluice.bino.bi/v1alpha1
kind: DataSource
metadata:
  name: web
spec:
  type: http
  # One view per URL: web.main.exports_countries_csv.
  urls:
    - "https://data.example.com/exports/countries.csv"
  headersRef: "secret://env/SLUICE_HTTP_HEADERS"

apiVersion: sluice.bino.bi/v1alpha1
kind: DataSource
metadata:
  name: warehouse
spec:
  type: motherduck
  database: analytics
  tokenRef: "secret://env/SLUICE_MOTHERDUCK_TOKEN"
  • postgres, mysql: the URL needs a user and a database. The default ports are 5432 and 3306. mysql also accepts mariadb://.
  • s3, http: each path or URL becomes one view in the schema main. Sluice reads nothing else.
  • s3, http: the extension selects the format: .parquet, .csv, .json, .ndjson, .jsonl or .xlsx. format sets parquet, csv, json or xlsx for every path.
  • s3: a path in allowedPaths can hold * or **, except for xlsx. ** covers every directory below. endpoint names an S3-compatible store.
  • http: no wildcards. Sluice sends the headers only to the hosts of urls.

Check the files

sluice datasource check --dir policies.d

NAME       TYPE        FACTORY  ERROR
----       ----        -------  -----
web        http        yes
warehouse  motherduck  yes
crm        mysql       yes
pg         postgres    yes
lake       s3          yes

5 checked, 0 failed

The command only reads the files. An invalid file ends it with status 1:

sluice datasource check --dir invalid

invalid/datasource-postgres.yaml: DataSource/pg: spec.connection: required for postgres/mysql
[exit status 1]

A running server tests every source at start and then every 30 seconds. GET /v1/ready shows the result. See Deployment.

Limits

  • Sluice attaches databases and files read-only. It reads s3 and http data through views. No setting changes this.
  • Sluice attaches sources only at start. Restart it after you add or change a DataSource.
  • A source that fails a check of its type stops the start. Examples: a missing sqlite or duckdb_file file, s3 without region, motherduck without tokenRef. datasource check and config validate miss these errors.
  • datasources.failFast: false skips such a source, and GET /v1/ready does not list it. A source that Sluice cannot attach still stops the start: a database that is down, or a broken duckdb_file file.
  • The health test checks that a sqlite file exists. It does not read the file. A file that is no database stays healthy, and queries on it fail with ERR_INTERNAL.
  • A source that goes away is seen by the next health test. Until then, queries on it can fail with ERR_INTERNAL.

See also