Testing policies¶

sluice policy test runs suites against a policy directory. A case has an identity, a SQL statement and the result you expect. It needs no server and no database.

Example¶

The policy directory holds an allow, a row filter and a column mask.

apiVersion: sluice.bino.bi/v1alpha1
kind: SqlAccessPolicy
metadata:
  name: allow-analysts
  priority: 0
spec:
  effect: allow
  match:
    any:
      - subjects: { groups: [analysts] }
        resources: { tables: ["*"] }
---
apiVersion: sluice.bino.bi/v1alpha1
kind: RowFilterPolicy
metadata:
  name: tenant-isolation
  priority: 50
spec:
  match:
    any:
      - resources: { tables: ["orders"] }
  filter:
    predicate:
      column: tenant_id
      op: Equals
      value: "{{ subject.tenant_id }}"
---
apiVersion: sluice.bino.bi/v1alpha1
kind: ColumnMaskPolicy
metadata:
  name: mask-email
  priority: 50
spec:
  match:
    any:
      - resources: { tables: ["customers"], columns: [email] }
  mask:
    type: partial
    args: { showFirst: 1, showLast: 0 }

The suite:

# The test runner has no database, so the suite lists the columns of the masked table.
schema:
  pg.public.customers: [id, email, tenant_id]

cases:
  - name: analyst reads orders with tenant filter
    identity: { subject: alice, groups: [analysts], claims: { tenant_id: acme } }
    sql: "SELECT id FROM pg.public.orders"
    expect:
      outcome: allow
      filters: ["pg.public.orders"]
      applied: ["RowFilterPolicy/tenant-isolation", "SqlAccessPolicy/allow-analysts"]
      rewrittenSqlContains: ["tenant_id = $1"]

  - name: non-analyst is denied by default
    identity: { subject: mallory, groups: [guests] }
    sql: "SELECT id FROM pg.public.orders"
    expect:
      outcome: deny

  - name: email is partially masked
    identity: { subject: alice, groups: [analysts] }
    sql: "SELECT id, email FROM pg.public.customers"
    expect:
      outcome: allow
      masks: ["pg.public.customers.email=partial"]
      rewrittenSqlContains: ["substr(sluice_src.email"]

sluice policy test policies.d --tests tests

PASS  analyst reads orders with tenant filter
PASS  non-analyst is denied by default
PASS  email is partially masked
PASS  star reads email masked via schema fixture

4 passed, 0 failed, 4 total

The fourth case comes from a second suite, shown below.

Suite format¶

A suite is a YAML file without apiVersion. The test runner ignores unknown keys: a typo such as outcom skips the check.

Key	Meaning
`schema`	Columns of masked tables. See below.
`cases[].name`	Printed in the report.
`cases[].identity.subject`	Subject id. Without it, the case has no subject.
`cases[].identity.groups`, `.claims`	Groups and claims. Claims can nest.
`cases[].identity.issuer`, `.email`	Issuer and email.
`cases[].request.headers`, `.userAgent`, `.remoteIp`	Headers, user agent and address of the request, for the `request.*` values and for `ipRanges`.
`cases[].sql`	One SQL statement.
`cases[].expect`	The checks. Every field is optional.

`expect` field	Compared with
`outcome`	`allow`, `deny` or `reject`.
`denyPolicy`	Name of the deny policy.
`rejections`	Reject rules that fired, as `policy/rule`.
`filters`	Tables with a row filter, as `catalog.schema.table` in lower case.
`masks`	Every mask on the tables read, as `table.column=type`. The column is the policy's pattern, such as `phone_*`.
`postMasks`	The masks that run after the query, same form.
`applied`	Every policy in `Enforce` mode that matched, as `Kind/Name`.
`rewrittenSql`	The whole rewritten SQL.
`rewrittenSqlContains`	Parts of the rewritten SQL.
`errorCode`	The error that must happen: `ACL_DENIED` for a deny, the rule code for a reject, else the rewrite error. The test runner ignores the `errorCode` of a deny policy.

Lists compare as sets, so the order is free. SQL checks ignore white space. An allow case fails when the rewrite fails, unless it sets errorCode.

- name: a deny policy
  identity: { subject: alice, groups: [analysts] }
  sql: "SELECT employee FROM shop.main.salaries"
  expect:
    outcome: deny
    denyPolicy: deny-salaries
    errorCode: ACL_DENIED

- name: a reject rule
  identity: { subject: alice, groups: [analysts] }
  sql: "SELECT id FROM shop.main.invoices UNION SELECT id FROM shop.main.orders"
  expect:
    outcome: reject
    rejections: ["reject-union/no-union"]
    errorCode: ACL_REJECTED

- name: a statement that the rewrite refuses
  identity: { subject: alice, groups: [analysts] }
  sql: "EXPLAIN SELECT email FROM shop.main.customers"
  expect:
    outcome: allow
    errorCode: ERR_UNSUPPORTED_SYNTAX

Schema fixtures for masked tables¶

The test runner has no database. For each masked table, list the columns under schema, in table order. The key is catalog.schema.table.

# Column names in the order of the real table.
schema:
  pg.public.customers: [id, email, tenant_id]

cases:
  - name: star reads email masked via schema fixture
    identity: { subject: alice, groups: [analysts] }
    sql: "SELECT * FROM pg.public.customers"
    expect:
      outcome: allow
      masks: ["pg.public.customers.email=partial"]
      rewrittenSqlContains: ["substr(sluice_src.email", "sluice_src.tenant_id FROM pg.public.customers sluice_src"]

Without the entry, the case fails and asks for a schema block. A table with only a row filter needs no entry, except under SELECT * in a query with a post-query mask.

A failing case¶

The second case of this suite is wrong.

cases:
  - name: an analyst reads orders
    identity: { subject: alice, groups: [analysts], claims: { tenantId: acme } }
    sql: "SELECT id FROM shop.main.orders"
    expect:
      outcome: allow

  # This case is wrong on purpose: no policy allows the guests group.
  - name: a guest reads orders
    identity: { subject: gus, groups: [guests] }
    sql: "SELECT id FROM shop.main.orders"
    expect:
      outcome: allow

sluice policy test policies.d --tests failing

PASS  an analyst reads orders
FAIL  a guest reads orders
        outcome: want "allow" got "deny"

1 passed, 1 failed, 2 total
[exit status 3]

With --json:

sluice policy test policies.d --tests failing --json

{
  "total": 2,
  "passed": 1,
  "failed": 1,
  "cases": [
    {
      "name": "an analyst reads orders",
      "passed": true
    },
    {
      "name": "a guest reads orders",
      "passed": false,
      "failures": ["outcome: want \"allow\" got \"deny\""]
    }
  ]
}
[exit status 3]

Flags and exit codes¶

Flag	Meaning
`--tests <path>`	Suite file or directory, without subdirectories. Default: `<policy-dir>/tests`.
`--strict`	Refuse unknown fields in policies.
`--json`	Print the report as JSON.

Exit status	When
0	All cases pass.
1	Bad flag, missing tests path or broken suite file.
3	A policy with an invalid field, a policy that does not compile, a failing case, or no case at all.

Limits¶

Only the YAML engine runs. OPA and RelationshipPolicy do not.
No SQL runs. A query on a missing column still passes.
SQL that does not parse reads no table. A deny case with a typo can then pass, unless it sets errorCode.
Not checked: timeouts, rate limits, budgets and the server row limit (limits.maxRows, max_rows). A case cannot expect a held query, but applied lists the ApprovalPolicy. A QueryRewritePolicy limit shows in rewrittenSql.
No authentication method: subjects.apiKeys never matches and subject.auth_method is empty.