How do I run a pipeline locally?

The CLI is not published yet; in the prototype it is built from source. Create a pipeline, check it, run it:

terminal

waddle init orders-sync          # writes pipeline.toml and pipeline.sql
waddle validate orders-sync/pipeline.toml
waddle run orders-sync/pipeline.toml --state orders-sync/state

The state file stores the watermark between runs. It is written only after the sink succeeded and the checks passed.

How do secrets work?

Configs never contain secrets. A connector names an environment variable (dsn_env) and the runner reads it at start. A config with an inline password field is rejected.

environment

export SRC_DSN='host=db.example.com user=reader dbname=app'
export WADDLE_S3_ENDPOINT=<account>.r2.cloudflarestorage.com   # S3 or R2 sinks and sources
export WADDLE_S3_KEY_ID=...
export WADDLE_S3_SECRET=...

What goes in the pipeline file?

Field	Meaning
`name`	Required. 1 to 64 characters of letters, digits, _ and -.
`sql`	Required. Path to the SQL file, relative to the config. No absolute paths, no "..".
`schedule`	Optional. Five-field cron expression, used by the hosted service.
`[source] kind = "file"`	url (https:// or s3://) and format (csv, json or parquet). Plain http and local paths are refused.
`[source] kind = "postgres"`	dsn_env (the name of an environment variable holding the connection string) and table (name or schema.name).
`[sink] kind = "file"`	url (https:// or s3://) and format.
`[sink] kind = "postgres"`	dsn_env, table, mode (append, replace or upsert) and key (required for upsert).
`[incremental]`	column (output column whose maximum becomes the next watermark) and initial (the value for the first run).
`[[checks]]`	rule = not_null or unique (column), row_count (min, max) or reconcile (tolerance, default 0).

Unknown fields are errors. Table and column names must be plain SQL identifiers.

What are the rules for the SQL?

The source is loaded as a table named src. Write one SELECT over it; the result is what the sink receives.
Incremental pipelines filter with the bound parameter $watermark. It is passed as a value, never pasted into the SQL, so it cannot inject code. The SQL must use it if [incremental] is set, and must not if it is not.
Comments are allowed, including a trailing one. Only one statement is accepted.
Your SQL runs after the machine is locked down: it cannot read files, reach the network, attach databases or load extensions.

What do the checks do?

not_null and unique inspect the query result before anything is written. A failure stops the run and leaves the sink and the watermark untouched.
row_count bounds the number of result rows. On an incremental pipeline a run with no new rows has zero, so omit min there.
reconcile runs after the write. For an upsert sink it checks that every key landed; for a replace sink or a file it compares row counts after reading the sink back.

How do I test a pipeline without a source?

waddle test pipeline.toml --sample sample.csv runs the SQL and the checks over a small CSV file, which the SQL reads as src. No source, sink or secret is used, and the same lockdown applies.

Quickstart and file format