Getting started

This walkthrough runs puddle in Docker with a SQLite metastore and a local warehouse, then creates a namespace and a table, writes rows and reads them back with PyIceberg. It takes a few minutes and needs only Docker and openssl. For running puddle for real, continue with Operating puddle.

The image

Puddle is distributed as a Docker image with the binary as the entrypoint:

$ docker run --rm ghcr.io/ragnard/puddle:latest --version
puddle main (commit d472c94cef6e7b1c2a4f09e3b8d5a6c7e1f20394)

--version reports the commit the binary was built from, and the image’s org.opencontainers.image.revision label carries the same. Every image is a build of main (--version says main; a build of any other commit, such as a pull request’s, says dev). A release tags one of those builds with a version without rebuilding it, so a released image reports main and its commit too; the release notes name the commit of each version.

The image is published for linux/amd64 and linux/arm64, and is based on distroless static-debian13:nonroot: puddle runs as uid/gid 65532, not root.

Choosing a version

There are two channels:

• Releases, for anything you keep. Now and then a build of main is released as a semantic version (0.x.y before the first public release), with a GitHub release. Versions only go up: a fix is a new release from main, never a backport.
• Builds of main, for trying what’s next. Every push to main that passes its checks is published.

The image’s tags:

Pin a release by its version or its digest, and verify its signature. To follow main, pin main by digest (ghcr.io/ragnard/puddle:main@sha256:...) and let a tool such as Renovate or Dependabot move the digest; see Upgrades. Releases are kept; builds of main are deleted once they are older than 90 days and 50 newer ones exist.

The docs of each release are at https://puddle.blahonga.net/<x.y.z>/, and /stable/ redirects to the newest; the releases page lists them. Every build of main has its docs too, at /builds/<commit>/ (/main/ redirects to the newest), marked as a development build.

1. Write a config

Puddle reads one YAML file. This one keeps the SQLite metastore and the warehouse in /var/lib/puddle, the image’s data directory, and requires a bearer token, read from the environment:

metastore:
  type: sqlite
  sqlite:
    path: /var/lib/puddle/puddle.db

warehouses:
  default:
    location: file:///var/lib/puddle/warehouse

authn:
  static-tokens:
    - token: ${PUDDLE_TOKEN}
      principal: admin

Without the authn: block puddle would accept anonymous requests and give every caller full access (it logs a warning at startup when it runs that way). The configuration reference covers every setting.

2. Start puddle

$ export PUDDLE_TOKEN=$(openssl rand -hex 32)
$ docker run -d --name puddle -p 8089:8089 \
    -e PUDDLE_TOKEN \
    -v puddle-data:/var/lib/puddle \
    -v "$PWD/puddle.yaml:/etc/puddle/puddle.yaml:ro" \
    ghcr.io/ragnard/puddle:latest -c /etc/puddle/puddle.yaml

puddle-data is a named volume: new, it takes the ownership of the image’s /var/lib/puddle (uid 65532), so puddle can write to it. A bind mount, or a volume mounted anywhere else, keeps its own ownership: make it writable by uid 65532 first (for example sudo chown -R 65532:65532 ./data). In Kubernetes, set fsGroup: 65532 in the pod’s securityContext.

Check that it is up. The readiness probe needs no token; the catalog API does:

$ curl -s localhost:8089/readyz
{"status":"ready"}
$ curl -s -H "Authorization: Bearer $PUDDLE_TOKEN" \
    'localhost:8089/api/catalog/v1/config?warehouse=default'
{"defaults":{},"overrides":{"prefix":"default"},"endpoints":[...]}
$ curl -s 'localhost:8089/api/catalog/v1/config?warehouse=default'
{"error":{"code":401,"message":"missing credentials",...,"type":"NotAuthorizedException"}}

3. Create a table and write to it

Engines talk to puddle for metadata only: they read and write data files directly in the warehouse’s storage. With a file:// warehouse inside a Docker volume, the engine has to see that volume too, so this runs PyIceberg in a container that mounts puddle-data, as uid 65532 so the files it writes belong to puddle’s user (puddle deletes them when a table is dropped with purge). It shares puddle’s network namespace, so the catalog is at localhost:8089.

import os

import pyarrow as pa
from pyiceberg.catalog import load_catalog

catalog = load_catalog(
    "puddle",
    type="rest",
    uri="http://localhost:8089/api/catalog",
    warehouse="default",
    token=os.environ["PUDDLE_TOKEN"],
)

catalog.create_namespace("demo")

rows = pa.table({
    "id": pa.array([1, 2, 3], pa.int64()),
    "name": pa.array(["a", "b", "c"], pa.string()),
})
table = catalog.create_table("demo.events", schema=rows.schema)
table.append(rows)

print(catalog.list_tables("demo"))
print(table.metadata_location)
print(catalog.load_table("demo.events").scan().to_arrow())

$ docker run --rm -i --user 65532:65532 \
    --network container:puddle \
    -v puddle-data:/var/lib/puddle \
    -e PUDDLE_TOKEN -e HOME=/tmp \
    ghcr.io/astral-sh/uv:python3.13-bookworm-slim \
    uv run --quiet --with 'pyiceberg[pyarrow]' python - < quickstart.py
[('demo', 'events')]
file:///var/lib/puddle/warehouse/6f0a2c1e-8b3d-4e57-9c1a-2d4b6e8f0a13/metadata/00002-4ec8d761-5205-42d1-84db-eb8ce639b09b.metadata.json
pyarrow.Table
id: int64
name: string
----
id: [[1,2,3]]
name: [["a","b","c"]]

The table now exists in the catalog. puddle’s log shows each request with its request_id, and an authz decision line for each action (docker logs puddle). Load the table over the REST API directly:

$ curl -s -H "Authorization: Bearer $PUDDLE_TOKEN" \
    localhost:8089/api/catalog/v1/default/namespaces/demo/tables/events
{"metadata":{...,"location":"file:///var/lib/puddle/warehouse/6f0a2c1e-8b3d-4e57-9c1a-2d4b6e8f0a13",...,"table-uuid":"6f0a2c1e-8b3d-4e57-9c1a-2d4b6e8f0a13",...},"metadata-location":"file:///var/lib/puddle/warehouse/6f0a2c1e-8b3d-4e57-9c1a-2d4b6e8f0a13/metadata/00002-....metadata.json"}

The metastore holds the table’s name, its location and the location of its current metadata file; the metadata, manifests and data files are in the warehouse, in a directory named after the table’s UUID (names aren’t part of paths, so renames don’t move anything).

Clean up with docker rm -f puddle && docker volume rm puddle-data.

Other engines

Any Iceberg REST client works. With an S3 warehouse (the usual setup for engines on other hosts; see warehouses), point the engine at puddle and give it the bearer token. Storage settings the warehouse config forwards (region, endpoint, path-style access) reach clients automatically; the S3 credentials either come from the engine’s own config, as below, or are vended by puddle per table.

Trino (etc/catalog/iceberg.properties):

connector.name=iceberg
iceberg.catalog.type=rest
iceberg.rest-catalog.uri=http://puddle.example.internal:8089/api/catalog
iceberg.rest-catalog.warehouse=default
iceberg.rest-catalog.security=OAUTH2
iceberg.rest-catalog.oauth2.token=<token>
fs.native-s3.enabled=true
s3.region=us-east-1
s3.aws-access-key=<key>
s3.aws-secret-key=<secret>
# With vended credentials instead of the two keys above:
# iceberg.rest-catalog.vended-credentials-enabled=true

Spark (spark-defaults.conf, with the iceberg-spark-runtime and iceberg-aws-bundle jars on the classpath):

spark.sql.extensions                      org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.puddle                  org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.puddle.type             rest
spark.sql.catalog.puddle.uri              http://puddle.example.internal:8089/api/catalog
spark.sql.catalog.puddle.warehouse        default
spark.sql.catalog.puddle.token            <token>
spark.sql.catalog.puddle.io-impl          org.apache.iceberg.aws.s3.S3FileIO
spark.sql.catalog.puddle.client.region    us-east-1
spark.sql.catalog.puddle.s3.access-key-id     <key>
spark.sql.catalog.puddle.s3.secret-access-key <secret>
# With vended credentials instead of the two keys above:
# spark.sql.catalog.puddle.header.X-Iceberg-Access-Delegation vended-credentials

The Docker Compose examples run Trino against puddle and an S3 store, with and without OPA and credential vending.

Verifying an image

Every image is built on main for a push, after its pull request passed the full suite (the Iceberg REST Compatibility Kit and the engine integration tests among them), and passes a lighter set of checks there before it is published: a smoke test, a vulnerability scan (no fixable HIGH or CRITICAL findings), the unit tests and one Compatibility Kit combination. It is signed with cosign keyless signing: the signature is tied to the main workflow’s GitHub Actions identity, not a long-lived key, and records the commit as the annotation org.opencontainers.image.revision. A release tags such an image without changing it, so its signature is the build’s. Verify with the commit it was built from (docker run --rm ghcr.io/ragnard/puddle:<tag> --version), using cosign v3 or later (the signature is stored in cosign’s new bundle format, which cosign v2 doesn’t find):

$ cosign verify ghcr.io/ragnard/puddle:<tag> \
    --certificate-identity https://github.com/ragnard/puddle/.github/workflows/main.yml@refs/heads/main \
    --certificate-oidc-issuer https://token.actions.githubusercontent.com \
    -a org.opencontainers.image.revision=<commit>

-a checks that the signature was made for that commit. The signature covers the multi-arch image index, so verify the tag or the index digest (ghcr.io/ragnard/puddle@sha256:...) that you deploy; each release’s GitHub release lists its digest, its commit and the exact commands. The image also carries a BuildKit SBOM (SPDX) and SLSA provenance attestation:

$ docker buildx imagetools inspect ghcr.io/ragnard/puddle:<tag> \
    --format '{{ json .Provenance }}'
$ docker buildx imagetools inspect ghcr.io/ragnard/puddle:<tag> \
    --format '{{ json .SBOM }}'