Credential vending

By default, an Iceberg client that opens a table reads and writes object storage using the catalog’s own long-lived credentials — anyone who can reach the catalog effectively has warehouse-wide access to the bucket.

With vending enabled, the catalog’s bucket keys stay inside puddle. Clients get short-lived, table-scoped tokens instead:

expire after a configurable TTL (default 1 hour), and are refreshed by the client before they do (see Refreshing credentials)
only grant access to the requested table’s prefix in the bucket
read-write for callers allowed to commit to the table (engines write data files with the token from the table load before they commit), read-only for everyone else
consumed transparently by PyIceberg, Spark, Trino, and other Iceberg clients — no client-side configuration

Three backends are supported:

AWS — real STS AssumeRole
RustFS — built-in STS on the S3 endpoint
Ceph (RGW) — STS AssumeRole, with a role created via radosgw-admin

The setup is the same shape in all three. Define a role granting the access you want vended tokens to be able to use — this is the upper bound. Give puddle long-lived credentials that are allowed to sts:AssumeRole that role. Puddle assumes the role per request and narrows the result with an inline session policy scoped to the table being accessed, so a token vended for table A can’t touch table B.

Puddle configuration

Add a vended-credentials block to a warehouse’s s3: config:

puddle.yaml

warehouses:
  prod:
    location: s3://my-warehouse
    s3:
      region: us-east-1
      endpoint: https://s3.amazonaws.com  # omit for AWS, set for RustFS/Ceph
      path-style-access: false            # true for RustFS/Ceph; AWS supports both
      access-key-id: ${PUDDLE_AWS_ACCESS_KEY_ID}
      secret-access-key: ${PUDDLE_AWS_SECRET_ACCESS_KEY}
      vended-credentials:
        enabled: true
        role-arn: arn:aws:iam::123456789012:role/IcebergTableAccess
        duration-seconds: 3600       # default; AWS allows 900..43200
        # Optional:
        # sts-endpoint: ""           # defaults to s3.endpoint, then SDK regional default
        # sts-region: ""             # defaults to s3.region
        # external-id: ""            # cross-account sts:ExternalId
        # read-policy-template: ""   # text/template, see Custom policy templates
        # write-policy-template: ""

When vended-credentials.enabled: true, s3.access-key-id and s3.secret-access-key become required — the catalog needs explicit credentials to call sts:AssumeRole. This is enforced at config-load time, so a misconfig fails at boot, not on the first request.

AWS

1. Create an IAM user (or role) for the catalog

This is the principal whose long-lived keys puddle uses to call sts:AssumeRole. If puddle runs on EC2/EKS, prefer an instance role or IRSA — the SDK chain picks them up when access-key-id / secret-access-key are left empty.

For a static-key setup (simplest), create a user puddle-catalog with no AWS-managed policies attached, then create an access key for it and put the values in s3.access-key-id / s3.secret-access-key.

2. Create the table-access role

Trust policy (who can assume it):

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "AWS": "arn:aws:iam::123456789012:user/puddle-catalog" },
    "Action": "sts:AssumeRole"
  }]
}

For cross-account, add an sts:ExternalId condition and set the matching value in vended-credentials.external-id:

"Condition": { "StringEquals": { "sts:ExternalId": "puddle-prod-2026" } }

Permissions policy on the role (the upper bound; puddle’s inline session policy narrows per table):

{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::my-warehouse/*" },
    { "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::my-warehouse" }
  ]
}

3. Allow the user to assume it

Attach this inline policy to puddle-catalog (or its instance role):

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": "sts:AssumeRole",
    "Resource": "arn:aws:iam::123456789012:role/IcebergTableAccess"
  }]
}

4. Wire it into puddle

puddle.yaml

s3:
  region: us-east-1
  access-key-id: AKIA...
  secret-access-key: ...
  vended-credentials:
    enabled: true
    role-arn: arn:aws:iam::123456789012:role/IcebergTableAccess

Leave sts-endpoint and sts-region unset — the AWS SDK picks the correct regional STS endpoint automatically.

RustFS

RustFS serves STS AssumeRole on its S3 listener and enforces the inline session policy directly. role-arn is nominal — any well-formed ARN is accepted; only the inline session policy gates access. STS needs no configuration.

1. Credentials with bucket access

The catalog’s credentials must hold at least the permissions you want vended tokens to be able to use — the inline session policy can only narrow, not widen. For a local setup, the root credentials (RUSTFS_ACCESS_KEY / RUSTFS_SECRET_KEY) are the fastest path to a working setup; in production, create a dedicated user with a policy limited to the warehouse bucket.

2. Wire it into puddle

puddle.yaml

warehouses:
  dev:
    location: s3://my-warehouse
    s3:
      region: us-east-1
      endpoint: http://rustfs.local:9000
      path-style-access: true
      access-key-id: puddle-catalog
      secret-access-key: long-random-secret
      vended-credentials:
        enabled: true
        # Placeholder ARN — RustFS accepts any well-formed ARN.
        role-arn: arn:aws:iam::000000000000:role/dummy
        duration-seconds: 900   # STS minimum

sts-endpoint defaults to s3.endpoint, which is the right value for RustFS. Leave it unset.

There are two working end-to-end examples: the trino-rustfs-opa-vending compose stack, and internal/fileio/s3/vending_integration_test.go, which spins up RustFS via testcontainers, configures vending against it, and asserts that vended tokens can read/write inside the table prefix and are rejected outside.

Ceph (RGW)

Ceph’s RGW supports STS AssumeRole against the S3 endpoint, but unlike RustFS the role must actually exist (created via radosgw-admin) and the calling user must hold sts:AssumeRole permission via a user policy.

1. Enable STS in `ceph.conf`

ceph.conf

[client.rgw.<id>]
rgw_sts_key = abcdefghijklmnop          # exactly 16 chars; signs session tokens
rgw_s3_auth_use_sts = true

Restart the RGW daemon after editing.

2. Create a real role

radosgw-admin role create \
  --role-name=IcebergTableAccess \
  --path=/ \
  --assume-role-policy-doc='{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Principal": { "AWS": ["arn:aws:iam:::user/puddle-catalog"] },
      "Action": ["sts:AssumeRole"]
    }]
  }'

Attach a permissions policy to the role (warehouse-wide upper bound):

radosgw-admin role-policy put \
  --role-name=IcebergTableAccess \
  --policy-name=AllowWarehouseAccess \
  --policy-doc='{
    "Version": "2012-10-17",
    "Statement": [
      { "Effect": "Allow",
        "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
        "Resource": "arn:aws:s3:::my-warehouse/*" },
      { "Effect": "Allow",
        "Action": "s3:ListBucket",
        "Resource": "arn:aws:s3:::my-warehouse" }
    ]
  }'

3. Grant the catalog user `sts:AssumeRole`

radosgw-admin user policy put \
  --uid=puddle-catalog \
  --policy-name=AllowAssumeIcebergRole \
  --policy-doc='{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam:::role/IcebergTableAccess"
    }]
  }'

4. Wire it into puddle

puddle.yaml

warehouses:
  prod:
    location: s3://my-warehouse
    s3:
      region: us-east-1
      endpoint: https://rgw.local:8080
      path-style-access: true
      access-key-id: <puddle-catalog access key>
      secret-access-key: <puddle-catalog secret key>
      vended-credentials:
        enabled: true
        role-arn: arn:aws:iam:::role/IcebergTableAccess
        duration-seconds: 3600

sts-endpoint defaults to s3.endpoint, which is correct for RGW.

Custom policy templates

The default session policy grants the minimum set of actions Iceberg clients need (s3:GetObject, s3:PutObject, s3:DeleteObject on arn:aws:s3:::<bucket>/<table-prefix>/*, and s3:ListBucket on the bucket with an s3:prefix condition of <table-prefix>/*). For a warehouse at the bucket root with a table at the bucket root, the object resource is arn:aws:s3:::<bucket>/* and listing is not restricted by prefix.

Override the policy via write-policy-template when you need to:

Tag uploads (s3:PutObjectTagging)
Allow s3:AbortMultipartUpload for large writers
Add condition keys (e.g. aws:SourceVpc, aws:SourceIp)
Restrict to specific storage classes

Templates use Go’s text/template. Available fields:

Field	Example	Bucket-root table
`.Bucket`	`my-warehouse`	`my-warehouse`
`.KeyPrefix`	`db/events_v1`	(empty)
`.KeyPattern`	`db/events_v1/*`	`*`

Values are JSON-escaped (without the surrounding quotes), so put them inside JSON strings, and prefer .KeyPattern over {{.KeyPrefix}}/* so bucket-root layouts render arn:aws:s3:::<bucket>/* rather than <bucket>//*:

{ "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject",
                 "s3:AbortMultipartUpload"],
      "Resource": "arn:aws:s3:::{{.Bucket}}/{{.KeyPattern}}" },
    { "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::{{.Bucket}}",
      "Condition": { "StringLike": { "s3:prefix": "{{.KeyPattern}}" } } } ] }

Bad template syntax fails at warehouse construction time, not at request time — misconfig is loud. A template that renders invalid JSON fails the request (500) instead of being sent to STS.

Puddle never vends for a table location that can’t be expressed safely in a policy: one containing the IAM wildcards * or ?, a $ (which starts an IAM policy variable), a control character, or a query or fragment. Such a table is still served, without credentials, and puddle logs a warning. Credentials are scoped to the table’s recorded location (see Locations), which stays the same when the table is renamed, also into another namespace, and when its namespace’s location changes; a table whose metadata names another location than the recorded one gets none, with a warning. Tables registered from an external warehouse get credentials scoped to their location inside it.

Read-only and read-write tokens

Which token a caller gets depends on whether it may commit to the table, as in Apache Polaris. On a table load, create (including stage-create) or register, puddle asks the authorizer whether the caller is allowed table.commit on that table:

allowed: a read-write token (write-policy-template, or the built-in policy above). Engines write data files with the token from the table load and only then commit, so a caller that can commit needs one.
denied: a read-only token (read-policy-template, or the built-in read policy: s3:GetObject on the table’s prefix and s3:ListBucket restricted to it). It can read the table’s metadata and data files and nothing else.

A commit’s response is always read-write: the caller was just allowed to commit. With the default authorization (allow-authenticated, or none) everyone may commit, so every token is read-write.

The check is a probe: it goes through the same policy as a real table.commit, but a reader loading a table is not a denied commit, so puddle logs it at debug level as event="authz.probe" rather than as an authz.decision, and flags it "probe": true in the OPA input. It only runs when the response actually vends credentials. If the authorizer fails (OPA unreachable), the request fails with a 500 rather than guessing.

A custom read-policy-template gets the same fields as the write template, and must still allow s3:GetObject on the table’s prefix or readers can’t read data files:

{ "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::{{.Bucket}}/{{.KeyPattern}}" },
    { "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::{{.Bucket}}",
      "Condition": { "StringLike": { "s3:prefix": "{{.KeyPattern}}" } } } ] }

The X-Iceberg-Access-Delegation request header is not consulted: every table response carries credentials when vending is enabled. Trino sends it and Spark doesn’t, so it can’t decide anything. Puddle only records it, to show which clients ask for credentials: as access_delegation on the request log line of a table load, and in the puddle_access_delegation_requests_total metric (see Observability).

Refreshing credentials

A job that runs longer than duration-seconds needs new credentials before the vended ones expire. Puddle serves the Iceberg REST endpoint for that, GET /v1/{prefix}/namespaces/{namespace}/tables/{table}/credentials, advertises it in /v1/config, and points clients at it: every table load, create, registration or commit response that vends credentials also carries, in its config,

{ "client.refresh-credentials-endpoint": "v1/<warehouse>/namespaces/<ns>/tables/<table>/credentials",
  "client.refresh-credentials-enabled": "true" }

The endpoint is relative to the catalog URI. Iceberg’s Java S3FileIO (Spark, Flink and other Java engines) uses it on its own: it fetches new credentials five minutes before the old ones expire. A stage-create response carries no refresh keys, since the table isn’t in the catalog until the create commits; the commit’s response does.

A refresh is authorized as table.load on the table and vends what a table load by the same caller would vend at that moment: read-write if the caller may table.commit the table (the same probe), read-only otherwise. So a role that loses commit permission gets read-only credentials at its next refresh. The response is the table’s storage-credentials list; it is empty when the warehouse doesn’t vend, or the table’s location can’t be vended for (see above). A missing table is a 404 NoSuchTableException. The planId parameter, which names a server-side scan plan, is rejected with a 400: puddle doesn’t plan scans.

Troubleshooting

AccessDenied from STS at startup, not at request time. Puddle does not call STS on boot — failures only surface on the first table-returning request. Check the catalog’s logs for the wrapped vending: AssumeRole: ... error.

AccessDenied on the vended token. The inline session policy is the intersection of (a) the role’s permissions policy and (b) the rendered template. If the role’s permissions policy doesn’t cover the table’s prefix, the session gets nothing — widen the role first.

STS returns InvalidParameterValue for DurationSeconds. STS’s minimum is 900s. Anything below that is rejected; puddle’s config validator also rejects it.

Wrong STS endpoint on RustFS/Ceph. Leave sts-endpoint empty — it defaults to s3.endpoint, which is where RustFS and RGW serve STS. Setting it to AWS’s https://sts.amazonaws.com will cause AssumeRole to fail.

Cross-account on AWS: AccessDenied even with the right role. The trust policy needs an sts:ExternalId condition that matches the value puddle sends. Set vended-credentials.external-id to match.

Authorization

Credential vending

Puddle configuration

AWS

1. Create an IAM user (or role) for the catalog

2. Create the table-access role

3. Allow the user to assume it

4. Wire it into puddle

RustFS

1. Credentials with bucket access

2. Wire it into puddle

Ceph (RGW)

1. Enable STS in ceph.conf

2. Create a real role

3. Grant the catalog user sts:AssumeRole

4. Wire it into puddle

Custom policy templates

Read-only and read-write tokens

Refreshing credentials

Troubleshooting

1. Enable STS in `ceph.conf`

3. Grant the catalog user `sts:AssumeRole`