Timeouts and purge

The settings are described in the configuration reference under server; this page is about what they mean for clients and for the files in the warehouse.

Request deadline

Every catalog request has a deadline: server.write-timeout minus a margin (10%, at most 5s), so 55s with the default 60s. Metastore queries, storage reads and writes, STS calls for vended credentials, and commit retries all stop at it.

• A request that runs out of time answers 503 ServiceUnavailableException naming the request id; the cause is logged under that id. Iceberg’s clients retry a 503 on idempotent requests (loads, lists), not on commits.
• A commit that hits the deadline while swapping the table pointer may have landed. Puddle checks; if it can’t tell, it answers 500 CommitStateUnknownException and keeps the new metadata file. The client then has to reload the table to see whether its commit is there, and must not clean up its files in the meantime.
• Commit retries after a concurrent change (up to 4) stop before the deadline; the client gets the 409 CommitFailedException and retries on its own.

Raise write-timeout if requests time out on slow storage or a slow STS; 0s removes the deadline (and the write timeout), which is for debugging only.

Dropping a table

DROP TABLE in an engine, or DELETE …/tables/{table} over the API, removes the table from the catalog. What happens to its files depends on whether the client asks for a purge (purgeRequested=true, for example Spark’s DROP TABLE … PURGE):

• Without purge, puddle removes the pointer only. All files stay in the warehouse; the table can be registered again from its last metadata file. Puddle doesn’t track them after that: deleting them is up to you. Dropping a view never deletes files.
• With purge, puddle removes the pointer, then deletes the files the table’s metadata names: data and delete files (unless the table sets gc.enabled=false), manifests, manifest lists, statistics files, older metadata files, and the current metadata file, last. Files nothing references (for example from failed writes) are not deleted, and on a file:// warehouse the emptied directories stay.

A purge deletes only files strictly under the table’s recorded location (its directory, wherever it has been renamed to). Anything else its metadata names (another table’s files, another warehouse, an external warehouse) is left in place and logged as purge: skipped files outside the table's location, and a table registered from an external warehouse keeps all its files. See Locations.

When the client gets its answer

Once the pointer is gone the drop has succeeded, and the answer is 204 no matter how the deletion goes. Puddle deletes the files after dropping the pointer and answers when the deletion is done, or when the request deadline arrives, whichever is first; in the second case (logged as purge: request ended before the purge finished) the deletion continues in the background, in the same replica. A client that disconnects doesn’t stop it either.

server.purge-timeout (default 10m, 0s for no limit) bounds each purge. Every purge logs one line when it ends, with the table, its metadata location, the number of files deleted and failed, the elapsed time, and the request_id of the drop:

• purge: finished: done. failed counts files that couldn’t be deleted (each also logged as purge: delete failed).
• purge: stopped before finishing; the remaining files are left in place: it ran out of purge-timeout, with the reason.
• purge: abandoned at shutdown; the remaining files are left in place: the replica was stopping and shutdown-timeout ran out first (see High availability).

A large table (many thousands of files on S3) can take longer than a request; that’s what the background deletion is for. Raise purge-timeout if purges log stopped before finishing.

Leftover files

Files a stopped or abandoned purge didn’t get to stay in the warehouse, and nothing in puddle refers to them any more. They are under the dropped table’s location; the log line’s metadata_location is under it, normally at <table location>/metadata/. To clean up, delete that location’s directory or prefix with your storage tools, after checking that no table created since the drop uses it: a drop frees its location, and a new table may be placed there.

If the current metadata file is still there (it is deleted last), you can instead register the table again from that metadata_location and drop it with purge once more.