DuckDB Shellfs Extension

The shellfs extension for DuckDB enables the use of Unix pipes for input and output. By appending a pipe character | to a filename, DuckDB will treat it as a series of commands to execute and capture the output. Conversely, if you prefix a filename with |, DuckDB will treat it as an output pipe.

Documentation

Full documentation, including installation, usage, the function reference, and cookbook examples, is available at:

https://query.farm/products/extensions/shellfs

Installation

install shellfs from community;
load shellfs;

Development

For instructions on building the extension from source and running its tests, see BUILDING.md.

Compress an output pipe once

When the command runs gzip, disable DuckDB's automatic compression explicitly. Otherwise a destination ending in .gz can cause DuckDB to compress the CSV before the pipe compresses it again.

COPY (SELECT * FROM range(3) AS t(n)) TO '| gzip > out.csv.gz'
  (FORMAT CSV, HEADER, COMPRESSION UNCOMPRESSED);
SELECT * FROM read_csv('out.csv.gz');