goldpan.adapters.parquet¶

The native format of goldpan and goldsluice: Parquet (specification, format version 1).

goldsluice writes it, goldpan reads it, and anything else can read it: the files use plain Arrow types, not extension types, so Polars, DuckDB, pandas and pyarrow open them without goldpan. The design is in GoldpanGoldsluice/2026-10-25 and 2026-10-26 (log repository); this docstring is the specification.

A dataset is a directory:

<dataset>/
    _goldpan.json                      the manifest
    <detector>/<series>_<dump>.parquet one file per series dump of a detector

The manifest is JSON: {"format": "goldpan-dataset", "format_version": 1, "streams": {<detector>: {"kind": "events", "main_array": true, "detector": {...}, "files": [relative paths]}}}. A stream is the table of one detector; only streams of kind events exist so far (time series and ancillary streams are added later without changing this). A single .parquet file is also a valid dataset for the reader.

A file is the table of one detector’s events, one row per event, with these columns:

series (string), dump (string), event (int64): the key that identifies an event (event numbers restart in every dump).
t_trigger (float64, optional): the trigger time in seconds, on the clock named in the metadata key clock.
traces.<channel>.<segment>: a fixed-size list column per channel and segment (the samples of one trace), any numeric dtype; segment is main for an ordinary trace. Channel and segment names contain no dot.
every other column is a quantity (an RQ): a number, a string, a list of numbers (variable length is fine), or a struct of these (for example a struct per channel with std and max).

File metadata (Parquet key-value, all strings): goldpan.format_version (an integer; a reader refuses a newer one), goldpan.detector (JSON: name, channels with name, role, unit, input, and segments with name, n_samples, pretrigger_samples, glitch_samples, sample_period_s, start_s) and goldpan.meta (JSON: provenance, units of the quantities, the clock). Compression is zstd; row groups hold a few hundred events, so that a reader can skip groups and read only some columns.

Versioning. An addition that old readers can ignore (a new quantity, a new metadata key) keeps the version; a change they cannot read raises goldpan.format_version. The sample dataset in python/tests/data/sample_dataset/ is read by the tests of goldpan and is the file that goldsluice’s tests read too.

Where the files go is not decided here. write(events, path) writes exactly where it is told and does not check that the folder is outside a Git repository: that guard, and the single output root (GOLDSLUICE_OUTPUT, see goldpan.dataset), belong to batch mode (goldsluice run, goldsluice.batch.check_output_location). Called by hand, keep the output outside the repository yourself.

Module Attributes

REVISION

Raised by one when a change of this reader changes what it returns for the same file (a fix), so that goldsluice run redoes the outputs made with the old one (goldpan.adapters.revision).

Functions

`iter_chunks`(path, *[, chunk, detector, channels])	Yield the events of the stream as `Events` of at most `chunk` events, file by file, without reading a file at once.
`read`(path, *[, detector, channels, events])	Read the stream of one detector of a dataset (or a single file) into one `Events`.
`stream_matches`(path, detector)	Whether the stream of `detector.name` in the dataset at `path` was written with this detector (its channels and its readout).
`write`(events, path, *[, row_group_size])	Write `events` into the dataset directory `path` (created if needed), one file per (series, dump).

goldpan.adapters.parquet.REVISION = 1¶: Raised by one when a change of this reader changes what it returns for the same file (a fix), so that goldsluice run redoes the outputs made with the old one (goldpan.adapters.revision). Not for a change that leaves the results alone.

goldpan.adapters.parquet.stream_matches(path, detector)[source]¶

Whether the stream of detector.name in the dataset at path was written with this detector (its channels and its readout).

None when there is nothing to disagree with: the dataset does not exist yet, or has no stream of that name. False is the case in which write refuses to extend the stream (a different channel map, or a different readout such as another pretrigger_samples).

Parameters:: detector (Detector)
Return type:: bool | None

goldpan.adapters.parquet.write(events, path, *, row_group_size=256)[source]¶

Write events into the dataset directory path (created if needed), one file per (series, dump).

A file of the same series dump is replaced; the manifest gets the stream of this detector. Events added to an existing stream must have the same detector (channels and readout). events.meta must be JSON-serializable. Safe to call from several threads of one process for different dumps (the manifest is updated under a lock, the data files are written outside it); two processes writing into one dataset at the same time are not protected.

Parameters:

events (Events)
row_group_size (int)

Return type:

None

goldpan.adapters.parquet.read(path, *, detector=None, channels=None, events=None)[source]¶

Read the stream of one detector of a dataset (or a single file) into one Events.

detector names the stream when the dataset has several; channels limits the trace columns that are read; events is a slice or a list of positions to keep (applied after reading: for a dataset larger than memory use iter_chunks).

Parameters:: detector (str | None)
Return type:: Events

goldpan.adapters.parquet.iter_chunks(path, *, chunk=256, detector=None, channels=None)[source]¶

Yield the events of the stream as Events of at most chunk events, file by file, without reading a file at once.

This is how a dataset larger than memory is processed; the functions of goldpan work on any chunk.

Parameters:

chunk (int)
detector (str | None)

Return type:

Iterator[Events]