goldpan.dataset

A dataset definition: which raw files, how to read them, and which processing to run on them. One TOML file, read by both packages: goldsluice runs the processing in batch, goldpan reads the pulses and the processed quantities back.

The file (r76.dataset.toml):

[dataset]
name = "r76-nai"
output = "r76-nai"                      # the processed quantities: a goldpan Parquet dataset (directory), keep it outside the repository
# output_format = "root"                # optional: RQ-style ROOT files instead, with our names (needs root_detector = N too)
# registry = ["../registry"]            # optional: folders of detector records (goldpan.registry), relative to this file
# metadata = "r76.series.csv"           # optional: run metadata per series (high voltage, trigger setting, ...), joined to the events when read

[[sources]]                             # where the pulses come from; as many as you like
name = "side2"
format = "idx"                          # any adapter: idx, midas, parquet, or one you registered
paths = ["~/idx/07221203_2025_F0001"]   # raw files or dump folders ...
glob = "~/idx/07221203_2025_F*"         # ... and/or a pattern (sorted)
detector = "NAI-S2"                     # the detector name (the adapter's detector_name)
readout = "DCRC_RUN76"                  # a preset of goldpan.config, or { pretrigger_samples = 1000, glitch_samples = 10, sample_period_s = 0.8e-6 }
[sources.read]                          # the other keyword arguments of the adapter's read(), verbatim
channels = { Side2pA = [0, 0], Side2pB = [0, 1] }
roles = { Side2pA = "phonon", Side2pB = "phonon" }

[[processing]]                          # the steps, in order; each adds quantities to the channels it names
algorithm = "peak"                      # a goldsluice algorithm
channels = "role:phonon"                # "all", "role:<role>", or a list of channel names
[processing.params]
glitch_samples = "@readout.glitch_samples"        # "@readout.<field>": taken from the readout of the data
pretrigger_samples = "@readout.pretrigger_samples"
amps_per_adc = 1.5e-9

Where the output goes. output may be a full path ("C:/data/goldsluice-output/r76-nai", or "~/goldsluice-output/r76-nai"), which is used as written. A relative output ("r76-nai") is put under the folder named by the environment variable GOLDSLUICE_OUTPUT when it is set, so the definition holds no path of your machine and the same file runs on every computer; when it is not set, a relative output is relative to the folder of the definition file, as before (and goldsluice run refuses it if that is inside a repository that does not ignore it).

An unfinished run. goldsluice run writes _goldsluice_status.json into the output folder (STATUS_FILE): {"state": "running", ...} when it starts and {"state": "complete", "started": ..., "finished": ..., "files": N, ...} when every raw file of the definition has been processed. If a run stops (an error, Ctrl-C, a crash) the state stays running, and Dataset.processed() warns that the output may be incomplete. An output folder made before this file existed has none and is read without a warning.

Steps that build on each other. A step reads the raw trace unless it has an input. A step whose algorithm gives a trace (a filter or a baseline subtraction: goldsluice algorithms lists them) has a name, writes no quantities, and a later step with input = "<name>" reads its result instead of the raw trace, for the channels the two steps share. A suffix is appended to the names of the quantities of a step, so that the same algorithm can run on both (adc_max, and adc_max_lp from the filtered trace). The quantities of a step must still be written once: two steps giving the same name is an error. Steps run in the order written, and an input is a step above:

[[processing]]
algorithm = "lowpass"                   # a filter: gives a trace, so it has a name
name = "lp"
channels = "role:phonon"
[processing.params]
cutoff_hz = 50e3
order = 4
sample_period_s = "@readout.sample_period_s"

[[processing]]
algorithm = "peak"
channels = "role:phonon"
input = "lp"                            # the peak of the filtered trace
suffix = "_lp"                          # adc_max_lp, current_max_lp
[processing.params]
...

Different steps for different sources. Every step runs on every source unless it has sources, a list of source names. Use it when the detectors need different processing; a step on role:phonon already adapts to each detector’s own channels, so name sources only when the steps differ. A step that reads a filter (input) must run on a subset of the sources the filter runs on, and every source needs at least one step. A source’s output is redone only when the steps that run on it change. Read back together, the sources show their different quantities as records of different fields (an awkward union), so pick one source’s rows by series before you use its fields:

[[sources]]
name = "nai"
detector = "NAI-S2"
...
[[sources]]
name = "ge"
detector = "GE-1"
...
[[processing]]
algorithm = "peak"
channels = "role:phonon"                # no `sources`: runs on both
...
[[processing]]
algorithm = "rise_crossing"
channels = "role:phonon"
sources = ["ge"]                        # only the raw files of the source "ge"
...

Run metadata. metadata is a reference to facts about whole series that an analysis needs for every event (high voltage, trigger setting, …): today a CSV file with a series column and any other columns (hv_v, trigger_value, trigger, …), relative to this file, one row per series; later it may name a database instead, and nothing else in the file changes. The values are joined at read time: Dataset.processed() (and so read()) adds one column per column of the table to the events, taken from the row of their series, so a change of a value never needs the data to be processed again. Nothing is written into the output and the fingerprint does not change. A series that has events but is not listed is an error that names it, and so is a column named like a quantity or a channel (processed(metadata=False) skips the join). The table is also at spec.run_metadata(), and goldpan.runmeta joins and groups it for events you have already. ROOT output is not read by processed(), so it has no metadata.

Writing a definition. dump(definition, path) writes a definition file from a mapping with the layout of the file (the same thing tomllib gives for it, so a program, such as a form in a browser, can build it from what the user chose) or from a DatasetSpec; dumps gives the text. It checks the definition the way load does before it writes anything, so a file it writes always loads, and load of it gives back the same definition. Comments are not kept and nothing is added, so write format_version yourself if the file should say which format it uses.

Nothing is assumed: the readout, the channels and every algorithm parameter are written down, in the file or in a registry it names. With registry (and/or the environment variable GOLDPAN_REGISTRY), a source whose detector is a name in the registry takes the readout, the [sources.read] defaults of its file format and, for ROOT output, root_detector from the detector’s record (goldpan.registry); whatever the file itself states wins, key by key, and a readout that disagrees with the registry’s is a warning. A detector name that is not in any registry is only a label, as before. Dataset.open(path) gives the same file to both packages:

ds = Dataset.open("r76.dataset.toml")
ev = ds.read()                          # the pulses, with the processed quantities attached (one Events)
q = ds.processed()                      # only the processed quantities, no pulses: small and fast
ev.quantities["Side2pA"]["current_max"]

With output_format = "root" the quantities are written as ROOT files in the layout of the collaboration’s RQ files (rqDir/eventTree and rqDir/zip<N>, see goldpan.adapters.rq_root.write), under our names (Side2pA_adc_max); root_detector is that N. Read them with goldpan.adapters.rq_root.read: Dataset.processed() reads the Parquet output only.

The processed quantities are stored one record per channel (quantities["Side2pA"]["adc_max"]), keyed by series, dump, event. python -m goldsluice run <file> produces them; reading them before that raises an error that says so.

Module Attributes

FORMAT_VERSION	registry (and a detector named in a registry supplies readout, [sources.read] defaults and root_detector).
STATUS_FILE	The file in an output folder that says whether the last goldsluice run finished (see output_status).
ENV_OUTPUT	The environment variable that holds the folder under which a relative output of a definition is placed.

Functions

dump(definition, path, *[, header])	Write a dataset definition to the file path (see dumps for what definition is and how it is checked; the check looks at the folder of path) and return it loaded from there, so the caller has what load(path) gives.
dumps(definition, *[, base, header])	The text of a dataset definition file, for definition: a mapping with the same layout as the file ({"dataset": {...}, "sources": [...], "processing": [...]}, what tomllib gives for it) or a DatasetSpec (its raw).
fingerprint(spec, source)	A short hash of what determines the processed quantities of source: how it is read and the processing steps that run on it (a step that names other sources does not count).
load(path)	Read and check a dataset definition file.
output_status(out)	The contents of _goldsluice_status.json in the output folder out ({"state": "running" | "complete", ...}), or None if the folder has no such file (it was made before the file existed, or is not a goldsluice output).
read_raw(source, path, *[, chunk])	The pulses of one raw file of source as Events read by its adapter: one Events, or with chunk pieces of at most that many events when the adapter can (iter_chunks; otherwise the whole file is one piece).
step_record(step)	A step as it is hashed and recorded in the provenance: name, input, suffix and sources appear only when the step has them, so a definition without them keeps the fingerprint it had before the keys existed.

Classes

Dataset(spec)	A dataset definition opened for reading: the pulses from its sources and the quantities processed from them.
DatasetSpec(name, output, sources, steps, ...)
Source(name, format, patterns, detector, ...)
Step(algorithm, channels, params[, name, ...])

goldpan.dataset.FORMAT_VERSION = 7

registry (and a detector named in a registry supplies readout, [sources.read] defaults and root_detector). 4: a relative output goes under $GOLDSLUICE_OUTPUT when that is set. 5: metadata, a reference (a CSV file) to run metadata per series, joined to the events when they are read. 6: steps may have a name, an input (the name of an earlier step whose result they read) and a suffix for their quantities. 7: a step may have sources, the names of the sources it applies to (without it a step applies to all of them).

Type:

The version of the dataset definition file; a reader refuses a newer one. 2

Type:

output_format and root_detector. 3

goldpan.dataset.STATUS_FILE = '_goldsluice_status.json'

The file in an output folder that says whether the last goldsluice run finished (see output_status).

goldpan.dataset.ENV_OUTPUT = 'GOLDSLUICE_OUTPUT'

The environment variable that holds the folder under which a relative output of a definition is placed.

class goldpan.dataset.Source(name: str, format: str, patterns: Tuple[str, ...], detector: str, readout: goldpan.config.PulseConfig | None, read: Mapping[str, Any], base: pathlib.Path, detector_record: goldpan.registry.DetectorRecord | None = None)

Bases: object

Parameters:

• name (str)

• format (str)

• patterns (Tuple[str, ...])

• detector (str)

• readout (PulseConfig | None)

• read (Mapping[str, Any])

• base (Path)

• detector_record (DetectorRecord | None)

name: str

format: str

patterns: Tuple[str, ...]

The paths and glob entries as written (with ~ and relative to the folder of the definition file when used).

detector: str

readout: PulseConfig | None

read: Mapping[str, Any]

Keyword arguments of the adapter’s read, as written in the file.

base: Path

The folder of the definition file, for relative paths.

detector_record: DetectorRecord | None = None

The registry’s record of detector, if a registry has one; read and readout already include what it supplies.

files()

The raw files or dump folders of the source, in order. ValueError if a pattern or path matches nothing.

Return type:

list

class goldpan.dataset.Step(algorithm: str, channels: Any, params: Mapping[str, Any], name: str | None = None, input: str | None = None, suffix: str = '', sources: Tuple[str, ...] | None = None)

Bases: object

Parameters:

• algorithm (str)

• channels (Any)

• params (Mapping[str, Any])

• name (str | None)

• input (str | None)

• suffix (str)

• sources (Tuple[str, ...] | None)

algorithm: str

channels: Any

"all", "role:<role>" or a tuple of channel names.

params: Mapping[str, Any]

name: str | None = None

What later steps call this one (input = "<name>"); needed by a step whose algorithm gives a trace (a filter).

input: str | None = None

The name of an earlier step whose resulting trace this step reads instead of the raw trace, or None.

suffix: str = ''

Appended to the names of the quantities this step writes (adc_max becomes adc_max_lp), so that the same algorithm can run on the raw trace and on a filtered one.

sources: Tuple[str, ...] | None = None

The names of the sources this step applies to, or None for all of them.

applies_to(source_name)

Does this step run on the raw files of the source called source_name? (Always, unless the step names its sources.)

Parameters:

source_name (str)

Return type:

bool

channel_names(detector)

The channels of detector this step applies to.

Return type:

list

resolve(config)

The parameters with every "@readout.<field>" replaced by that field of config.

Parameters:

config (PulseConfig)

Return type:

dict

class goldpan.dataset.DatasetSpec(name: str, output: pathlib.Path, sources: Tuple[goldpan.dataset.Source, ...], steps: Tuple[goldpan.dataset.Step, ...], path: pathlib.Path, raw: Mapping[str, Any], output_format: str = 'parquet', root_detector: int | None = None, registry: goldpan.registry.Registry | None = None, metadata: pathlib.Path | None = None)

Bases: object

Parameters:

• name (str)

• output (Path)

• sources (Tuple[Source, ...])

• steps (Tuple[Step, ...])

• path (Path)

• raw (Mapping[str, Any])

• output_format (str)

• root_detector (int | None)

• registry (Registry | None)

• metadata (Path | None)

name: str

output: Path

sources: Tuple[Source, ...]

steps: Tuple[Step, ...]

path: Path

The definition file.

raw: Mapping[str, Any]

The parsed TOML, kept for provenance.

output_format: str = 'parquet'

"parquet" (a goldpan dataset) or "root" (RQ-style ROOT files, goldpan.adapters.rq_root.write).

root_detector: int | None = None

The number N of the zip<N> tree of the ROOT output; required with output_format = "root" (written in the file, or the number of the registry’s record of the detector), otherwise None.

registry: Registry | None = None

The registry the definition names (and the environment variable adds); empty if there is none.

metadata: Path | None = None

The file the definition names as its run metadata ([dataset] metadata), or None. It is not opened until it is used.

run_metadata()

The run metadata as a table ({column: array}, see goldpan.runmeta.read_table), or None if the definition names none. The file is read now, not when the definition is loaded, so a run that does not need it works without it.

source(name=None)

Parameters:

name (str | None)

Return type:

Source

goldpan.dataset.load(path)

Read and check a dataset definition file. Nothing is opened: the raw files need not exist yet.

Return type:

DatasetSpec

goldpan.dataset.dumps(definition, *, base='.', header='')

The text of a dataset definition file, for definition: a mapping with the same layout as the file ({"dataset": {...}, "sources": [...], "processing": [...]}, what tomllib gives for it) or a DatasetSpec (its raw). The definition is checked first, exactly as load checks a file, as if the file were in the folder base (relative registry and metadata are looked for there), so text that load would refuse is never produced, and load(dump(...)) gives back the same definition. header is written as comment lines at the top. Keys are written as given and nothing is added, not even format_version. Tables go in the order dataset, sources, steps, with read and params as sub-tables; comments of an earlier file are not kept.

Parameters:

header (str)

Return type:

str

goldpan.dataset.dump(definition, path, *, header='')

Write a dataset definition to the file path (see dumps for what definition is and how it is checked; the check looks at the folder of path) and return it loaded from there, so the caller has what load(path) gives. A definition that does not pass the check is not written, and an existing file at path is left alone.

Parameters:

header (str)

Return type:

DatasetSpec

goldpan.dataset.step_record(step)

A step as it is hashed and recorded in the provenance: name, input, suffix and sources appear only when the step has them, so a definition without them keeps the fingerprint it had before the keys existed.

Parameters:

step (Step)

Return type:

dict

goldpan.dataset.fingerprint(spec, source)

A short hash of what determines the processed quantities of source: how it is read and the processing steps that run on it (a step that names other sources does not count). The batch run stores it in every output file and skips a file whose fingerprint has not changed.

Parameters:

• spec (DatasetSpec)

• source (Source)

Return type:

str

goldpan.dataset.read_raw(source, path, *, chunk=None)

The pulses of one raw file of source as Events read by its adapter: one Events, or with chunk pieces of at most that many events when the adapter can (iter_chunks; otherwise the whole file is one piece).

Parameters:

• source (Source)

• chunk (int | None)

Return type:

Iterator[Events]

goldpan.dataset.output_status(out)

The contents of _goldsluice_status.json in the output folder out ({"state": "running" | "complete", ...}), or None if the folder has no such file (it was made before the file existed, or is not a goldsluice output).

Return type:

dict | None

class goldpan.dataset.Dataset(spec)

Bases: object

A dataset definition opened for reading: the pulses from its sources and the quantities processed from them.

Parameters:

spec (DatasetSpec)

classmethod open(path)

Return type:

Dataset

processed(*, detector=None, metadata=True)

The processed quantities of a detector, without pulses (an Events with no traces): one record per event. With the run metadata of the definition (metadata) each event also gets its series’ values as columns, joined now (metadata=False leaves them out).

Parameters:

• detector (str | None)

• metadata (bool)

Return type:

Events

iter_events(source=None, *, processed=True)

One Events per raw file of source: its pulses, with the processed quantities attached unless processed=False.

Parameters:

• source (str | None)

• processed (bool)

Return type:

Iterator[Events]

read(source=None, *, processed=True)

All the events of source as one Events (pulses, and the processed quantities unless processed=False).

Parameters:

• source (str | None)

• processed (bool)

Return type:

Events