goldsluice.batch

Batch mode: run the processing of a dataset definition (goldpan.dataset) over all its raw files.

python -m goldsluice run r76.dataset.toml # process every raw file that is new or changed python -m goldsluice check r76.dataset.toml # read the definition and list what would be done

or from Python, goldsluice.batch.run("r76.dataset.toml"). For every raw file the pulses are read in pieces by the source’s adapter (goldpan.adapters), every step is applied to the channels it names with the Rust kernels, and the quantities (one record per channel, keyed by series, dump and event, plus what the adapter knows about each event such as trigger_type, without the pulses) are written into the output dataset of the definition, a goldpan Parquet dataset that goldpan.dataset.Dataset.processed() reads (or, with output_format = "root", RQ-style ROOT files under our names, written by goldpan.adapters.rq_root.write and read with rq_root.read; needs goldsluice[root]). A ledger, _goldsluice.json in the output folder, records for each raw file what was done, with what, and how long it took, so that a second run only does what changed: a file is skipped only if its fingerprint (the definition), its stamp (the raw file) and the revisions of the algorithms and of the reader that made the output are unchanged (force=True redoes everything). Every output file also carries its provenance in its metadata (goldsluice key).

Reading a dump dominates the time, and it is the same work for every dump, so run(..., workers=N) (--workers N) processes N raw files at once, each in its own thread (the readers, the kernels and the writers let go of the interpreter lock while they work): each file is read, reduced and written by one thread; only the small shared files (the manifest of a Parquet dataset, the ledger) are updated one at a time. Every file needs its dump in memory, about a gigabyte for a dump of 3000 events, so choose N by the memory of the machine as well as by its cores.

On a hard disk the readers fight for the head and the decoding waits for the disk, so a thread of its own reads the next raw files from the disk, one after the other, into the cache of the operating system a few files ahead of the workers (readahead, by default two more files than there are workers; readahead=0 turns it off). It reads only; the workers still decode and reduce.

goldpan does the reading and the definition, so batch mode needs it (pip install goldsluice[batch]).

A step may build on an earlier one (name, input, suffix in goldpan.dataset): a step whose algorithm gives a trace (KIND = "trace", a filter or a baseline subtraction) keeps its result for the channels it names, in memory for the chunk being processed, and a step with input = "<name>" reads that trace instead of the raw one. The ledger fingerprint and the provenance include name, input and suffix only where they are written.

A changed detector, and a run that did not finish. An output folder keeps the detector it was made with (the channels and the readout, pretrigger_samples among them) and will not take files made with another one, so that two readouts never share a folder. run says so before it processes anything. replace=True (--replace) is the way on: it moves the whole old output folder aside, to <output>.old-<date>-<time> (nothing is deleted), and starts a clean one, so a stopped run leaves a folder that has only files of the new detector, and the next run carries on in it. It does nothing when the detector is the same. Every run writes _goldsluice_status.json into the output folder: {"state": "running", ...} when it starts and {"state": "complete", ...} when every raw file has been processed; if the run stops the state stays running, and goldpan.dataset.Dataset.processed() warns that the output may be incomplete.

Processed data must not end up in a Git repository: check and run refuse an output folder that lies inside a Git working tree and whose data files that tree does not ignore (check_output_location; allow_output_in_repo=True, or --allow-output-in-repo on the command line, overrides it).

Functions

check(spec_path, *[, allow_output_in_repo])

Read the definition and return what run would look at, as [(source name, path), ...]; raises on any problem (an unknown algorithm, a pattern that matches nothing, a bad readout, an output folder inside a Git working tree that does not ignore it).

check_output_location(out[, output_format])

Raise ValueError if the output folder out is inside a Git working tree that does not ignore the data files written there (_probe.parquet, or _probe.root for ROOT output): results must not be committed by accident.

run(spec_path, *[, chunk, force, backend, ...])

Process the dataset defined in spec_path; returns the Report (what was done and the timing).

Classes

FileReport(path, skipped, n_events, read_s, ...)

Report(name, files, total_s, workers)

class goldsluice.batch.FileReport(path: str, skipped: bool = False, n_events: int = 0, read_s: float = 0.0, compute_s: dict = <factory>, write_s: float = 0.0)[source]

Bases: object

Parameters:
path: str
skipped: bool = False
n_events: int = 0
read_s: float = 0.0
compute_s: dict
write_s: float = 0.0
class goldsluice.batch.Report(name: str, files: list = <factory>, total_s: float = 0.0, workers: int = 1)[source]

Bases: object

Parameters:
name: str
files: list
total_s: float = 0.0
workers: int = 1
property n_events: int
goldsluice.batch.check_output_location(out, output_format='parquet')[source]

Raise ValueError if the output folder out is inside a Git working tree that does not ignore the data files written there (_probe.parquet, or _probe.root for ROOT output): results must not be committed by accident. An output outside any repository, or inside one that ignores it (the processed/ rule of this repository’s .gitignore), passes. When Git cannot answer (it is not installed, or it fails on that folder) nothing is refused.

Parameters:
Return type:

None

goldsluice.batch.check(spec_path, *, allow_output_in_repo=False)[source]

Read the definition and return what run would look at, as [(source name, path), ...]; raises on any problem (an unknown algorithm, a pattern that matches nothing, a bad readout, an output folder inside a Git working tree that does not ignore it).

Parameters:

allow_output_in_repo (bool)

Return type:

list

goldsluice.batch.run(spec_path, *, chunk=1000, force=False, backend='rust', allow_output_in_repo=False, workers=1, readahead=None, replace=False, progress=<built-in function print>)[source]

Process the dataset defined in spec_path; returns the Report (what was done and the timing). chunk is the number of events read at once (it bounds the memory used by the pulses); backend="python" runs the mirrors, for comparison. workers is the number of raw files processed at the same time (default 1, one after the other): the results are the same, only faster, and every file in flight holds its dump in memory (see the module documentation). readahead is how many raw files ahead of the workers a thread reads from the disk into the cache of the operating system (default workers + 2; 0 turns it off): it matters on a hard disk. An existing Parquet output made with a different detector (channels or readout) is refused before anything is processed; with replace=True it is moved aside to <output>.old-<date>-<time> (kept, not deleted) and a clean output is started. The run writes its state (running, then complete) to _goldsluice_status.json in the output folder.

Parameters:
Return type:

Report