Where does the raw and processed (RQ) k100 data for R76 actually live at UMN? The paths Processing.xlsx records no longer exist

Note 8 · Author(s): Anthony Villano · 13 Oct 2026 · Status: In progress

The question

We have had trouble locating the raw k100 data in general, and the Run 68 and Run 76 data specifically, on the UMN side. NSDF (Note 1) holds raw dumps for 130 of the 235 cataloged Run76 series, but says nothing about where the processed (BatRoot “RQ”) data lives, or about Run68 at all. The group keeps a processing log, Processing.xlsx, in /project/chocula/k100/run076 on cdms3.spa.umn.edu; this note reads it and checks whether the locations it records still exist.

Method

This note has no notebook to pin: everything below comes from reading one file and running read-only ls/find commands over SSH as user avillano on cdms3.spa.umn.edu, on 13 Oct 2026. Unlike a notebook pinned to a commit, these results reflect the state of that filesystem at that moment and could change; rerun the same commands to check them again.

The file read was /project/chocula/k100/run076/Processing.xlsx (owned by fritts, last modified 16 Dec 2022, 41,895 bytes, MD5 04d9ddc0ee43fb516c3c026d9b914788). It was not committed to this repository: it is someone else’s internal processing log, not raw detector data, and the repository already has an unrelated file of the same name (data_analysis/R76/series_lists/Processing.xlsx, MD5 b7ee6cfc0b5cabf91a75241f78129d22, different content) which this note does not touch.

What Processing.xlsx records

Processing.xlsx is a BatRoot (cdmsbats) processing log and template: nine sheets, but only three distinct Run76 series (several sheets are the same series split by dump range across different machines for parallel processing, not nine different series). Each sheet sets the raw-data and processed-output locations as shell environment variables before running BatRoot.

The pattern, if it still held: both raw and processed data live under a fixed path with the series name appended — raw at /data/chocula/fritts/data/k100proc/midasraw/byseries/<series>, processed at /data/chocula/fritts/data/k100proc/midasrq/byseries/<series>, both under user fritts’s area. The directory also has an analysis macro, r76_analysis_hvcal.c (9 Dec 2022), whose datapath variable hardcodes the same .../fritts/data/k100proc/midasrq/byseries/ path and reads umn*.root files from it with a TChain.

What is actually on cdms3 today

None of the six paths above exist. Not just the per-series byseries/<series> folders — the fritts directory itself is gone from /data/chocula (there is no top-level /data/chocula/fritts at all; fritts only owns a couple of unrelated small folders elsewhere under /data/chocula, such as docker_files and ristani). So this is not a renamed series folder or a typo in the spreadsheet: the entire personal processing area the log and the analysis macro depend on is absent.

Other users’ copies of the k100 pipeline

The k100proc layout (midasraw/byseries, midasrq/byseries, midasfilterFiles) is a shared convention, not unique to fritts: a search for every directory named k100proc under /project/chocula (depth 3, so any user's personal area) found it under neog, will, shi and acharya too, and none of these k100proc directories, anywhere under /project/chocula, contain our three series — checked by testing for each series under every such directory’s midasraw/byseries and midasrq/byseries, not by sampling a few users by name.

There is also a group pipeline tool, /project/chocula/cdmssoft/k100Pipe, with scripts copyRawToRemote.sh and copyRQToRemote.sh for copying raw and RQ data to a remote host by series and time range — evidence that at least some k100 data is routinely copied off cdms3 to somewhere else, which this note has not tracked down.

Reading

• The three series' recorded RQ output is gone from where the log says it was, and is not in any k100proc directory found under /project/chocula (every user’s, not a sample of a few). This does not prove the RQ data no longer exists anywhere — only that it is not where Processing.xlsx says, and not in any readable k100proc area on this filesystem today.
• The loss is structural, not a path change. The whole fritts area is missing, not just the series subfolders, so this looks like an account or directory removal (for example when someone leaves the group) rather than a reorganization of the k100 pipeline.
• This only speaks to these three series. Note 1 catalogs roughly 130 Run76 series with raw dumps on NSDF; this note has checked RQ availability for 3 of them (all off NSDF, i.e. UMN-only in this log) and raw/RQ availability at UMN for those same 3. It says nothing about Run68, and nothing about the other ~230 Run76 series.

Next steps

• Ask the UMN group (fritts, or whoever administers cdms3) directly whether that account's files were migrated or archived, and where.
• Check whether copyRawToRemote.sh / copyRQToRemote.sh point at a known remote host that might hold a copy.
• Repeat this same check for Run68, which this note did not touch.
• If the RQ data for these three series is confirmed gone, the fallback is reprocessing from NSDF's raw dumps with BatRoot, for whichever of the 130 available series are needed.
• Finish the broader run076/run68 name search across /project that did not complete in time for this note.