AI Agents

NSDF-Data Analysis Notes

Analysis of University of Minnesota R68/R76 SuperCDMS-type data accessed through the National Science Data Fabric and nsdf-cli.

Student guides

Routine maintenance: keep your setup up to date

The project keeps changing: new releases bring new code and guides, and sometimes a new version of Python or of the packages the notebooks use. Every time the project lead announces a new release, and whenever the environment check warns you, follow Updating your setup (student guide 7). It takes about 10 minutes. Your first-analysis notebook runs the check for you; you can also run it yourself with python check_environment.py in the python folder.

Updating your setup: one page for Windows, macOS and Linux. What changed in each release: release notes.

New to computing? Start here: do the setup guide first, then the first analysis. Each guide is a web page written step by step, with checkpoints and a Copy button on every command. A PDF version, to download or print, is linked at the top of each page.

No.GuideTimeWorks on
Get started
1Setting up your computerInstall the tools and get the project running. Do this first.about 1 hourWindows macOS Linux
2Your first analysisOpen a notebook and make your first plots from detector data.about 2 hoursWindows macOS Linux
8Git in five minutesBranches, commits and pull requests, in plain words.5 minutesAny computer
9Git in VS CodeOptional: the same Git steps with buttons instead of typed commands.–Windows macOS Linux
Keep up to date
7Updating your setupAfter every release announcement, or when the environment check warns you.about 10 minutesWindows macOS Linux
Write and publish a note
3Writing a noteWrite your results as a note.–Any computer
4Publishing a noteGet your note onto the project site through a pull request.–Any computer
5Publishing a note with an AI assistantThe same, with an AI assistant doing the Git steps.–Any computer
6Working in your browser with GitHub CodespacesNo install: work from a web page.–In your browser
★Cheat-sheet: from note to live pageOne page, for after you have read guides 3 and 4.1 pageAny computer

Guides 1, 2 and 7 have a version for Windows, macOS and Linux on one page: the page picks your computer from your browser, and a switch at the top changes it (it remembers your choice). Each page links to the PDF for the selected computer.

Developer guides

For people who work on the repository itself. Each is one web page, with a PDF version linked inside it.

Library reference

The shared python/ library, generated from the code: every module, function, signature and status (done or under development).

Analysis Notes

Each note summarizes one investigation, pins the notebooks it rests on to the commits that produced the results, and ends with a status and next steps. Newest first. The links in a note show each file as it was at the commit named in the link, which may be older than the latest version, and the file may have moved since. To find where a file is now, see the table in data_analysis/README.md.

Status legend

StatusDefinition
CompleteNote is in a complete and finalized state.
In-progressNote is still being worked on.
Superseded (reference/link)A newer note has taken its place. This note stays up for reference and links to the newer one.
AbandonedNote was created, but, for whatever reason, is left incomplete without the intent for completion in the future.
BrainstormNote is used for brainstorming or putting down ideas, without any analysis results or conclusions.

Notebooks and deprecation

Notes marked Complete are left as they are, apart from small updates where a change in the python/ library requires one, such as a renamed function. The Jupyter notebooks they rest on are handled separately: when a library change breaks a notebook, it is either updated for the change or, if it no longer runs and is not worth fixing, marked deprecated with a warning cell at the top, like this:

Example of the marker (illustrative only; no notebook is marked deprecated at present):

DeprecationWarning: example_notebook.ipynb is deprecated (2026-09-25).
  It no longer runs: excursion_band(low, high) was renamed ratio_band in python/pulse_cuts.py.
  Kept as a record of the analysis; the current version is in Note 2.
Cell to paste at the top of a deprecated notebook 
<div style="border-left: 4px solid #d9534f; background: #fdd; color: #3b1c1c; padding: 10px 14px; font-family: monospace; white-space: pre-wrap;"><b>DeprecationWarning:</b> this notebook is deprecated (YYYY-MM-DD).
  It no longer runs: WHAT CHANGED, e.g. excursion_band(low, high) was renamed ratio_band in python/pulse_cuts.py.
  Kept as a record of the analysis; the current version is WHERE.</div>
Note # Link Date Author Title Description Keywords Status
10 Open note → 2026-10-21 Anthony How can we normalize our pulses to QET current? Want a refresher on how to analyze the SQUID readout chain for SuperCDMS detectors and include tests (e.g. IbIs) that allow us to get the absolutely normalized TES output current. Scan the UMN Wiki for records of such tests for R76 data. pulse, normalization, current, squid In progress
9 Open note → 2026-10-12 Buddy Holly (A. Villano) What is the current best PSD estimate from series 07221203_2025? I created a sample of clean randoms selected in the same way as Note 2 and produced a noise PSD for 07221203_2025. The randoms selection is updated/improved but the technique is the same. Since there are no known real pulses in those noise traces this is a true characterization of our noise environment. PSD, Noise, Randoms, PuBe Complete (2026-10-20),
revised (2026-10-23)
8 Open note → 13 Oct 2026 Anthony Where does the raw and processed (RQ) k100 data for R76 actually live at UMN? The Processing.xlsx paths no longer exist Processing.xlsx (cdms3.spa.umn.edu, fritts' processing log) records raw and RQ locations for 3 R76 series under /data/chocula/fritts/data/k100proc. None of the 6 paths exist today, and the whole fritts directory is gone from /data/chocula. A search of every k100proc directory under /project/chocula (not just fritts') found none of the 3 series anywhere. UMN, chocula, k100proc, BatRoot, RQ data, raw data, data location, Run68 In progress
7 Open note → 23 Sept 2026 Anthony Origin of the sharp peaks in the average noise PSD; the 408 kHz line The sharp peaks in the 14-trace average PSD are Detector 1 lines: two of the 14 traces are from Detector 1, whose median spectrum has a comb of lines that Detector 0’s lacks. Only a ~408 kHz line is common to both detectors (present in ~72% of Detector 0 traces); its source is unidentified. PSD, line noise, Detector 1, 408 kHz, mixed detectors In progress
6 Open note → 23 Sept 2026 Anthony excursion_band_AI: removing pulse-like traces from the noise selection, and the good_noise archive Adds a pulse-shape veto to the Tight cut (192 → 179 traces). The 13 removed traces are unambiguous pulses or drifts, and the mean PSD drops by a factor of ~20–165 below ~4 kHz. The selection is archived in an HDF5 file. excursion_band_AI, looks_like_pulse, HDF5 archive, PSD, under development In progress
5 Open note → 22 Sept 2026 Anthony Good triggered pulses on Detector 0 / Channel 0 (series 07221203_2025, dumps 2–5) A tightened cut (bstd < p10, ratio > 9, dev < 20) picks 10 of 11,642 pooled traces, but only 3 contain a visible pulse; the other 7 are selected by baseline drift. The 3 real pulses start at different samples (~1650, ~3600, ~3900). triggered pulses, baseline drift, pulse onset, trigger time In progress
4 Open note → 22 Sept 2026 Anthony python/ pulse library: development history and validation How the shared library grew from 20 to 37 functions over 22–25 Sept, how the refactored notebooks were checked against their earlier results, and the 52 unit tests. Structure and function tables are in Note 1a. library, validation, refactoring, unit tests, history In progress
3 Open note → 21 Sept 2026 Anthony Noise PSD from selected traces (Channel 0, Detectors 0 and 1) Time-domain overlay, FFT, and average power spectrum of 14 selected Channel 0 traces (12 from Detector 0, 2 from Detector 1) on a real-Hz axis (1.25 MHz sample rate). The selection is not the 41-trace cut of Note 2 and shares no events with it. PSD, FFT, noise, sample rate, glitch, selection bias Superseded (reference/link) by Note 6
2a Open note → 25 Sept 2026 Anthony Why does bstd_1000 have so many more high-bstd traces than bstd? (series 07221203_2025, dump 1) bstd and bstd_1000 against event number for dump 1. Before event 11000 the 1000-sample window has about twice as many high-bstd traces (the random-pulse expectation). From event 11071 the Unknown-trigger segments are quiet in 500 samples and loud in 1000, because their pulses rise together just after sample 500; these are most of the lined-up pulses in the Note 2 high tail. Most of them reach half their maximum about 836 us into the trace (histogram of the new half_max_time_us). Also compares the 179-trace noise selection with and without a randoms cut (141 randoms, 38 real triggers): the two parts agree in baseline scatter and in spectrum above 10 kHz; a leading glitch in a few traces can distort mean spectra. bstd, pretrigger window, trigger type, time order, dump 1 In progress
2 Open note → 14 Sept 2026 Anthony Noise-pulse selection on Detector 0 / Channel 0 (series 07221203_2025, dump 1) Building a clean-noise selection from 1517 traces. A naive low-ratio cut selects pulse-contaminated baselines; a 500-sample pretrigger window isolates the genuinely quiet population. noise selection, pretrigger window, baseline contamination, dev/bstd ratio, extreme-value statistics Complete (2026-09-23)
1a Open note → 22 Sept 2026 Anthony python/ library structure: modules, function tables, and status markers How the shared pulse library is organized (samples on the last axis, one config, quantities / cuts / operations / I/O / archive), with a table per category listing every function’s purpose and its done or under-development status. 37 functions: 29 done, 8 under development. library, structure, PulseConfig, status tagging, function inventory In progress
1 Open note → 14 Sept 2026 Anthony Run76 series catalog: non-PuBe series and availability on nsdf-cli Cross-checked the 235 Run76 series in the run log against nsdf-cli’s list and the live NSDF server: 130 PuBe series (38,566 dumps, 1,550 GiB) are available, and none of the other 105 are (42 Am, 2 Na-22, 21 with no source recorded, and 40 PuBe, nearly all DCRC3-triggered). Unchanged since the table was first made. Run76, DataSeriesList, nsdf-cli, live probe, Am, PuBe, DCRC3, availability Complete (2026-09-14)