Loading data onto a SLAC jupyter notebook

It is recommended to use uproot to load the data, so it will need to be imported at the start of your notebook. When opening one .root file this works fine on any version of SLAC's CDMS kernels, but if we wish to open multiple dumps with uproot.concatenate() then we will need uproot version at least 4, which is only available on recent kernels. However recent kernels, such as CDMS V04-08, do not have the correct version of awkward to support uproot.

To update awkward use the following code once in your notebook.

!pip install awkward==1.8.0 --user

Then to ensure the correct version of awkward is imported, we remove the path to the old version, do this after all other imports.

import uproot
import sys
sys.path.remove('/cvmfs/sft.cern.ch/lcg/views/LCG_98py3cu10/x86_64-centos7-gcc8-opt/lib/python3.7/site-packages')
import awkward
awkward.__version__

The last line should output the correct version of awkward.

Now all of the dumps in the data can be loaded with uproot.concatenate(), note that uproot works with individual TTrees, not the entire TChain.

rqdir = "/gpfs/slac/staas/fs1/g/supercdms/tf/umn/rq/"
series='07220531_1214'
eventTree = uproot.concatenate(rqdir+series+"/*.root:rqDir/eventTree")
zip1 = uproot.concatenate(rqdir+series+"/*.root:rqDir/zip1")

The outputted data will be in the form of an awkward array. Since this can take some time, it is recommended to store the output into a pickle to be loaded instead. For example,

pickle.dump(eventTree, open(f'{series}_eventTree.pkl', "wb"))
pickle.dump(zip1, open(f'{series}_zip1.pkl', "wb"))

The data can then be converted into numpy arrays. For example,

crand = numpy.array(eventTree['EventCategory'], dtype=bool)
ncrand = numpy.invert(crand)
PTOFamps = numpy.array(zip1['PTOFamps'])

or can be filled into a pyroot histogram.

Here is an example notebook showing more details and some calibration steps: r76_calibration.zip

Using pyroot can yield simpler code which parallels root code.

import ROOT
rqdir = "/gpfs/slac/staas/fs1/g/supercdms/tf/umn/rq/"
series='07220531_1214/'
e = ROOT.TChain("rqDir/eventTree")
z = ROOT.TChain("rqDir/zip1")
ROOT.TChain.Add(e,rqdir+series+"umn*root")
ROOT.TChain.Add(z,rqdir+series+"umn*root")
ROOT.TChain.AddFriend(z,e)

But the TChain structure can be limiting, for example, I have not found a good way to fit to a histogram created from the TChain while also using TCuts. In ROOT this can be done with

z->Draw("PTOF1amps>>h(300,0.2e-6,1e-6)",!crand)
h->Fit("gaus","","",0.34e-6,0.40e-6)

And in pyroot, one might try something like

c = ROOT.TCanvas("canvas","title",800,600)
ROOT.TChain.Draw(z,"PTOF1amps>>h(300,0.2e-6,1e-6)",ncrand)
ROOT.TChain.Fit(z,"gaus","PTOF1amps>>h(300,0.2e-6,1e-6)")
c.Draw()

but TChain's fit function does not accept TCuts.

Raw data can be loaded in using rawio and be stored in a pickle, but a large amount of raw data can be two much for the SLAC SDF and will take a long time and likely crash your server. For example, the following code will load all dumps up to a specified dump number into a folder rawdata/.

import rawio.IO as io
import os
import pickle
from tqdm import tqdm
from glob import glob
import pandas
rawdir = "/gpfs/slac/staas/fs1/g/supercdms/tf/umn/raw/"
series='07191124_1501'

if os.path.exists(f'rawdata/{series}.pkl'):
    events = pickle.load(open(f'rawdata/{series}.pkl', "rb"))
else:  
    dumplist=sorted(glob(rawdir+series+"/07*.mid.gz"))
    events_list = []
    for i,dump in tqdm(enumerate(dumplist)):
        if i==2:
            break
        events_dump=io.getRawEvents("",dump)
        events_list.append(events_dump) 
    events = pandas.concat(events_list)
    pickle.dump(events, open(f'rawdata/{series}.pkl', "wb"))

The data can then be handled like a dictionary.