The procedure described here is current as of 2022 Jul 1. This works on the old SL5 machines (cdms1, cdms3, cdms-04-1, cdms-04-2, cdms-04-3); we haven't gotten cdmsbats entirely working on CentOS. This method does not use Condor to distribute the jobs. Condor doesn't work very well currently.

First you may need to follow some of the instructions below to 1) get cdmsbats installed and working and/or 2) set up directories for output data.

To start the processing jobs you can use the spreadsheet below, modified as necessary:

Processing spreadsheet

Starting on the Noise tab, enter the run ID and the names of the configuration files on the upper left. Automatic lines are generated in Column D. (These lines are specific to Matt Fritts' directory location for cdmsbats and for processed data and should be changed here and on the Dumps tab if you are using other directories.) Copy and paste the lines from Column D into any SL5 machine to process the noise. This should take about 15 seconds. It will produce a file “testnoise.txt” which you can check to see if it worked or whether there were errors.

Then go to the Dumps tab. You can enter numbers for First Dump and Last Dump. Then copy the automatically-generated lines in Column D to process those dumps. (The way I have it set up is to divide the dumps into 5 chunks and run jobs in parallel on the 5 SL5 computers. If you want to do the same, enter # Of Dumps in the appropriate place, then start by entering 1 for First Dump. The Last Dump will be automatically calculated. Copy and paste, then to process the next chunk enter 1 greater than the Last Dump value as First Dump to get the next chunk of 5.) Files with names like “test15.txt” will be generated for each dump - you can check these to see the progress of the processing jobs.

The procedure below was written by Anthony on 2016 Feb 22 and reflects the current setup for MIDAS data. Generally data is processed using the autoprocessing but if you want to do some processing different than the standard you can follow this procedure.

The old procedure is kept here for posterity: obsolete_processing_procedure

for now this is done by contacting Anthony. You can skip this step if you do not intend to use /data/chocula/k100 as your data output directory.

 #setup k100 processing
 PATH=$PATH:/local/cdms/k100Pipe/quickProc/condor

 cp -r /local/cdms/k100Pipe/quickProc/condor/environments/ .
 cp -r /local/cdms/k100Pipe/quickProc/condor/templates/ .
 

Note: under templates you may need to edit pycondor-NoiseProc and pycondor-SeriesProc to include certain machine requirements.

Follow the format in the environments/ directory, there are several examples. The named cdmsbats input files must be in the directory for the version of cdmsbats that you're using, like:

 cdmsbats/UserSettings/BatRootSettings/processing/

for processing inputs, or

 cdmsbats/UserSettings/BatRootSettings/analysis/

for config inputs

If you leave the first line in the environment file, env:prefixdir:, the same, the processed data will overwrite whatever data exists in our standard data directory. Alternatively you can modify this line to write the processed data to a directory of your choosing. You will also need a symlink in that directory to the raw data. You'll also need to put a subdirectory for the noise processing in that directory. So do something like the following:

 mkdir /data/chocula/MyProcessedDataDirectory
 cd /data/chocula/MyProcessedDataDirectory
 ln -s /data/chocula/k100/midasraw/ midasraw
 mkdir midasfilterFiles
 mkdir midasfilterFiles/all

Then in the environment file you intend to use, change the first line to

 env:prefixdir: /data/chocula/MyProcessedDataDirectory

It may also make sense to change the last line, condor:outdir:, to a different location to store the condor logs for your jobs.

 condorProc.py --help

For example, say if I want to process all dumps and do the noise for a series:

 condorProc.py --series 07160222_1059 --donoise --dumps a --procenv environments/pymidasQ1024P2048.env

 condor_q -wide

Next time only step 7)-8) is necessary, and possibly step 5) if you're modifying the cdmsbats configuration.

There is currently no official copy of the processing code. Processing is usually done from Anthony's copy of the code. Sometimes this is not desirable for a manual processing job since Anthony's copy of the code may be in a development state. Also if you want to test some modification of the processing code, you will want your own copy. (Note that if you want to develop modifications and fixes to cdmsbats you should use established git protocol.)

Steps for creating a copy of the processing code:

You'll need an account on nero at Stanford. If you don't have one you should ask Steve Yellin to set one up.

In your ~/.ssh/config file make sure you also have the following stanza:

 # stanford - needed for git with submodules in nero:/data/git
 Host nero
         Hostname nero.stanford.edu
         User <username>
         GSSAPIAuthentication yes
         GSSAPIDelegateCredentials yes
         StrictHostKeyChecking no

 git clone --recursive nero:/data/git/Reconstruction/cdmsbats.git

 make

or

 make all

The most current version of cdmsbats and BatCommon don't compile on our system. A fix is to delete the IOlibrary directory from BatCommon. If you are in the UMNdev branch in both cdmsbats and BatCommon, and if there is no IOlibrary directory in BatCommon, it should compile.

The environment file mentioned above points to two files in which the user can include custom settings. They are located in

config file: <blockquote>cdmsbats/UserSettings/BatRootSettings/analysis</blockquote> and processing file: <blockquote>cdmsbats/UserSettings/BatRootSettings/processing</blockquote>

The config file is used to specify various properties of the detectors, DAQ, and readout systems. E.g. Number of DIBs, trace lengths, digitizer bins, gains, bias. In effect, many things that describe the state in which we took data as well as processing parameters such as butterworth filter parameters and template normalizations.

The processing file is used to specify what types of algorithms are used in processing the raw data. I.e. what rq's you want to calculate.

Remember, when initiating processing using condor, you must use the environment file to specify which config and processing files you want to use!

The OF and NSOF algorithms in CDMSbats require pulse template files. These are root files containing histograms of pulse traces for the various charge and phonon channels. There are many different templates used for processing different detector/dib configurations. The system of organizing these is to have a single folder at <blockquote>cdmsbats/PulseTemplates/files</blockquote> containing all the template root files, each file containing the TH1 templates for a single DIB, with root file names starting with e.g Z1* for ZIP1 templates. These template files must have a very specific structure. They must contain a TDirectory with the name of the zip for which they're being used. e.g. “zip1”, “zip2”. Within that directory, there must be a set of TH1's of template traces, each for a specific channel. The names of these templates must correspond to the type of detector for which they will be used. E.g. for detector type 4, where all dibs are considered separate detectors, they should be in folders named PA, PB, etc. QI, QO, QIX, etc. For detector type 700, generally used for iZips, where 3 physical dibs are actually considered one detector, they should be in a single TDirectory “zip1”, and named PAS1, PAS2.. QIS1 etc. Note that the detector type is specified by DET_TYPE in the processing file mentioned above. Also note that the template lengths must match the trace length set in Midas which should also be the sum of P(Q)_PRETRIGGER P(Q)_POSTTRIGGER in the processing file.

To tell the processing which of the many template files to actually use, there are a set of lists maintained in <blockquote>cdmsbats/PulseTemplates/lists</blockquote>

These are also organized by ZIP, with names like “Z1_Templates.list”. Their contents are a tab-separated table: <blockquote> 11105000000 19999999999 Z1_Templates_R133 % R133 templates

60000000000 69999999999 Z1_Templates_DCRC_4096 % Queens data templates w/ 3998 bins

70000000000 71005040000 Z1_Templates_DCRC_3998 % UMN data templates w/ 3998 bins

71005040001 71202032032 Z1_Templates_DCRC_4096 % UMN data w/ 4096 bins</blockquote>

Each line gives a range of series numbers and a template file to apply to any processing of data within that range. Make sure that the leading digit of the series listed corresponds to the proper facility where the data was taken.

If you're monkeying around with processing files, configuration files, or templates, chances are you've broken something and cdmsbats won't run properly the first(or tenth) time through. To hunt down the source of your errors, use the output files generated by condor. In your environment file, there is a parameter “condor:outdir:” that specifies the location of the condor output files. These files are named like <blockquote>[series number]_[“Noise” or “RQ”]_condor.phys.[“log”, “out”, or “err”].[condor job id].txt</blockquote> Depending on what you broke, there may be useful errors in either the Noise or RQ files. Note that Noise is processed first, so if your jobs are failing, start there. The “condor job id” is the number displayed when you run condor_q. The most recent jobs run will have the highest job number for a given series.

After finding your error, you should check the config or processing file settings to see if you've changed something related to the error. Checking against other, working, config/processing files can give some insight. If you aren't able to figure out what's breaking from there, it may be worth using grep in cdmsbats/BatCommon to find the place in code where it dies.