So far, the best way of batch processing K100 data is to use the condor system of the physics department. Below I will outline the steps to use this system. To make it a complete guide, data copying and organizing are also touched.

Data copying from the DAQ machine and the physics cluster can be done by running a rsync job periodically. To avoid typing the ssh login password to the DAQ machine every time you launch a rsync job, you can set passwordless login on the DAQ machine. I also wrapped the rsync job in a bash script so I don't have to type the long command every time I need to copy data. The content of the script is

rsync -vv -r -t -W -e ssh --progress Administrator@vuk-01.spa.umn.edu:/cygdrive/i/data/pulseData/07* /data/masherbrum/k100/raw/byseries
/data/snowdon/soft/processing/organizeRawData.py

The CDMS data processing package, cdmsbats, needs to know where the raw data is located to do the processing. This location is stored in a shell variable BATROOT_RAWDATA. To avoid changing it every time when a new series is processed, this variable was kept a constant /data/masherbrum/k100/raw/all (or /data/snowdon/k100/raw/all), instead, links to all the raw data files are made in the directory. This is what the script organizeRawData.py in the second line of the above shell script does. The content of organizeRawData.py I'm using is posted below.

!/usr/bin/env python
 
import subprocess
import time
import sys
import os
import re
 
#locations  = ['/data/snowdon/k100/raw/byseries', '/data/vuk1/cdms/k100/raw/byseries']
locations  = ['/data/masherbrum/k100/raw/byseries']
 
raw_all_dir = '/data/masherbrum/k100/raw/all'
 
# Remove bad links
links = os.listdir(raw_all_dir)
links.sort()
for link in links:
    fullpath = os.path.join(raw_all_dir, link)
    if not os.path.exists(fullpath):
        links.remove(link)
        os.remove(fullpath)
        print 'Removed broken link', fullpath
 
 
# find all raw files that are not in links 
for raw_dir in locations:
    # delete file smaller than 2kB
    subprocess.call('find ' + raw_dir +' -user $USER -name *_F* -size -2k -exec  rm \'{}\' \\;', shell=True)
    # change the permissions of directories and files
    subprocess.call('find ' + raw_dir +' -user $USER -type d ! -perm -070 -exec chmod \'775\' \'{}\' \\;', shell=True)
    subprocess.call('find ' + raw_dir +' -user $USER -type f ! -perm -060 -exec chmod \'664\' \'{}\' \\;', shell=True)
    # change groups
    subprocess.call('find ' + raw_dir +' -user $USER ! -group hep -exec chgrp \'hep\' \'{}\' \\;', shell=True)
    # make links for the new raw files
    for root, dir, name in os.walk(raw_dir):
        for filename in name:
            if re.match(r'^\d{8}\_\d{4}\_F\d{4}', filename) and filename not in links:
                print 'Created link for file', os.path.join(root,filename)
                links.append(filename)
                os.symlink(os.path.join(root,filename), os.path.join(raw_all_dir,filename))

Now that the raw data is ready, we can setup cdmsbats and use it to process the data. The installation steps of cdmsbats can be found in a different note. After installation, the shell variables that cdmsdbats needs need to be set. I put all the commands in a bash script below

#!/bin/bash
 
# =========== setup things to run root and cdmsbats  ==========
if  [ $HOSTTYPE = "x86_64" ]; then
  . /data/snowdon/soft/x86_64/root_v5.26.00/bin/thisroot.sh
    export BATROOTDIR=/data/snowdon/soft/x86_64/cdmsbats_110412/BatRoot
    export PATH=$PATH:$BATROOTDIR:$BATROOTDIR/noise
else # 32-bit systems
  . /data/snowdon/soft/root/bin/thisroot.sh
    export BATROOTDIR=/data/snowdon/soft/cdmsbats/BatRoot
    export PATH=$PATH:$BATROOTDIR:$BATROOTDIR/noise
fi
 
# ---------- cdmsbats data related variables ----------
export MODE='prod'
 
if [ $MODE = 'prod' ]; then
    export BATROOT_RAWDATA=/data/masherbrum/k100/raw/all
    export BATROOT_NOISEFILES=/data/masherbrum/k100/filterfiles
    export BATROOT_RQDATA=/data/masherbrum/k100/rq/all
    export BATROOT_GPIBFILES=/data/masherbrum/k100
    export BATROOT_AUXFILES=/data/masherbrum/k100
fi

Then we can use the following command to process the data

# generate filter files
# NoiseGen series#, file#, # of events to process, processing file, config file
NoiseGen 07110916_2109 1 500 processingUMNData.G42 configUMNData.G42L
# reduce raw data to RQ data
# BatRoot series#, file#, # of events to process, processing file, config file
BatRoot 07110916_2109 1 10000 processingUMNData.G42 configUMNData.G42L

Condor system only does the processing job management. All the above information are still needed in the processing. What condor does is to setup all the variables cdmsbats needs to know and start the NoiseGen or BatRoot program. All the necessary information is contained in the condor job file. I posted examples of the condor job files for NoiseGen and BatRoot below.

# condor job file
+CondorGroup = "phys"
Universe        = vanilla
Executable      = /data/snowdon/soft/x86_64/cdmsbats_110412/BatRoot/noise/NoiseGen
GetEnv      = true
environment = "ROOTSYS=/data/snowdon/soft/x86_64/root_v5.26.00 LIBPATH=/data/snowdon/soft/x86_64/root_v5.26.00/lib LD_LIBRARY_PATH=/data/snowdon/soft/x86_64/root_v5.26.00/lib:/opt/intel_fc_80/lib:/opt/intel_fc_80/lib BATROOTDIR=/data/snowdon/soft/x86_64/cdmsbats_110412/BatRoot BATROOT_RAWDATA=/data/masherbrum/k100/raw/all BATROOT_NOISEFILES=/data/masherbrum/k100/filterfiles BATROOT_RQDATA=. BATROOT_GPIBFILES=/data/masherbrum/k100 BATROOT_AUXFILES=/data/masherbrum/k100"

Notification = error
Notify_user = zhang@physics.umn.edu
#Rank = KFlops
#Requirements = (Arch=="X86_64") && (KFlops>1000000) && (OPSYS_CPU_SSE2 == TRUE)
Requirements = (Arch=="X86_64")

OutputDir   = /scratch/masherbrum/zhang/condor
Error       = $(OutputDir)/condor.phys.err.$(Cluster)-$(Process).txt
Log         = $(OutputDir)/condor.phys.log.$(Cluster)-$(Process).txt
Output      = $(OutputDir)/condor.phys.out.$(Cluster)-$(Process).txt

should_transfer_files = YES
when_to_transfer_output = ON_EXIT

Arguments = "07110718_2110 1 500 processingUMNData.G101 configUMNData.G101"
Queue
Arguments = "07110718_2330 1 500 processingUMNData.G101 configUMNData.G101"
Queue

Some explanations of the file

+CondorGroup = “phys” defines the class ID CondorGroup. This is a class ID added by the physics department. Here I run my jobs on the “phys” cluster where I can get a decent number of x86_64 linux computers. Other options could be “cdms” or “ligo” or something else. “cdms” or “ligo” has much fewer machines. For more details, please see the physics website.

Executable = /data/snowdon/soft/x86_64/cdmsbats_110412/BatRoot/noise/NoiseGen specifies the program you want to run.

GetEnv = true and environment = xxx define the environmental variables needed by the executable. Shell variables are separated by space. Note in BATROOT_RQDATA=., the dot (.) means the directory where you submit the condor job. This means you should always submit you condor job from the directory where the RQ data is located. To make it more flexible, you can use BATROOT_RQDATA=/data/masherbrum/k100/rq/all instead.

Notification = error and Notify_user = zhang@physics.umn.edu tell condor to send you email notices when errors happened. You should use your own email address instead.

Requirements = (Arch==“X86_64”) tell condor only schedule jobs on x64 machines cause cdmsbats was compiled for 64-bit machines.

OutputDir = /scratch/masherbrum/zhang/condor tells condor where to put the error , output, and log files. You should use the directory which you have write permissions.

should_transfer_files = YES and when_to_transfer_output = ON_EXIT tell condor to use file transfer mechanism. This may not be necessary. I use this simply hopping to reduce the load of the computer where the raw and rq data are stored. It seems most of the load is from reading raw data rather than writing the rq data. The effect of this may not be significant.

Arguments = “07110718_2110 1 500 processingUMNData.G101 configUMNData.G101” defines the arguments the executable will take.

Queue tells condor to launch 1 job of the executable with the arguments.

Arguments = “07110718_2110 1 500 processingUMNData.G101 configUMNData.G101” and Queue define a specific job. You can have as many jobs as you want. The above example has two jobs defined.

The condor job files for BatRoot are in the exactly same way. Here is an example

+CondorGroup = "phys"
Universe        = vanilla
Executable      = /data/snowdon/soft/x86_64/cdmsbats_110412/BatRoot/BatRoot
GetEnv      = true
environment = "ROOTSYS=/data/snowdon/soft/x86_64/root_v5.26.00 LIBPATH=/data/snowdon/soft/x86_64/root_v5.26.00/lib LD_LIBRARY_     PATH=/data/snowdon/soft/x86_64/root_v5.26.00/lib:/opt/intel_fc_80/lib:/opt/intel_fc_80/lib BATROOTDIR=/data/snowdon/soft/x86_64/cdmsbats_110412/BatRoot BATROOT_RAWDATA=/data/masherbrum/k100/raw/all BATROOT_NOISEFILES=/data/masherbrum/k100/filterfiles BATROOT_RQDATA=. BATROOT_GPIBFILES=/data/masherbrum/k100 BATROOT_AUXFILES=/data/masherbrum/k100"

Notification = error
Notify_user = zhang@physics.umn.edu
#Rank = KFlops
#Requirements = (Arch=="X86_64") && (KFlops>1000000) && (OPSYS_CPU_SSE2 == TRUE)
Requirements = (Arch=="X86_64")

OutputDir   = /scratch/masherbrum/zhang/condor
Error       = $(OutputDir)/condor.phys.err.$(Cluster)-$(Process).txt
Log         = $(OutputDir)/condor.phys.log.$(Cluster)-$(Process).txt
Output      = $(OutputDir)/condor.phys.out.$(Cluster)-$(Process).txt

should_transfer_files = YES
when_to_transfer_output = ON_EXIT

Arguments = "07110718_2110 1 10000 processingUMNData.G101 configUMNData.G101"
Queue
Arguments = "07110718_2110 2 10000 processingUMNData.G101 configUMNData.G101"
Queue

Some useful commands of condor

# submit a condor job
condor_submit filename
# check condor jobs
condor_q
# check details of running jobs
condor_q -run
# more details of running jobs:q

condor_q -run -l
# remove jobs
condor_rm username
# change job properties, eg.
condor_qedit 1.1 CondorGroup '"cdms"'

I put Condor job generating, submitting, and monitoring together into a single script autoProcess.py. The ways to use it

# process unprocessed data between two series (end points are inclusive)
autoProcess.py au 07110916_2109 07110916_2109 processingUMNData.G42 configUMNData.G42L
# process all data between two series (end points are inclusive). Existing RQ data will be overwritten.
autoProcess.py a 07110916_2109 07110916_2109 processingUMNData.G42 configUMNData.G42L
# process data from a sereis # list in a file. Existing rq file will be overwritten (? need to check)
autoProcess.py f /home/grad/zhang/cdms/analysis/umnprocessing/g16k_nbias processingUMNData.G42 configUMNData.G42L

A single script does data copying, organizing, condor job creating, submitting, and monitoring. Eg.

/home/grad/zhang/cdms/analysis/umnprocessing/k100cp.sh
cd /data/masherbrum/k100/rq/all
autoProcess.py au 07110916_2109 07110916_2109 processingUMNData.G42 configUMNData.G42L

Snowdon or masherbrum doesn't have enough disk space to store all the raw data. Only relatively new data are on these machines. All the raw data are compressed and archived in /data/vuk1. cdmsbats can handle .gz file directly. This won't make the processing more difficult.

PARAMETER_INTEGER	DET_TYPE 		DETECTOR 1-2		= 4    # ZIPS
#DO_PROCESSING	DETECTOR 1-2	=	0
#WRITE_RQ	DETECTOR 1-2	=	0
DO_PROCESSING	DETECTOR 1-2	=	1
WRITE_RQ	DETECTOR 1-2	=	1
DO_PHONON_ALGORITHM     InflectionTime          DETECTOR 1-2	=	0
DO_PHONON_ALGORITHM	PipeFitPhonon 		DETECTOR 1-2	=	0 # algorithm not yet optimized for TF
DO_PHONON_ALGORITHM	OptimalFilterPhonon	DETECTOR 1-2	=	1
DO_PHONON_ALGORITHM	RTFTWalkPhonon 		DETECTOR 1-2	=	0
DO_PHONON_ALGORITHM	PulseIntegral 		DETECTOR 1-2	=	1
DO_PHONON_ALGORITHM     NoiseSelector           DETECTOR 1-2    	=    	0
DO_PHONON_ALGORITHM     ConstFreqRTFTWalkPhonon  DETECTOR 1-2     =    	1 
DO_PHONON_TOT		RTFTWalkPhonon	 	DETECTOR 1-2	=	0
DO_CHARGE_ALGORITHM     F5ChargeX               DETECTOR 1-2	=	0
DO_CHARGE_ALGORITHM	OptimalFilterChargeX	DETECTOR 1-2	=	1
DO_CHARGE_ALGORITHM	RTFTWalkCharge 		DETECTOR 1-2	=	1
DO_CHARGE_ALGORITHM	ChargeNoiseCovariance 	DETECTOR 1-2	=	0 # only for special studies, located in NoiseBuilder

Old

PARAMETER_DOUBLE        Q_CHISQ_THRESH          DETECTOR 1,2 =          999999 999999  #turned off for TF

New

PARAMETER_DOUBLE        Q_CHISQ_THRESH          DETECTOR 1,2 =      1e-4        1e-4 # Soudan detectors are around 5e-5

Can cdmsbats handle negative pulses ?

Find two identical pulse with different polarity and compare the output of cdmsbats. Or compare the raw pulses of two identical RQ's, which is easier. Only look at the first 1000 events

The answer is no.

Checked out cdmsbats from CVS and renamed as cdmsbats_110309. Since we are going to switch to 64-bit Linux soon, this version was put in /data/snowdon/soft/x86_64 and compiled as 64-bit version. The version cdmsbats_101117 was copied from the one we are using at the moment and recompiled as 64-bit version. The version cdmsbats_110309 use functional forms as templates and it is much easier to try different trace lengths.

Because SCL5 only has gcc4.1.2, only matlab 2008a is supported to compile mex code. make a script named 'matlab' in your ~/bin directory and include the following lines

#!/bin/bash
. /data/snowdon/soft/x86_64/root/bin/thisroot.sh
/local/site/pkg/matlab2008a/bin/matlab

Then start matlab, assuming CAP was installed, you need to compile 64-bit mex files for CAP. On the matlab command line, run the commands

>> CAP_user_setup
>> CAP_make_binaries
>> CAP_make_binaries_root

Exit matlab and restart matlab, CAP should be ready for you on 64-bit Linux.

We could not see the Am/Ba lines on the data we took these days. I guess this was because of there were too many pileups. Using a shorter trace length should reduce the pileups. Below I attached Lauren's email about how to deal with this.

Hi Jianjie, pileup usually shows up clearly in the chisq vs qsum plot - it will be a line shooting up to high chisq's at low energy (because the energy is incorrectly estimated and usually winds up smaller than it should be. since the fit is bad, the chisq is large). You can also view the pulses using the method you've been using to check the noise pulses - running the noise generating code on the files you are analyzing with the flag for storing pulses on. Its a big kludgy, but it will allow you to look at the pulses to check for obvious pileup problems. Bruno is currently working on incorporating cdmsbats into “get_tiny”. My understanding is that its almost working. When it is, you'll be able to run that on your test facility data to study the pulses more easily.

As for your other question regarding the trace length, if you are running the latest version of cdmsbats, you can choose to use a functional form instead of a template. For that you set “CALC_PHONON_TEMPLATE” and “CALC_CHARGE_TEMPLATE” to true in the processing config file (see the processingSoudanData.ITHybrid for an example). You can tweak the functional forms in the analysis config (again look at ITHybrid for an example). You also set the digitization length and trigger time in the analysis config as you have been doing. Since its a functional form it won't be as good as a template, but you should still be able to see the lines. For example, we are using this option for the DQ and we've seen the lines for mZIP and iZIP. If you are playing around with the trace length a lot, this might be a better option for you so that you don't have to generate a new template each time you change the trace length. If you need to run an older version of cdmsbats, then you will need to make a template to match the trace length, as we've done in the past.

Lauren

Inverted pulse data can be processed after changing the gain in cdmsbats from the default value 1 to -1. The changes are:

EventBuilder.cxx: Line 183 added line

183 tempBasicPulseCalc.SetPulseNorm(-1); // added by jz

NoiseBuilder.cxx: changed the gain from 1 to -1 on the following lines

1310      double normalization = -1;
1530      double normalization = -1;

The version which can process inverted pulses is saved as /data/snowdon/soft/x86_64/cdmsbats_101117_ng

It seems to me cdmsbats will ignore the configuration files if it doesn't find the isr file. This is unexpected.