Obsolete: data was taken on vuk-01 with the LabVIEW DAQ but not since switching to the MIDAS DAQ. See Manual Processing Procedure for the current procedure.

All raw data is found in /home/Administrator/pulseData on vuk-01. To login type

<blockquote> ssh Administrator@vuk-01 </blockquote>

With the password [redacted] Once you are in the pulseData directory, look for the new data you just made. Keep in mind that there be a throwaway file that occurs before the flashing.

Raw data is automatically copied over to /data/chocula/k100/raw/byseries every hour via Allison's crontab. It also changes the group of the data so to cdms. However, computer/network errors sometimes keep this process from being completed. In which case data will have to be copied over manually. In case someone else must someday take over the copying task, the crontab file is as follows:

<blockquote> 40 * * * * ~/K100_Data_Processing/./vuk01_copy_data.sh > ~/K100_Data_Processing/vuk01_auto_copy.txt 2> ~/K100_Data_Processing/vuk01_auto_copy.err 50 * * * * find /data/chocula/k100/raw/byseries -maxdepth 2 -user kennedy -exec chgrp cdms '{}' \; -exec chmod g+w '{}' \; </blockquote>

Where vuk01_copy_data.sh contains:

<blockquote> #!/bin/sh

file=$(date +07%y%m%d)

rsync -avz Administrator@vuk-01.spa.umn.edu:/cygdrive/i/data/pulseData/${file}* /data/chocula/k100/raw/byseries/ </blockquote>

Once the raw data has been copied to the correct directory on chocula, it can be processed. Instructions below are for iZIPv6 data taken with three DCRCs.

1. Get the processing files from /data/chocula/k100/k100Processing.tar.gz and place them in a directory of your choosing (for example ~/K100_Data_Processing).
2. Go into the quickProc directory. Type source doIZIPS.sh and hit enter. Then type export PATH=$PATH:./ .
3. To process a file of your choosing, say 07150710_1212 for example, type procOneNoise.sh 07150709_1213. After running this script is completed,you can check for any errors that might have occurred by looking at the output files produced in the procfile directory. In the given example, the output files would be labelled 07150709_1213_noise1_proc.err and 07150709_1213_noise1_proc.txt. If no error occurred, then continue on.
4. Log in to cdms1, cdms2, or cdms3 by typing ssh -Y username@cdms1 for example. Then return to the quickproc directory in your processing directory.
5. Type procOneSeries.sh 07150709_1213 for whatever series you are currently processing. Although the command line prompt comes up after entering this command, the processing is still running in the background and will continue for a couple hours. To see if you still have processing running type ps -u username | grep Bat and check if any output is printed. DO NOT PROCESS MORE THAN A COUPLE SERIES AT ONCE ON THE SAME COMPUTER! You can also monitor progress by looking at the output file - for example: tail procfiles/07150709_1213_01proc.txt (one file per raw data file)
6. Once completed, the processed data will be in /data/chocula/k100/rq/byseries/ .

To process HV data (using the HV adapter board, data read out with two DCRCs):

1. Do Step 1 from the previous instructions. Go into the quickProc directory.
2. The file “doHV.sh” may not be included. If not, create “doHV.sh” with the lines below. Make this file executable with chmod a+x doHV.sh
   1. - #!/bin/sh
   2. - export CDMSBATS_PROC_FILE=processingUMNDataHV.oZIP
   3. - export CDMSBATS_CONFIG_FILE=configUMNDataHV.oZIP
3. Type source doHV.sh and hit enter. Then type export PATH=$PATH:./ . Note: the instructions below are identical to those in the previous instructions.
4. To process a file of your choosing, say 07150904_1654 for example, type procOneNoise.sh 07150904_1654. After running this script is completed,you can check for any errors that might have occurred by looking at the output files produced in the procfile directory. In the given example, the output files would be labelled 07150904_1654_noise1_proc.err and 07150904_1654_noise1_proc.txt. If no error occurred, then continue on.
5. Log in to cdms1, cdms2, or cdms3 by typing ssh -Y username@cdms1 for example. Then return to the quickproc directory in your processing directory.
6. Type procOneSeries.sh 07150904_1654 for whatever series you are currently processing. Although the command line prompt comes up after entering this command, the processing is still running in the background and will continue for a couple hours. To see if you still have processing running type ps -u username | grep Bat and check if any output is printed. DO NOT PROCESS MORE THAN A COUPLE SERIES AT ONCE ON THE SAME COMPUTER! You can also monitor progress by looking at the output file - for example: tail procfiles/07150904_1654_01proc.txt (one file per raw data file)
7. Once completed, the processed data will be in /data/chocula/k100/rq/byseries/ .

To parallel-process using CONDOR:

1. clone the repository (move any directories named 'k100Pipe' first): git clone <user>@vegemite.spa.umn.edu:/data/giacdms1/villaa/repositories/k100Pipe
2. Go into the repository to the directory k100Pipe/quickProc/condor/.
3. execute this command to process a series and its noise: ./condorProc.py –donoise –series 07151006_0703 –dumps a &
4. or this command to reprocess the dumps but not process the noise: ./condorProc.py –series 07151006_0703 –dumps a &
5. you can also process only certain dumps, say 1,2,4,5,6: ./condorProc.py –donoise –series 07151006_0703 –dumps 1,2,4-6 &
6. In order to see the jobs running on condor, go to the machine you issued the command on and do “condor_q”

Once data has been copied to snowdon for processing, copy relevant data to chocula as well. At the start of each run create the directory /data/chocula/k100/runXXX/raw (XXX=run number) and then copy the data from snowdon to chocula using the cp command. Once this is completed and you are sure both snowdon and chocula contain copies of relevant raw data, then delete that data from vuk-01 to make room for more data taking. CLEAR THE DISK, FEAR THE DISK!

If you wish to view raw pulses, go to

/data/snowdon/soft/cdmsbats_120210/BUILD/bin

to use BatViewer. As an example:

./BatViewer data/masherbrum/k100/raw/byseries/07130217_0332/07130217_0332_F0002 21000

The last number is the event number.

Here is the old guide to processing data! DON'T DO IT THIS WAY ANYMORE!!

All raw data is found in /home/Administrator/pulseData on vuk-01. To login type

<blockquote> ssh Administrator@vuk-01 </blockquote>

With the password [redacted] Once you are in the pulseData directory, look for the new data you just made. Keep in mind that there be a throwaway file that occurs before the flashing. Write down all the useful files and group them according to bias.

Once you know what data you are looking at, go to /data/snowdon/soft/processing on our regular computer system (not vuk-01). Edit the file k100cp.sh. This file copies the data from vuk into snowden. You will see numerous rsync lines, with many commented out. The only thing you need to touch in this file is to make sure the directory of the data being copied has the correct date. You can have multiple lines, one for each day of data. It is ok if you are copying over data that has already been copied or that has already/won't be processed.

Next open biasn and biasp. These files should contain the names of all the data files you wish to process, with the positive (+/-) data files going into biasp and the negative (-/+) files going into biasn. Put in the data files you wrote down into the correct file. If you wish you can now vim autoprocess.sh. This is the script that first runs k100cp.sh to copy the files, and then runs autoProcess.py with the biasp and biasn files. In autoprocess.sh, there are a series of lines of the form

<blockquote>autoProcess.py au 07110914_1939 07110915_0019 processingUMNData.G42 configUMNData.G42p</blockquote>

or

<blockquote>autoProcess.py f $DIR/biasp processingUMNData.G106_3D configUMNData.G106_ph3_p2</blockquote>

These call autoProcess to either process files listed after (such as “au 07110914_1939 07110915_0019”), or to open a txt file and process all the files listed inside (such as “f $DIR/biasp”). After listing the raw data files, include the processing and configuration files, such as the processing and configuration files for G42 and G106a shown above. Many of these files have already been made for multiple detectors and multiple set ups (charge only or phonon and charge, 2 Dibs or 3 Dibs, etc.). You can access the processing files through /data/snowdon/soft/x86_64/cdmsbats_110412/BatRoot/configuration/ and the configuration files through /data/snowdon/soft/x86_64/cdmsbats_110412/BatRoot/configuration/analysis. If a file does not exist for the detector/set up you are using, pick the closest setup and alter the file as needed. Remember to save the new file with a name that clearly indicates what it is for.

Once everything is ready you simply type

<blockquote> ./autoprocess.sh </blockquote>

in the /data/snowdon/soft/processing directory.

You will be prompted to type in the vuk-01 password for each rsync line in k100cp.sh and if you wait too long there will be an error and it won't actually copy the files, so watch it carefully. Otherwise that is it!, your files should now be processed!

Once data has been copied to snowdon for processing, copy relevant data to chocula as well. At the start of each run create the directory /data/chocula/k100/runXXX/raw (XXX=run number) and then copy the data from snowdon to chocula using the cp command. Once this is completed and you are sure both snowdon and chocula contain copies of relevant raw data, then delete that data from vuk-01 to make room for more data taking. CLEAR THE DISK, FEAR THE DISK!

If you wish to view raw pulses, go to

/data/snowdon/soft/cdmsbats_120210/BUILD/bin

to use BatViewer. As an example:

./BatViewer data/masherbrum/k100/raw/byseries/07130217_0332/07130217_0332_F0002 21000

The last number is the event number.