Guide to Automated Processing and Data Database

The online database can be found at http://www.hep.umn.edu/cdms/cdms_restricted/K100/run_table/runTable_couch/run_table.html.

0) Get the necessary directories, scripts etc. by cloning them from Anthony. To do this, go to the directory you want this all to be in (I chose /home/user1/[username]/K100_Data_Processing/ but you can choose whatever) and type

git clone [username]@vegemite.spa.umn.edu:/data/giacdms1/villaa/repositories/k100Pipe

You should now have a filled k100Pipe directory. If you haven't done this in a while, you might want to type git pull in the K100Pipe directory, to get the most up-to-date scripts.

1) Go to directory /home/user1/[username]/K100_Data_Processing/k100Pipe/quickProc/condor or wherever you have set up the k100 Pipe directory.

2) Type python to start python in command line

3) Type

import seriesDatabase as sd 

(Note: the name sd is not required here, just be consistent)

4) Set the environment by typing

sd.setTheEnvironment()

5) Connect to the server by typing server,

server,db=sd.dbConnect()

Now you can access the database! If you want to view certain databases, use the views Anthony set up. These include 'seriesfull', 'paradigm', 'proc', 'std', 'bg', 'ba', 'cf', 'stdwbias','stdkeydiag','runs'. You can learn more about the different view by looking at views.txt in the condor directory.

To look at the full database with one of these views (proc for example) type the following:

for row in db.view('_design/application/_view/proc',include_docs=True):
  print row.key,row.value

Note that the indentation of the print statement is important. You need at least one space before the print statement for Python to know it is part of the for loop. Typing this in will give you a list of output that looks like such

[u'Midas', u'07151112_1653'] {u'_id': u'07151112_1653_proc'}

The first value is the type of DAQ, in this case Midas. The second value is the series number. The value in curly brackets in the processing id, or proc_id. This value is unique for each item in the database (i.e. there is one for each series). Now lets try another. If you want to look at the series database type the following:

for row in db.view('_design/application/_view/std',include_docs=True):
  print row.key,row.value

Again, indentation is important. This is the standard view of the series database. The output should look something like this

07151215_1328 [u'07151215_1328', u'UMN', 55, None, 1000, 31, u'bg', u'good']

This view first shows you the series, then in brackets it shows you the series, the facility, the 'run number', 'events per file', randoms, duration, 'data type', and diagnosis.

Using the Database: Looking at an individual Entry Now lets look at an individual entry in a database. Let's say I want to look at the processing entry for the series 07151112_1653. Since I want to look at this series' entry in the processing database, and not the series database, I need to use it's processing id. Fortunately that id is simply the series name_proc. So to read the entry, I first define it as such:

test_doc=db[u'07151112_1653_proc']

Then type print test_doc to see it. The output looks as such:

<Document u'07151112_1653_proc'@u'7-7d0469a4a763de6f6a9ee517c149a827' {u'on disk': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13], u'paradigms': {u'vanilla': {u'proc rq': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13], u'paradigm id': u'8cf56e4c5d6f4e115b3b48a1a82a85a7', u'noise': True, u'proc rrq': [], u'reproc': False}}, u'doc type': u'proc', u'series': u'07151112_1653'}>

This is the entire entry from the processing database for a given series. It starts with the id for this particular entry '07151112_1653_proc'. It then shows the number of revisions (7), and has a random string of letters and numbers that represents the key for the current revision. It then lists the various keys ('on disk', 'paradigms', 'doc type'), and their corresponding values for this entry. For example, 'on disk' has a list value [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13], and 'paradigms' has a long curly bracket list as its value, with other keys inside.

Now let's say we want to look at the series database entry for this particular series. Just as before, we need the entry id. For the series database, the entry id is just the series number.

test_doc=db[u'07151112_1653']
print test_doc

this should return the output

<Document u'07151112_1653'@u'16-6d3097d9b5d6ab0400d513aaed17a300' {u'run number': 54, u'facility': u'UMN', u'default_diagnosis': True, u'series': u'07151112_1653', u'detector type': u'IZIP', u'bias': {u'detector 1': {u'side 2': 0.0, u'side 1': 0.0}}, u'proc id': u'07151112_1653_proc', u'randoms': 1000, u'DAQ': u'Midas', u'diagnosis': u'good', u'duration': 26, u'data type': u'88-Y'}>

As before, the entry starts with the id for this particular entry and the number of revisions (this time 16) and the revision key. It then lists keys relevant to the series ('run number', 'facility', 'bias', 'diagnosis', and 'data type' to name a few), and their values for this particular entry.

Now let's day we want to edit an entry in our database, for example, lets say we want to change the diagnosis from our last series to bad.

1) First we need to import json. Type import json

2) Then we need to load the relevant entry with json. Type test_doc=db[u'07151112_1653'] followed by dict=json.loads(json.dumps(test_doc))

3) If you type dict now, you will see the same dictionary again, in a slightly different format. In this example, it looks like this:

{u'run number': 54, u'facility': u'UMN', u'default_diagnosis': True, u'series': u'07151112_1653', u'_rev': u'16-6d3097d9b5d6ab0400d513aaed17a300', u'detector type': u'IZIP', u'bias': {u'detector 1': {u'side 2': 0.0, u'side 1': 0.0}}, u'proc id': u'07151112_1653_proc', u'randoms': 1000, u'DAQ': u'Midas', u'diagnosis': u'good', u'duration': 26, u'data type': u'88-Y', u'_id': u'07151112_1653'}

But now we can make changes to it. To make changes copy the output from typing dict and change the relevant portions. Notice that not only do I want to change the diagnosis key to 'bad', but I also want to change the 'default_diagnosis' key to False, indicating that it was not an automatic diagnosis. Set your database entry ( in this case db[u'07151112_1653']) equal to the new dictionary with the changes made, as shown below

dict={u'run number': 54, u'facility': u'UMN', u'default_diagnosis': False, u'series': u'07151112_1653', u'_rev': u'16-6d3097d9b5d6ab0400d513aaed17a300', u'detector type': u'IZIP', u'bias': {u'detector 1': {u'side 2': 0.0, u'side 1': 0.0}}, u'proc id': u'07151112_1653_proc', u'randoms': 1000, u'DAQ': u'Midas', u'diagnosis': u'bad', u'duration': 26, u'data type': u'88-Y', u'_id': u'07151112_1653'}

db[u'07151112_1653']=dict

4) Now if I type db[u'07151112_1653'] again, I will see that my changes have been made. You will also notice that the number of revisions has increased.

But there is an easier way if you just want to change a few things. Let's say I want to go back and make my diagnosis 'good' again, but I don't want to resave the whole entry. The first two steps are the same.

1) First we need to import json. Type import json

2) Then we need to load the relevant entry with json. Type test_doc=db[u'07151112_1653'] followed by dict=json.loads(json.dumps(test_doc))

3) To change just one key value in the entry, type dict[key]=newvalue. For example, if I want to change my diagnosis back to good I would type dict['diagnosis']='good'. If you want to change a key value that is nested in another key, such as bias you include both keys such as dict[outer_key][outer_key]=newvalue. For example,if I wanted to change the bias to +/- 6 V I would type the following: dict['bias']['detector 1']= {u'side 2': -6.0, u'side 1': 6.0}.

4) Once dict has been changed as you like, set your database entry equal to dict by typing

db[u'07151112_1653']=dict

If you print db[u'07151112_1653'] now, you will see that the changes have been made and the revision has increased.

This is more for my own benefit than anyone else's, but to look at the dumps for a given processing paradigm (for example, the standard paradigm, referred to as 'vanilla') you can use the following function:

proc_list=sd.readProcListFromDatabase(u'vanilla',server,db)

Then typing print proc_list will return the list of all entries for the paradigm 'vanilla'

And typing print proc_list[0] will return the first value from the list of all entries.

It will have the output [u'07150827_1512', 1, [1, 2, 3, 4], [1, 2, 3, 4], 0] where the first value is the series, the second is a noise processing boolean (1=noise processed, 0= noise not processed), the third entry is a list of all dumps in the raw series directory, the fourth entry is a list of all dumps that have been processed with the vanilla paradigm, and the final entry is a boolean for whether the data needs to be reprocessed (1=yes, 0=no).

1) Type exit() to exit python. That's it, you're done.

There is more that could be added to this, but this is a good jumping off point. Let me know if you have questions, or if I got anything wrong.

The autoprocessing script auto_process.py, is responsible for updating the database with current raw and processed data as well as submitting condor processing jobs in an organized fashion. It was designed to be versatile, dealing appropriately with MIDAS and Labview data, and allowing the user to indicate which processing type to use. The code works using the following steps.

1. Using the date range provided, script checks raw data directory and compares raw data to data listed in the database and/or runfile.
2. Selects portion of data that is unprocessed to be processed.
3. Updates database and/or runfile with any new raw data, as well as the data that is to be processed.
4. Send processing jobs to Condor.

Several tags can be called with the auto_process.py, listed in the table below.

The auto_process.py code is most useful when run periodically using the cron utility. To edit your list of cron jobs, type crontab -e. There is plenty of info online about the details of running cron jobs, but an example of how to run the code every 5 mins is shown below. Every 5 mins, cron executes these commands in a shell script. The first two commands source the user's bash settings and move to a directory where manual processing is usually done (i.e. the place where environment files are kept.) Then the auto_process command is called. The auto_process is called in runfile mode as opposed to database mode, so a local runfile is specified. Each command is ended with, “ >/dev/null 2>&1” which prevents all the command line output and errors from being piped to the user's email.

<blockquote>*/5 * * * * source /home/mast/.bashrc >/dev/null 2>&1; cd /home/mast/cdms/processing/; python /home/mast/cdms/K100_Data_Processing/k100Pipe/autoProcPy/auto_process.py 171207-* –runFile autoProcRunFiles/run066_runFile.txt –procenv environments/pymidas_HV_HV1nm.env >/dev/null 2>&1</blockquote>