To copy data to SLAC you will need a SLAC Unix account.

1. register as SLAC user:
   https://oraweb4.slac.stanford.edu/apex/epnprod/f?p=134:1
2. once you get a SLAC ID (SID) you can get a SLAC computing account (I guess Unix is enough, you don't need Windows). Go to:
   http://www2.slac.stanford.edu/comp/slacwide/account/account.html
   read 1) and 4)
   print, fill, and sign 2) and 3)
   scan the filled forms at 2 and 3 and send them to Tina Cartaro, cartaro@slac.stanford.edu.
3. After you get the account you have 30 days to complete the CS101, Cyber Security 101 training from
   http://training.slac.stanford.edu/web-training.asp
   (with windows password)
   or, if you choose not to have a windows passord:
   https://slactraining.skillport.com/skillportfe/login.action
   where the SID is the number assigned to you when you first register as a user and the password is [redacted]

On the SLAC cluster our raw data is located at

/nfs/slac/g/supercdms/tf/umn/

If you have a SLAC user account, one of the servers you can use to access the cluster is

rhel6-64.slac.stanford.edu

Update 2024:

centos7.slac.stanford.edu

use

rsync -avz

to copy raw data directories to SLAC

example:

rsync -avz 07210727_1333 07210728_1139 07210728_1347 07210728_1625 07210729_1431 07210802_1113 07210803_1134 07210803_1141 07210803_1151 07210803_1156 07210804_1419 07210805_1058 07210806_1250 07210807_1702 07210807_1710 07210809_1032 07210809_1049 07210809_1615 07210810_1100 07210811_1036  rhel6-64.slac.stanford.edu:/nfs/slac/g/supercdms/tf/umn/raw/

At SLAC there is a data catalog that we can register our series in. So far this has not been implemented, but this section gives some notes about how to do this.

To set up the code to interact with the data catalog through Python, follow these instructions on the command line. You will need an account on Nero at Stanford to access the code repository.

 pip install request --user
 svn co svn://svn.slac.stanford.edu/srs/org-srs-datacat/branches/0.4/client/python
 cd python
 python setup.py install --user
 cd ..
 git clone ssh://scdms@nero.stanford.edu:26/data/git/DataHandling/DataCat.git
 cd DataCat
 python setup.py install --user

Here's a link to an online version of the catalog: http://srs.slac.stanford.edu/DataCatalog/?experiment=CDMS

Here's a link with instructions and info about interacting with the catalog: https://confluence.slac.stanford.edu/display/CDMS/SuperCDMS+Data+Catalog

Depending on how you want to setup your copying routine, what might make sense is to create a python script which can determine all the relevant metadata from your file, and call that on each file from the copying step. You should do the registration probably both for where it is at UMN (if you’re also keeping it) and where you put it at slac, and you should try to do the SLAC-side registration after the file is copied. Something like this would work as a bash script:

 files=$@ #all files passed to copy script
 newpath=/path/to/slac/location
 for file in $files
 do
    rsync $file user@rhel6-64.slac.stanford.edu:$newpath/$file
    registerDataset.py $newpath/$file $implicitly tells the catalog that the file is at slac, and determines metadata
 done

Hi Matt,

I’ve added an AddTFData.py script and the related datacat object to the package, if you run git pull and re-run the setup script it should be all set and you can add the query commands where indicated in the script. I’ve set it up so it commits datasets passed to the script as arguments so you can embed it in your copy script. Let me know if you need help getting this working.

Best, Noah

A Google Shared Drive called “UMN CDMS” was created on Matt Fritts' UMN Google account. It is currently shared with Vuk Mandic and Nick Mast.

The software “rclone” is installed on cdms1, cdms2, and cdms3.

First run “rclone config” to create a remote path to your UMN Google Drive account. For “name” choose whatever you like, such as “google-storage”. For “Storage” select the “Google Drive” option (probably this will be Option #12). For “Scope” select the “full access” option #1. For everything else that asks for a value just leave it blank. When asked “Use auto config?” choose “yes”. This will open a browser to a Google sign-in - sign in with your UMN ID then allow rclone access. Continuing with the config, when asked “Configure this as a team drive?” choose “yes” and you should see a list of your shared drives: choose “UMN CDMS”.

Here's a log of my configuration:

rclone_config_example

After config is finished, you can do things like this:

rclone lsd google-storage:/

The above command is a top-level list of the shared Google drive. To copy a series of raw data to the Google drive, you can do for example

rclone copy /data/chocula/k100/midasraw/byseries/07151209_1516 google-storage:/07151209_1516

Note that in the above you want to copy the directory name at the end, so that a new directory will be created in the Google drive, otherwise it will just copy the files into the main directory.

Similarly you can copy a series of raw data back to your current local directory like this:

rclone copy google-storage:/07151209_1516 ./07151209_1516

Here's an example of listing the contents of the Google drive:

rclone lsd google-storage:/