*This page is no longer relevant to BaBar users as they can no longer access the RAL Tier A*
Information regarding the RAL xrootd setup and book keeping can be found here
BaBar RAL Tier A
- RAL CSF Linux
- First Time BaBar Users
- Getting a RAL Account
- Changing your Password
- Setting up the BaBar user
- Your Home Area and Quotas
- Tier A
- CM2 data at RAL
- PID and tracking tables
- Job Submission
- Job Status
- Grid job submission
- Book Keeping Tools
- Mirror of the BaBar website
- Archiving your data
- Old documents
RAL CSF Linux
The Linux farms are managed by the RAL e-Science Centre. BaBar users are no longer able to access the Tier A farm. The farm is now restricted to Simulation Production (SP). Users wanting to run jobs or perform data analysis should contact to their AWG convenor who will inform you where to log on to.
For SP, babar machines may be accessed by Secure Shell (ssh) (telnet access has been turned off). Full documentation can be found in the CSF User Guide. It should be noted that the CSF User Guide was not written with BaBar in mind. In the case of minor difference between the documentation and BaBar procedures, information in these pages takes precedence.
To get an account for the Linux machines (both farms), read what steps need to be taken on obtaining a RAL account and fill in the online web form.
First time users should follow the following steps to get going at RAL:
Getting an Account
To obtain a RAL account please follow the instructions here
At RAL your AFS and computer passwords (CSF) are independent. To change your AFS password type
To change your CSF password (password required to log into machines) type
Setting up the BaBar user
Your home directory is
/home/csf/username on NFS.
Most users won't have to do any special setup.
If your RAL CSF account is registered for a different experiment
bfactory is not your default group
(eg. you are working on both BaBar and CMS), then you should either
bfactory as your default the HEPiX group :
csf> mkdir ~/.hepix
csf> echo "bfactory" > ~/.hepix/preferred-group
- Add the following to ~/.login :
if ( -r /usr/local/lib/hepix/central_login.csh ) then
- Add the following to ~/.cshrc :
if ( -r /usr/local/lib/hepix/central_env.csh ) then
- Logout and log back it. You should now be able to access the BaBar environment.
Your Home Area and Quotas
Unlike SLAC your home area is on NFS disk as opposed to AFS. Therefore you do not require an AFS token to write to this disk. The top of your home directory has the path:
To keep track of the disk space you are using in your home area you can use the command "quota". For example:
Disk quotas for user olaiya (uid 27527):
Filesystem blocks quota limit grace files quota limit grace
289444 650000 700000 8607 0 0
46108 50000 150000 1 0 0
Where the usage, quota and limit numbers have units of kBytes
We have finished the process of expanding the Linux farm which has access to a total 75 TB of disk
space for BaBar usage. Most of this disk is being used to store CM2 data whilst roughly 3 TB is being used for user and AWG scratch disk. Regarding CPU at present, there are approximately 452 dual processor machines running Scientific Linux 3 (SL3). Of these processors, 284 are 1.4 GHz PIIIs, 8 are 2.66 GHz Xeons and 160 are 2.8 GHz Xeons. The number of CPUs allocated to the farm may vary very slightly from time to time.
Their setup is as follows:
- Access to the front ends of these machines can be achieved by Secure Shell (ssh) login to
babar.rl.ac.uk (also known as
- To transfer data off site please use the machine
- Data transfer rates are quicker on this machine. There is more infomation on this page regarding methods for transferring data off site
These are Scientific Linux 3 machine for debugging and the
submission of jobs. Please DO NOT log into
csfc.rl.ac.uk as these are not BaBar machines and consequently may not be setup correctly.
- Submit your jobs using bbrbsub and they will run on the Scientific Linux 3 worker machines.
- Four disks have been brought on line (
/stage/xrootd-data4-7/). One to serve as a working area (
/stage/xrootd-data6/) and the others are for SP.
BaBar users should be aware that the following points apply to disk and data under the /stage/xrootd-dataX
1. The AWG volumes are for business use only
2. The AWG volumes provide a business function
3. There should be no expectation of privacy
Users are able to access their work area in a similar fashion to SLAC, via the path
$BFROOT/work/o/olaiya/ for example. This is linked to an NFS disk, so an AFS token is not required. Furthermore it should be noted that neither of these disks are backed up, nor is there any plan to do so. People should start to use their
$BFROOT/work/u/user/ area for test release binaries, as they do at SLAC.
newrel -s $BFROOT/work/u/user -t analysis-24 Beta_a24
People should be considerate with their use of disk space in this area. Using large amounts of data will fill up the disk and cause other people's jobs to crash.
- AFS token passing via ssh has been enabled on these machines so if you login from a computer that already has an AFS token (SLAC, CERN etc.), that same token will also be available at RAL (see below).
AWG Disk Adminsitration
At RAL there is a mirror of the CVS repository for all BaBar
code at SLAC. The clone is updated nightly (actually at about 1 am UK time) and
so the latest versions, as well as all previous releases, should always be
available. All the major programs maintained at SLAC are copied to RAL.
- Information for AWG conveners and RAL BaBar support on managing the AWG disks can be found here:
Important: So updates to BaBar packages are not scattered between RAL and SLAC, updates are not allowed in the RAL CVS repository. It is locked for writing and you should see warnings if you try to access the RAL repository. Updates must be made directly to SLAC CVS were they will be copied to RAL the following day. To check packages into or update directly from the SLAC repository you can use the command "bcvs". You must have a SLAC AFS token for this to work. Examples of bcvs commands in your release directory:
- klog firstname.lastname@example.org or kslac (get SLAC AFS token)
- bcvs update Q2BUser (update package Q2BUser from SLAC)
- bcvs add mypackage (add your new package to the SLAC repository)
- bcvs ci mypackage (commit your new package to the SLAC repository)
- bcvs (list all the possible bcvs commands)
CM2 data at RAL
Information on the latest releases can be found
here or look under $BFROOT/dist/releases.
Pid and Tracking Tables
The PID and tracking tables are copied over every night. However you should be using the database access method.
You can access your SLAC home directory from RAL via AFS,
however you will need a SLAC AFS token.
If you login from a computer that already has an AFS token
and use a version of ssh that supports AFS token passing, then that
same token will also be available at RAL.
SLAC, CERN, and many other HEP sites have this version - you can check
If the -k option is listed then your ssh supports token passing.
Otherwise, you can obtain a SLAC token with
or the shortcut,
csfe ~ > kslac
csfe ~ > ls /afs/slac.stanford.edu/u/ec/adye
You can also use your RAL AFS area so your RAL files are easily accessible
from elsewhere. Your AFS directory is of the form
/afs/rl.ac.uk/user/u/username (eg. /afs/rl.ac.uk/user/a/adye).
Note that when you log in you do not obtain an AFS token automatically,
so you should use
klog before you try to write there
(by default, read access is universal).
RAL uses PBS (qsub) as opposed to LSF (bsub) at SLAC. However at RAL, we try to minimise this difference for BaBar users with the aid of the command
bbrbsub. Jobs can be submitted to the farms from a SL3 frontend by specifying the
prod PBS batch queue (this feeds into the
The "-c hh:mm" option specfies the same amount of CPU time (cput) and per-process CPU time (pcput) requested for the job (the cput and pcput requested are required to be the same for a job). This "-c" option has the advantage over the equally valid qsub option "-l cput=hh:mm:ss,pcput=hh:mm:ss" of being simpler and the same as the SLAC bsub option. You will notice that with the "-c" option you cannot specify the amount of seconds requested, which is no significant loss as CPU requests at the level of minutes is a more than sufficient precison.
bbrbsub -q prod -c hh:mm file.job
Also, as the submission queue
prod is the default for Linux, omitting the -q option works equally well. Therefore, one can simply submit a job as follows
We also have an SL4 express queue for testing one or two short jobs. One can submit jobs to the SL4 express queue as follows:
bbrbsub -c hh:mm file.job
bbrbsub -q express file.job
bbrbsub is a wrapper script for the PBS
qsub command. We recommend Babar users use
bbrbsub as opposed to
qsub as it takes care of a lot of the differences between
bbrbsub has the following properties:
1. Makes the batch job execute in directory where execution starts
2. Transfers environment variables (with no failure due to character limits)
3. Keeps stdout and stderr on batch node until job ends
4. Enforces use of the
prod feeder queue which directs jobs to the
5. Provides a job name of binary/script but limited to 15 letters
6. Allows multiple arguments to be passed to the script on the command line (e.g.
bbrbsub -c hh:mm file.job arg1 arg2 arg3 arg4)
bbrbsub uses the same PBS flags as
qsub and as with qsub AFS tokens are not passed with the job. However at RAL this is not such a problem as you can use your NFS mounted disk, where an AFS token is not required.It should be noted that bbrbsub does not recognise PBS directives like qsub (i.e commands such as '#PBS -j oe' in the job file). See man page of qsub for details of qsub options that bbrbsub passes straight through.
This is a simple script that functions just as
qdel but also offers the additonal feature of deleting all your jobs in the queue.
bbrbkill (jobnumber) (same as qdel)
bbrbkill 0 (deletes all your jobs)
Batch queue scheduling
Details of the batch job scheduling are
One additional control (not currently documented there, 24/Jan/06) is the
maximum CPU time you specify on your job with the "-l cput=<time>" option.
The maximum amount of time a job can run on a batch machine is 48 realtime hours. This information can be obtained with the command:
qmgr -c "l q sl3p"
Your jobs may start earlier and you might be
able to run more at the same time if you make a good estimate of the CPU
time they will take (how much of an effect this will be depends on the mix of
other jobs in the system). In the long run you will, however, not be penalised
by using an estimate that is too long. Also take care not to make the
CPU time estimate too tight as a job running above the limit will get
To estimate the CPU time a job requires you can run a test job (with a
high limit) and look at the stdout at the end of the batch job. Remember
to add a safety margin as the job will be killed prematurely if it runs
for too long.
The batch machines at RAL have local disk available for you to write the output of your jobs. The disk is accessible via the $WORKDIR variable on the batch machines. The advantage of using this area is that it is usually much faster to write ntuples etc to the local disk and then copy the final file to /stage/babar-user1 at the end of the job. Not only does it speed up the I/O of your jobs but it also reduces the NFS load on the user and AWG disks. More information on the $WORKDIR variable for the batch machines can be found here
To see what jobs are running on the farm you can use the command
To check the status of your jobs you can run
To see the number of jobs running in the different queues at RAL type
Keep in mind that BaBar user will have only running jobs in the sl3p queue.
To look at the output of jobs that are currently running you can use the
Grid job submission
The RAL Tier A will accept LCG jobs from users with an LCG certificate in the BaBar Virtual Organisation (among others).
See the BaBar Grid
Registration page for details of obtaining a certificate, registering it with LCG, and adding it to the BaBar VO.
Grid submission for BaBar analysis jobs is still experimental, but is being worked on.
Book Keeping Tools
The book keeping tools used at SLAC are also available at RAL. To access the relevant database for the RAL site the flag --dbsite=ral must be specified. For example to see what skims are at RAL you can run
To produce onpeak 4 body tcl files each identifying 200000 events you can run the command
- BbkDatasetTcl --dbsite=ral
- BbkDatasetTcl --dbsite=ral BFourBody-OnPeak-R14-BlackDiamond -t 200000 --splitruns
Mirror of BaBar website
An up to date mirror the BaBar website can
be found at RAL here
Archiving your data
If you no longer need your data on the home filesystem (
/home/csf), user data disk
/stage/babar-user1), or AWG data disk (
please consider freeing up the space by deleting it -
perhaps after copying it back to your desktop machine or local institution, or
archiving it to tape at RAL.
All questions BaBar questions specific to the RAL Tier 1A center should be posted on the hypernews:
RAL Tier A Centre and UK Farms.
If these do not get a response then contact
Tim Adye or
http://hepunx.rl.ac.uk/BFROOT/csflnx.html last modified
5th Jan 2009 by
Emmanuel Olaiya, <E.Olaiya@rl.ac.uk>
Tim Adye, <T.J.Adye@rl.ac.uk>