What is refrepp?

FRE post-processing is time- and I/O-intensive, and susceptible to intermittent and unforeseeable failures (I/O timeouts, etc...) refrepp is a script that allows you to find such failures (causing "holes" in the archived output) and submit the smallest possible sequence of frepp jobs to repair them.

What refrepp does

The current frepp runs years in sequence... the 1981 job submits 1982 and so on. If you hit a "chunk" boundary, that job is longer, as it does the chunk stuff.

Working with the example of a 5-year chunk, and let's assume 1983 failed but 1981, 82, 84 and 85 are available.

The most efficient implementation of refrepp would detect the 1983 hole, resubmit a frepp -t 1983 (which by default is one year, then submit frepp -t 1985 to complete the chunk. (This will repeat the one-year processing for 1985 but that can't be helped, at least in the current frepp).

1983 should finish running before you launch 1985... this is done by using qsub -hold_jid, which declares a dependency between one job and another.

If 1981 and 1983 are both missing, they can both be launched concurrently.

The refrepp script does the following:

• detects the frepp configuration, currently by parsing the output from freppcheck.
• detects the holes: by looking for files matching the expected filenames in the archive directory.
• if requested, submits a sequence of frepp jobs to replace all the missing files.

Syntax:

Usage: refrepp [options] exp [exp ...]
  Options:
      -x|--xmlfile xmlFile (default XML file \$cwd/rts.xml)
      -c|--component <c>   (process only component <c>, default all)
      -d|--dir <dir>       (where scripts are stored, default \$rootDir/scripts/postProcess)
      -b|--begin <year>    (year to begin processing: should line up with chunk boundaries!),
      -e|--end   <year>    (year to end processing),
      -s|--submit          submit the script as a batch job.
      -h|--help            print help message.
      -v|--verbose         verbose messages.
  At least one experiment from the xmlFile must be listed as an argument.

Notes

• refrepp currently calls freppcheck and frepp, they must be in your \$path. Add /home/fms/bin at the end of your \$path if needed.
• refrepp assumes that chunks line up at the first year being processed (see the -b flag). Please set this carefully. For instance, if your run begins in 1860 and post-processing uses 5, 20 and 100 year chunks, refrepp -b can take values of 1860, 1960, ... Any other value will provide incorrect results.
• refrepp -s might submit a lot of jobs. A wise user will call refrepp first without -s, see how many jobs refrepp is threatening to submit, and then proceed.

Download

Use the version in /home/fms/bin or, if you're one of the ones who downloads your own copy of FRE (FRE-loader?) you can check out the version with the tag fre_20080701.