Usage: para-trickle-make [options] -help para-trickle-make [options] -manyids [make-options] < foo.ids para-trickle-make [options] -quick 'quick-command > quick.log' This program launches one or more jobs under using parasol, as described below. Base Options (you must choose one): -help : print usage -manyids : create joblist for foo.ids using make-options -quick : create joblist with only the single given command Options for all formats: -cluster : the target cluster on which parasol is to be invoked must be one of {condor | farm | kilo | pita | moai} or unique abbreviation DEFAULT: farm -max_jobs : maximum number of jobs allowed on the parasol queue DEFAULT: (25 if -cluster condor. else no limit) -runmins : if specified, the maximum minutes of cpu runtime for each job. Note: 1440 = 1 day. DEFAULT: -check : number of seconds to sleep between attempts to push more jobs to the parasol queue. DEFAULT: 250 -rmvdir : if specified, (and if no jobs crash) the parasol job-directory that is created will be removed after all jobs have finished. -createonly: if specified, this program will exit without pushing any jobs after building the joblist and executing para create, allowing you to manually execute para push, para try, etc. -crashstop: if specified, this program will check for crashed jobs after each push of jobs (as controlled by -max_jobs). As soon as a crashed job is detected, it will issue para stop to remove any queued jobs and will exit. if (and only if) -wait is also specified, the checking will continue until all jobs have run or the first crashed job is detected. -wait : if specified, this program will not complete until all pushed jobs have completed (unless crashstop is also specified, in which case this program will exit at detection of the first crashed job). -verbose : add some informative messages -terse : suppress all except error messages Make-Options for -manyids format: -makefile : name of the makefile that is to be executed for each of the entries in foo.ids DEFAULT: /projects/compbio/experiments/models.97/Makefile.models97 -targets : list of makefile targets to be used with the makefile. Note: either quote the list OR use the option repeatedly: -targets 'foo baz zom' -targets foo -targets baz -targets zom DEFAULT: -modelsdir : directory under which, for each of the foo.ids, a separate subdirectory is to be found (or created if necessary) DEFAULT: /projects/compbio/experiments/models.97/pdb -use_subdir/ Sets use-subdir to 1 -nouse_subdir: Sets use-subdir to 0 If use_subdir=1, then under the modelsdir, the foo.ids subdirectories should be in two levels, with the first level consisting of the some letters of the id. Two other variables control which letters: subdir_skip and subdir_length. The first subdir_skip letters of the id are skipped, and the next subdir_length letters are used. if use_subdir=0, the subdirectories are simply the ids (e.g. 1a73A, 2ct4B) DEFAULT: 1 -subdir_skip : How many letters of id are to be skipped to construct subdirectory name when use_subdir is set. Specifying subdir_skip implies use_subdir. DEFAULT: 0 -subdir_length : How many letters of id are to be used (after skipping subdir_skip letters) to construct subdirectory name when use_subdir is set. Specifying subdir_length implies use_subdir. DEFAULT: 2 -2letter/ : Special case of -use_subdir, implying subdir_skip=0 and subdir_length=2. -no2letter : Equivalent to -nouse_subdir if 2letter is set, then under the modelsdir, the foo.ids subdirectories should be in two levels, with the first level consisting of the first 2 letters of the id. (e.g. 1a/1a73A, 2c/2ct4B etc.) -se2log : if specified, STDERR will be redirected to the logfile. In that case, the 'para problems' command will only retrieve a message pointing you to the logfile. You may want to use this option if your makefile does not redirect STDERR from the programs it calls to their own logfiles. This program creates a job-directory, a joblist, and a shell script (to be called by the joblist entries). It then invokes parasol to run the joblist until all jobs have been pushed onto the queue. The job-directory is based on the user and process id, and is found in a common parent directory (dependent on the cluster). It has the form: -cluster farm: /projects/compbio/tmp/para-trickle-make-farm-uid-procid-random -cluster condor: /projects/compbio/tmp/para-trickle-make-condor-uid-procid-random -cluster kilo: /cluster/store9/tmp/para-trickle-make-kilo-uid-procid-random -cluster pita: /cluster/store9/tmp/para-trickle-make-pita-uid-procid-random -cluster moai: /projects/compbio/tmp/para-trickle-make-moai-uid-procid-random If -quick is specified, the joblist has a single entry and the shell script contains the quick-command. If -manyids is specified, 1) ids are read from standard in and the parasol joblist is created with a line for each id: para.exec.csh id-directory id 2) the shell script 'para.exec.csh' is created to run in each id-directory from the joblist, placing the output of the makefile for each id in the common job-directory: cd id-directory; make -f makefile -k targets > job-directory/id.log WARNING: This program calls the routines in para_cluster_jobs.pm, which use ssh. To avoid hangs waiting for login passwords, you must do one of the following: Establish passphraseless ssh access to the machines in the desired cluster. This can be done with keys and a config file in your .ssh directory. (deprecated alternative) You should have a .shosts or .rhosts file that includes the name of the computer you are submitting from (for example, "bray.cse.ucsc.edu karplus") NOTE regarding STDERR and 'para problems': Unless -se2log is specified, the -manyids option does not redirect STDERR from the makefile to the id.log, only STDOUT. This allows parasol to get STDERR and report it using 'para problems'. The same practice can be followed when using the -quick option. NOTE on manually checking parasol progress: A README file is created in the job-directory which has more information on how to check the progress of the jobs that have been submitted to parasol.