Tutorial - Running a spatial domain simulation

This document contains a tutorial and training exercise for getting started using the VIC model across a spatial domain and making use of the ITaP Research Computing cluster system resources. It assumes that you have already completed the basic tutorial: Tutorial - Running a time series simulation, and that you have a working knowledge of Linux/Unix, and that you have access to a Purdue Linux cluster.

If you are a member of PHIG then you can request access to the conte cluster through Dr. Keith A, Cherkauer.
If you are a member of another research group with access to conte or any other cluster, you can request access through your advisor.
If you are enrolled in ABE 65100 Environmental Informatics you should have access to the scholar cluster through the class.

Step 1: Setting up the model for the simulation tutorial

Log onto the correct Linux cluster system (scholar for class, brown or radon for research) using your Purdue career account and password.
Make a new directory (if necessary) called "VICtutorial".
Change into the new directory.
While in the VICtutorial directory download the tar file using the wget command: wget ftp://ftp.ecn.purdue.edu/hydrodat/TutorialFiles/VicModelSpatialTutorial.tgz.
Untar and uncompress the file using tar (tar -xvzf VicModelSpatialTutorial.tgz)
This will create and populate a directory called "SpatialTutorial", when the tar is complete, change into the new directory for the rest of the tutorial.

Step 2: Compiling the model

Change to the VIC_4.1.2.k.Purdue sub-directory that was created within the SpatialTutorial directory.
Use the command make full to compile all components of the VIC model required for this tutorial. If make full does not work; module load gcc.
Before leaving this directory, use the make clean command to remove all of the ".o" object files that are no longer needed.

Step 3: Setting up and running the model to run using multiple cluster nodes

Before completing these steps, it is recommended that you make a working copy of the tutorial files in your scratch drive. Please refer to Step 4: Setting up your Workspace in Tutorial - Running a time series simulation, if you need help doing this. Copying files to and starting simulations from the scratch drive will make sure that you do not overwrite critical files in the source directories, and give you access to a very large disk system (~100 TB available depending on current use) that is optimized for use with the high-speed network of the computer cluster. Your home drive storage is much more limited, and the data depot is not designed for heavy processing loads.

Change into the scratch drive directory from which you will be running the simulations.

Create the output directory "Results"

{prompt} mkdir Results
{prompt} ls
FORCING_DATA/ GLOBAL_FILE/ MODEL_PARAMS/ Results/ VIC_4.1.2.k.Purdue/

You are now prepared to run the script vic_SLURM_setup.csh that will work out how to split the larger spatial simulation across multiple compute nodes for parallel evaluation.
```
{prompt} vic_SLURM_setup.csh
Usage: vic_SLURM_setup.csh jobdir global CPUs [option flags]
```
The default queue (phig) has 1 computer node with 128 cores each available, the VIC model will will be split over all available cores, while extra jobs will wait on the queue until a core is available.

Warning

Do not include a '/' after jobdir, and do not use the current directory, as jobdir is cleared during this process.

Options flags include:

Option Flag	Description
`-L #`	Number of layers (3 is default)
`-X <name>`	Specify the executable version of the VIC model to use for simulations (default vicNl)
`-Q <name>`	Specify the SLURM queue name to use for model simulations (default phig)
`-W <hh:mm:ss>`	Specify the walltime to use for model simulations (default 12:00:00)
`-S`	Spatial Snow is being used
`-F`	Spatial Frost is being used
`-E`	Excess Ice is being used
`-J`	July Average Temperature is being used
`-C`	The VIC-CropSyst model is being used

Note

If you type the command at the prompt and get the response vic_SLURM_setup.csh: Command not found. then your environment is not setup to access shared programs stored on /depot/phig/apps. Talk with Dr. Cherkauer for help in getting things setup.

To run the program you need to provide:
1. jobdir - this should be a separate directory that will contain only the simulation submission scripts and the SLURM log files that will capture the stdout and stderr streams from each submitted job. The script will create the directory if it does not already exist, otherwise the contents of the directory will be deleted. So DO NOT USE an existing directory with critical information (such as the model parameter directory), as it will be cleared each time you run this script.
2. global - this is the global control file that would control the simulation if you were to run it all on a single CPU. The process of splitting a single run across several compute nodes will create multiple copies of the global file with the extension .XXXX, where XXXX represents the simulation set number 1 to CPUs.
3. CPUs - the number of compute nodes over which you will split the simulation. If using the cherkaue queue to run on brown, then the maximum number of CPUs available is 3 nodes times 16 CPUs each = 48 CPUs. If using the scholar queue on the scholar cluster, then you have access to 16 nodes with 20 CPUs each, so 320 CPUs. If you request fewer CPUs than are available, you are leaving CPUs for other users (sharing is good, especially for class). If you request more CPUs than are available then your extra jobs will sit on the queue until CPUs come available, with the queue directing traffic.
  
  Note
  
  The SLURM queue system operates with a complex system of rules, but fundamentally is a first-in, first-out queue, so that the first job onto the queue will be first off of the queue. It differs because: (a) some queues have higher priority. The phig queue is paid for using project money, so is higher priority than the debug or standby queues, which are available to all users, so jobs submitted to the phig queue will move to the front of the line; (b) jobs that require multiple nodes may allow smaller requests (fewer nodes, shorter walltimes) past while the SLURM queue clears the resources requested of the larger job, so sometimes a job asking for a single node or willing to share CPUs on a single node will bypass a larger job waiting for all of its resources to come on-line at once. Use the myqueue command to see what queues are available to you on the current cluster system.
4. executable - optional parameter that sets the location of the vic model executable to use for these simulations. You will use this for the tutorial to force the selection of the vic model version you compiled in Step 2.
5. option flags - these warn the script about model options that will add to the length of the simulation time. The setup script tries to estimate the walltime for each simulation set based on the number of grid cells and the extra options selected, the splitting into simulation sets also takes into account factors such as the number of vegetation types per grid cell so that it sets up model simulation sets that will run in about the same time. This is a work in progress, so adjustment of final walltime in your scripts may be required to get jobs to finish.
  
  Note
  
  The walltime is the time you request from the queue system to be on a remote node. It is used by the queue system to schedule job requests. Short jobs may be moved in front of large jobs requiring lots of CPUs that are waiting on other longer jobs to finish. This keeps smaller jobs moving through the queue, while the system clears resources for the larger requests. When requesting walltime for your submissions, you want to be as close as possible to the actual time needed to complete the job so taht it can be effectively scheduled. However, if you underestimate the time required, your job will be killed when walltime is reached.

Run the vic_SLURM_setup.csh script with the following options.

{prompt} vic_SLURM_setup.csh TestSims GLOBAL_FILE/FullRedRiver_ponding_global 25 -X VIC_4.1.2.k.Purdue/vicNl
{prompt} ls TestSims/
vicsend.csh.0001* vicsend.csh.0007* vicsend.csh.0013* vicsend.csh.0019* vicsend.csh.0025*
vicsend.csh.0002* vicsend.csh.0008* vicsend.csh.0014* vicsend.csh.0020* vicsend.csh.0026*
vicsend.csh.0003* vicsend.csh.0009* vicsend.csh.0015* vicsend.csh.0021*
vicsend.csh.0004* vicsend.csh.0010* vicsend.csh.0016* vicsend.csh.0022*
vicsend.csh.0005* vicsend.csh.0011* vicsend.csh.0017* vicsend.csh.0023*
vicsend.csh.0006* vicsend.csh.0012* vicsend.csh.0018* vicsend.csh.0024*

The script has now created 26 submission scripts, the first 25 have 52 cells each, while the 26th script will run the remaining 10 cells. Now view the contents of any of these files:

{prompt} cat TestSims/vicsend.csh.0001
#!/bin/sh -l

# FILENAME:
/scratch/brown/c/cherkaue/SpatialTutorial/TestSims/vicsend.csh.0001

#SBATCH -A phig
#SBATCH -N 1 -n 1
#SBATCH --exclusive=user
#SBATCH -t 02:08:00
#SBATCH --job-name=TestSims.0001
#SBATCH -e TestSims
#SBATCH -o TestSims

# Change to the directory from which this script was originally called.
cd $SLURM_SUBMIT_DIR

# Start VIC model simulation.
VIC_4.1.2.k.Purdue/vicNl -g
GLOBAL_FILE/FullRedRiver_ponding_global.0001

You can see that the files are scripts and include multiple commands for the SLURM queue embedded in the file header (lines that start with #SBATCH):
1. FILENAME - declares the name of the current script so that messages include the actual script name
2. -A phig - sets the queue to be used to phig.
3. -N 1 -n 1 - limits requested resources to a single cpu (ppn) on a single node (nodes).
4. --exclusive=user - turns on the sharing policy so that other requests to the queue for a single CPU can be sent to the same node, otherwise the request for a single CPU on a single node will still lock up the entire node (leaving the other CPUs unavailable until the job ends).
5. -t 02:08:00 - requested walltime in hh:mm:ss, mm:ss, or ss. Note that hh and hh:mm are not available, minutes and seconds must be present to define hours.
6. --job-name=TestSims.0001 - provides a job name to the queue system, keeping it short makes it possible to read when checking queue status.
7. -e and -o - define the output directory used for the stderr (-e) and stdout (-o) log files. Here they will be stored in the same directory as the submission script to reduce clutter in the parent directory.
The first command cd $SLURM_SUBMIT_DIR changes the directory to the location from which the script was submitted. When scripts are started by SLURM queue, they log in just like you do, so they start in your home directory. The SLURM_SUBMIT_DIR environment variable remembers where you were when you started the script, giving you a shortcut command back to where you want execution to happen. Note that is you try running this script on a front-end system rather than submit it via SLURM to the compute node, the environment variable $SLURM_SUBMIT_DIR will not be set and you will get and error and/or the cd command will go to your home directory (e.g., "cd").
The final command in the script will start the model, using the executable you defined and a global file representing the current simulation set.

Now you are ready to submit all of the jobs you just created as follows:

{prompt} for file in TestSims/vicsend.csh.00??; do sbatch $file; done
4897672.brown-adm.rcac.purdue.edu
4897673.brown-adm.rcac.purdue.edu
4897674.brown-adm.rcac.purdue.edu
4897675.brown-adm.rcac.purdue.edu
4897676.brown-adm.rcac.purdue.edu
...
4897697.brown-adm.rcac.purdue.edu

Because the SLURM queue options are defined in the header of each submission script, all you have to do is send the script to the queue using the sbatch command. Here I used a foreach loop to submit all of the scripts at the same time. Each completion of the sbatch command results in the Job ID for the current submission being written to the screen, so in this case "4897672.brown-adm.rcac.purdue.edu". The numbers indicate the order in which jobs were submitted, and may not be sequential if someone else submits jobs while your are being submitted.

You can use the squeue command to check the status of your jobs:

[cherkaue@brown-fe03] $ squeue -A phig
JOBID USER ACCOUNT NAME NODES CPUS TIME_LIMIT ST TIME
11909167 cherkaue phig TestSims.0001 1 1 02:08:00 R 00:13:38
11906209 cherkaue phig TestSims.0002 1 1 02:08:00 R 00:13:32

This indicates that the first two jobs are "R"unning, and have been running for 00:13:38 and 00:13:32, respectively. Having given the system a short job name, I can see which jobs are currently running, which are still "PD" (pending) and which are no longer on the list. I can see that the walltime for all jobs was set to 02:08:00. How long jobs are queued depends on system usage. Once submitted to the queue system, jobs will remain until they get a chance to run, so you can log out and return later and the queue system will handle all submission processes. When squeue returns with no listed jobs, then your simulations have been completed.

Step 5: Checking model output

Of course, just because jobs are no longer on the queue does not mean that everything ran correctly, so here are a few things I check before moving on to the processing steps.

First, check that you have the right number of output files

{prompt} wc MODEL_PARAMS/SoilParams.wFS
1285 68105 656545 MODEL_PARAMS/SoilParams.wFS

{prompt} ls -1 Results/fluxes* | wc
1284 1284 41088

{prompt} ls -1 Results/fdepth* | wc
1284 1284 41088

{prompt} ls -1 Results/snow* | wc
1284 1284 38520

{prompt} ls -1 Results/lake* | wc
1284 1284 38520

At the end of the simulation there are 1,284 output files with the prefixes fluxes, fdepth, snow and lake. There are 1,285 lines in the soil file. If you do a head on the soil file, you will see that it has a header, so that means there should be 1,285 - 1 = 1,284 output files from this simulation. So the number of output files checks out!
Next do a long list (e.g., ls -l * | less or dir) on each set of files, scrolling quickly through them to make sure that they are all about the same size. If you use the binary output file format, then all files must be of exactly the same length, so you can use a script to do an automated check of file sizes. If you have zipped the output files, do not look for similarities in file size, since file size reflects compression efficiency not the brownnts of the files.

If you are finding missing files, or files that are not complete, you will want to check the log files for the submitted scripts. These will be in the TestSims directory, with names like slurm-11909167.out. Where the final 8 digit number is the SLURM queue Job ID (refer to the output from sbatch and squeue).

The file may contain "Warning: no access to tty (Bad file descriptor)." and "Thus no job control in this shell.", This warning message just indicates that the script was not run in an interactive shell environment, which is what you wanted.

More importantly the file will have most of the output from the VIC model, plus any errors thrown by the SLURM queue.

Look at the end of the file first, to see if there are messages related to walltime being exceeded. If this was the case, then the setup script underestimated the submission walltime and you will need to edit the submission scripts and try again.

Note

Editing the multiple files produced by this method can be a headache using an interactive editor, but can be done much more easily using the sed command line editor or for more complex editing tasks using an awk script. Here is an example of changing the walltime from 02:08:00 to 04:00:00 using the sed editior:

{prompt} head TestSims/vicsend.csh.0001
#!/bin/sh -l
# FILENAME: /scratch/brown/c/cherkaue/SpatialTutorial/TestSims/vicsend.csh.0001
#SBATCH -A phig
#SBATCH -N 1 -n 1
#SBATCH --shared
#SBATCH -t 02:08:00
#SBATCH --job-name=TestSims.0001
#SBATCH -e TestSims
#SBATCH -o TestSims

# Change to the directory from which this script was originally called.

{prompt} for file in TestSims/vicsend.csh.00??; do sed -i.bck 's/#SBATCH -t 02:08:00/#SBATCH -t 04:00:00/' $file; done

{prompt} ls TestSims/
vicsend.csh.0001*     vicsend.csh.0006.bck* vicsend.csh.0012*     vicsend.csh.0017.bck* vicsend.csh.0023*
vicsend.csh.0001.bck* vicsend.csh.0007*     vicsend.csh.0012.bck* vicsend.csh.0018*     vicsend.csh.0023.bck*
vicsend.csh.0002*     vicsend.csh.0007.bck* vicsend.csh.0013*     vicsend.csh.0018.bck* vicsend.csh.0024*
vicsend.csh.0002.bck* vicsend.csh.0008*     vicsend.csh.0013.bck* vicsend.csh.0019*     vicsend.csh.0024.bck*
vicsend.csh.0003*     vicsend.csh.0008.bck* vicsend.csh.0014*     vicsend.csh.0019.bck* vicsend.csh.0025*
vicsend.csh.0003.bck* vicsend.csh.0009*     vicsend.csh.0014.bck* vicsend.csh.0020*     vicsend.csh.0025.bck*
vicsend.csh.0004*     vicsend.csh.0009.bck* vicsend.csh.0015*     vicsend.csh.0020.bck* vicsend.csh.0026*
vicsend.csh.0004.bck* vicsend.csh.0010*     vicsend.csh.0015.bck* vicsend.csh.0021*     vicsend.csh.0026.bck*
vicsend.csh.0005*     vicsend.csh.0010.bck* vicsend.csh.0016*     vicsend.csh.0021.bck* 
vicsend.csh.0005.bck* vicsend.csh.0011*     vicsend.csh.0016.bck* vicsend.csh.0022*
vicsend.csh.0006*     vicsend.csh.0011.bck* vicsend.csh.0017*     vicsend.csh.0022.bck*

{prompt} head TestSims/vicsend.csh.0001
#!/bin/sh -l
# FILENAME: /scratch/brown/c/cherkaue/SpatialTutorial/TestSims/vicsend.csh.0001
#SBATCH -A phig
#SBATCH -N 1 -n 1
#SBATCH --exclusive=user
#SBATCH -t 04:00:00
#SBATCH --job-name=TestSims.0001
#SBATCH -e TestSims
#SBATCH -o TestSims

# Change to the directory from which this script was originally called.

{prompt} rm TestSims/*.bck

The sed command "s" does a search for the string "#SBATCH -t 02:08:00" and replaces it with the string "#SBATCH -t 04:00:00". The "--i" flag option tells sed to edit the file in place rather than writing the edited version to stdout, while the ".bck" that follows the flag option tells sed to save the original file with the ".bck" extension before editing. I like this option, because I can always recover if something goes wrong with my sed command, but clean up is also easy if the sed command works as expected. The new submission scripts now have walltimes set to 4 hours, which will be reflected the next time they are submitted.

Look through the VIC model messages to see if there are any errors associated with missing files, or with run time errors. If there are runtime errors, I suggest requesting an interactive session on a node where you can run the script directly and diagnose the problem as with any other VIC model application.
```
{prompt} sinteractive -t 12:00:00 -A phig # this requests an entire
compute node for 12 hours of interactive use.
```
Note

Add a "-X" to the command if you need access to X-windows the the session. This shell will lock until the session starts at which point the prompt will change to indicate that you are no longer on a front-end node. The interactive session will last until you exit it, or your walltime is reached.
Once the problems have been resolved, any or all jobs can be resubmitted to the SLURM queue to complete the submission. As job log files include the Job ID, every resubmission will create a new set of log files, so it is up to you to determine if you want to keep them all or remove them as you finish with them (e.g., rm TestSims/TestSims.????.[eo]???????).

Step 6: Evaluating model output

Output files for this tutorial were created with file headers defining their contents. You can view the general contents of the file using the GetVicHeader tool, the output from a snow file is shown here because it includes fewer columns of data.

{prompt} GetVicHeader Results/snow_47.4375_-98.3125
Column 0: YEAR 0 0 (Average Value)
Column 1: MONTH 0 0 (Average Value)
Column 2: DAY 0 0 (Average Value)
Column 3: OUT_SWE 0 0 (End Value)
Column 4: OUT_SNOW_DEPTH 0 0 (End Value)
Column 5: OUT_SNOW_CANOPY 0 0 (End Value)
Column 6: OUT_ADVECTION 0 0 (Average Value)
Column 7: OUT_DELTACC 0 0 (Average Value)
Column 8: OUT_SNOW_FLUX 0 0 (Average Value)
Column 9: OUT_RFRZ_ENERGY 0 0 (Average Value)

Time Step: 24
File start date: 01/01/1961 - 0
Number of Layers: 0
Number of Soil Thermal Nodes: 0
Number of Elevation Bands: 0
Number of Frost Fronts: 0
Number of Lake Nodes: 0
Number of Columns: 10

Since this is an ASCII file the columns between the variable name and the aggregation method are both unused and displayed as 0s. If this was a binary file format, then they would display the output variable type and multiplier. The information at the bottom of the file regarding number of layers and nodes reflects what is in the current output file. Since this snow file contains no data for soil layers, thermal nodes, elevation bands, etc. those values are all set to 0. If you looked for example at the output from a flux file, it would have the number of layers set to 3.

Now lets calculate some summary statistics and generate some spatial plots. The program VicOutputASMStats will calculate (A)nnual, (M)onthly, and (S)easonal statistics on raw VIC model output, assuming that those output files have the VIC model header at the start. The program can handle ASCII and Binary formats, and will produce GMT grid files or XYZ files of the results.

{prompt} VicOutputASMStats

Usage: VicOutputASMStats <file list file> <Output ArcGrid:
TRUE/FALSE> <output prefix> <grid resolution> <column list
file> <start date> <end date> <V|A|S|M|W|D> <OVERWRITE:
TRUE/FALSE> <PrtAllPeriods: TRUE/FALSE>

This program produces either an ARC/INFO ASCII grid file

(output ArcGrid = T) or an XYZ file (Output ArcGrid = F) of the
average of the selected data column for the given time period.

<file list> is a file containing the full grid file name and location, 
    latitude and longitude of the grid cell, for each grid cell to be
    included.

<output prefix> is the prefix (path and start of file name) for the
    output files that will be generated by this program. A suffix
    will be added to all file names to separate individual output
    for each variable, and for each tperiod (year, season, month,
    etc). Files containing multi-year average statistics for annual,
    seasonal and monthly periods will use "9999" for
    the date in the file name.

<grid resolution> is the resolution in degrees of the desired output
    grid.

<column list file> is a multi-column ASCII file that lists the column
    name for each column to be output. Followed by the statistic
    to be computed and a thresehold if required by the statistic.

Statistic options include:
    Mean value 'mean',
    Cumulative value 'sum',
    Standard Deviation 'stdev',
    Maximum value 'max',
    Minimum value 'min',
    First value 'first',
    Last value 'last',
    Last day over thres. 'ldayo' <thres>,
    Last day under thres. 'ldayu' <thres>,
    First day over thres. 'fdayo' <thres>,
    First day under thres. 'fdayu' <thres>,
    Days over threshold 'othres' <thres>, and
    Days under threshold 'uthres' <thres>.

NOTE: To output potential evapotranspiration, include "OUT_PE"
    as a variable. To output total runoff (runoff + baseflow),
    include "OUT_TOTAL_RUNOFF" as a variable.

<start date> and <end date> are the starting and ending dates of the
    period of interest in MMDDYYYY format (MM = month, DD = day,
    YYYY = year - date must be 8 characters).
<V|A|S|M|W|D> export Annual a(V)erage, (A)nnual, (S)easonal, (M)onthly,
    (W)eekly or (D)aily grids for all years in the file
        - Annual uses given start date to start year;
        - Seasonal uses Winter = DJF, Spring = MAM, Summer = JJA, and
          Autumn = SON;
        - Weekly parses data into 7 day weeks starting with the given
          start date.

<OVERWRITE> if set to TRUE then the output grid files will be
    overwritten. The default is to replace values in existing
    files with new data.
<PrtAllPeriods> if set to TRUE then the program will write all output
    files (all years, all seasons, etc), if set to FALSE only
    annual averages will be written.

For this tutorial, we will walk through the steps to generate maps of annual average preciptation (OUT_PREC) and evapotranspiration (OUT_EVAP).

First generate a file list using this awk script

{prompt} ls -1 Results/fluxes_* | awk '{ split( $1, tmpval, "_" ); print $1, tmpval[2], tmpval[3] }' > fluxes_file_list
{prompt} head fluxes_file_list
Results/fluxes_45.4375_-94.0625 45.4375 -94.0625
Results/fluxes_45.4375_-94.1875 45.4375 -94.1875
Results/fluxes_45.4375_-94.3125 45.4375 -94.3125
Results/fluxes_45.4375_-94.4375 45.4375 -94.4375
Results/fluxes_45.4375_-94.5625 45.4375 -94.5625
Results/fluxes_45.4375_-94.6875 45.4375 -94.6875
Results/fluxes_45.4375_-94.8125 45.4375 -94.8125
Results/fluxes_45.4375_-94.9375 45.4375 -94.9375
Results/fluxes_45.4375_-95.0625 45.4375 -95.0625
Results/fluxes_45.4375_-95.1875 45.4375 -95.1875

Now you need to create a file to control the calculation of summary statistics. Start by creating a directory to hold the output and control files for the summary statistics, I called it PlotData (e.g., "mkdir PlotData"). Then create a file in the new directory call AnnualStatControl, and copy the following text into the control file. Precipitation (OUT_PREC), evapotranspiration (OUT_EVAP) and runoff plus baseflow (OUT_TOTAL_RUNOFF) are summed for each year and then the sums are averaged for all years. For the liquid soil moisture of the three soil layers (OUT_SOIL_LIQ_0, 1, and 2), the last value for each year is captured, and then averaged for all years.
```
OUT_PREC sum
OUT_EVAP sum
OUT_TOTAL_RUNOFF sum
OUT_SOIL_LIQ_0 last
OUT_SOIL_LIQ_1 last
OUT_SOIL_LIQ_2 last
```
Now let's run the program to calculate summary statistics for a 30 (water) year period starting in October 1970 and running through October 1999. Output will be as ArcGIS ASCII grid files, and we will not print all of the annual grids, only the final annual average values will be output.
```
> VicOutputASMStats fluxes_file_list TRUE PlotData/ANNUAL 0.125
> PlotData/AnnualStatControl 10011970 10011999 A FALSE FALSE
```

When the program is complete, the new files will all be written in the PlotData directory with the prefix ANNUAL. The date is replaced with "9999" to indicate that the file contains annual average values.

{prompt} ls PlotData/
ANNUAL_9999_OUT_EVAP_sum.asc ANNUAL_9999_OUT_SOIL_LIQ_1_last.asc AnnualStatControl
ANNUAL_9999_OUT_PREC_sum.asc ANNUAL_9999_OUT_SOIL_LIQ_2_last.asc
ANNUAL_9999_OUT_SOIL_LIQ_0_last.asc
ANNUAL_9999_OUT_TOTAL_RUNOFF_sum.asc

A similar analysis can be run to compute the same statistics on a seasonal basis. Only the output file prefix and the statistic type are changed but now statistics are computed for the pre-defined seasons. Because these may line up directly with the date range selected, the program will adjust the start and end dates so that the analysis captures full season periods.
```
> VicOutputASMStats fluxes_file_list TRUE PlotData/SEASONAL 0.125
> PlotData/AnnualStatControl 10011970 10011999 S FALSE FALSE
```

This creates more files in the PlotData directory, this time with the season (WIN)ter, (SPR)ring, (SUM)mer, and (AUT)umn indicated as part of the file name.

{prompt} ls PlotData/
{danpatch:[~/Projects/SpatialTutorial]} ls PlotData/
ANNUAL_9999_OUT_EVAP_sum.asc               SEASONAL_9999_SPR_OUT_SOIL_LIQ_1_last.asc
ANNUAL_9999_OUT_PREC_sum.asc               SEASONAL_9999_SPR_OUT_SOIL_LIQ_2_last.asc
ANNUAL_9999_OUT_SOIL_LIQ_0_last.asc        SEASONAL_9999_SPR_OUT_TOTAL_RUNOFF_sum.asc
ANNUAL_9999_OUT_SOIL_LIQ_1_last.asc        SEASONAL_9999_SUM_OUT_EVAP_sum.asc
ANNUAL_9999_OUT_SOIL_LIQ_2_last.asc        SEASONAL_9999_SUM_OUT_PREC_sum.asc
ANNUAL_9999_OUT_TOTAL_RUNOFF_sum.asc       SEASONAL_9999_SUM_OUT_SOIL_LIQ_0_last.asc
AnnualStatControl                          SEASONAL_9999_SUM_OUT_SOIL_LIQ_1_last.asc
SEASONAL_9999_AUT_OUT_EVAP_sum.asc         SEASONAL_9999_SUM_OUT_SOIL_LIQ_2_last.asc
SEASONAL_9999_AUT_OUT_PREC_sum.asc         SEASONAL_9999_SUM_OUT_TOTAL_RUNOFF_sum.asc
SEASONAL_9999_AUT_OUT_SOIL_LIQ_0_last.asc  SEASONAL_9999_WIN_OUT_EVAP_sum.asc
SEASONAL_9999_AUT_OUT_SOIL_LIQ_1_last.asc  SEASONAL_9999_WIN_OUT_PREC_sum.asc
SEASONAL_9999_AUT_OUT_SOIL_LIQ_2_last.asc  SEASONAL_9999_WIN_OUT_SOIL_LIQ_0_last.asc
SEASONAL_9999_AUT_OUT_TOTAL_RUNOFF_sum.asc SEASONAL_9999_WIN_OUT_SOIL_LIQ_1_last.asc
SEASONAL_9999_SPR_OUT_EVAP_sum.asc         SEASONAL_9999_WIN_OUT_SOIL_LIQ_2_last.asc
SEASONAL_9999_SPR_OUT_PREC_sum.asc         SEASONAL_9999_WIN_OUT_TOTAL_RUNOFF_sum.asc
SEASONAL_9999_SPR_OUT_SOIL_LIQ_0_last.asc

Step 7: Plotting Summary Statistics

This section makes use of the Generic Mapping Tools (GMT), more complete tutorials are available through the GMT web site.

Start by loading the full GMT module on the cluster system using module load gmt.

Note

This is GMT version 5.x, which implements a newer command control system that is more in line with modern programming standards. All commands are now accessed as sub-calls to a centralized "gmt" command, so what used to be "psxy", is now "gmt psxy". Links are currently installed with GMT 5.x to allow for backwards compatability (a call to "psxy" will still work), but these are expected to be phased out in later releases of the software package.
Make two new directories, PlotScripts and Postscript to hold plotting scripts and GMT plot output, respectively. Do this from the SpatialTutorial directory where you also ran all of your simulations.

Create a plot script called PlotAnnualAveragePrecipitation.csh by opening a blank file in an editor and copying the text from the box as follows:

#!/bin/env csh
# Created on January 16, 2017
# by Keith Cherkauer
#
# This script plots a single ArcGIS ASCII floating point grid
# representing annual average precipitation for the Red River
# using GMT 5.x commands.
#

# load the gmt module
module load gmt

set InData = PlotData/ANNUAL_9999_OUT_PREC_sum.asc
set OutFile = Postscript/AnnualPrecip.eps
gmt gmtset FORMAT_FLOAT_OUT %.0f

# Reformat a floating point ESRI ASCII raster file into a GMT NetCDF file
gmt grdconvert PlotData/ANNUAL_9999_OUT_PREC_sum.asc=ef PlotData/ANNUAL_9999_OUT_PREC_sum.nc -V

# Create a color scale for the current plot (uses normal distribution to create 9 bands)
gmt grd2cpt -Ccool -S10 PlotData/ANNUAL_9999_OUT_PREC_sum.nc > ! PlotData/ANNUAL_9999_OUT_PREC_sum.cpt

# Draw the current plot
set CellSize = `gmt grdinfo -I PlotData/ANNUAL_9999_OUT_PREC_sum.nc`
set Range = `gmt grdinfo $CellSize PlotData/ANNUAL_9999_OUT_PREC_sum.nc`
gmt grdimage -JX5 $Range PlotData/ANNUAL_9999_OUT_PREC_sum.nc -CPlotData/ANNUAL_9999_OUT_PREC_sum.cpt -P -K -Y2 >! $OutFile

# plot the color scale on the same figure
gmt gmtset FORMAT_FLOAT_OUT %.0f # don't leave this as it affects many things
gmt psscale -D2.5/-0.5/5/0.25h -CPlotData/ANNUAL_9999_OUT_PREC_sum.cpt -O -B:"Annual Precipitation (mm)": >> $OutFile
gmt gmtset FORMAT_FLOAT_OUT %.12g # returns to original value

This will result in the figure below, where purple indicates higher annual precipitation and blue lower annual precipitation. The color scale developed here is broken out using the normal distribution so that colors will be equal in area covered, but not necessarily equal in range. This is fine for a single plot, but a problem when trying to compare multiple variables.
Similar plots can be generated for other variables.