Running using param. dict.¶
[This note-book is in oceantracker/tutorials_how_to/]
The earlier notebook showed how to set parameters and run using the helper class. Here we exploit the flexibility to run Oceantracker directly using a parameter dictionary built in code or read from a file.
Build param. dict. with code¶
The below extends the minimal_example, shows two ways to build a parameter dictionary and different ways to run OceanTracker, from code or command line.
# build a more complex dictionary of parameters using code
params={
'output_file_base' :'param_test1', # name used as base for output files
'root_output_dir':'output', # output is put in dir 'root_output_dir'/'output_file_base'
'time_step' : 120, # 2 min time step as seconds
'reader':{'input_dir': '../demos/demo_hindcast', # folder to search for hindcast files, sub-dirs will, by default, also be searched
'file_mask': 'demoHindcastSchism*.nc', # the file mask of the hindcast files
},
# add release locations from two points,
# particle_release_groups are a list of one or more release groups
# (ie locations where particles are released at the same times and locations)
'release_groups': {
'my_release_point' :{'points':[[1595000, 5482600],
[1599000, 5486200]], # must be an N by 2 or 3 or list, convertible to a numpy array
'release_interval': 3600, # seconds between releasing particles
'pulse_size': 10, # number of particles released each release_interval
},
'my_polygon_release': {'class_name': 'oceantracker.release_groups.polygon_release.PolygonRelease', # use a polygon release
'points':[ [1597682.1237, 5489972.7479],
[1598604.1667, 5490275.5488],
[1598886.4247, 5489464.0424],
[1597917.3387, 5489000],
[1597300, 5489000], [1597682.1237, 5489972.7479]],
'release_interval': 7200, # seconds between releasing particles
'pulse_size': 20, # number of particles released each release_interval
},
},
'resuspension' : {'critical_friction_velocity': .005}, # only re-suspend particles if friction vel. exceeds this value
# velocity_modifiers are a set of velocities added to water velocity give in hydrodynamic model's
'velocity_modifiers' : { # here a fall velocity with given mean and variance is added to the computation
'fall_velocity': {'class_name' : 'oceantracker.velocity_modifiers.terminal_velocity.TerminalVelocity',
'value': -0.001,'variance': 0.0002}
},
}
# write params to build on for later examples
from oceantracker.util import json_util, yaml_util
json_util.write_JSON('./example_param_files/param_test1.json', params)
yaml_util.write_YAML('./example_param_files/param_test1.yaml', params)
Build param. dict. from template¶
Use a provided template, which pre-defines most top level parameters that can be used, then set required parameters using assignments. Reproducing the above in code…
from oceantracker import main
# repeat above using a template and set dict keys values by assignments
params = main.param_template() # get a copy of the template
params['output_file_base'] ='param_test1'
params['root_output_dir']= 'output' # output is put in dir 'root_output_dir'/'output_file_base'
params['time_step']= 120 # 2 min time step as seconds
params['reader']['input_dir']= '../demos/demo_hindcast' # folder to search for hindcast files, sub-dirs will, by default, also be searched
params['reader']['file_mask']= 'demoHindcastSchism*.nc' # the file mask of the hindcast files
params['release_groups']['my_release_point'] = {
'points':[[1595000, 5482600],
[1599000, 5486200]], # must be an N by 2 or 3 or list, convertible to a numpy array
'release_interval': 3600, # seconds between releasing particles
'pulse_size': 10, # number of particles released each release_interval
}
params['release_groups']['my_polygon_release'] = {
'class_name': 'oceantracker.release_groups.polygon_release.PolygonRelease', # use a polygon release
'points':[ [1597682.1237, 5489972.7479],
[1598604.1667, 5490275.5488],
[1598886.4247, 5489464.0424],
[1597917.3387, 5489000],
[1597300, 5489000], [1597682.1237, 5489972.7479]],
'release_interval': 7200, # seconds between releasing particles
'pulse_size': 20, # number of particles released each release_interval
}
params['resuspension']['critical_friction_velocity']= .005
params['velocity_modifiers']['my_fall_velocity']= { # here a fall velocity with given value and variance is added to the computation
'class_name' : 'oceantracker.velocity_modifiers.terminal_velocity.TerminalVelocity',
'value': -0.001,'variance': 0.0002
}
# show the full template as json
import json
print( json.dumps(params, indent=4))
{
"add_date_to_run_output_dir": null,
"backtracking": null,
"block_dry_cells": null,
"case_output_file_tag": null,
"debug": null,
"max_particles": null,
"max_run_duration": null,
"max_warnings": null,
"minimum_total_water_depth": null,
"multiprocessing_case_start_delay": null,
"numba_function_cache_size": null,
"open_boundary_type": null,
"output_file_base": "param_test1",
"processors": null,
"profiler": null,
"root_output_dir": "output",
"run_as_depth_averaged": null,
"screen_output_time_interval": null,
"time_step": 120,
"use_random_seed": null,
"user_note": null,
"write_output_files": null,
"write_tracks": null,
"z0": null,
"dispersion": {},
"field_group_manager": {},
"interpolator": {},
"particle_group_manager": {},
"reader": {
"input_dir": "../demos/demo_hindcast",
"file_mask": "demoHindcastSchism*.nc"
},
"resuspension": {
"critical_friction_velocity": 0.005
},
"solver": {},
"tracks_writer": {},
"event_loggers": {},
"fields": {},
"particle_concentrations": {},
"particle_properties": {},
"particle_statistics": {},
"release_groups": {
"my_release_point": {
"points": [
[
1595000,
5482600
],
[
1599000,
5486200
]
],
"release_interval": 3600,
"pulse_size": 10
},
"my_polygon_release": {
"class_name": "oceantracker.release_groups.polygon_release.PolygonRelease",
"points": [
[
1597682.1237,
5489972.7479
],
[
1598604.1667,
5490275.5488
],
[
1598886.4247,
5489464.0424
],
[
1597917.3387,
5489000
],
[
1597300,
5489000
],
[
1597682.1237,
5489972.7479
]
],
"release_interval": 7200,
"pulse_size": 20
}
},
"status_modifiers": {},
"time_varying_info": {},
"trajectory_modifiers": {},
"velocity_modifiers": {
"my_fall_velocity": {
"class_name": "oceantracker.velocity_modifiers.terminal_velocity.TerminalVelocity",
"mean": -0.001,
"variance": 0.0002
}
}
}
### Show parameters in yaml format
yaml format has no brackets/braces and relies on tab indenting to nest items
# show the full template in yaml format
import yaml
print( yaml.dump(params))
add_date_to_run_output_dir: null
backtracking: null
block_dry_cells: null
case_output_file_tag: null
debug: null
dispersion: {}
event_loggers: {}
field_group_manager: {}
fields: {}
interpolator: {}
max_particles: null
max_run_duration: null
max_warnings: null
minimum_total_water_depth: null
multiprocessing_case_start_delay: null
numba_function_cache_size: null
open_boundary_type: null
output_file_base: param_test1
particle_concentrations: {}
particle_group_manager: {}
particle_properties: {}
particle_statistics: {}
processors: null
profiler: null
reader:
file_mask: demoHindcastSchism*.nc
input_dir: ../demos/demo_hindcast
release_groups:
my_polygon_release:
class_name: oceantracker.release_groups.polygon_release.PolygonRelease
points:
- - 1597682.1237
- 5489972.7479
- - 1598604.1667
- 5490275.5488
- - 1598886.4247
- 5489464.0424
- - 1597917.3387
- 5489000
- - 1597300
- 5489000
- - 1597682.1237
- 5489972.7479
pulse_size: 20
release_interval: 7200
my_release_point:
points:
- - 1595000
- 5482600
- - 1599000
- 5486200
pulse_size: 10
release_interval: 3600
resuspension:
critical_friction_velocity: 0.005
root_output_dir: output
run_as_depth_averaged: null
screen_output_time_interval: null
solver: {}
status_modifiers: {}
time_step: 120
time_varying_info: {}
tracks_writer: {}
trajectory_modifiers: {}
use_random_seed: null
user_note: null
velocity_modifiers:
my_fall_velocity:
class_name: oceantracker.velocity_modifiers.terminal_velocity.TerminalVelocity
mean: -0.001
variance: 0.0002
write_output_files: null
write_tracks: null
z0: null
Run OceanTracker from parameters¶
There are several ways to run OceanTracker
By coding
build parameters in code and run
or coding to read parameter file and then run
Without coding
run from command line with parameter file which is built by editing a json/yaml text file
Note:
There are many ways to run the code, eg. with IDE like Pycharm, Visual Studio Code. It can also, as here, be run in iPython notebooks. However the way notebooks are implemented can sometimes result in issues:
errors when running Oceantracker a second time or other unexpected behavior, due to shared memory space, fix by reloading the kernel
if using note books on Windows, it is not possible to run Oceantracker cases in parallel, without a work around given in a later “how to”.
Run with code built params.¶
Is line below!
# run oceantracker using param dict built in cells above
from oceantracker import main
case_info_file_name = main.run(params)
# case_info file is the name of a json file useful in plotting results
main: -------------------------------------------------------------------------- main: OceanTracker- preliminary setup main: Python version: 3.10.9 | packaged by conda-forge | (main, Jan 11 2023, 15:15:40) [MSC v.1916 64 bit (AMD64)] main: - found hydro-model files of type SCHISIM main: - sorted hyrdo-model files in time order, 0.102 sec main: >>> Note: output is in dir= e:H_Local_driveParticleTrackingoceantrackertutorials_how_tooutputparam_test1 main: >>> Note: to help with debugging, parameters as given by user are in "param_test1_raw_user_params.json" C000: -------------------------------------------------------------------------- C000: Starting case number 0, param_test1 at 2023-06-27T13:58:29.031855 C000: -------------------------------------------------------------------------- C000: - built node to triangles map, 0.814 sec C000: - built triangle adjacency matrix, 0.272 sec C000: - found boundary triangles, 0.000 sec C000: - built domain and island outlines, 1.569 sec C000: - calculated triangle areas, 0.000 sec C000: Finished grid setup C000: - set up release_groups, 1.030 sec C000: - built barycentric-transform matrix, 0.453 sec C000: - initial set up of core classes, 0.471 sec C000: - final set up of core classes, 0.001 sec C000: - created particle properties derived from fields, 0.003 sec C000: >>> Warning: When using a terminal velocity, ensure time step is small enough that vertical displacement is a small fraction of the water depth, ie vertical Courant number < 1 C000: >>> Note: No open boundaries requested, as run_params["open_boundary_type"] = 0 C000: Hint: Requires list of open boundary nodes not in hydro model, eg for Schism this can be read from hgrid file to named in reader params and run_params["open_boundary_type"] = 1 C000: -------------------------------------------------------------------------- C000: - Starting param_test1, duration: 0 days 23 hrs 0 min 0 sec C000: - Initialized Solver Class, 0.000 sec C000: 00% step 0000:H0000b00-01 Day +00 00:00 2017-01-01 00:30:00: Rel.: 40: Active:00040 M:00040 S:00000 B:00000 D:000 O:00 N:000 Buffer:0040 - 0% step time = 6894.9 ms C000: - Reading-file-00 demoHindcastSchism3D.nc, steps in file 24, steps available 000:023, reading 24 of 48 steps, for hydo-model time steps 00:23, from file offsets 00:23, into ring buffer offsets 000:023 C000: - read 24 time steps in 0.5 sec C000: - opening tracks output to : param_test1_tracks_compact.nc C000: 04% step 0030:H0001b01-02 Day +00 01:00 2017-01-01 01:30:00: Rel.: 60: Active:00060 M:00054 S:00000 B:00006 D:000 O:00 N:000 Buffer:0060 - 0% step time = 3081.1 ms C000: 09% step 0060:H0002b02-03 Day +00 02:00 2017-01-01 02:30:00: Rel.: 100: Active:00100 M:00092 S:00000 B:00008 D:000 O:00 N:000 Buffer:0100 - 0% step time = 61.6 ms C000: 13% step 0090:H0003b03-04 Day +00 03:00 2017-01-01 03:30:00: Rel.: 120: Active:00120 M:00098 S:00011 B:00011 D:000 O:00 N:000 Buffer:0120 - 0% step time = 62.5 ms C000: 17% step 0120:H0004b04-05 Day +00 04:00 2017-01-01 04:30:00: Rel.: 160: Active:00160 M:00130 S:00012 B:00018 D:000 O:00 N:000 Buffer:0160 - 0% step time = 62.6 ms C000: 22% step 0150:H0005b05-06 Day +00 05:00 2017-01-01 05:30:00: Rel.: 180: Active:00180 M:00144 S:00013 B:00023 D:000 O:00 N:000 Buffer:0180 - 0% step time = 64.7 ms C000: 26% step 0180:H0006b06-07 Day +00 06:00 2017-01-01 06:30:00: Rel.: 220: Active:00220 M:00177 S:00013 B:00030 D:000 O:00 N:000 Buffer:0220 - 0% step time = 66.8 ms C000: 30% step 0210:H0007b07-08 Day +00 07:00 2017-01-01 07:30:00: Rel.: 240: Active:00240 M:00196 S:00012 B:00032 D:000 O:00 N:000 Buffer:0240 - 0% step time = 66.3 ms C000: 35% step 0240:H0008b08-09 Day +00 08:00 2017-01-01 08:30:00: Rel.: 280: Active:00280 M:00216 S:00011 B:00053 D:000 O:00 N:000 Buffer:0280 - 0% step time = 67.2 ms C000: 39% step 0270:H0009b09-10 Day +00 09:00 2017-01-01 09:30:00: Rel.: 300: Active:00300 M:00232 S:00000 B:00068 D:000 O:00 N:000 Buffer:0300 - 0% step time = 69.5 ms C000: 43% step 0300:H0010b10-11 Day +00 10:00 2017-01-01 10:30:00: Rel.: 340: Active:00340 M:00246 S:00000 B:00094 D:000 O:00 N:000 Buffer:0340 - 0% step time = 69.8 ms C000: 48% step 0330:H0011b11-12 Day +00 11:00 2017-01-01 11:30:00: Rel.: 360: Active:00360 M:00225 S:00000 B:00135 D:000 O:00 N:000 Buffer:0360 - 0% step time = 69.5 ms C000: 52% step 0360:H0012b12-13 Day +00 12:00 2017-01-01 12:30:00: Rel.: 400: Active:00400 M:00266 S:00000 B:00134 D:000 O:00 N:000 Buffer:0400 - 0% step time = 70.7 ms C000: 57% step 0390:H0012b12-13 Day +00 13:00 2017-01-01 13:30:00: Rel.: 420: Active:00420 M:00300 S:00004 B:00116 D:000 O:00 N:000 Buffer:0420 - 0% step time = 71.6 ms C000: 61% step 0420:H0014b14-15 Day +00 14:00 2017-01-01 14:30:00: Rel.: 460: Active:00460 M:00328 S:00014 B:00118 D:000 O:00 N:000 Buffer:0460 - 0% step time = 72.5 ms C000: 65% step 0450:H0015b15-16 Day +00 15:00 2017-01-01 15:30:00: Rel.: 480: Active:00480 M:00297 S:00062 B:00121 D:000 O:00 N:000 Buffer:0480 - 0% step time = 73.3 ms C000: 70% step 0480:H0016b16-17 Day +00 16:00 2017-01-01 16:30:00: Rel.: 520: Active:00520 M:00285 S:00066 B:00169 D:000 O:00 N:000 Buffer:0520 - 0% step time = 71.6 ms C000: 74% step 0510:H0017b17-18 Day +00 17:00 2017-01-01 17:30:00: Rel.: 540: Active:00540 M:00234 S:00076 B:00230 D:000 O:00 N:000 Buffer:0540 - 0% step time = 71.0 ms C000: 78% step 0540:H0018b18-19 Day +00 18:00 2017-01-01 18:30:00: Rel.: 580: Active:00580 M:00326 S:00076 B:00178 D:000 O:00 N:000 Buffer:0580 - 0% step time = 71.5 ms C000: 83% step 0570:H0019b19-20 Day +00 19:00 2017-01-01 19:30:00: Rel.: 600: Active:00600 M:00341 S:00073 B:00186 D:000 O:00 N:000 Buffer:0600 - 0% step time = 73.9 ms C000: 87% step 0600:H0020b20-21 Day +00 20:00 2017-01-01 20:30:00: Rel.: 640: Active:00640 M:00390 S:00069 B:00181 D:000 O:00 N:000 Buffer:0640 - 0% step time = 75.1 ms C000: 91% step 0630:H0021b21-22 Day +00 21:00 2017-01-01 21:30:00: Rel.: 660: Active:00660 M:00402 S:00014 B:00244 D:000 O:00 N:000 Buffer:0660 - 0% step time = 76.1 ms C000: 96% step 0660:H0022b22-23 Day +00 22:00 2017-01-01 22:30:00: Rel.: 700: Active:00700 M:00356 S:00004 B:00340 D:000 O:00 N:000 Buffer:0700 - 0% step time = 75.2 ms C000: 100% step 0689:H0022b22-23 Day +00 22:58 2017-01-01 23:28:00: Rel.: 700: Active:00700 M:00400 S:00000 B:00300 D:000 O:00 N:000 Buffer:0700 - 0% step time = 75.0 ms C000: >>> Note: No open boundaries requested, as run_params["open_boundary_type"] = 0 C000: Hint: Requires list of open boundary nodes not in hydro model, eg for Schism this can be read from hgrid file to named in reader params and run_params["open_boundary_type"] = 1 C000: >>> Warning: When using a terminal velocity, ensure time step is small enough that vertical displacement is a small fraction of the water depth, ie vertical Courant number < 1 C000: - Triangle walk summary: Of 990,448 particles located 0, walks were too long and were retried, of these 0 failed after retrying and were discarded C000: -------------------------------------------------------------------------- C000: - Finished case number 0, param_test1 started: 2023-06-27 13:58:29.030855, ended: 2023-06-27 13:58:46.174430 C000: Elapsed time =0:00:17.143575 C000: -------------------------------------------------------------------------- main: >>> Note: run summary with case file names "param_test1_runInfo.json" main: >>> Note: output is in dir= e:H_Local_driveParticleTrackingoceantrackertutorials_how_tooutputparam_test1 main: >>> Note: to help with debugging, parameters as given by user are in "param_test1_raw_user_params.json" main: >>> Note: run summary with case file names "param_test1_runInfo.json" main: -------------------------------------------------------------------------- main: OceanTracker summary: elapsed time =0:00:17.307615 main: Cases - 0 errors, 1 warnings, 2 notes, check above main: Main - 0 errors, 0 warnings, 3 notes, check above main: --------------------------------------------------------------------------
# plot animation of results
from matplotlib import pyplot as plt
from oceantracker.post_processing.plotting.plot_tracks import animate_particles
from oceantracker.post_processing.read_output_files import load_output_files
from IPython.display import HTML # show animation in note book
# read particle track data into a dictionary using case_info_file_name
tracks = load_output_files.load_track_data(case_info_file_name)
ax= [1591000, 1601500, 5478500, 5491000] # area to plot
# animate particles
anim = animate_particles(tracks, axis_lims=ax,title='Fall vel.+ re-sus., grey part. are on bottom when flows too weak to resuspend',
show_dry_cells=True, show_grid=True, show=False) # use ipython to show video, rather than matplotlib plt.show()
# this is slow to build!
HTML(anim.to_html5_video())
Run by reading param. file¶
Run from command line¶
Run without coding from command lin bu using a parameter file pre-built in a text editor.
From within an activated oceantracker conda environment, run command line below.
On Windows, do this within an anaconda/miniconda prompt window with an activated environment.
eg. run “run_oceantracker.py” script in the oceantracker/oceantracker directory with command
python ../oceantracker/run_oceantracker.py ./example_param_files/param_test1.json
Options when running at command line¶
These allow
redefining the input and output dirs given within parameter file, which may have been built for a different location
limiting the run duration or the number of parallel cases during testing
the full arguments are below
!python ../oceantracker/run_oceantracker.py -h
usage: run_oceantracker.py [-h] [--input_dir INPUT_DIR]
[--root_output_dir ROOT_OUTPUT_DIR]
[--processors PROCESSORS] [--duration DURATION]
[--cases CASES] [-debug]
param_file
positional arguments:
param_file json or yaml file of input parameters
options:
-h, --help show this help message and exit
--input_dir INPUT_DIR
overrides dir for hindcast files given in param file
--root_output_dir ROOT_OUTPUT_DIR
overrides root output dir given in param file
--processors PROCESSORS
overrides number of processors in param file
--duration DURATION in seconds, overrides model duration in seconds of all
of cases, useful in testing
--cases CASES only runs first "cases" of the case_list, useful in
testing
-debug gives better error information, but runs slower, eg
checks Numba array bounds