Running with helper class¶
[This note-book is in oceantracker/tutorials_how_to/]
Oceantracker is designed so that both coders and non-coders can get almost the same level of adaptability to their specific needs. Also to streamline running many cases across distributed computers, along with enabling online particle tracking as a service.
To achieve these capabilities, it uses parameters to both change the settings to meet the users needs, but also to build the computational pipeline for the users specific needs. This flexibility in the computational pipeline is achieved using “class” parameters.
Classes perform specific roles in the pipeline. Parameters tell Oceantracker which version of core classes the user wants to use for each role, or add optional classes. Eg. add multiple “release_groups” classes, which may release particles at points, or within a polygon, at different times, rates and locations, all within the same computational run.
Simply by changing the parameters, users can load their own version of a class, including the “solver” class which orchestrates the time stepping and operations by other classes. There are base classes for all the major roles within the computational pipeline. Classes can be adapted by inheriting the bases class, or one of its children, and overwriting some of the methods to alter how the class performs its role.
To make the ideas of settings and classes easier to adopt, the below uses methods of a helper class to add settings and classes to the pipeline. The notebook E_run_using_parameter_dictionaries.ipynb, shows how run directly from parameter dictionaries, which are built in code or read from a json or yaml file.
Parameters using helper¶
There are two types of parameters:
Settings parameters
These are set with one or more calls to helper method, ot.setting(…), where … are keyword arguments. eg “time_step”, the model time step in seconds is set using, time_step= 1800.
There are default values for most settings. A full set of settings and their defaults is at all settings
Class role parameters
Classes add specific tasks to the computational pipeline, eg. how to release particles, write output, particle suspension etc. These are added using helper method ot.add_class(class_role, ….). The first argument is a “class_role”. Each class has its own specific settings, which are set using keyword arguments of helper method. Eg. ot.add_class(‘reader’, input_dir= ‘my_hindcast_dir’, ….) adds a reader class and tells is it which folder contains the hindcast files.
A full set of classes and their default settings is at parameter_ref
Normally there must be a keyword argument “class_name” setting. This is used to import the required class into the computational pipeline, with its specific settings. This “class_name” is a string, used to import the class at setup (so is the same as that used to import any python class within a module).
The “class_name” setting is normally required. A few class_roles have a default class_name, eg. in minimal_example “release_groups” assumes a point_release, and the reader’s “class_name” is chosen by looking at variable names within the hindcast’s netcdf file.
There are two types of “class_role”.
When
Only a single class is required, these have singular “class_role”, eg.”reader” and “solver”.
One or more classes can be added. These have plural “class_role” (eg. can add multiple “release_groups”). Users must give a unique name to each one.
Documentation for all parameters/settings and their default values.
Add link here
Extend the minimal example¶
The below extends the minimal_example, it adds:
release at random locations within a polygon, at random water depths (the default)
a particle fall velocity
particles on the bottom are re-suspended if friction velocity exceeds a critical value.
# build parameters using helper class
from oceantracker.main import OceanTracker
ot = OceanTracker() # make an instance of the helper class
# one or more settings can be set by calls to os.settings
ot.settings(output_file_base='param_test1',# name used as base for output files
root_output_dir='output' # output is put in dir 'root_output_dir'/'output_file_base'
)
# ot.settings can be used more than once to add more settings
ot.settings(time_step =120) # 2 min model time step as seconds
# add a compulsory reader class
# no class_name setting is required
# as will detect that it needs a a schism reader class
ot.add_class('reader', # class_role is reader
input_dir= '../demos/demo_hindcast', # folder to search for hindcast files, sub-dirs will, by default, also be searched
file_mask = 'demoHindcastSchism*.nc', # the file mask of the hindcast files
)
# add some release groups
# release_groups are one or more release groups
# (ie locations where particles are released at the same times and locations)
# there must be at least one release group
# add release locations from two points,
ot.add_class('release_groups', # class_role is release_groups
name ='my_release_points1',
# the required name setting, is used to refer to this release group internally anf in postprocsing
points= [[1595000, 5482600],
[1599000, 5486200]], # must be an N by 2 or 3 or list, convertible to a numpy array
release_interval= 3600, # seconds between releasing particles
pulse_size = 10, # number of particles released each release_interval
# no class_name setting, so assumes a point release
)
# add a second release group, from random locations within a polygon
ot.add_class('release_groups', # class_role is release_group
name ='my_polygon_release1',
class_name= 'oceantracker.release_groups.polygon_release.PolygonRelease', # use a polygon release
# this time is a polygon , so below points are polygon cords
points = [ [1597682.1237, 5489972.7479],
[1598604.1667, 5490275.5488],
[1598886.4247, 5489464.0424],
[1597917.3387, 5489000],
[1597300, 5489000],
[1597682.1237, 5489972.7479]
],
release_interval= 7200, # seconds between releasing particles
pulse_size = 20, # number of particles released each release_interval
)
# alter default re-suspension class's default settings
ot.add_class('resuspension', critical_friction_velocity = .005) # only re-suspend particles if friction vel. exceeds this value
# add a class to modify the particle velocity
# velocity_modifiers are a set of velocities added to water velocity given in the hydrodynamic model's
# here a fall velocity with a given value is added to the computation
ot.add_class('velocity_modifiers', # class_role is velocity_modifiers
name ='my_fall_velocity',
class_name = 'oceantracker.velocity_modifiers.terminal_velocity.TerminalVelocity',
value= -0.001, # mean terminal vel < 0 for falling
# optionally variance can also be use to give each particle its own fall velocity
)
helper: --------------------------------------------------------------------------
helper: Starting OceanTracker helper class
Running OceanTracker¶
There are several ways to run OceanTracker
By coding
user “class helper” method to build parameters (as above) then run
build parameters dictionary in code then run
read parameter file and then run
Without coding
run from command line with parameter file which is built by editing a json/yaml text file
Here we use the “class helper” method approach, the other ways to run directly using parameter dictionaries outlined in E_run_using_parameter_dictionaries.ipynb, add link…
Note:
There are many ways to run the code, eg. with IDE like Pycharm, Visual Studio Code. It can also, as here, be run in iPython notebooks. However the way notebooks are implemented can sometimes result in issues.
See … for more details
Below runs oceantracker using the helper class.
See all parameters¶
# first see the parameters build using the helper class instance
# this is a dictionary ot.params
print(ot.params) # ugly!
# use json.dumps() to make it pretty
import json
print(json.dumps(ot.params, indent=4))
# lots of params are shown
# as filling a template given by from user callable main.param_template()
# null or None are optional ones which will use defaults
{'root_output_dir': 'output', 'add_date_to_run_output_dir': None, 'output_file_base': 'param_test1', 'time_step': 120, 'screen_output_time_interval': None, 'backtracking': None, 'run_as_depth_averaged': None, 'debug': None, 'minimum_total_water_depth': None, 'write_output_files': None, 'max_run_duration': None, 'max_particles': None, 'processors': None, 'max_warnings': None, 'use_random_seed': None, 'numba_function_cache_size': None, 'multiprocessing_case_start_delay': None, 'profiler': None, 'user_note': None, 'case_output_file_tag': None, 'write_tracks': None, 'z0': None, 'open_boundary_type': None, 'block_dry_cells': None, 'dispersion': {}, 'field_group_manager': {}, 'interpolator': {}, 'particle_group_manager': {}, 'reader': {'input_dir': '../demos/demo_hindcast', 'file_mask': 'demoHindcastSchism*.nc'}, 'resuspension': {'critical_friction_velocity': 0.005}, 'solver': {}, 'tracks_writer': {}, 'event_loggers': {}, 'fields': {}, 'particle_concentrations': {}, 'particle_properties': {}, 'particle_statistics': {}, 'release_groups': {'my_release_points1': {'points': [[1595000, 5482600], [1599000, 5486200]], 'release_interval': 3600, 'pulse_size': 10}, 'my_polygon_release1': {'class_name': 'oceantracker.release_groups.polygon_release.PolygonRelease', 'points': [[1597682.1237, 5489972.7479], [1598604.1667, 5490275.5488], [1598886.4247, 5489464.0424], [1597917.3387, 5489000], [1597300, 5489000], [1597682.1237, 5489972.7479]], 'release_interval': 7200, 'pulse_size': 20}}, 'status_modifiers': {}, 'time_varying_info': {}, 'trajectory_modifiers': {}, 'velocity_modifiers': {'my_fall_velocity': {'class_name': 'oceantracker.velocity_modifiers.terminal_velocity.TerminalVelocity', 'value': -0.001}}}
{
"root_output_dir": "output",
"add_date_to_run_output_dir": null,
"output_file_base": "param_test1",
"time_step": 120,
"screen_output_time_interval": null,
"backtracking": null,
"run_as_depth_averaged": null,
"debug": null,
"minimum_total_water_depth": null,
"write_output_files": null,
"max_run_duration": null,
"max_particles": null,
"processors": null,
"max_warnings": null,
"use_random_seed": null,
"numba_function_cache_size": null,
"multiprocessing_case_start_delay": null,
"profiler": null,
"user_note": null,
"case_output_file_tag": null,
"write_tracks": null,
"z0": null,
"open_boundary_type": null,
"block_dry_cells": null,
"dispersion": {},
"field_group_manager": {},
"interpolator": {},
"particle_group_manager": {},
"reader": {
"input_dir": "../demos/demo_hindcast",
"file_mask": "demoHindcastSchism*.nc"
},
"resuspension": {
"critical_friction_velocity": 0.005
},
"solver": {},
"tracks_writer": {},
"event_loggers": {},
"fields": {},
"particle_concentrations": {},
"particle_properties": {},
"particle_statistics": {},
"release_groups": {
"my_release_points1": {
"points": [
[
1595000,
5482600
],
[
1599000,
5486200
]
],
"release_interval": 3600,
"pulse_size": 10
},
"my_polygon_release1": {
"class_name": "oceantracker.release_groups.polygon_release.PolygonRelease",
"points": [
[
1597682.1237,
5489972.7479
],
[
1598604.1667,
5490275.5488
],
[
1598886.4247,
5489464.0424
],
[
1597917.3387,
5489000
],
[
1597300,
5489000
],
[
1597682.1237,
5489972.7479
]
],
"release_interval": 7200,
"pulse_size": 20
}
},
"status_modifiers": {},
"time_varying_info": {},
"trajectory_modifiers": {},
"velocity_modifiers": {
"my_fall_velocity": {
"class_name": "oceantracker.velocity_modifiers.terminal_velocity.TerminalVelocity",
"value": -0.001
}
}
}
Start run using helper¶
# now run oceantracker
# as helper "ot" has set params above, simply run it
case_info_file_name = ot.run()
# output now in folder "root_output_dir"/"output_file_base"
# in this example output is in directory output/param_test1'
# case_info_file_name is the name of a json file with useful for post processing,
# eg it holds output file names to assist in reading and plotting data
print('case file name=',case_info_file_name)
helper: - Starting run using helper class main: -------------------------------------------------------------------------- main: OceanTracker version 0.4.01.003 2023-07-14 - preliminary setup main: Python version: 3.10.10 | packaged by conda-forge | (main, Mar 24 2023, 20:00:38) [MSC v.1934 64 bit (AMD64)] main: - found hydro-model files of type SCHISIM main: - sorted hyrdo-model files in time order, 0.087 sec main: >>> Note: output is in dir= e:H_Local_driveParticleTrackingoceantrackertutorials_how_tooutputparam_test1 main: >>> Note: to help with debugging, parameters as given by user are in "param_test1_raw_user_params.json" C000: -------------------------------------------------------------------------- C000: Starting case number 0, param_test1 at 2023-07-17T08:48:53.601451 C000: -------------------------------------------------------------------------- C000: - built node to triangles map, 0.669 sec C000: - built triangle adjacency matrix, 0.235 sec C000: - found boundary triangles, 0.000 sec C000: - built domain and island outlines, 1.360 sec C000: - calculated triangle areas, 0.000 sec C000: Finished grid setup C000: - set up release_groups, 0.939 sec C000: - built barycentric-transform matrix, 0.457 sec C000: - initial set up of core classes, 0.476 sec C000: - final set up of core classes, 0.000 sec C000: - created particle properties derived from fields, 0.003 sec C000: >>> Warning: When using a terminal velocity, ensure time step is small enough that vertical displacement is a small fraction of the water depth, ie vertical Courant number < 1 C000: >>> Note: No open boundaries requested, as run_params["open_boundary_type"] = 0 C000: Hint: Requires list of open boundary nodes not in hydro model, eg for Schism this can be read from hgrid file to named in reader params and run_params["open_boundary_type"] = 1 C000: -------------------------------------------------------------------------- C000: - Starting param_test1, duration: 0 days 23 hrs 0 min 0 sec C000: - Initialized Solver Class, 0.000 sec C000: - Reading-file-00 demoHindcastSchism3D.nc, steps in file 24, steps available 000:023, reading 24 of 24 steps, for hydo-model time steps 00:23, from file offsets 00:23, into ring buffer offsets 000:023 C000: - read 24 time steps in 0.5 sec C000: - opening tracks output to : param_test1_tracks_compact.nc C000: 00% step 0000:H0000b00-01 Day +00 00:00 2017-01-01 00:30:00: Rel.: 40: Active:00040 M:00038 S:00000 B:00002 D:000 O:00 N:000 Buffer:0040 - 0% step time = 10518.2 ms C000: 04% step 0030:H0001b01-02 Day +00 01:00 2017-01-01 01:30:00: Rel.: 60: Active:00060 M:00054 S:00000 B:00006 D:000 O:00 N:000 Buffer:0060 - 0% step time = 2.8 ms C000: 09% step 0060:H0002b02-03 Day +00 02:00 2017-01-01 02:30:00: Rel.: 100: Active:00100 M:00089 S:00000 B:00011 D:000 O:00 N:000 Buffer:0100 - 0% step time = 3.3 ms C000: 13% step 0090:H0003b03-04 Day +00 03:00 2017-01-01 03:30:00: Rel.: 120: Active:00120 M:00093 S:00011 B:00016 D:000 O:00 N:000 Buffer:0120 - 0% step time = 2.8 ms C000: 17% step 0120:H0004b04-05 Day +00 04:00 2017-01-01 04:30:00: Rel.: 160: Active:00160 M:00125 S:00012 B:00023 D:000 O:00 N:000 Buffer:0160 - 0% step time = 3.3 ms C000: 22% step 0150:H0005b05-06 Day +00 05:00 2017-01-01 05:30:00: Rel.: 180: Active:00180 M:00143 S:00015 B:00022 D:000 O:00 N:000 Buffer:0180 - 0% step time = 2.9 ms C000: 26% step 0180:H0006b06-07 Day +00 06:00 2017-01-01 06:30:00: Rel.: 220: Active:00220 M:00165 S:00015 B:00040 D:000 O:00 N:000 Buffer:0220 - 0% step time = 3.2 ms C000: 30% step 0210:H0007b07-08 Day +00 07:00 2017-01-01 07:30:00: Rel.: 240: Active:00240 M:00179 S:00012 B:00049 D:000 O:00 N:000 Buffer:0240 - 0% step time = 2.9 ms C000: 35% step 0240:H0008b08-09 Day +00 08:00 2017-01-01 08:30:00: Rel.: 280: Active:00280 M:00218 S:00012 B:00050 D:000 O:00 N:000 Buffer:0280 - 0% step time = 3.2 ms C000: 39% step 0270:H0009b09-10 Day +00 09:00 2017-01-01 09:30:00: Rel.: 300: Active:00300 M:00224 S:00000 B:00076 D:000 O:00 N:000 Buffer:0300 - 0% step time = 2.9 ms C000: 43% step 0300:H0010b10-11 Day +00 10:00 2017-01-01 10:30:00: Rel.: 340: Active:00340 M:00225 S:00000 B:00115 D:000 O:00 N:000 Buffer:0340 - 0% step time = 3.3 ms C000: 48% step 0330:H0011b11-12 Day +00 11:00 2017-01-01 11:30:00: Rel.: 360: Active:00360 M:00215 S:00000 B:00145 D:000 O:00 N:000 Buffer:0360 - 0% step time = 2.9 ms C000: 52% step 0360:H0012b12-13 Day +00 12:00 2017-01-01 12:30:00: Rel.: 400: Active:00400 M:00242 S:00000 B:00158 D:000 O:00 N:000 Buffer:0400 - 0% step time = 3.2 ms C000: 57% step 0390:H0013b13-14 Day +00 13:00 2017-01-01 13:30:00: Rel.: 420: Active:00420 M:00293 S:00006 B:00121 D:000 O:00 N:000 Buffer:0420 - 0% step time = 3.0 ms C000: 61% step 0420:H0014b14-15 Day +00 14:00 2017-01-01 14:30:00: Rel.: 460: Active:00460 M:00299 S:00016 B:00145 D:000 O:00 N:000 Buffer:0460 - 0% step time = 3.2 ms C000: 65% step 0450:H0015b15-16 Day +00 15:00 2017-01-01 15:30:00: Rel.: 480: Active:00480 M:00284 S:00062 B:00134 D:000 O:00 N:000 Buffer:0480 - 0% step time = 3.1 ms C000: 70% step 0480:H0016b16-17 Day +00 16:00 2017-01-01 16:30:00: Rel.: 520: Active:00520 M:00271 S:00066 B:00183 D:000 O:00 N:000 Buffer:0520 - 0% step time = 3.4 ms C000: 74% step 0510:H0017b17-18 Day +00 17:00 2017-01-01 17:30:00: Rel.: 540: Active:00540 M:00220 S:00077 B:00243 D:000 O:00 N:000 Buffer:0540 - 0% step time = 3.0 ms C000: 78% step 0540:H0018b18-19 Day +00 18:00 2017-01-01 18:30:00: Rel.: 580: Active:00580 M:00326 S:00077 B:00177 D:000 O:00 N:000 Buffer:0580 - 0% step time = 3.6 ms C000: 83% step 0570:H0019b19-20 Day +00 19:00 2017-01-01 19:30:00: Rel.: 600: Active:00600 M:00324 S:00075 B:00201 D:000 O:00 N:000 Buffer:0600 - 0% step time = 3.2 ms C000: 87% step 0600:H0020b20-21 Day +00 20:00 2017-01-01 20:30:00: Rel.: 640: Active:00640 M:00379 S:00071 B:00190 D:000 O:00 N:000 Buffer:0640 - 0% step time = 3.5 ms C000: 91% step 0630:H0021b21-22 Day +00 21:00 2017-01-01 21:30:00: Rel.: 660: Active:00660 M:00411 S:00016 B:00233 D:000 O:00 N:000 Buffer:0660 - 0% step time = 3.2 ms C000: 96% step 0660:H0022b22-23 Day +00 22:00 2017-01-01 22:30:00: Rel.: 700: Active:00700 M:00349 S:00006 B:00345 D:000 O:00 N:000 Buffer:0700 - 0% step time = 3.4 ms C000: 100% step 0689:H0022b22-23 Day +00 22:58 2017-01-01 23:28:00: Rel.: 700: Active:00700 M:00368 S:00000 B:00332 D:000 O:00 N:000 Buffer:0700 - 0% step time = 3.8 ms C000: >>> Note: No open boundaries requested, as run_params["open_boundary_type"] = 0 C000: Hint: Requires list of open boundary nodes not in hydro model, eg for Schism this can be read from hgrid file to named in reader params and run_params["open_boundary_type"] = 1 C000: >>> Warning: When using a terminal velocity, ensure time step is small enough that vertical displacement is a small fraction of the water depth, ie vertical Courant number < 1 C000: - Triangle walk summary: Of 970,028 particles located 0, walks were too long and were retried, of these 0 failed after retrying and were discarded C000: -------------------------------------------------------------------------- C000: - Finished case number 0, param_test1 started: 2023-07-17 08:48:53.601451, ended: 2023-07-17 08:49:10.975352 C000: Elapsed time =0:00:17.373901 C000: -------------------------------------------------------------------------- main: >>> Note: run summary with case file names "param_test1_runInfo.json" main: >>> Note: output is in dir= e:H_Local_driveParticleTrackingoceantrackertutorials_how_tooutputparam_test1 main: >>> Note: to help with debugging, parameters as given by user are in "param_test1_raw_user_params.json" main: >>> Note: run summary with case file names "param_test1_runInfo.json" main: -------------------------------------------------------------------------- main: OceanTracker summary: elapsed time =0:00:17.529012 main: Cases - 0 errors, 1 warnings, 2 notes, check above main: Helper- 0 errors, 0 warnings, 0 notes, check above main: Main - 0 errors, 0 warnings, 3 notes, check above main: -------------------------------------------------------------------------- case file name= e:H_Local_driveParticleTrackingoceantrackertutorials_how_tooutputparam_test1param_test1_caseInfo.json
Basic plots of tracks¶
also see … for more on plotting notebook
# plot animation of results
from matplotlib import pyplot as plt
from oceantracker.post_processing.plotting.plot_tracks import animate_particles
from oceantracker.post_processing.read_output_files import load_output_files
from IPython.display import HTML # show animation in note book
# read particle track data into a dictionary using case_info_file_name
tracks = load_output_files.load_track_data(case_info_file_name)
ax= [1591000, 1601500, 5478500, 5491000] # area to plot
# animate particles
anim = animate_particles(tracks, axis_lims=ax,title='Fall vel.+ re-sus., grey part. are on bottom when flows too weak to resuspend',
show_dry_cells=True, show_grid=True, show=False) # use ipython to show video, rather than matplotlib plt.show()
# this is slow to build!
HTML(anim.to_html5_video())
