Running with helper class

[This note-book is in oceantracker/tutorials_how_to/]

Oceantracker is designed so that both coders and non-coders can get almost the same level of adaptability to their specific needs. Also to streamline running many cases across distributed computers, along with enabling online particle tracking as a service.

To achieve these capabilities, it uses parameters to both change the settings to meet the users needs, but also to build the computational pipeline for the users specific needs. This flexibility in the computational pipeline is achieved using “class” parameters.

Classes perform specific roles in the pipeline. Parameters tell Oceantracker which version of core classes the user wants to use for each role, or add optional classes. Eg. add multiple “release_groups” classes, which may release particles at points, or within a polygon, at different times, rates and locations, all within the same computational run.

Simply by changing the parameters, users can load their own version of a class, including the “solver” class which orchestrates the time stepping and operations by other classes. There are base classes for all the major roles within the computational pipeline. Classes can be adapted by inheriting the bases class, or one of its children, and overwriting some of the methods to alter how the class performs its role.

To make the ideas of settings and classes easier to adopt, the below uses methods of a helper class to add settings and classes to the pipeline. The notebook E_run_using_parameter_dictionaries.ipynb, shows how run directly from parameter dictionaries, which are built in code or read from a json or yaml file.

Parameters using helper

There are two types of parameters:

  1. Settings parameters

These are set with one or more calls to helper method, ot.setting(…), where … are keyword arguments. eg “time_step”, the model time step in seconds is set using, time_step= 1800.

There are default values for most settings. A full set of settings and their defaults is at all settings

  1. Class role parameters

Classes add specific tasks to the computational pipeline, eg. how to release particles, write output, particle suspension etc. These are added using helper method ot.add_class(class_role, ….). The first argument is a “class_role”. Each class has its own specific settings, which are set using keyword arguments of helper method. Eg. ot.add_class(‘reader’, input_dir= ‘my_hindcast_dir’, ….) adds a reader class and tells is it which folder contains the hindcast files.

A full set of classes and their default settings is at parameter_ref

Normally there must be a keyword argument “class_name” setting. This is used to import the required class into the computational pipeline, with its specific settings. This “class_name” is a string, used to import the class at setup (so is the same as that used to import any python class within a module).

The “class_name” setting is normally required. A few class_roles have a default class_name, eg. in minimal_example “release_groups” assumes a point_release, and the reader’s “class_name” is chosen by looking at variable names within the hindcast’s netcdf file.

There are two types of “class_role”.

When

  1. Only a single class is required, these have singular “class_role”, eg.”reader” and “solver”.

  2. One or more classes can be added. These have plural “class_role” (eg. can add multiple “release_groups”). Users must give a unique name to each one.

Documentation for all parameters/settings and their default values.

Add link here

Extend the minimal example

The below extends the minimal_example, it adds:

  • release at random locations within a polygon, at random water depths (the default)

  • a particle fall velocity

  • particles on the bottom are re-suspended if friction velocity exceeds a critical value.

# build parameters using helper class
from oceantracker.main import OceanTracker

ot = OceanTracker() # make an instance of the helper class

# one or more settings can be set by calls to os.settings
ot.settings(output_file_base='param_test1',# name used as base for output files
            root_output_dir='output' #  output is put in dir   'root_output_dir'/'output_file_base'
            )
# ot.settings can be used more than once to add more settings
ot.settings(time_step =120) #  2 min model time step as seconds

# add a compulsory reader class
# no class_name setting is required
# as will detect that it needs a a schism reader class
ot.add_class('reader', #  class_role is reader
            input_dir= '../demos/demo_hindcast',  # folder to search for hindcast files, sub-dirs will, by default, also be searched
            file_mask = 'demoHindcastSchism*.nc',    # the file mask of the hindcast files
            )

# add some release groups
# release_groups are one or more release groups
#     (ie locations where particles are released at the same times and locations)
# there must be at least one release group

# add  release locations from two points,
ot.add_class('release_groups',  #  class_role is release_groups
            name ='my_release_points1',
                # the required name setting, is used to refer to this release group internally anf in postprocsing
            points= [[1595000, 5482600],
                    [1599000, 5486200]],      # must be an N by 2 or 3 or list, convertible to a numpy array
                    release_interval= 3600,           # seconds between releasing particles
                    pulse_size = 10,                   # number of particles released each release_interval
                    # no class_name setting, so assumes a point release
                    )
# add a second release group, from random locations within a polygon
ot.add_class('release_groups', #  class_role is release_group
            name ='my_polygon_release1',
            class_name= 'oceantracker.release_groups.polygon_release.PolygonRelease', # use a polygon release
            # this time is a polygon , so below points are polygon cords
            points = [  [1597682.1237, 5489972.7479],
                        [1598604.1667, 5490275.5488],
                        [1598886.4247, 5489464.0424],
                        [1597917.3387, 5489000],
                        [1597300, 5489000],
                        [1597682.1237, 5489972.7479]
                    ],
            release_interval= 7200,    # seconds between releasing particles
            pulse_size = 20,                   # number of particles released each release_interval
            )

# alter default re-suspension class's default settings
ot.add_class('resuspension', critical_friction_velocity = .005) # only re-suspend particles if friction vel. exceeds this value

# add a class to modify the particle velocity
# velocity_modifiers are a set of velocities added to  water velocity given in the hydrodynamic model's
# here a fall velocity with a given  value is added to the computation
ot.add_class('velocity_modifiers',  #  class_role is velocity_modifiers
             name ='my_fall_velocity',
            class_name = 'oceantracker.velocity_modifiers.terminal_velocity.TerminalVelocity',
            value= -0.001, # mean terminal vel < 0 for falling
            # optionally variance can also be use to give each particle its own fall velocity
            )
helper: --------------------------------------------------------------------------
helper: Starting OceanTracker helper class

Running OceanTracker

There are several ways to run OceanTracker

  1. By coding

    • user “class helper” method to build parameters (as above) then run

    • build parameters dictionary in code then run

    • read parameter file and then run

  2. Without coding

    • run from command line with parameter file which is built by editing a json/yaml text file

Here we use the “class helper” method approach, the other ways to run directly using parameter dictionaries outlined in E_run_using_parameter_dictionaries.ipynb, add link…

Note:

There are many ways to run the code, eg. with IDE like Pycharm, Visual Studio Code. It can also, as here, be run in iPython notebooks. However the way notebooks are implemented can sometimes result in issues.

See … for more details

Below runs oceantracker using the helper class.

See all parameters

# first see the parameters build using the helper class instance
# this is a dictionary ot.params
print(ot.params)  # ugly!

# use json.dumps() to make it pretty
import json
print(json.dumps(ot.params, indent=4))

# lots of params are shown
# as filling a template given by from user callable main.param_template()
# null or None are optional ones which will use defaults
{'root_output_dir': 'output', 'add_date_to_run_output_dir': None, 'output_file_base': 'param_test1', 'time_step': 120, 'screen_output_time_interval': None, 'backtracking': None, 'run_as_depth_averaged': None, 'debug': None, 'minimum_total_water_depth': None, 'write_output_files': None, 'max_run_duration': None, 'max_particles': None, 'processors': None, 'max_warnings': None, 'use_random_seed': None, 'numba_function_cache_size': None, 'multiprocessing_case_start_delay': None, 'profiler': None, 'user_note': None, 'case_output_file_tag': None, 'write_tracks': None, 'z0': None, 'open_boundary_type': None, 'block_dry_cells': None, 'dispersion': {}, 'field_group_manager': {}, 'interpolator': {}, 'particle_group_manager': {}, 'reader': {'input_dir': '../demos/demo_hindcast', 'file_mask': 'demoHindcastSchism*.nc'}, 'resuspension': {'critical_friction_velocity': 0.005}, 'solver': {}, 'tracks_writer': {}, 'event_loggers': {}, 'fields': {}, 'particle_concentrations': {}, 'particle_properties': {}, 'particle_statistics': {}, 'release_groups': {'my_release_points1': {'points': [[1595000, 5482600], [1599000, 5486200]], 'release_interval': 3600, 'pulse_size': 10}, 'my_polygon_release1': {'class_name': 'oceantracker.release_groups.polygon_release.PolygonRelease', 'points': [[1597682.1237, 5489972.7479], [1598604.1667, 5490275.5488], [1598886.4247, 5489464.0424], [1597917.3387, 5489000], [1597300, 5489000], [1597682.1237, 5489972.7479]], 'release_interval': 7200, 'pulse_size': 20}}, 'status_modifiers': {}, 'time_varying_info': {}, 'trajectory_modifiers': {}, 'velocity_modifiers': {'my_fall_velocity': {'class_name': 'oceantracker.velocity_modifiers.terminal_velocity.TerminalVelocity', 'value': -0.001}}}
{
    "root_output_dir": "output",
    "add_date_to_run_output_dir": null,
    "output_file_base": "param_test1",
    "time_step": 120,
    "screen_output_time_interval": null,
    "backtracking": null,
    "run_as_depth_averaged": null,
    "debug": null,
    "minimum_total_water_depth": null,
    "write_output_files": null,
    "max_run_duration": null,
    "max_particles": null,
    "processors": null,
    "max_warnings": null,
    "use_random_seed": null,
    "numba_function_cache_size": null,
    "multiprocessing_case_start_delay": null,
    "profiler": null,
    "user_note": null,
    "case_output_file_tag": null,
    "write_tracks": null,
    "z0": null,
    "open_boundary_type": null,
    "block_dry_cells": null,
    "dispersion": {},
    "field_group_manager": {},
    "interpolator": {},
    "particle_group_manager": {},
    "reader": {
        "input_dir": "../demos/demo_hindcast",
        "file_mask": "demoHindcastSchism*.nc"
    },
    "resuspension": {
        "critical_friction_velocity": 0.005
    },
    "solver": {},
    "tracks_writer": {},
    "event_loggers": {},
    "fields": {},
    "particle_concentrations": {},
    "particle_properties": {},
    "particle_statistics": {},
    "release_groups": {
        "my_release_points1": {
            "points": [
                [
                    1595000,
                    5482600
                ],
                [
                    1599000,
                    5486200
                ]
            ],
            "release_interval": 3600,
            "pulse_size": 10
        },
        "my_polygon_release1": {
            "class_name": "oceantracker.release_groups.polygon_release.PolygonRelease",
            "points": [
                [
                    1597682.1237,
                    5489972.7479
                ],
                [
                    1598604.1667,
                    5490275.5488
                ],
                [
                    1598886.4247,
                    5489464.0424
                ],
                [
                    1597917.3387,
                    5489000
                ],
                [
                    1597300,
                    5489000
                ],
                [
                    1597682.1237,
                    5489972.7479
                ]
            ],
            "release_interval": 7200,
            "pulse_size": 20
        }
    },
    "status_modifiers": {},
    "time_varying_info": {},
    "trajectory_modifiers": {},
    "velocity_modifiers": {
        "my_fall_velocity": {
            "class_name": "oceantracker.velocity_modifiers.terminal_velocity.TerminalVelocity",
            "value": -0.001
        }
    }
}

Start run using helper

# now run oceantracker
# as helper "ot" has set params above, simply run it

case_info_file_name = ot.run()

# output now in folder "root_output_dir"/"output_file_base"
# in this example output is in directory  output/param_test1'

# case_info_file_name is the name of a json file with useful for post processing,
# eg it holds output file names to assist in reading and plotting data
print('case file name=',case_info_file_name)
helper:   - Starting run using helper class
main: --------------------------------------------------------------------------
main:  OceanTracker version 0.4.01.003 2023-07-14 - preliminary setup
main:      Python version: 3.10.10 | packaged by conda-forge | (main, Mar 24 2023, 20:00:38) [MSC v.1934 64 bit (AMD64)]
main:   - found hydro-model files of type SCHISIM
main:       -  sorted hyrdo-model files in time order,        0.087 sec
main:     >>> Note: output is in dir= e:H_Local_driveParticleTrackingoceantrackertutorials_how_tooutputparam_test1
main:     >>> Note: to help with debugging, parameters as given by user  are in "param_test1_raw_user_params.json"
C000: --------------------------------------------------------------------------
C000: Starting case number   0,  param_test1 at 2023-07-17T08:48:53.601451
C000: --------------------------------------------------------------------------
C000:       -  built node to triangles map,   0.669 sec
C000:       -  built triangle adjacency matrix,       0.235 sec
C000:       -  found boundary triangles,      0.000 sec
C000:       -  built domain and island outlines,      1.360 sec
C000:       -  calculated triangle areas,     0.000 sec
C000:   Finished grid setup
C000:       -  set up release_groups,         0.939 sec
C000:       -  built barycentric-transform matrix,    0.457 sec
C000:       -  initial set up of core classes,        0.476 sec
C000:       -  final set up of core classes,          0.000 sec
C000:       -  created particle properties derived from fields,       0.003 sec
C000: >>> Warning: When using a terminal velocity, ensure time step is small enough that vertical displacement is a small fraction of the water depth, ie vertical Courant number < 1
C000: >>> Note: No open boundaries requested, as run_params["open_boundary_type"] = 0
C000:       Hint: Requires list of open boundary nodes not in hydro model, eg for Schism this can be read from hgrid file to named in reader params and run_params["open_boundary_type"] = 1
C000: --------------------------------------------------------------------------
C000:   - Starting param_test1,  duration: 0 days 23 hrs 0 min 0 sec
C000:       -  Initialized Solver Class,      0.000 sec
C000:   - Reading-file-00  demoHindcastSchism3D.nc, steps in file  24, steps  available 000:023, reading  24 of 24 steps,  for hydo-model time steps 00:23,  from file offsets 00:23,  into ring buffer offsets 000:023
C000:       -  read  24 time steps in  0.5 sec
C000:   - opening tracks output to : param_test1_tracks_compact.nc
C000: 00% step 0000:H0000b00-01 Day +00 00:00 2017-01-01 00:30:00: Rel.:      40: Active:00040 M:00038 S:00000  B:00002 D:000 O:00 N:000 Buffer:0040 -  0% step time = 10518.2 ms
C000: 04% step 0030:H0001b01-02 Day +00 01:00 2017-01-01 01:30:00: Rel.:      60: Active:00060 M:00054 S:00000  B:00006 D:000 O:00 N:000 Buffer:0060 -  0% step time =  2.8 ms
C000: 09% step 0060:H0002b02-03 Day +00 02:00 2017-01-01 02:30:00: Rel.:     100: Active:00100 M:00089 S:00000  B:00011 D:000 O:00 N:000 Buffer:0100 -  0% step time =  3.3 ms
C000: 13% step 0090:H0003b03-04 Day +00 03:00 2017-01-01 03:30:00: Rel.:     120: Active:00120 M:00093 S:00011  B:00016 D:000 O:00 N:000 Buffer:0120 -  0% step time =  2.8 ms
C000: 17% step 0120:H0004b04-05 Day +00 04:00 2017-01-01 04:30:00: Rel.:     160: Active:00160 M:00125 S:00012  B:00023 D:000 O:00 N:000 Buffer:0160 -  0% step time =  3.3 ms
C000: 22% step 0150:H0005b05-06 Day +00 05:00 2017-01-01 05:30:00: Rel.:     180: Active:00180 M:00143 S:00015  B:00022 D:000 O:00 N:000 Buffer:0180 -  0% step time =  2.9 ms
C000: 26% step 0180:H0006b06-07 Day +00 06:00 2017-01-01 06:30:00: Rel.:     220: Active:00220 M:00165 S:00015  B:00040 D:000 O:00 N:000 Buffer:0220 -  0% step time =  3.2 ms
C000: 30% step 0210:H0007b07-08 Day +00 07:00 2017-01-01 07:30:00: Rel.:     240: Active:00240 M:00179 S:00012  B:00049 D:000 O:00 N:000 Buffer:0240 -  0% step time =  2.9 ms
C000: 35% step 0240:H0008b08-09 Day +00 08:00 2017-01-01 08:30:00: Rel.:     280: Active:00280 M:00218 S:00012  B:00050 D:000 O:00 N:000 Buffer:0280 -  0% step time =  3.2 ms
C000: 39% step 0270:H0009b09-10 Day +00 09:00 2017-01-01 09:30:00: Rel.:     300: Active:00300 M:00224 S:00000  B:00076 D:000 O:00 N:000 Buffer:0300 -  0% step time =  2.9 ms
C000: 43% step 0300:H0010b10-11 Day +00 10:00 2017-01-01 10:30:00: Rel.:     340: Active:00340 M:00225 S:00000  B:00115 D:000 O:00 N:000 Buffer:0340 -  0% step time =  3.3 ms
C000: 48% step 0330:H0011b11-12 Day +00 11:00 2017-01-01 11:30:00: Rel.:     360: Active:00360 M:00215 S:00000  B:00145 D:000 O:00 N:000 Buffer:0360 -  0% step time =  2.9 ms
C000: 52% step 0360:H0012b12-13 Day +00 12:00 2017-01-01 12:30:00: Rel.:     400: Active:00400 M:00242 S:00000  B:00158 D:000 O:00 N:000 Buffer:0400 -  0% step time =  3.2 ms
C000: 57% step 0390:H0013b13-14 Day +00 13:00 2017-01-01 13:30:00: Rel.:     420: Active:00420 M:00293 S:00006  B:00121 D:000 O:00 N:000 Buffer:0420 -  0% step time =  3.0 ms
C000: 61% step 0420:H0014b14-15 Day +00 14:00 2017-01-01 14:30:00: Rel.:     460: Active:00460 M:00299 S:00016  B:00145 D:000 O:00 N:000 Buffer:0460 -  0% step time =  3.2 ms
C000: 65% step 0450:H0015b15-16 Day +00 15:00 2017-01-01 15:30:00: Rel.:     480: Active:00480 M:00284 S:00062  B:00134 D:000 O:00 N:000 Buffer:0480 -  0% step time =  3.1 ms
C000: 70% step 0480:H0016b16-17 Day +00 16:00 2017-01-01 16:30:00: Rel.:     520: Active:00520 M:00271 S:00066  B:00183 D:000 O:00 N:000 Buffer:0520 -  0% step time =  3.4 ms
C000: 74% step 0510:H0017b17-18 Day +00 17:00 2017-01-01 17:30:00: Rel.:     540: Active:00540 M:00220 S:00077  B:00243 D:000 O:00 N:000 Buffer:0540 -  0% step time =  3.0 ms
C000: 78% step 0540:H0018b18-19 Day +00 18:00 2017-01-01 18:30:00: Rel.:     580: Active:00580 M:00326 S:00077  B:00177 D:000 O:00 N:000 Buffer:0580 -  0% step time =  3.6 ms
C000: 83% step 0570:H0019b19-20 Day +00 19:00 2017-01-01 19:30:00: Rel.:     600: Active:00600 M:00324 S:00075  B:00201 D:000 O:00 N:000 Buffer:0600 -  0% step time =  3.2 ms
C000: 87% step 0600:H0020b20-21 Day +00 20:00 2017-01-01 20:30:00: Rel.:     640: Active:00640 M:00379 S:00071  B:00190 D:000 O:00 N:000 Buffer:0640 -  0% step time =  3.5 ms
C000: 91% step 0630:H0021b21-22 Day +00 21:00 2017-01-01 21:30:00: Rel.:     660: Active:00660 M:00411 S:00016  B:00233 D:000 O:00 N:000 Buffer:0660 -  0% step time =  3.2 ms
C000: 96% step 0660:H0022b22-23 Day +00 22:00 2017-01-01 22:30:00: Rel.:     700: Active:00700 M:00349 S:00006  B:00345 D:000 O:00 N:000 Buffer:0700 -  0% step time =  3.4 ms
C000: 100% step 0689:H0022b22-23 Day +00 22:58 2017-01-01 23:28:00: Rel.:     700: Active:00700 M:00368 S:00000  B:00332 D:000 O:00 N:000 Buffer:0700 -  0% step time =  3.8 ms
C000: >>> Note: No open boundaries requested, as run_params["open_boundary_type"] = 0
C000:       Hint: Requires list of open boundary nodes not in hydro model, eg for Schism this can be read from hgrid file to named in reader params and run_params["open_boundary_type"] = 1
C000: >>> Warning: When using a terminal velocity, ensure time step is small enough that vertical displacement is a small fraction of the water depth, ie vertical Courant number < 1
C000:   -  Triangle walk summary: Of  970,028 particles located  0, walks were too long and were retried,  of these  0 failed after retrying and were discarded
C000: --------------------------------------------------------------------------
C000:   - Finished case number   0,  param_test1 started: 2023-07-17 08:48:53.601451, ended: 2023-07-17 08:49:10.975352
C000:       Elapsed time =0:00:17.373901
C000: --------------------------------------------------------------------------
main:     >>> Note: run summary with case file names   "param_test1_runInfo.json"
main:     >>> Note: output is in dir= e:H_Local_driveParticleTrackingoceantrackertutorials_how_tooutputparam_test1
main:     >>> Note: to help with debugging, parameters as given by user  are in "param_test1_raw_user_params.json"
main:     >>> Note: run summary with case file names   "param_test1_runInfo.json"
main: --------------------------------------------------------------------------
main: OceanTracker summary:  elapsed time =0:00:17.529012
main:       Cases -   0 errors,   1 warnings,   2 notes, check above
main:       Helper-   0 errors,   0 warnings,   0 notes, check above
main:       Main  -   0 errors,   0 warnings,   3 notes, check above
main: --------------------------------------------------------------------------
case file name= e:H_Local_driveParticleTrackingoceantrackertutorials_how_tooutputparam_test1param_test1_caseInfo.json

Basic plots of tracks

also see … for more on plotting notebook

# plot animation of results
from matplotlib import pyplot as plt
from oceantracker.post_processing.plotting.plot_tracks import animate_particles
from oceantracker.post_processing.read_output_files import  load_output_files
from IPython.display import HTML # show animation in note book

# read particle track data into a dictionary using case_info_file_name
tracks = load_output_files.load_track_data(case_info_file_name)

ax= [1591000, 1601500, 5478500, 5491000]  # area to plot
# animate particles
anim = animate_particles(tracks, axis_lims=ax,title='Fall vel.+  re-sus., grey part. are on bottom when flows too weak to resuspend',
                         show_dry_cells=True, show_grid=True, show=False) # use ipython to show video, rather than matplotlib plt.show()

# this is slow to build!
HTML(anim.to_html5_video())
info/how_to/B_running_with_helper_class_files%5CB_running_with_helper_class_7_1.png