Output files#
[This note-book is in oceantracker/tutorials_how_to/]
After running OceanTacker, output files are in the files are the folder given by parameters ./“root_output_dir”/“output_file_base”. Note that the type of slashes is platform dependant (Linux & Mac: “/”, Windows: “”). Hint: Browsers (e.g. Firefox or Chrome) will display json files in expandable structure in case your editor doesn’t
The core files - that are present for any comleted run - are:
raw_users_params.json, a copy of the parameters as supplied by the user, useful in debugging or re-running.
caseInfo.json files, have all the output file names, plus information about the run and data useful in plotting. Eg.
full set of working parameters used to run the model
output_files, the list of all output files created by (path) names separated by type.
basic information about the hindcast, eg start date, end date, time step, …
basic information from each class used in the computational pipeline
run-time timings of parts of code to identify if and why the code might run slow
hindcast_info.json has a catalog of information about all hindcast file variables (dims, sizes etc.) plus which files hold each variable
run_log.txt has a copy of the screen output form the run
gridXXX.nc is a netcdf containing information on the hydrodynamical-model grids and other information, which is used in the plotting routines and sometimes useful in the analysis/post-processing
Optional output files that are written depending on what kind of model output you have been asking for are:
tracksXXX.nc holds the particle tracks in a netcdf file, see below for code example on reading the tracks
**stats_*.nc** contain the output of “on-the-fly” statistics (see notebook “G”)
**_events.nc** a netcdf output from events classes, which only writes output when events occur, eg. a particle entering or exiting given polygons.
Note, that all time variables are represented in either the “posix” format (counting seconds since 1970-01-01) or as a ISO 8601 datatime string (e.g. 2026-01-21T11:23:11-12:00) Distances are generally expressed in the hydrodynamical model unit. I.e. either in degrees longitude & latitude, or meters.
# Notes for debugging if the scripts below fail:
# * These scripts assume that you already installed oceantracker. If you didn't take a look at https://oceantracker.github.io/oceantracker/_build/html/info/installing.html
# * Paths in this directory are relative to the location of the ipython notebook.
# I.e. On Linux or Mac, running a cell with "!ls" should return a list containing the notebook you are running.
# to show show a list of output files after running minimal_example:
# Note:
# To make this tutorial run on all operating system we us the 'os' package to handle file paths
# If you don't care for that you can just use path/to/your/files as usual
import os
import glob
output_dir = os.path.join('output', 'minimal_example')
for f in glob.glob(os.path.join(output_dir, '*')):
print(f)
output/minimal_example/run_log.txt
output/minimal_example/raw_user_params.json
output/minimal_example/grid000.nc
output/minimal_example/hindcast_info.json
output/minimal_example/release_groups.nc
output/minimal_example/tracks_rectangular_000.nc
output/minimal_example/caseInfo.json
Reading particle tracks#
The below shows how to read the netcdf particle track output file into a python dictionary. The track file has a record of each of the particle properties, plus other useful information. The netcdf tracks file has a compact format, the below code reads this file into rectangular numpy [ time, particle] arrays. Key variables are :
tracks[‘x’]- the particle locations as a rectangular array of position vectors. With dimensions [ time, particle, vector component]. So that the 2D location is
x, y = tracks['x'][:,:,0], tracks['x'][:,:,1]
and for a 3D model the vertical position is z= tracks[‘x’][:,:,2]
tracks[‘time’] - array of time in seconds since 1970-1-1
tracks[‘date’] - array of dates as numpy datetime64[s]
tracks[‘status’] - the numerical codes of particle status, eg moving, dead etc as [ time, particle] array. The values of these status codes are also in the dictionary, eg tracks[‘status_stranded_by_tide’] = 3.
tracks[‘age’] - time series of each particles age, ie time since release in seconds.
tracks[‘IDrelease_group’] and tracks[‘IDpulse’] - id’s of which release group particles where released from and which pulse within that group. These are based indices and arrays of size particle, the total number of particles released during the run.
The index within the “particle” dimension, is the individual particleID, ordered from first to last release across the entire model run.
Note: For very large track files reading may fail, eg where variables exceed the 2-4Gb numpy array limit in Windows. To avoid this rerun using the tracks_writer setting “time_steps_per_per_file” to split the track file into files with given number of time steps.
# example of reading tracks file
# read netcdf into dictionary
import os
from oceantracker.read_output.python import read_ncdf_output_files
# here we will use the "read_ncdf_output_files" functions from oceantracker
# they help with some of the IO management (e.g. concatenating split track files)
# but what we show below could also be achieved using the netCDF4 library directly
# or for some slightly fancier output which is helpful for some initial data exploration
# I'd recommend using xarray's open_dataset()
output_dir = os.path.join('output', 'minimal_example')
tracks_file = os.path.join(output_dir, 'tracks_rectangular_000.nc')
tracks = read_ncdf_output_files.read_tracks_file(tracks_file)
print('Track data:')
print('===========')
_ = [print(item) for item in tracks.keys()]
loading oceantracker read files
prelim: Starting package set up
Reading rectangular track file "tracks_rectangular_000.nc"
Track data:
===========
variable_attributes
x
particles_written_per_time_step
x0
status_last_good
time_released
dry_cell_index
time_step_range
time
IDrelease_group
num_part_released_so_far
status
user_release_groupID
age
ID
hydro_model_gridID
water_depth
IDpulse
tide
global_attributes
dimensions
variables
date
# to read the hydro-dynamic grid file, useful in plotting
grid_file = os.path.join(output_dir, 'grid000.nc')
grid = read_ncdf_output_files.read_grid_file(grid_file)
print('Grid data:')
print('==========')
_ = [print(item) for item in grid.keys()]
Grid data:
==========
index_note
created
geographic_coords_used
node_typeID_interior
node_typeID_island_boundary
node_typeID_domain_boundary
node_typeID_open_boundary
node_typeID_land
x
triangles
triangle_area
adjacency
node_type
is_boundary_triangle
node_to_tri_map
tri_per_node
bc_transform
water_depth
domain_outline_nodes
domain_outline_x
domain_masking_polygon
island_outline_nodes
island_outline_nodes_packed_ranges
grid_outline
# to caseInfo contains the paths to all the output files,
# and in most cases it is sufficient to to use the load_tracks_data
# and to point it at the case info file.
# This will then load all tracks and grids which can then be plotted more easily
from oceantracker.read_output.python import load_output_files
import os
case_info_file = os.path.join('output', 'minimal_example', 'caseInfo.json')
tracks_plot = load_output_files.load_track_data(case_info_file)
_ = [print(item) for item in tracks_plot]
Merging rectangular track files
Reading rectangular track file "tracks_rectangular_000.nc"
particles_written_per_time_step
time_step_range
time
num_part_released_so_far
x
x0
status
status_last_good
age
ID
IDrelease_group
user_release_groupID
IDpulse
hydro_model_gridID
time_released
water_depth
tide
dry_cell_index
grid
particle_status_flags
particle_release_groups
axis_lim