Implement scores model-obs RMSE - #2373
Conversation
e6aa6f8 to
e9617d9
Compare
Total coverage: 93% (HTML report)Name Stmts Miss Branch BrPart Cover --------------------------------------------------------------------------------------------------- src/CSET/__init__.py 105 0 14 0 100% src/CSET/_common.py 156 0 54 0 100% src/CSET/cset_workflow/app/fetch_fcst/bin/fetch_data.py 117 28 26 0 78% src/CSET/cset_workflow/app/fetch_nimrod/bin/fetch_nimrod.py 81 8 28 11 83% src/CSET/cset_workflow/app/finish_website/bin/finish_website.py 79 2 8 2 95% src/CSET/cset_workflow/app/parbake_recipes/bin/parbake.py 29 0 8 0 100% src/CSET/cset_workflow/app/send_email/bin/send_email.py 25 0 4 0 100% src/CSET/cset_workflow/lib/python/jinja_utils.py 17 0 6 0 100% src/CSET/extract_workflow.py 103 1 26 1 98% src/CSET/graph.py 44 0 14 0 100% src/CSET/operators/__init__.py 89 0 26 0 100% src/CSET/operators/_atmospheric_constants.py 9 0 0 0 100% src/CSET/operators/_colormaps.py 249 3 72 4 98% src/CSET/operators/_stash_to_lfric.py 3 0 0 0 100% src/CSET/operators/_utils.py 190 8 72 6 95% src/CSET/operators/ageofair.py 142 7 64 5 94% src/CSET/operators/aggregate.py 77 1 22 1 98% src/CSET/operators/aviation.py 61 0 18 0 100% src/CSET/operators/collapse.py 155 8 72 3 93% src/CSET/operators/constraints.py 113 7 50 3 93% src/CSET/operators/convection.py 38 4 10 2 88% src/CSET/operators/ensembles.py 27 0 14 0 100% src/CSET/operators/feature.py 44 0 10 0 100% src/CSET/operators/filters.py 67 2 30 0 98% src/CSET/operators/fluxes.py 41 0 10 0 100% src/CSET/operators/humidity.py 135 0 52 0 100% src/CSET/operators/imageprocessing.py 57 0 16 0 100% src/CSET/operators/mesoscale.py 18 0 2 0 100% src/CSET/operators/misc.py 172 1 72 3 98% src/CSET/operators/plot.py 1117 155 412 71 83% src/CSET/operators/power_spectrum.py 98 3 30 3 95% src/CSET/operators/precipitation.py 204 2 92 2 99% src/CSET/operators/pressure.py 41 0 12 0 100% src/CSET/operators/read.py 430 21 182 16 94% src/CSET/operators/regrid.py 147 1 70 3 98% src/CSET/operators/scoreswrappers.py 181 16 52 6 90% src/CSET/operators/temperature.py 121 0 32 0 100% src/CSET/operators/transect.py 63 0 24 0 100% src/CSET/operators/wind.py 46 3 10 2 91% src/CSET/operators/write.py 15 0 6 0 100% src/CSET/recipes/__init__.py 104 0 28 0 100% src/CSET/sample_data/__init__.py 0 0 0 0 100% --------------------------------------------------------------------------------------------------- TOTAL 5010 281 1750 144 93% |
Some example plots showing RMSE collapses in the temporal coordinate for each model, and one plot collapsed in the spatial coordinates. |
James Warner (jwarner8)
left a comment
James Warner (jwarner8)
left a comment
There was a problem hiding this comment.
Great stuff, few suggestions/possible optimisations, mostly for readability and documentation
Main things I'd suggest would be that new tests need to be added for new scoreswrappers operators, and also rename recipe to remove 'difference' , as these plots do not show the difference in RMSE skill between models. Instead, RMSE is calculated independently for each model against the observations, and each model's RMSE is plotted as a separate line on the same figure. The output therefore represents model-versus-observation scores, not inter-model score differences.
| return cube | ||
|
|
||
|
|
||
| def spatial_pcolormesh_plot_multi_model( |
There was a problem hiding this comment.
Could we not modify spatial_pcolormesh_plot and allow it to take cube or cubelist, and just loop over
_spatial_plot( "pcolormesh", cube, filename, sequence_coordinate, stamp_coordinate, **kwargs )
if cubelist? Saves writing another function
There was a problem hiding this comment.
I do actually feel like there's an argument in favour of splitting functions like this. This is a fairly trivial case but to me it feels cleaner and easier to read i.e the logic in each function is much simpler, rather than introducing conditionals. Regardless, I did make the change in my local branch but just pushing back a bit and maybe something to think about going forward as a lot of the functions in cset are multipurpose, which can be confusing!
| @@ -0,0 +1,54 @@ | |||
| category: Surface Observation Time Series | |||
| title: "$VARNAME scores rmse spatial plot for observation points $SUBAREA_NAME" | |||
| description: Extracts and plot a spatial plot of $VARNAME for all times based on mean of matched model and observation points. | |||
There was a problem hiding this comment.
Good to have some brief description of rmse, link to scores here, as this is what the user will see on the webpage. You could probably copy docstring in scoreswrappers function for this
Co-authored-by: James Warner <62252918+jwarner8@users.noreply.github.com>
Co-authored-by: James Warner <62252918+jwarner8@users.noreply.github.com>
Co-authored-by: James Warner <62252918+jwarner8@users.noreply.github.com>
James Warner (jwarner8)
left a comment
James Warner (jwarner8)
left a comment
There was a problem hiding this comment.
Looks good, minor comments
| @@ -0,0 +1,77 @@ | |||
| category: Surface Model vs Observation | |||
| title: "$VARNAME scores rmse spatial plot for model vs observation points $SUBAREA_NAME" | |||
| description: Extracts and plot a spatial plot of the RMSE of $VARNAME for all times based on mean of matched model and observation points. | |||
There was a problem hiding this comment.
| description: Extracts and plot a spatial plot of the RMSE of $VARNAME for all times based on mean of matched model and observation points. | |
| description: | | |
| Extracts and plot a spatial plot of the RMSE of $VARNAME for all times based on mean of matched model and observation points. |
There was a problem hiding this comment.
follows syntax of other recipes (e.g. https://github.com/MetOffice/CSET/blob/main/src/CSET/recipes/verification/surface_difference_scores_RMSE.yaml), otherwise it gets bunched together in the web.
2 participants
Contribution checklist
Aim to have all relevant checks ticked off before merging. See the developer's guide for more detail.