<jupyter_start><jupyter_text>Model skill assessment Simple comparisonSometimes all your need is a simple comparison of two time series. The `fmskill.compare()` method does just that.<jupyter_code>from mikeio import Dfs0
import fmskill<jupyter_output><empty_output><jupyter_text>The modelCan be either a dfs0 or a DataFrame.<jupyter_code>fn_mod = 'data/SW/ts_storm_4.dfs0'
df_mod = Dfs0(fn_mod).read(items=0).to_dataframe()<jupyter_output><empty_output><jupyter_text>The observationCan be either a dfs0, a DataFrame or a PointObservation object.<jupyter_code>fn_obs = 'data/SW/eur_Hm0.dfs0'<jupyter_output><empty_output><jupyter_text>compare()The compare() method will return an object that can be used for scatter plots, skill assessment, time series plots etc.<jupyter_code>c = fmskill.compare(fn_obs, df_mod)
c.plot_timeseries();<jupyter_output><empty_output><jupyter_text>Systematic vs random errors![](images/systematic_random_error.png)A model is an simplified version of a natural system, such as the ocean, and as such does not reflect every detail of the natural system.In order to validate if a model does capture the essential dynamics of the natural system, it can be helpful to classify the mismatch of the model and observations in two broad categories:* systematic errors* random errorsA quantitativate assesment of a model involves calculating one or more model score, skill metrics, which in varying degrees capture systematic errors, random errors or a combination. Metrics**Bias** is an indication of systematic error. In the left figure above, the model has negative bias (modelled wave heights are lower thatn observed). Thus it is an indication that the model can be improved.**Root Mean Square Error** (rmse) is a combination of systematic and random error. It is a common metric to indicate the quality of a calibrated model, but less useful to understand the potential for further calibration since it captures both systematic and random errors.**Unbiased Root Mean Square Error** (urmse) is the unbiased version of Root Mean Square Error. Since the bias is removed, it only captures the random error.For a complete list of possible metrics, see the [Metrics section in the FMSkill docs](https://dhi.github.io/fmskill/api.htmlmetrics). To get a quantitative model skill, we use the .skill() method, which returns a table (similar to a DataFrame).<jupyter_code>c.skill()<jupyter_output><empty_output><jupyter_text>The default is a number of common metrics, but you are free to pick your favorite metrics.<jupyter_code>c.skill(metrics=["mae","rho","lin_slope"])<jupyter_output><empty_output><jupyter_text>A very common way to visualize model skill is to use a scatter plot.The scatter plot includes some additional features such as a 2d histogram, a Q-Q line and a regression line, but the appearance is highly configurable.<jupyter_code>c.scatter()
c.scatter(binsize=0.5, 
          show_points=False,
          xlim=[0,6], ylim=[0,6],
          title="A calibrated model!")<jupyter_output><empty_output><jupyter_text>Taylor diagramA taylor diagram is a way to combine several statistics in a single plot, and is very useful to compare the skill of several models, or observations in a single plot.<jupyter_code>c.taylor()<jupyter_output><empty_output><jupyter_text>Elaborate comparison<jupyter_code>import numpy as np
from fmskill import PointObservation
from fmskill import ModelResult, Connector
fn = 'data/SW/HKZN_local_2017_DutchCoast.dfsu'
mr = ModelResult(fn, name='HKZN_local', item=0)
mr.dfs
o1 = PointObservation('data/SW/HKNA_Hm0.dfs0', item=0, x=4.2420, y=52.6887, name="HKNA")
o2 = PointObservation("data/SW/eur_Hm0.dfs0", item=0, x=3.2760, y=51.9990, name="EPL")
o1.hist();
o1.plot();<jupyter_output><empty_output><jupyter_text>Connecting<jupyter_code>con = Connector([o1, o2], mr)
con.observations
con.plot_observation_positions();
cc = con.extract()
cc
cc.skill().style(precision=2)
cc["EPL"].skill(metrics="mean_absolute_error")
cc["HKNA"].plot_timeseries(figsize=(10,5));
cc["EPL"].scatter(figsize=(8,8), show_hist=True) #, ylim=[2,8], xlim=[1,10])
cc["EPL"].hist(bins=20)
cc["HKNA"].scatter(binsize=0.25, cmap="viridis")
con.plot_temporal_coverage();<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09782093794410232, 'fraction_numerical': 0.016816674561819043, 'mean_word_length': 6.541071428571429, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '422900', 'license': 'MIT', 'n_tokens_mistral': 1372, 'n_tokens_neox': 1310, 'n_words': 442, 'path': 'mini_book/model_skill.ipynb', 'repo_name': 'DHI/book-learn-mikeio-fmskill', 'size': '4222'}