<jupyter_start><jupyter_text>Load error measurements using `CurveMeasurements`. See `notebooks/measurements.ipynb` for more about reading error measurements.<jupyter_code>curvems = CurveMeasurements()
curvems.load_from_json('../data/no_pretr_ft.json')
print(curvems)<jupyter_output>--
num_train_samples: 25
test_errors: [74.04, 74.48, 74.07, 74.4, 75.95, 74.97, 74.02, 75.8, 74.6, 74.27]
num_ms: 10
--
num_train_samples: 50
test_errors: [57.59, 58.45, 58.13, 58.44, 59.46, 58.14, 58.71, 58.22]
num_ms: 8
--
num_train_samples: 100
test_errors: [45.01, 43.64, 45.06, 44.63]
num_ms: 4
--
num_train_samples: 200
test_errors: [35.36, 35.2]
num_ms: 2
--
num_train_samples: 400
test_errors: [27.92]
num_ms: 1
--<jupyter_text>Load default config. Modify `config.yaml` directly or update parameters once loaded.<jupyter_code>cfg = OmegaConf.load('../lc/config.yaml')
print('-'*20)
print('Default config')
print('-'*20)
print(OmegaConf.to_yaml(cfg))

cfg.gamma_search = False
print('-'*20)
print('Modified config')
print('-'*20)
print(OmegaConf.to_yaml(cfg))
curve_estimator = LearningCurveEstimator(cfg)
curve, objective = curve_estimator.estimate(curvems)
print('Quality of the fit:',objective)
curve.print_summary(cfg.N)<jupyter_output>Quality of the fit: 16.561
------------------------------
Learning curve summary
------------------------------
error_400: 28.2575
beta_400: 16.0257
gamma: -0.5
alpha: 12.2318
eta: 320.5148<jupyter_text>Searching for gamma leads to better fit. To enable gamma search set `gamma_search` to `True` (default). When gamma search is disabled, `curve_estimator.estimate()` uses `cfg.gamma` to estimate the curve.<jupyter_code>cfg.gamma_search = True
curve, objective = curve_estimator.estimate(curvems)
print('Quality of the fit:',objective)
curve.print_summary(cfg.N)<jupyter_output>Quality of the fit: 6.2808
------------------------------
Learning curve summary
------------------------------
error_400: 27.9133
beta_400: 18.2284
gamma: -0.41
alpha: 5.6835
eta: 259.2864<jupyter_text>Use `curve_estimator.plot()` to visualizes the learning curve and the error measurements.<jupyter_code>curve_estimator.plot(curve,curvems,label='No Pretr; Ft')<jupyter_output><empty_output><jupyter_text>You may also want to visualize the variance estimates. We recommend using the smoothed variance estimate but you can switch to using sample variance for curve estimation by setting `cfg.variance_type='sample'`. See `notebooks/variance.ipynb` for details on smooth variance estimation.<jupyter_code>curve_estimator.err_mean_var_estimator.visualize(curvems)<jupyter_output><empty_output><jupyter_text>Plot multiple curves for easy comparison.<jupyter_code>plot_metadata = [
    ['../data/no_pretr_linear.json','No Pretr; Lin','r','--'],
    ['../data/no_pretr_ft.json','No Pretr; Ft','g','-'],
    ['../data/pretr_linear.json','Pretr; Lin','b','--'],
    ['../data/pretr_ft.json','Pretr; Ft','m','-']
]
for (json_path,label,color,linestyle) in plot_metadata:
    curvems.load_from_json(json_path)
    curve, _ = curve_estimator.estimate(curvems)
    curve_estimator.plot(curve,curvems,label,color,linestyle)<jupyter_output><empty_output><jupyter_text>What if you don't have all recommended error measurements?Technically, it is possible to use just 2 training set sizes to estimate the learning curve but the results may be susceptible to high variance. For instance, here we estimate the learning curve using only measurements on training set sizes of 400 and 200. Note that below, we plot all error measurements and not just the ones used to estimate the curve.<jupyter_code>import copy
for (json_path,label,color,linestyle) in plot_metadata:
    curvems.load_from_json(json_path)
    curvems_filtered = copy.deepcopy(curvems)
    curvems_filtered.curvems = [errms for errms in curvems if errms.num_train_samples in [400,200]]
    curve, _ = curve_estimator.estimate(curvems_filtered)
    curve_estimator.plot(curve,curvems,label,color,linestyle)<jupyter_output><empty_output><jupyter_text>However, results improve considerably when using 3 training set sizes. Below, we estimate learning curve using measurements on training sets of sizes 400, 200, and 100.<jupyter_code>for (json_path,label,color,linestyle) in plot_metadata:
    curvems.load_from_json(json_path)
    curvems_filtered = copy.deepcopy(curvems)
    curvems_filtered.curvems = [errms for errms in curvems if errms.num_train_samples in [400,200,100]]
    curve, _ = curve_estimator.estimate(curvems_filtered)
    curve_estimator.plot(curve,curvems,label,color,linestyle)<jupyter_output><empty_output><jupyter_text>It may be possible to use much smaller training set sizes to compute learning curves as shown below. Note that the errors predicted by the curve at 200 and 400 training set sizes, which were not used to estimate the curve, are reasonably accurate.<jupyter_code>for (json_path,label,color,linestyle) in plot_metadata:
    curvems.load_from_json(json_path)
    curvems_filtered = copy.deepcopy(curvems)
    curvems_filtered.curvems = [errms for errms in curvems if errms.num_train_samples in [100,50,25]]
    curve, _ = curve_estimator.estimate(curvems_filtered)
    curve_estimator.plot(curve,curvems,label,color,linestyle)<jupyter_output><empty_output><jupyter_text>Quick and Lazy ApproachIt is possible to compute a decent learning curve with only 3 error measurments - 1 for each of full, half, and quarter dataset sizes. In this case, simply set `cfg.v_1` to a reasonable value and proceed as before. This is our recommended approach if you are in a rush. See `notebooks/basic_lazy_usage.ipynb` for more details on this quick and lazy approach.<jupyter_code>cfg.v_1 = 10
for (json_path,label,color,linestyle) in plot_metadata:
    curvems.load_from_json(json_path)
    curvems_filtered = copy.deepcopy(curvems)
    curvems_filtered.curvems = [errms for errms in curvems if errms.num_train_samples in [400,200,100]]
    for errms in curvems_filtered:
        # Through away all but 1 error measurment per train set size
        errms.test_errors = [errms.test_errors[0]]
        errms.num_ms = 1
    curve, _ = curve_estimator.estimate(curvems_filtered)
    curve_estimator.plot(curve,curvems,label,color,linestyle)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12229523962173425, 'fraction_numerical': 0.041993909280333384, 'mean_word_length': 5.172106824925816, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 38, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '3612', 'license': 'Apache-2.0', 'n_tokens_mistral': 2305, 'n_tokens_neox': 2033, 'n_words': 540, 'path': 'notebooks/curve.ipynb', 'repo_name': 'allenai/learning-curve', 'size': '6239'}