<jupyter_start><jupyter_text>Get started with TomateA little easily reproducible example to show some of Tomate features: loading data from a file, plotting it, doing some computations, and writing data to disk.This is more a showcase than a proper tutorial, reading the [documentation](https://tomate.readthedocs.io) is always recommended :).I am using the Sea Surface Temperature from this dataset https://podaac.jpl.nasa.gov/dataset/AMSRE_L3_SST_1DEG_1MO,you can get the same file by running:`wget https://podaac-opendap.jpl.nasa.gov/opendap/allData/modis/L3/aqua/11um/v2014.0/9km/monthly/2019/A20191822019212.L3m_MO_NSST_sst_9km.nc` Creating the databaseWe have a single NetCDF file, we can use `nc.scan_file` to create our database. For more realistic setups with multiple files, some more workis necessary: see the [tutorial](https://tomate.readthedocs.io/en/latest/tutorial.html).<jupyter_code>import matplotlib.pyplot as plt
%matplotlib inline

from tomate.scan_library.nc import scan_file
db = scan_file("A20191822019212.L3m_MO_NSST_sst_9km.nc")

print(db)<jupyter_output>Data object
Class: Data, Bases: DataDisk, DataCompute, DataPlot 

Variables:
palette [rgb, eightbitcolor] 
qual_sst (VariableMasked)  [lat, lon] 
sst (VariableMasked)  [lat, lon] 

Data available: 
Scope: available
var: palette, qual_sst, sst [3]
lat: -89.96 - 89.96 [2160]
lon: -179.96 - 179.96 [4320]
rgb: 0.00 - 2.00 [3]
eightbitcolor: 0.00 - 255.00 [256]

Data not loaded

No data selected

1 Filegroups:
	: palette, qual_sst, sst<jupyter_text>This has printed some information about the database: the type of database, the variables object available, and what part of the data is available, loaded or selected.We also have access to data and variables attributes through the VariablesInfo object (or VI):<jupyter_code>print(db.vi, '\n')

print(db.vi.date_created)
print(db.vi['sst'].units)<jupyter_output>Variables: palette, qual_sst, sst
Attributes: display_scale, display_min, display_max, long_name, scale_factor, _FillValue, valid_min, standard_name, _var_class, add_offset, datatype, valid_max, units
Infos: product_name, instrument, title, project, platform, temporal_range, processing_version, date_created, history, l2_flag_names, time_coverage_start, time_coverage_end, start_orbit_number, end_orbit_number, map_projection, latitude_units, longitude_units, northernmost_latitude, southernmost_latitude, westernmost_longitude, easternmost_longitude, geospatial_lat_max, geospatial_lat_min, geospatial_lon_max, geospatial_lon_min, latitude_step, longitude_step, sw_point_latitude, sw_point_longitude, geospatial_lon_resolution, geospatial_lat_resolution, geospatial_lat_units, geospatial_lon_units, spatialResolution, number_of_lines, number_of_columns, measure, suggested_image_scaling_minimum, suggested_image_scaling_maximum, suggested_image_scaling_type, suggested_image_scaling_applied, _lastM[...]<jupyter_text>Loading and plotting some dataNow we load everything from the file. We then plot the variable 'sst'.<jupyter_code>db.load()
fig, ax = plt.subplots()
im = db.imshow(ax, 'sst')<jupyter_output><empty_output><jupyter_text>We reload only a subpart of the data from the file, with longitudes ranging from 60W to 10E, and latitudes from 50S to 50N.Note this discard and replaces the data previously loaded.Again, we plot what we just loaded.<jupyter_code>db.load_by_value(lon=slice(-60, 10), lat=slice(-50, 50))
fig, ax = plt.subplots()

im = db.imshow(ax, 'sst')
im.add_colorbar()
im.set_labels()<jupyter_output><empty_output><jupyter_text>Let's compute the mean of the variable 'sst' over a small region. For that, we need to find the indices corresponding to that region. The `subset` function will find all indices between two values.<jupyter_code>lat_slice = db.loaded.lat.subset(-30., -15.)
lon_slice = db.loaded.lon.subset(8., 25.)
mean = db.mean('sst', dims=['lon', 'lat'], lat=lat_slice, lon=lon_slice)

print(mean)<jupyter_output>18.433271<jupyter_text>We can also directly plot an averaged variable, let's plot the meridional average of the SST. This is very useful for Hovmöller plots (for those use `imshow_avg`)!<jupyter_code>fig, ax = plt.subplots()

line = db.plot_avg(ax, 'sst', avg_dims=['lon'], axes=['sst', 'lat'])
line.set_labels()<jupyter_output><empty_output><jupyter_text>Viewing dataOnce loaded, the data array is directly accessible for each variable as:<jupyter_code>sst = db['sst'].data
sst = db['sst'][:]<jupyter_output><empty_output><jupyter_text>But you can also retrieve parts of the data using the `view` function, from a variable, or the database object itself (to retrieve different variables at once).<jupyter_code>sst = db['sst'].view(lat=slice(0, 10), lon=slice(0, 40, 2))
print(sst.shape)

sst = db.view(var='sst', lat=slice(0, 10), lon=slice(0, 40, 2))
print(sst.shape)

array = db.view(var=['sst', 'qual_sst'], lat=slice(0, 10), lon=slice(0, 40, 2),
                stack='force')
print(array.shape)<jupyter_output>(10, 20)
(10, 20)
(2, 10, 20)<jupyter_text>`view_by_value` does the same thing, but select by value and not index.<jupyter_code>sst = db.view_by_value('sst', lat=slice(-10, 10), lon=[-40, -35, -30, -25])
print(sst.shape)<jupyter_output>(242, 4)<jupyter_text>Selecting dataYou can similarly select data, eventually change the selection and use it in some functions.<jupyter_code># Do the selection
db.select(var='sst', lat=slice(0, 100))

# Reduce the selection
db.selected.slice(lon=slice(0, 50))

# Expand the selection
db.add_to_selection(var='qual_sst')

# Use the selection
sst = db.view_selected(var='sst')
print(sst.shape)<jupyter_output>(100, 50)<jupyter_text>Again, we can select by value. Note that `*_by_value` functions can receive value.s, ranges of values, **and** indices by  appending `_idx` to the name of the dimension.<jupyter_code>db.select_by_value(var='sst', lat=slice(-50., -41.8), lon_idx=slice(0, 50))
print(db.view_selected().shape)<jupyter_output>(100, 50)<jupyter_text>Adding variablesLet's add another variable. This will just be the SST with different units.<jupyter_code>from tomate.var_types import VariableMasked

db.add_variable('sst_k', dims=['lat', 'lon'], var_class=VariableMasked, attrs={'units': 'degree_K'})

db['sst_k'] = db['sst'][:] + 273.15<jupyter_output>INFO:tomate:Allocating [1202, 842] of type None for sst_k<jupyter_text>Writing to diskWe can now write our new variable to disk (along with old variable).We need first to add some information for this new variable to the `Filegroup` object, which takes care of data loading and writing.<jupyter_code>db.filegroups[0].add_variable('sst_k')
db.write('new_file.nc', file_kw={'clobber': True})<jupyter_output>INFO:tomate:Opening new_file.nc
INFO:tomate:Laying eightbitcolor values, extent 0.00 - 255.00
INFO:tomate:Laying rgb values, extent 0.00 - 2.00
INFO:tomate:Laying lat values, extent -50.04 - 50.04
INFO:tomate:Laying lon values, extent -60.04 - 10.04
INFO:tomate:Inserting variable palette
INFO:tomate:Taking keys from palette: [:, :]
INFO:tomate:Placing it in file at [:, :].
INFO:tomate:Inserting variable qual_sst
INFO:tomate:Taking keys from qual_sst: [:, :]
INFO:tomate:Placing it in file at [:, :].
INFO:tomate:Inserting variable sst
INFO:tomate:Taking keys from sst: [:, :]
INFO:tomate:Placing it in file at [:, :].
INFO:tomate:Inserting variable sst_k
INFO:tomate:Taking keys from sst_k: [:, :]
INFO:tomate:Placing it in file at [:, :].
{'fraction_non_alphanumeric': 0.11720226843100189, 'fraction_numerical': 0.03294625978935998, 'mean_word_length': 6.57361963190184, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 44, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '329329', 'license': 'MIT', 'n_tokens_mistral': 2804, 'n_tokens_neox': 2567, 'n_words': 716, 'path': 'examples/get_started.ipynb', 'repo_name': 'Descanonge/tomate', 'size': '7406'}