<story><title>Historical weather data API for machine learning, free for non-commercial</title><url>https://openmeteo.substack.com/p/historical-weather-for-machine-learning</url></story><parent_chain></parent_chain><comment><author>meteo-jeff</author><text>Some technical background:&lt;p&gt;Open-Meteo offers free weather APIs for a while now. Archiving data was not an option, because forecast data alone required 300 GB storage.&lt;p&gt;In the past couple of weeks, I started to look for fast and efficient compression algorithms like zstd, brotli or lz4. All of them, performed rather poor with time-series weather data.&lt;p&gt;After a lot of trial and error, I found a couple of pre-processing steps, that improve compression ratio a lot:&lt;p&gt;1) Scaling data to reasonable values. Temperature has an accuracy of 0.1° at best. I simply round everything to 0.05 instead of keeping the highest possible floating point precision.&lt;p&gt;2) A temperature time-series increases and decreases by small values. 0.4° warmer, then 0.2° colder. Only storing deltas improves compression performance.&lt;p&gt;3) Data are highly spatially correlated. If the temperature is rising in one &amp;quot;grid-cell&amp;quot;, it is rising in the neighbouring grid cells as well. Simply subtract the time-series from one grid-cell to the next grid-cell. Especially this yielded a large boost.&lt;p&gt;4) Although zstd performs quite well with this encoded data, other integer compression algorithms have far better compression and decompression speeds. Namely I am using FastPFor.&lt;p&gt;With that compression approach, an archive became possible. One week of weather forecast data should be around 10 GB compressed. With that, I can easily maintain a very long archive.</text></comment>
<story><title>Historical weather data API for machine learning, free for non-commercial</title><url>https://openmeteo.substack.com/p/historical-weather-for-machine-learning</url></story><parent_chain></parent_chain><comment><author>m0llusk</author><text>This is how we could defeat a rogue AI:  Distract it by talking about the weather.</text></comment>