<commit_before># forecast

Parse specialized log file. Show histogram of tomorrow's maximum temperature
for the included ip addresses.

By default, random locations and forecasts are returned soas to not hit the ip-to-location and location-to-weather services too hard. Append '--live' to command to get real-time data.

A weather api will be needed, which may be acquired by signing up on https://openweathermap.com/

## Usage

There are two ways to run this:

- [memory version](doc/memory.md), uses a memory repository for holding the ip and location information,
- [aerospike version](doc/aero.md), uses an aerospike repository.
- `datascript version`, uses, well, a datascript repository.
Usage is identical to the memory version, except use `run-datascript.sh` in place of `run`.

## Logfiles

Three example logfiles are provided in the data directory:

- __logfile__, one invalid ip address, ten good addresses, and three duplicate ip addresses. Histogram should have temperatures for only the ten valid, unique addresses.
- __logfile2__, has the ten valid addresses from _logfile_ plus ten more. Processing _logfile_ and _logfile2_ multiple times should leave a histogram with 20 entries.
- __logfile-big__, has 4082 entries.

The logfile for this program (how meta) is data/log.log.

# How It Works

![how it works](how-it-works.jpg)

_Note: the following description explains the aerospike storage version,
but the memory and datascript repositories operate similarly._

Aerospike's `test` namespace contains two sets: `ip` and `location`.

1. `load-logfile.sh` extracts ip addresses from the log file.
For each discovered ip address, it checks to see if that ip address
has already been entered in the `ip` set, and inserts it if not
found with a `state` of "new".
<commit_msg>Correct image link.<commit_after># forecast

Parse specialized log file. Show histogram of tomorrow's maximum temperature
for the included ip addresses.

By default, random locations and forecasts are returned soas to not hit the ip-to-location and location-to-weather services too hard. Append '--live' to command to get real-time data.

A weather api will be needed, which may be acquired by signing up on https://openweathermap.com/

## Usage

There are two ways to run this:

- [memory version](doc/memory.md), uses a memory repository for holding the ip and location information,
- [aerospike version](doc/aero.md), uses an aerospike repository.
- `datascript version`, uses, well, a datascript repository.
Usage is identical to the memory version, except use `run-datascript.sh` in place of `run`.

## Logfiles

Three example logfiles are provided in the data directory:

- __logfile__, one invalid ip address, ten good addresses, and three duplicate ip addresses. Histogram should have temperatures for only the ten valid, unique addresses.
- __logfile2__, has the ten valid addresses from _logfile_ plus ten more. Processing _logfile_ and _logfile2_ multiple times should leave a histogram with 20 entries.
- __logfile-big__, has 4082 entries.

The logfile for this program (how meta) is data/log.log.

# How It Works

![how it works](doc/how-it-works.jpg)

_Note: the following description explains the aerospike storage version,
but the memory and datascript repositories operate similarly._

Aerospike's `test` namespace contains two sets: `ip` and `location`.

1. `load-logfile.sh` extracts ip addresses from the log file.
For each discovered ip address, it checks to see if that ip address
has already been entered in the `ip` set, and inserts it if not
found with a `state` of "new".

{'fraction_non_alphanumeric': 0.06748466257668712, 'fraction_numerical': 0.0050195203569436695, 'mean_word_length': 4.493108728943338, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2029852', 'n_tokens_mistral': 982, 'n_tokens_neox': 932, 'n_words': 504}