<jupyter_start><jupyter_text>Introduction to Stream DatabasesStream databases are the building block of data sets in Sentenai. A Stream database represent something as simple as a single data feed, or as complex as an entire production line on a factory floor. Data streams in a stream database are organized into a tree-like hierarchy of relationships. This hierarchy is purely organizational in nature, but splitting complex flat data sets into a hierarchical organization can have substantial performance benefits. Here's an example of a flat dataset called "Activity" like you'd typically see in a relational database:| Timestamp | Location | Sensor | Temperature | Humidity| --- | --- | --- | --- | --- || 2020-01-01T00:00:00Z | Boston | 53 | 48 | .83| 2020-01-01T00:00:00Z | Providence | 48 | 43 | .55In Sentenai you might instead arrange your dataset:```- Activity  - Boston    - 53      - Temperature      - Humidity  - Providence    - 48      - Temperature      - Humidity```This organization implies there are natural filters you'd want to apply:`Activity/Boston/53/Temperature` is equivalent to filtering a table on Location and Sensor id.  Managing data in a Stream DatabaseStream databases feature transactional data logging. Any streams within the stream graph can be updated in a single transaction, provided the update applies to the same time interval across all updated streams.So if we initialize a new database `test-2':<jupyter_code>db = sentenai.init('test-2')<jupyter_output><empty_output><jupyter_text>We can update the entire database with each insertion without requiring any up-front schema declaration of stream instantiation:<jupyter_code>db.log[datetime(2020, 1, 1): datetime(2020, 1, 1, 1): 'update-1'] = {
    'Boston': {
        '53': {
            'Temperature': 48.0,
            'Humidity': 0.83,
        },
    },
    'Providence': {
        '48': {
            'Temperature': 43.0,
            'Humidity': 0.55,
        }
    }
}<jupyter_output><empty_output><jupyter_text>Now that we've inserted our first update into the database's `log`, we can see for ourselves the structure using the `.graph` property of the database.<jupyter_code>db.graph.show()<jupyter_output>test-2
├── Boston
│   └── 53
│       ├── Humidity
│       └── Temperature
└── Providence
    └── 48
        ├── Humidity
        └── Temperature<jupyter_text>Each node in this tree is a stream of either events or values. A stream of values is just a stream of events with a value attached, so in essence every node in the tree is the same.  Inserting bulk data In addition to the basic pattern of insertion into the log with `[ start : (optional) end : (optional) id ]`, you can also stream large amounts of data over a single connection using Python's `with` keyword and a streaming context manager. To start up the streaming connection, type:```with db.log as log:```This creates a new streaming connection, called `log`, for us to use to insert data. This connection can actively manage asynchronous uploading and buffering data, so you can feel free to iterate over very large files without worrying about them being loaded fully into memory. Here's a basic example:<jupyter_code>import random
with db.log as log:
    for i in range(50):
        db.log[datetime(2020, 1, 1, 2, i) : : ] = {
            'Boston': {'53': {
                'Humidity' : random.random(),
                'Temperature': random.random() * 100
            }}}<jupyter_output><empty_output><jupyter_text>Retrieving an update by unique idUpdates can be retrieved by id:<jupyter_code>db.log['update-1']<jupyter_output><empty_output><jupyter_text>Deleting an updateIf an individual update has been given a unique id, it can be deleted from the log by its id:<jupyter_code>del db.log['update-1']
try:
    print(db.log['update-1'])
except KeyError as k:
    print(k)<jupyter_output>'update not found.'<jupyter_text>Retrieving updates by timeYou can retrieve a set of updates by time slice: `db.log[ start : end : limit ]`. All arguments are optional. To retrieve all updates you can do `db.log[:]`. To get the first 5 updates, do:<jupyter_code>for x in db.log[ : : 5]:
    print(x)<jupyter_output>{'start': numpy.datetime64('2020-01-01T02:00:00'), 'end': None, 'data': {'Boston': {'53': {'Humidity': 0.9294579174551353, 'Temperature': 22.32584915785809}}}}
{'start': numpy.datetime64('2020-01-01T02:01:00'), 'end': None, 'data': {'Boston': {'53': {'Humidity': 0.6331413173773333, 'Temperature': 81.71983424155601}}}}
{'start': numpy.datetime64('2020-01-01T02:01:00'), 'end': None, 'data': {'Boston': {'53': {'Humidity': 0.030784589936583617, 'Temperature': 96.44885425146505}}}}
{'start': numpy.datetime64('2020-01-01T02:01:00'), 'end': None, 'data': {'Boston': {'54': {'Humidity': 55}}}}
{'start': numpy.datetime64('2020-01-01T02:02:00'), 'end': None, 'data': {'Boston': {'53': {'Humidity': 0.3031839372221987, 'Temperature': 23.21499324972436}}}}<jupyter_text>Negative limit values reverse the retrieval order. To get the last five updates, you can do:<jupyter_code>for x in db.log[ : : -5]:
    print(x)<jupyter_output>{'start': numpy.datetime64('2020-01-01T02:49:00'), 'end': None, 'data': {'Boston': {'53': {'Humidity': 0.7662644674096308, 'Temperature': 93.03395578163271}}}}
{'start': numpy.datetime64('2020-01-01T02:49:00'), 'end': None, 'data': {'Boston': {'53': {'Humidity': 0.5353517813206717, 'Temperature': 8.141972824783416}}}}
{'start': numpy.datetime64('2020-01-01T02:48:00'), 'end': None, 'data': {'Boston': {'53': {'Humidity': 0.04041089532843878, 'Temperature': 94.67279770679082}}}}
{'start': numpy.datetime64('2020-01-01T02:48:00'), 'end': None, 'data': {'Boston': {'53': {'Humidity': 0.10329406728415713, 'Temperature': 77.18615457054389}}}}
{'start': numpy.datetime64('2020-01-01T02:47:00'), 'end': None, 'data': {'Boston': {'53': {'Humidity': 0.16096466815504118, 'Temperature': 90.58414373050245}}}}<jupyter_text>Properties of a Stream DatabaseA stream database has several properties that are useful when writing programs.<jupyter_code># Name
print(db.name)
# Origin
db.origin
# number of updates
len(db)<jupyter_output><empty_output><jupyter_text>Accessing Streams in a Stream DatabaseSo far we've seen how to manage data in a stream database, but we haven't yet seen how to work with that data.Stream databases are dict-like objects, so we can access streams in the same way we would access a value in a dict:<jupyter_code>for key in db:
    print(db[key])<jupyter_output>test-2/Boston
test-2/Providence<jupyter_text>The `.keys()` and `.items()` methods are also supported. For paths that are multiple levels deep you can access them one of two ways:<jupyter_code>print(db['Boston']['43'])
print(db['Boston', '43'])<jupyter_output>test-2/Boston/43
{'fraction_non_alphanumeric': 0.1349911190053286, 'fraction_numerical': 0.08821788040260509, 'mean_word_length': 4.687710437710438, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 39, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '509175', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2507, 'n_tokens_neox': 2065, 'n_words': 733, 'path': 'tutorial/Databases.ipynb', 'repo_name': 'Sentenai/python-documentation', 'size': '6756'}