<jupyter_start><jupyter_text>An LSF file is a binary file of concatenated IMC messages.This notebook demonstrates some of the included functionality for parsing and exporting LSF files.**Warning:** *Pyimc parses the LSF files under the assumption that the IMC specification is the same as the one it was compiled against.*<jupyter_code>import pyimc
from pyimc.lsf import LSFReader, LSFExporter

lsf_path ='./Data.lsf'<jupyter_output><empty_output><jupyter_text>LSFReader usage The LSFReader class can be instantiated using the 'with' statement to automatically handle file opening/closing.<jupyter_code># Print first three messages in LSF file
# save_index: a message index is generated, speeds up subsequent reads
with LSFReader(lsf_path, save_index=True) as lsf:
    for i, msg in zip(range(3), lsf.read_message()):
        # Print outputs the message type, destination/source imc address, timestamp and fields 
        print('Message', i)
        print(msg)
        print()<jupyter_output>Message 0
LoggingControl
00D1:47 -> FFFF:FF
2020/02/11 08:35:21
    op: 1
    name: 20200211/083521_test_plan

Message 1
QueryEntityInfo
00D1:47 -> 00D1:00
2020/02/11 08:35:21
    id: 0

Message 2
QueryEntityInfo
00D1:47 -> 00D1:01
2020/02/11 08:35:21
    id: 1<jupyter_text>The convenience function "read" returns a generator of messages of a certain type instead of using the "with" command.Can be iterated over or converted to a list<jupyter_code>msgs = list(LSFReader.read(lsf_path, types=[pyimc.EstimatedState]))
last_msg = msgs[-1]
print(last_msg)<jupyter_output>EstimatedState
00D1:28 -> FFFF:FF
2020/02/11 08:36:40
    lat: 0.718814
    lon: -0.151952
    height: -0.168725
    x: -147.467
    y: 72.6613
    z: 0.285875
    phi: -0.000253006
    theta: 0.235803
    psi: -1.3983
    u: 1.08701
    v: 0.000631064
    w: 0.0149416
    vx: 0.182613
    vy: -1.04441
    vz: -0.239383
    p: -0.00140179
    q: 5.05482e-05
    r: 0.000335344
    depth: 0.0805944
    alt: 6.02733<jupyter_text>LSFExporter usage This class reads LSF files and exports it to a Pandas DataFrame.<jupyter_code>import pandas as pd
from IPython.display import display
pd.set_option('display.max_columns', 30)

exp = LSFExporter(lsf_path)
df = exp.export_messages(imc_type=pyimc.EstimatedState)

# Print first five rows
display(df.head())
import numpy as np

# Index by timestamp instead
df_t = df.set_index('timestamp')

# Get all rows between two timestamps
t_start = np.datetime64('2020-02-11 08:36:00')
t_end = np.datetime64('2020-02-11 08:36:10')
df_slice = df_t[t_start:t_end]
display(df_slice)
# Output data to a csv file
df_slice.to_csv('output.csv')
# Read another message type using the same exporter (re-use metadata)
df2 = exp.export_messages(imc_type=pyimc.Announce)

# Remove all messages not originating from lauv-simulator-1
df2 = df2[df2['src'] == 'lauv-simulator-1']
display(df2.head(2))

# Alternatively, supply a lambda predicate to the export_messages function (avoids converting unnecessary messages to pandas)
imc_id = exp.get_node_id('lauv-simulator-1')
df3 = exp.export_messages(imc_type=pyimc.Announce, condition=lambda msg: msg.src == imc_id)
display(df3.head(2))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09924859110832812, 'fraction_numerical': 0.09423919849718222, 'mean_word_length': 4.480274442538594, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '495448', 'license': 'MIT', 'n_tokens_mistral': 1340, 'n_tokens_neox': 1143, 'n_words': 326, 'path': 'examples/lsf_parsing.ipynb', 'repo_name': 'mquinaz/pyimc', 'size': '3194'}