<commit_before># Cassandra system.log parser

This is a rule-based Cassandra `system.log` parser.  The [systemlog.py](systemlog.py) file contains
a set of rules that define how to parse the system.log. The `parse_log` generator applies
these rules and yields a dictionary containing a single event from the log at a time.
It takes as input another generator which should yield one line from the log at a time.
This works perfectly with `fileinput` from the python standard library.

## log_to_json

The [log_to_json](log_to_json) script parses system.log and outputs events in JSON format with one
event per line.  It takes a list of log files on the command line and parses them.
If no arguments are supplied it will attempt to parse stdin. This can be used to parse
a live log file by piping from tail: `tail -f /var/log/cassandra/system.log | log_to_json`.

## cassandra_ingest

The [cassandra_ingest](cassandra_ingest) script parses system.log and inserts each event into the
logparse.systemlog table defined in [systemlog.cql](systemlog.cql). It takes a list of log files
on the command line and parses them.  If no arguments are supplied it will attempt 
to parse stdin. This can be used to parse a live log file by piping from tail: 
`tail -f /var/log/cassandra/system.log | cassandra_ingest`.

The systemlog table contains a standard set of fields that are common to each event,
as well as a set of collections of various types, and fields that do not exist in 
the table are automatically inserted into the appropriate collection depending on
the type of data it contains. This allows event-specific fields to be saved 
automatically.  The naming convention of these collections allows them to be treated 
as dynamic fields when the table is indexed in DSE's implementation of Solr.

The [cassandra_store.py](cassandra_store.py) file contains the code to save the log data into Cassandra.
It requires the [DataStax Python Driver](https://github.com/datastax/python-driver).
It should be installed by running `pip install cassandra`. The script currently assumes
that Cassandra is running on `localhost` on the default CQL port 9042.

## Solr indexing

The logparse.systemlog table can be indexed using the Solr implementation from 
[DataStax Enterprise](http://docs.datastax.com/en/datastax_enterprise/4.7//datastax_enterprise/newFeatures.html).
A [schema.xml](solr/schema.xml) and [solrconfig.xml](solr/solrconfig.xml) are provided
in the [solr](solr) directory along with the [add-schema.sh](solr/add-schema.sh) script 
which will upload the Solr schema to DSE.  

Once indexed in Solr, the log events can be subsequently analyzed and visualized using 
[Banana](https://github.com/LucidWorks/banana).  Banana is a port of Kibana 3.0 to Solr.
Several pre-made dashboards are saved in json format in the [banana](banana) subdirectory. 
These can be loaded using the Banana web UI.

Setup Instructions:

1. Clone https://github.com/LucidWorks/banana to $DSE_HOME/resources/banana.<commit_msg>fixed pip command instructions to install python driver<commit_after># Cassandra system.log parser

This is a rule-based Cassandra `system.log` parser.  The [systemlog.py](systemlog.py) file contains
a set of rules that define how to parse the system.log. The `parse_log` generator applies
these rules and yields a dictionary containing a single event from the log at a time.
It takes as input another generator which should yield one line from the log at a time.
This works perfectly with `fileinput` from the python standard library.

## log_to_json

The [log_to_json](log_to_json) script parses system.log and outputs events in JSON format with one
event per line.  It takes a list of log files on the command line and parses them.
If no arguments are supplied it will attempt to parse stdin. This can be used to parse
a live log file by piping from tail: `tail -f /var/log/cassandra/system.log | log_to_json`.

## cassandra_ingest

The [cassandra_ingest](cassandra_ingest) script parses system.log and inserts each event into the
logparse.systemlog table defined in [systemlog.cql](systemlog.cql). It takes a list of log files
on the command line and parses them.  If no arguments are supplied it will attempt 
to parse stdin. This can be used to parse a live log file by piping from tail: 
`tail -f /var/log/cassandra/system.log | cassandra_ingest`.

The systemlog table contains a standard set of fields that are common to each event,
as well as a set of collections of various types, and fields that do not exist in 
the table are automatically inserted into the appropriate collection depending on
the type of data it contains. This allows event-specific fields to be saved 
automatically.  The naming convention of these collections allows them to be treated 
as dynamic fields when the table is indexed in DSE's implementation of Solr.

The [cassandra_store.py](cassandra_store.py) file contains the code to save the log data into Cassandra.
It requires the [DataStax Python Driver](https://github.com/datastax/python-driver).
It should be installed by running `pip install cassandra-driver`. The script currently assumes
that Cassandra is running on `localhost` on the default CQL port 9042.

## Solr indexing

The logparse.systemlog table can be indexed using the Solr implementation from 
[DataStax Enterprise](http://docs.datastax.com/en/datastax_enterprise/4.7//datastax_enterprise/newFeatures.html).
A [schema.xml](solr/schema.xml) and [solrconfig.xml](solr/solrconfig.xml) are provided
in the [solr](solr) directory along with the [add-schema.sh](solr/add-schema.sh) script 
which will upload the Solr schema to DSE.  

Once indexed in Solr, the log events can be subsequently analyzed and visualized using 
[Banana](https://github.com/LucidWorks/banana).  Banana is a port of Kibana 3.0 to Solr.
Several pre-made dashboards are saved in json format in the [banana](banana) subdirectory. 
These can be loaded using the Banana web UI.

Setup Instructions:

1. Clone https://github.com/LucidWorks/banana to $DSE_HOME/resources/banana.
{'fraction_non_alphanumeric': 0.06232435113241858, 'fraction_numerical': 0.002975698462555794, 'mean_word_length': 4.576036866359447, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 8}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1905383', 'n_tokens_mistral': 1733, 'n_tokens_neox': 1688, 'n_words': 822}