<jupyter_start><jupyter_text>Splunk - Data Connector  DescriptionThe data provider library of msticpy provides functions to allow for the defining of data sources, connectors to them and queries for them as well as the ability to return query result from the defined data sources. For more information, check documentation- Data Provider Library: https://msticpy.readthedocs.io/en/latest/data_acquisition/DataProviders.htmlIn this notebooks will demonstrate Splunk data connector feature of msticpy. This feature is built on-top of the [Splunk Enterprise SDK for Python](https://dev.splunk.com/enterprise/docs/devtools/python/sdk-python/).  Installation<jupyter_code>#!pip install --upgrade msticpy<jupyter_output><empty_output><jupyter_text>AuthenticationAuthentication for the Splunk data provider is handled by specifying credentials directly in the connect call or specifying the credentials in msticpy config file.For more information on how to create new user with approapriate roles and permissions, follow Splunk Docs [Addandeditusers](https://docs.splunk.com/Documentation/Splunk/8.0.5/Security/Addandeditusers) and [Aboutusersandroles](https://docs.splunk.com/Documentation/Splunk/8.0.5/Security/Aboutusersandroles). The user should have permission to at least run its own searches or more depending upon the actions to be performed by user.Once you created user account with the appropriate roles, you will require the following details to specify while connecting- host = "localhost"(Splunk server FQDN hostname to connect, for locally installed splunk, you can specify localhost)- port = 8089 (Splunk REST API )- username = "admin" (username to connect to Splunk instance)- password = "<PASSWORD>" (password of the userspecified in username)Once you have details, you can specify it in `msticpyconfig.yaml` as shown in below example<jupyter_code>'''
SplunkApp:
  Args:
    host: "{Splunk server FQDN or localhost}"
    port: "{default 8089}"
    username: "{username with search permissions to connect}"
    password: "{<PASSWORD>}"
'''
#Check we are running Python 3.6
import sys
MIN_REQ_PYTHON = (3,6)
if sys.version_info < MIN_REQ_PYTHON:
    print('Check the Kernel->Change Kernel menu and ensure that Python 3.6')
    print('or later is selected as the active kernel.')
    sys.exit("Python %s.%s or later is required.\n" % MIN_REQ_PYTHON)

#imports
import pandas as pd
import msticpy.nbtools as nbtools

#data library imports
from msticpy.data.data_providers import QueryProvider

print('Imports Complete')<jupyter_output>Imports Complete<jupyter_text>Instantiating a query providerYou can instantiate a data provider for Splunk by specifying the credentials in connect or in msticpy config file.  If the details are correct and authentication is successful, it will show connected.<jupyter_code>splunk_prov = QueryProvider('Splunk')
splunk_prov.connect(host=<hostname>, username=<username>, password=<password>)<jupyter_output>connected<jupyter_text>Listing available  queriesUpon connecting to the Splunk data environment, we can take a look what query options available to us by ruinning `QUERY_PROVIDER.list_queries()`For more informatio, refer documentation : [Listing available queries](https://msticpy.readthedocs.io/en/latest/data_acquisition/DataProviders.htmllisting-available-queries).This will display all the saved searches from the connected splunk instance and also few custom queries to do common operations such as list datatypes, list saved searches, alerts, audittrail informaion.<jupyter_code>splunk_prov.list_queries()<jupyter_output><empty_output><jupyter_text>In order to get help for query , refer documentation - [Getting Help for a query](https://msticpy.readthedocs.io/en/latest/data_acquisition/DataProviders.htmlgetting-help-for-a-query)<jupyter_code>splunk_prov.SplunkGeneral.get_events_parameterized('?')<jupyter_output>Query:  get_events_parameterized
Data source:  Splunk
Generic parameterized query from index/source

Parameters
----------
add_query_items: str (optional)
    Additional query clauses
    (default value is: | head 100)
end: datetime (optional)
    Query end time
    (default value is: 08/26/2017:00:00:00)
index: str (optional)
    Splunk index name
    (default value is: *)
project_fields: str (optional)
    Project Field names
    (default value is: | table TimeCreated, host, EventID, EventDescripti...)
source: str (optional)
    Splunk source type
    (default value is: *)
start: datetime (optional)
    Query start time
    (default value is: 08/25/2017:00:00:00)
timeformat: str (optional)
    Datetime format to use in Splunk query
    (default value is: "%Y-%m-%d %H:%M:%S.%6N")
Query:
 search index={index} source={source} timeformat={timeformat} earliest="{start}" latest="{end}" {project_fields} {add_query_items}<jupyter_text>Running pre-defined queryIn order to run pre-defined query , refer documentation - [Running an pre-definedfined query](https://msticpy.readthedocs.io/en/latest/data_acquisition/DataProviders.htmlrunning-an-pre-defined-query)<jupyter_code>splunk_prov.SplunkGeneral.get_events_parameterized(
    index="botsv2",
    source="WinEventLog:Microsoft-Windows-Sysmon/Operational",
    start="2017-08-25 00:00:00.000000",
    end="2017-08-25 10:00:00.000000"
)<jupyter_output><empty_output><jupyter_text>By-default, splunk query results are limited to 100. you can specify count=0 argument to return all the results.  Running a Ad-hoc Splunk queryYou can also define a splunk query and run it via splunk provider<jupyter_code>splunk_query = '''
search index="blackhat" sourcetype="network" earliest=0 
| table TimeGenerated, TotalBytesSent
'''
df = splunk_prov.exec_query(splunk_query)
df.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08867167044859486, 'fraction_numerical': 0.017105952173154127, 'mean_word_length': 5.69392523364486, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 32, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '414414', 'license': 'MIT', 'n_tokens_mistral': 1775, 'n_tokens_neox': 1637, 'n_words': 556, 'path': 'docs/notebooks/Splunk-DataConnector.ipynb', 'repo_name': 'dg2kjb/msticpy', 'size': '5751'}