<jupyter_start><jupyter_text>GRR Colab<jupyter_code>%load_ext grr_colab.ipython_extension
import grr_colab<jupyter_output><empty_output><jupyter_text>Specifying GRR Colab flags:<jupyter_code>grr_colab.flags.FLAGS.set_default('grr_http_api_endpoint', 'http://localhost:8000/')
grr_colab.flags.FLAGS.set_default('grr_admin_ui_url', 'http://localhost:8000/')
grr_colab.flags.FLAGS.set_default('grr_auth_api_user', 'admin')
grr_colab.flags.FLAGS.set_default('grr_auth_password', '<PASSWORD>')<jupyter_output><empty_output><jupyter_text>Magics API GRR magics allow to search for clients and then to choose a single client to work with. The results of magics are represented as pandas dataframes unless they are primitives.  Searching clients You can search for clients by specifying username, hostname, client labels etc. The results are sorted by the last seen column.<jupyter_code>df = %grr_search_clients -u admin
df[['online', 'online.pretty', 'client_id', 'last_seen_ago', 'last_seen_at.pretty']]<jupyter_output><empty_output><jupyter_text>There is a shortcut for searching for online only clients directly so that you don't need to filter the dataframe.<jupyter_code>df = %grr_search_online_clients -u admin
df[['online', 'online.pretty', 'client_id', 'last_seen_ago', 'last_seen_at.pretty']]<jupyter_output><empty_output><jupyter_text>Every datetime field has two representations: the original one that is microseconds and the pretty one that is pandas timestamp.<jupyter_code>df[['last_seen_at', 'last_seen_at.pretty']]<jupyter_output><empty_output><jupyter_text>Setting current clients To work with a client you need to select a client first. It means that you are able to work only with a single client simultaneously using magic commands (there is no such restriction for Python API). To set a client you need either a hostname (works in case of one client set up for that hostname) or a client ID which you can get from the search clients dataframe.<jupyter_code>client_id = df['client_id'][0]
%grr_set_client -c {client_id}

%grr_id<jupyter_output><empty_output><jupyter_text>An attempt to set a client with a hostname that has multiple clients will lead to an exception.  Requesting approvals If you don't have valid approvals for the selected client, you will get an error while attempting to run a flow on it. You can request an approval with magic commands specifying the reason and list of approvers.<jupyter_code>%grr_request_approval -r "For testing" -a admin<jupyter_output><empty_output><jupyter_text>This function will not wait until the approval is granted. If you need your code to wait until it's granted, use `grr_request_approval_and_wait` instead.  Exploring filesystem In addition to the selected client, working directory is also saved. It means that you can use relative paths instead of absolute. Note that the existence of directories is not checked and you will not get an error if you try to cd into directory that does not exist. Initially you are in the root directory.<jupyter_code>%grr_pwd
%grr_cd tmp/foo/bar
%grr_pwd
%grr_cd ../baz
%grr_pwd<jupyter_output><empty_output><jupyter_text>You can ls the current directory and any other directories specified by relative and absolute paths. **Note**. The most file-related magics start flows and fetch live data from the client. It means that the client has to be online in order for them to work.<jupyter_code>df = %grr_ls
df<jupyter_output><empty_output><jupyter_text>Stat mode has two representations: number and UNIX-style:<jupyter_code>df[['st_mode', 'st_mode.pretty']]
%grr_ls ../baz/dir2
%grr_ls /tmp/foo<jupyter_output><empty_output><jupyter_text>To see some metadata of a file you can just call `grr_stat` function.<jupyter_code>%grr_stat file1<jupyter_output><empty_output><jupyter_text>You can use globbing for stat:<jupyter_code>%grr_stat "file*"<jupyter_output><empty_output><jupyter_text>You can print the first bytes of a file:<jupyter_code>%grr_head file1 -c 30<jupyter_output><empty_output><jupyter_text>Alghough there is no offset in original bash `head` command you can specify offset in `grr_head`:<jupyter_code>%grr_head file1 -c 30 -o 20<jupyter_output><empty_output><jupyter_text>Some of the functions like `grr_head` and `grr_ls` have `--cached` (`-C` for short) option which indicates that no calls to the client should be performed. In this case the data will be fetched from the cached data on the server. Server cached data is updated only during calls to the client so it is not always up-to-date but accessing it is way faster.<jupyter_code>%grr_ls /tmp/foo/baz -C
%grr_head file1 -C<jupyter_output><empty_output><jupyter_text>Grepping files is also possible. `--fixed-string` (`-F` for short) option indicates that pattern to search for is not a regular expression. `--hex-string` (`-X` for short) option allows to pass hex strings as a pattern.<jupyter_code>%grr_grep "line" file1
%grr_grep -F "line" file1
%grr_grep -X "6c696e65" file1<jupyter_output><empty_output><jupyter_text>There is a shortcut for `--fixed-strings` option. Globbing is also available here.<jupyter_code>%grr_fgrep "line" "file*"
%grr_fgrep -X "6c696e65" file1<jupyter_output><empty_output><jupyter_text>If the file is too large and you'd like to download it then use `wget`:<jupyter_code>%grr_wget file1<jupyter_output><empty_output><jupyter_text>You can also download a cached version:<jupyter_code>%grr_wget file1 -C<jupyter_output><empty_output><jupyter_text>You can specify path type with `--path-type` flag (`-P` for short) for all filesystem related magics. The available values are `os` (default), `tsk`, `ntfs`, `registry`.<jupyter_code>%grr_ls -P os -C<jupyter_output><empty_output><jupyter_text>System information Names of the functions are the same as in bash for simplicity. Printing hostname of the client:<jupyter_code>%grr_hostname<jupyter_output><empty_output><jupyter_text>Getting network interfaces info:<jupyter_code>ifaces = %grr_ifconfig<jupyter_output><empty_output><jupyter_text>For mac address fields there are also two columns: one with the original bytes type but not representable and pretty one with string representation of mac address.<jupyter_code>ifaces[['mac_address', 'mac_address.pretty']][1:]<jupyter_output><empty_output><jupyter_text>If a field contains a collection then the cell in the dataframe is represented as another dataframe. IP address fields also have two representations.<jupyter_code>ifaces['addresses'][1]<jupyter_output><empty_output><jupyter_text>For `uname` command only two options are available: `--machine` that prints the machine architecture and `--kernel-release`.<jupyter_code>%grr_uname -m
%grr_uname -r<jupyter_output><empty_output><jupyter_text>To get the client summary you can simply call interrogate flow.<jupyter_code>df = %grr_interrogate
df[['client_id', 'system_info.system', 'system_info.machine']]<jupyter_output><empty_output><jupyter_text>There is also possible to get info about processes that are running on client machine:<jupyter_code>ps = %grr_ps
ps[:5]<jupyter_output><empty_output><jupyter_text>To fetch some system information you can also use osquery. Osquery tables are also converted to dataframes.<jupyter_code>%grr_osqueryi "SELECT pid, name, cmdline, state, nice, threads FROM processes WHERE pid >= 440 and pid < 600;"<jupyter_output><empty_output><jupyter_text>Running YARA for scanning processes is also available.<jupyter_code>import os 

pid = os.getpid()
data = "dadasdasdasdjaskdakdaskdakjdkjadkjakjjdsgkngksfkjadsjnfandankjd"
rule = 'rule TextExample {{ strings: $text_string = "{data}" condition: $text_string }}'.format(data=data)

df = %grr_yara '{rule}' -p {pid}
df[['process.pid', 'process.name', 'process.exe']]<jupyter_output><empty_output><jupyter_text>Configuring flow timeout The default flow timeout is 30 seconds. It's time the function waits for a flow to complete. You can configure this timeout with `grr_set_flow_timeout` specifying number of seconds to wait. For examples, this will set the timeout to a minute:<jupyter_code>%grr_set_flow_timeout 60<jupyter_output><empty_output><jupyter_text>To tell functions to wait for the flows forever until they are completed:<jupyter_code>%grr_set_no_flow_timeout<jupyter_output><empty_output><jupyter_text>To set timeout to default value of 30 seconds:<jupyter_code>%grr_set_default_flow_timeout<jupyter_output><empty_output><jupyter_text>Setting timeout to 0 tells functions not to wait at all and exit immediately after the flow starts.<jupyter_code>%grr_set_flow_timeout 0<jupyter_output><empty_output><jupyter_text>In case timeout is exceeded (or you set 0 timeout) you will se such error with a link to Admin UI.  Collecting artifacts You can first list all the artifacts that you can collect:<jupyter_code>df = %grr_list_artifacts
df[:2]<jupyter_output><empty_output><jupyter_text>To collect an artifact you just need to provide its name:<jupyter_code>%grr_collect "DebianVersion"<jupyter_output><empty_output><jupyter_text>Python API  Getting a client Using Python API you can work with multiple clients simultaneously. You don't need to select a client to work with, instead you simply get a client object. Use `search` method to search for clients. You can specify `ip`, `mac`, `host`, `version`, `user`, and `labels` search criteria. As a result you will get a list of client objects so that you can pick one of them to work with.<jupyter_code>clients = grr_colab.Client.search(user='admin')
clients
clients[0].id<jupyter_output><empty_output><jupyter_text>If you know a client ID or a hostname (in case there is one client installed for this hostname) you can get a client object using one of these values:<jupyter_code>client = grr_colab.Client.with_id('C.dc3782aeab2c5b4c')<jupyter_output><empty_output><jupyter_text>Client properties There is a bunch of simple client properties to get some info about the client. Unlike magic API this API returns objects but not dataframes for non-primitive values. Getting the client ID:<jupyter_code>client.id<jupyter_output><empty_output><jupyter_text>Getting the client hostname:<jupyter_code>client.hostname<jupyter_output><empty_output><jupyter_text>Getting network interfaces info:<jupyter_code>client.ifaces[1:]
client.ifaces[1].ifname<jupyter_output><empty_output><jupyter_text>This is a collection of interface objects so you can iterate over it and access interface object fields:<jupyter_code>for iface in client.ifaces:
  print(iface.ifname)<jupyter_output>enp0s31f6
lo<jupyter_text>Getting the knowledge base for the client: You can also access its fields:<jupyter_code>client.knowledgebase
client.knowledgebase.os_release<jupyter_output><empty_output><jupyter_text>Getting an architecture of a machine that client runs on:<jupyter_code>client.arch<jupyter_output><empty_output><jupyter_text>Getting kernel version string:<jupyter_code>client.kernel<jupyter_output><empty_output><jupyter_text>Getting a list of labels that are associated with this client:<jupyter_code>client.labels<jupyter_output><empty_output><jupyter_text>First seen and last seen times are saved as datetime objects:<jupyter_code>client.first_seen
client.last_seen<jupyter_output><empty_output><jupyter_text>Requesting approvals As in magics API here you also need to request an approval before running flows on a client. To do this simply call `request_approval` method providing a reason for the approval and list of approvers.<jupyter_code>client.request_approval(approvers=['admin'], reason='Test reason')<jupyter_output><empty_output><jupyter_text>This method does not wait until the approval is granted. If you need to wait, use `request_approval_and_wait` method that has the same signature.  Running flows To set the flow timeout use `set_flow_timeout` function. 30 seconds is the default value. 0 means exit immediately after the flow started. You can also reset timeout and set it to a default value of 30 seconds.<jupyter_code># Wait forever
grr_colab.set_no_flow_timeout()

# Exit immediately
grr_colab.set_flow_timeout(0)

# Wait for one minute
grr_colab.set_flow_timeout(60)

#Wait for 30 seconds
grr_colab.set_default_flow_timeout()<jupyter_output><empty_output><jupyter_text>Below are examples of flows that you can run. Interrogating a client:<jupyter_code>summary = client.interrogate()
summary.system_info.system<jupyter_output><empty_output><jupyter_text>Listing processes on a client:<jupyter_code>ps = client.ps()
ps[:1]
ps[0]
ps[0].exe<jupyter_output><empty_output><jupyter_text>Listing files in a directory. Here you need to provide the absolute path to the directory because there is no state.<jupyter_code>files = client.ls('/tmp/foo/baz')
files
for f in files:
  print(f.pathspec.path)<jupyter_output>/tmp/foo/baz/dir1
/tmp/foo/baz/dir2
/tmp/foo/baz/file1
/tmp/foo/baz/file2<jupyter_text>Recursive listing of a directory is also possible. To do this specify the max depth of the recursion.<jupyter_code>files = client.ls('/tmp/foo', max_depth=3)
files
for f in files:
  print(f.pathspec.path)<jupyter_output>/tmp/foo/bar
/tmp/foo/baz
/tmp/foo/baz/dir1
/tmp/foo/baz/dir2
/tmp/foo/baz/file1
/tmp/foo/baz/file2
/tmp/foo/baz/dir2/dir3<jupyter_text>Globbing files:<jupyter_code>files = client.glob('/tmp/foo/baz/file*')
files<jupyter_output><empty_output><jupyter_text>Grepping files with regular expressions:<jupyter_code>matches = client.grep(path='/tmp/foo/baz/file*', pattern=b'line')
matches
for match in matches:
  print(match.pathspec.path, match.offset, match.data)
matches = client.grep(path='/tmp/foo/baz/file*', pattern=b'\x6c\x69\x6e\x65')
matches<jupyter_output><empty_output><jupyter_text>Grepping files by exact match:<jupyter_code>matches = client.fgrep(path='/tmp/foo/baz/file*', literal=b'line')
matches<jupyter_output><empty_output><jupyter_text>Downloading files:<jupyter_code>client.wget('/tmp/foo/baz/file1')<jupyter_output><empty_output><jupyter_text>Osquerying a client:<jupyter_code>table = client.osquery('SELECT pid, name, nice FROM processes WHERE pid < 5')
table
header = ' '.join(str(col.name).rjust(10) for col in table.header.columns)
print(header)
print('-' * len(header))
for row in table.rows:
  print(' '.join(map(lambda _: _.rjust(10), row.values)))<jupyter_output>name       nice        pid
--------------------------------
   systemd          0          1
  kthreadd          0          2
    rcu_gp        -20          3
rcu_par_gp        -20          4<jupyter_text>Listing artifacts:<jupyter_code>artifacts = grr_colab.list_artifacts()
artifacts[0]<jupyter_output><empty_output><jupyter_text>To collect an artifact you just need to provide its name:<jupyter_code>client.collect('DebianVersion')<jupyter_output><empty_output><jupyter_text>Running YARA:<jupyter_code>import os 

pid = os.getpid()
data = "dadasdasdasdjaskdakdaskdakjdkjadkjakjjdsgkngksfkjadsjnfandankjd"
rule = 'rule TextExample {{ strings: $text_string = "{data}" condition: $text_string }}'.format(data=data)

matches = client.yara(rule, pids=[pid])
print(matches[0].process.pid, matches[0].process.name)<jupyter_output>63438 python3<jupyter_text>Working with files You can read and seek files interacting with them like fith usual python files.<jupyter_code>with client.open('/tmp/foo/baz/file1') as f:
  print(f.readline())
with client.open('/tmp/foo/baz/file1') as f:
  for line in f:
    print(line)
with client.open('/tmp/foo/baz/file1') as f:
  print(f.read(22))
  f.seek(0)
  print(f.read(22))
  print(f.read())<jupyter_output>b'This is the first line'
b'This is the first line'
b'\nThis is the second line\nThis is the third LINE\n'<jupyter_text>Cached data To fetch server cached data use `cached` property of a client object. You can list files in directory (recursively also) and read and download files as above:<jupyter_code>files = client.cached.ls('/tmp/foo/baz')
files
files = client.cached.ls('/tmp/foo/baz', max_depth=2)
files
with client.cached.open('/tmp/foo/baz/file1') as f:
  for line in f:
    print(line)
client.cached.wget('/tmp/foo/baz/file1')<jupyter_output><empty_output><jupyter_text>You can also refresh filesystem metadata that is cached on the server by calling `refresh` method (that will refresh the contents of the directory and not its subdirectories):<jupyter_code>client.cached.refresh('/tmp/foo/baz')<jupyter_output><empty_output><jupyter_text>To refresh a  directory recursively specify `max_depth` parameter:<jupyter_code>client.cached.refresh('/tmp/foo/baz', max_depth=2)
### Path types<jupyter_output><empty_output><jupyter_text>To specify path type, just use one of the client properties: `client.os` (the same as just using `client`), `client.tsk`, `client.ntfs`, `client.registry`.<jupyter_code>client.os.ls('/tmp/foo')
client.os.cached.ls('/tmp/foo')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09859575739468181, 'fraction_numerical': 0.008246190618464296, 'mean_word_length': 6.600363306085377, 'pattern_counts': {'":': 0, '<': 258, '<?xml version=': 0, '>': 257, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '65', 'dir': 'jupyter-structured-clean-dedup', 'id': '143601', 'license': 'Apache-2.0', 'n_tokens_mistral': 5638, 'n_tokens_neox': 5490, 'n_words': 1614, 'path': 'colab/examples/demo.ipynb', 'repo_name': 'BA7JCM/grr', 'size': '16730'}