<jupyter_start><jupyter_text>S2 - Command Line Tools  General Remarks- script data access for common tasks (bash, python)- frequent tasks should be a single command- learn a scripting language (C/C++/Java are too verbose)  Use UNIX- Rich set of data mangling tools available- Can work with streams (not only files)- Does not hide the details (lilke Windows UIs)- Was designed to do data handling jobs- Pipes to compose tools- E.g. can use ssh to connect to remote servers (get CPU stats from remote)  Use CSV files- Widely supported (Spreadsheets, Excel, Programming Languages, DBs, GitHub!)- Portable (if you stick to ASCII or utf-8)- Human readable- Good tooling (cat, grep, csvkit)- Can be used like SQL tables (e.g. Join tables for many-many relations)- XML/JSON are describing trees (not lists or tables)- Use compression (zcat or cat | gzip -d to read and | gzip > out.csv)- Use proper csv (not '\t' separators and the like)  Tools  Handling text streams with UNIX tools<jupyter_code># cat: print output to stdout
# !cat DataSets/LogDB.csv # long output
# Print only first 10 lines
# Pipe: connect stdout to stdin stream of next command
!cat DataSets/LogDB.out | head
# Print only first 3 lines
!seq 10 | head -n 3
# Print the first n-3 lines
!seq 10 | head -n-3
# Last 3 lines
!seq 10 | tail -n 3
# Last n-3 lines: skip first 2 lines
!seq 10 | tail -n+3
# Filter matching lines with grep
# Count lines with wc -l
!cat DataSets/LogDB.out | grep GET | wc -l<jupyter_output>3335<jupyter_text>perl -pe```-e: Allows you to provide the program as an argument rather    than in a file. You don't want to have to create a script    file for every little Perl one-liner.-p: Places a printing loop around your command so that it acts on each    line of standard input. Used mostly so Perl can beat the    pants off awk in terms of power AND simplicity :-)        ```* Very powerful regular expressions* Sane, widely known syntax<jupyter_code># Parse Log Files with perl -pe
!cat DataSets/LogDB.out | perl -pe 's/.*"(.*)".*/\1/' | head


# Full csv conversion:
# !cat DataSets/LogDB.out | perl -pe 's/^.*"(.*?) (.*)? (.*)?" (\d+) (\d+) (\d+.\d+)$/\1,\2,\4,\5,\6/' |  head<jupyter_output>POST /doc/0 HTTP/1.1
PUT /doc/1 HTTP/1.1
POST /doc/2 HTTP/1.1
PUT /doc/1 HTTP/1.1
GET /doc/3 HTTP/1.1
GET /doc/4 HTTP/1.1
PUT /doc/1 HTTP/1.1
GET /doc/3 HTTP/1.1
GET /doc/4 HTTP/1.1
PUT /doc/1 HTTP/1.1
-p destination: Broken pipe
cat: write error: Broken pipe<jupyter_text>csvkit (http://csvkit.readthedocs.org/en/0.9.1/)* Set of tools to handle csv files at the command line* Python based<jupyter_code># pretty print CSV files with csvlook
#
# -H no headder row provided
!csvlook -H DataSets/cpu.out.csv | head
# Print summary statistics for each column
!csvstat -H DataSets/cpu.out.csv
# convert between different formats

#!csvformat -h

# to "-T" tab separated output, "-U 1" quote all
!echo "Convert csv to tsv with quoting"
!cat DataSets/cpu.out.csv | head -n 3 | csvformat -T -U 1

!echo "\n\nConvert back to csv"
!cat DataSets/cpu.out.csv | head -n 3 | csvformat -T -U 1 | csvformat -t -u 1
# Select columns from dataset
!cat DataSets/cpu.out.csv | head -n 3 | csvcut -c 1-3
# Need SQL features?

# csfjoin - join two csv tables
# csvsql  - import to database<jupyter_output><empty_output><jupyter_text>JSON Handling* jq http://stedolan.github.io/jq/* like xpath, xidel for XML<jupyter_code># pretty print
!cat DataSets/HistogramAPI.json | jq '.' | head
# Extract Histogram and convert to csv
!cat DataSets/HistogramAPI.json | jq '.[0][2] | to_entries | .[] | [.key, .value] | @csv' --raw-output | csvformat -U 0 | head<jupyter_output>0.5,1
0.59,2
1.7,1
10,193
100,1
11,209
12,223
120,1
13,176
14,163
{'fraction_non_alphanumeric': 0.11856642414443547, 'fraction_numerical': 0.032066828348154136, 'mean_word_length': 3.504854368932039, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '191579', 'license': 'MIT', 'n_tokens_mistral': 1373, 'n_tokens_neox': 1277, 'n_words': 518, 'path': 'work/S2 - Command Line Tools.ipynb', 'repo_name': 'agladkov/DataScience4Ops', 'size': '3711'}