<jupyter_start><jupyter_text>subprocess- The subprocess module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes.  [Documentation: subprocess][1][1]:<jupyter_code>from subprocess import run, Popen, PIPE<jupyter_output><empty_output><jupyter_text>1. Run system command<jupyter_code># Run command `ls -l`
r1 = run(['ls', '-l'], stdout=PIPE)  # Need to separate command & argument (flag)
r2 = run('ls -l', shell=True)  # Shell mode, not recommanded due to security concern like "shell injection"
print(r1.stdout)
print("Output for r1 is captured, r2 is not.")<jupyter_output>b'total 36\n-rw-rw-r-- 1 qz qz 5383 Jun 17 21:55 decorator.ipynb\n-rw-rw-r-- 1 qz qz 1070 Jun 15 03:44 LICENSE\n-rw-rw-r-- 1 qz qz 4011 Jun 17 23:49 lru_cache.ipynb\n-rw-rw-r-- 1 qz qz  140 Jun 17 23:49 README.md\n-rw-rw-r-- 1 qz qz 7591 Jun 15 03:45 sched.ipynb\n-rw-rw-r-- 1 qz qz 5661 Jun 18 23:29 subprocess.ipynb\n'
Output for r1 is captured, r2 is not.<jupyter_text>1.0. shell injection example<jupyter_code># https://docs.python.org/3/library/subprocess.html#security-considerations
filename = 'somefile; ls -l'
command = 'ls -l {}'.format(filename)
print('No quote: {}'.format(command))
from shlex import quote  # Avoid shell injection
command = 'ls -l {}'.format(quote(filename))
print('Use quote: {}\n'.format(command))


# need to quote(command) before split
remote_command = 'ssh home {}'.format(quote(command))
print('Quoted command: {}'.format(remote_command))
from shlex import split
remote_command = split(remote_command)
print('First split: {}'.format(remote_command))
command = split(remote_command[-1])
print('Second split: {}'.format(command))<jupyter_output>No quote: ls -l somefile; ls -l
Use quote: ls -l 'somefile; ls -l'

Quoted command: ssh home 'ls -l '"'"'somefile; ls -l'"'"''
First split: ['ssh', 'home', "ls -l 'somefile; ls -l'"]
Second split: ['ls', '-l', 'somefile; ls -l']<jupyter_text>2. Popen & communicate<jupyter_code># Popen: Execute child program in a new process
# PIPE: Special value that can be used as the stdin, 
#   stdout or stderr argument to Popen and indicates 
#   that a pipe to the standard stream should be opened.
# communicate(): returns tuple (stdout_data, stderr_data)
p = Popen(['echo', "Hello World"], stdout=PIPE)
out, err = p.communicate()  
print("Standard output: {}".format(out))
print("Standard error: {}".format(err))<jupyter_output>Standard output: b'Hello World\n'
Standard error: None<jupyter_text>2.0. Standard Error<jupyter_code>import subprocess
try:
    subprocess.run(['false'], check=True)
except subprocess.CalledProcessError as err:
    print('ERROR:', err)<jupyter_output>ERROR: Command '['false']' returned non-zero exit status 1.<jupyter_text>3. Pipe `|`<jupyter_code>p1 = Popen(["ls", "-l"], stdout=PIPE)
p2 = Popen(["grep", "subprocess.*"], stdin=p1.stdout, stdout=PIPE)  # stdout to null, set stdout to: subprocess.DEVNULL
p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p2 exits.
out, err = p2.communicate()
print("Standard output: {}".format(out))
print("Standard error: {}".format(err))<jupyter_output>Standard output: b'-rw-rw-r-- 1 qz qz 5661 Jun 18 23:29 subprocess.ipynb\n'
Standard error: None
{'fraction_non_alphanumeric': 0.15176908752327747, 'fraction_numerical': 0.03227808814400993, 'mean_word_length': 5.23404255319149, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '569819', 'license': 'MIT', 'n_tokens_mistral': 1228, 'n_tokens_neox': 1145, 'n_words': 349, 'path': 'src/subprocess.ipynb', 'repo_name': 'fordham-msqf-official/practical-pythonic', 'size': '3222'}