<jupyter_start><jupyter_text>Working with arbitrary subkernels * **Difficulty level**: easy* **Time need to lean**: 20 minutes or less* **Key points**:  * SoS works with any Jupyter kernel  * `%cd` changes directory of all subkernels  * `%expand` expands expressions in the cell input before sending it to subkernels  * `%capture` captures output from subkernels and save them into Python variables  * `%render` renders output from subkernels in different formats A SoS kernel is a master kernel that can start, stop, and interact with any Jupyter kernel, which are called *subkernels* by SoS. Although most of the times SoS just passes user input to subkernels and sendes outputs from subkernels to the frontend (notebook), you can use a few SoS magics to modify user inputs before they are sent to the subkernel, and process outputs before they are sent to the frontend.     Change workding directory (magic `%cd`) When working with multiple kernels in a notebook, it could be confusing if kernels have different working directory. The rule of thumb in SoS are as follows:First, **subkernels starts at the current working directory of the SoS kernel**<jupyter_code>import os
os.getcwd()
getwd()<jupyter_output><empty_output><jupyter_text>Second, **changing working directory in one subkernel will not affect other living kernels**<jupyter_code>setwd('..')
getwd()
os.getcwd()<jupyter_output><empty_output><jupyter_text>Third, **magic `%cd` changes working directory of all subkernels**<jupyter_code>%cd /Users/bpeng1/sos/sos-docs
os.getcwd()
getwd()<jupyter_output><empty_output><jupyter_text>Expand input (magic `%expand`) Script in SoS cells are by default sent to SoS or the subkernels in verbatim. For example,<jupyter_code>print("A parameter {par} is specified.")<jupyter_output>A parameter {par} is specified.<jupyter_text>With variable `par` defined in the SoS kernel,<jupyter_code>par = 100<jupyter_output><empty_output><jupyter_text>the `%expand` treats the content of the following cell as a [Python f-string](https://www.python.org/dev/peps/pep-0498/) and expands expressions inside braces:<jupyter_code>%expand
print("A parameter {par} is specified.")<jupyter_output>[1] "A parameter 100 is specified."<jupyter_text>If the script contains `{ }`, which is quite common in R, you can double the braces<jupyter_code>%expand
if ({par} > 50) {{
    print("A parameter {par} greater than 50 is specified.");
}}<jupyter_output>[1] "A parameter 100 greater than 50 is specified."<jupyter_text>If there are multiple braces, it is obviously better to use a different sigil, such as `${ }` to interpolate the script<jupyter_code>%expand ${ }
if (${par} > 50) {
    print("A parameter ${par} greater than 50 is specified.");
}<jupyter_output>[1] "A parameter 100 greater than 50 is specified."<jupyter_text>Although not the topic of this tutorial, it is worth mentioning that the usage of the `%expand` magic is the same as the `expand` option of SoS actions so that you can convert the above script that was executed in a R session to an R action in a SoS workflow as follows:<jupyter_code>R: expand='${ }'
if (${par} > 50) {
    print("A parameter ${par} greater than 50 is specified.");
}<jupyter_output>[1] "A parameter 100 greater than 50 is specified."<jupyter_text>Expand input in subkernels (magic `%expand --in`) If a language module supports the `expand` interface (not all of them does), you can expand the content of a cell in specified kernel. This is most naturally used in a [`markdown` kernel](https://github.com/vatlab/markdown-kernel) when the expanded text are treated as markdown text. By default the markdown kernel processes the input text literally:<jupyter_code>* Hello, the value of a is {a}<jupyter_output><empty_output><jupyter_text>With the `%expand` magic, you can expand expressions with variables in SoS kernel (default).<jupyter_code>a = 5
%expand
Hello, the value of a is {a}<jupyter_output><empty_output><jupyter_text>If you have variables in R,<jupyter_code>a <- 100<jupyter_output><empty_output><jupyter_text>You can expand the content of the cell using variables (and expressions) in R:<jupyter_code>%expand --in R
Hello, the value of a**2 is {a**2}<jupyter_output><empty_output><jupyter_text>If you are more familiar with delimiters in RMarkdown, you can specify the delimiter as option of `%expand`.<jupyter_code>%expand `r ` --in R
Hello, the value of a**2 is `r a**2`<jupyter_output><empty_output><jupyter_text>Capture cell output (magic `%capture`) Magic `%capture` all or part of the output of a cell to a SoS variable. To understand how this magic works, you will need to understand [how Jupyter  works](https://jupyter-client.readthedocs.io/en/stable/messaging.html). Briefly, after a cell is executed, the kernel sends one or more of messages `stream`, `display_data`, and other controlling messages before it sends `execute_result` to conclude the execution. The `stream` message type can contain standard output (`stdout`) and standard error output (`stderr`), and the `display_data` message can contain a lot more complex data under a dictionary with keys `text/html`, `text/plain`, `text/markdown` etc, and the frontend will decide how to display these messages.  Determine what to captureThe `%capture` magic can capture the following types of information| name | message ||-- | -- || `stdout` | `stdout` of `stream` messages || `stderr` | `stderr` of `stream` messages || `text` | `text/plain` of `display_data` or `execute_result` messages || `html` | `text/html` of `display_data` or `execute_result` messages || `markdown` | `text/markdown` of `display_data` or `execute_result` messages || `error` | `evalue` of `error` message || `raw` | All above messages |The first step to capture output from a cell is to determine what types of messages are sent by the cell. If you are uncertain, you can open the console panel (right click and select `New Console for Notebook` if you are using Jupyter Lab), and use the `%capture` magic without option (or with the `raw` option).<jupyter_code>%capture
echo "I am from Bash"<jupyter_output>I am from Bash<jupyter_text>The messages that has been returned by the cell will be displayed in the console window```python[('stream', {'name': 'stdout', 'text': 'I am from Bash\n'})]```for this cell, from which you can see that the message is of type `stdout`. You can then specify the `stdout` type,<jupyter_code>%capture stdout
echo "I am from Bash"<jupyter_output>I am from Bash<jupyter_text>The captured result is by default saved to a variable `__captured` in the SoS kernel:<jupyter_code>__captured<jupyter_output><empty_output><jupyter_text>You can use option `-t` (`--to`) to assign the name of the variable<jupyter_code>%capture stdout --to bash_output
echo "I am from Bash"
bash_output<jupyter_output><empty_output><jupyter_text>As a more complex example, the following cell runs a SPARQL query and returns multiple messages.<jupyter_code>%capture

%format json
%display table 
%endpoint http://dbpedia.org/sparql

PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX foaf:  <http://xmlns.com/foaf/0.1/>
SELECT ?person ?name
WHERE {
    ?person a foaf:Person .
    ?person rdfs:label ?name
    FILTER regex(?name,"da Vinci","i")
    FILTER langMatches(lang(?name),"en")
} LIMIT 2<jupyter_output><empty_output><jupyter_text>The `__captured` variable shows all returned messages<jupyter_code>__captured<jupyter_output><empty_output><jupyter_text>You can then save the content of `text/html` to a variable `html_table`<jupyter_code>%capture html --to html_table

%format json
%display table 
%endpoint http://dbpedia.org/sparql

PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX foaf:  <http://xmlns.com/foaf/0.1/>
SELECT ?person ?name
WHERE {
    ?person a foaf:Person .
    ?person rdfs:label ?name
    FILTER regex(?name,"da Vinci","i")
    FILTER langMatches(lang(?name),"en")
} LIMIT 2<jupyter_output><empty_output><jupyter_text>which contains the `text/html` data of two messages<jupyter_code>print(html_table)<jupyter_output><div class="krn-spql"><div class="magic">Return format: JSON</div><div class="magic">Display: table</div><div class="magic">Endpoint set to: http://dbpedia.org/sparql</div></div><div class="krn-spql"><table><tr class=hdr><th>person</th>
<th>name</th></tr><tr class=odd><td class=val><a href="http://dbpedia.org/resource/Paul_Da_Vinci" target="_other">http://dbpedia.org/resource/Paul_Da_Vinci</a></td>
<td class=val>Paul Da Vinci</td></tr><tr class=even><td class=val><a href="http://dbpedia.org/resource/Leonardo_da_Vinci" target="_other">http://dbpedia.org/resource/Leonardo_da_Vinci</a></td>
<td class=val>Leonardo da Vinci</td></tr></table><div class="tinfo">Total: 2, Shown: 2</div></div><jupyter_text>Then, if you would like to process the output programatically, you can use one of the many powerful Python modules<jupyter_code>from bs4 import BeautifulSoup
soup = BeautifulSoup(html_table, 'html.parser')
soup.find_all('a')<jupyter_output><empty_output><jupyter_text>Capture formatted content If the output of a cell is well-formatted, it is possible to capture the output as variables in a type other than `str`.For example, if you would like to capture the size of some files from a few notebook files. Instead of using Python scripts, you could possibly use a shell command as follows<jupyter_code>!ls -l ex*.ipynb  | awk '{{printf("%s,%d\n", $10, $6)}}'<jupyter_output><empty_output><jupyter_text>The output is well formatted so you can capture it in csv format as follows<jupyter_code>%capture stdout --as csv --to notebooks
!ls -l ex*.ipynb  | awk '{{printf("%s,%d\n", $10, $6)}}'<jupyter_output><empty_output><jupyter_text>The resulting variable is a Pandas DataFrame but unfortunately treated the first data line as header, which is not entirely correct here.<jupyter_code>notebooks<jupyter_output><empty_output><jupyter_text>The `%capture` magic can capture data in `text` (default), `json`, `csv`, and `tsv` format, and can append to instead of replacing an existing variable (option `-a`). Please refer to the [SoS Magics reference](sos_magics.html) or command `%capture -h` for a comlete list of options.  Render cell output (magic `%render`) The `%render` magic intercepts the output of a cell, convert it to certain format before displaying it in the notebook. The format can be any format supported by the [`IPython.display` module](https://ipython.org/ipython-doc/3/api/generated/IPython.display.html) and is default to `Markdown`. For example, if you have a dataset<jupyter_code>data = [('John', 'Smith', 50),
        ('Eve', 'Jackson', 35)
       ]<jupyter_output><empty_output><jupyter_text>You can format it in HTML format<jupyter_code>res = '''
<table>
  <tr>
    <th>Firstname</th>
    <th>Lastname</th> 
    <th>Age</th>
  </tr>
'''
for first_name, last_name, age in data:
    res += f'''
  <tr>
    <td>{first_name}</td>
    <td>{last_name}</td> 
    <td>{age}</td>
  </tr>
'''
res += '</table>'

print(res)<jupyter_output><table>
  <tr>
    <th>Firstname</th>
    <th>Lastname</th> 
    <th>Age</th>
  </tr>

  <tr>
    <td>John</td>
    <td>Smith</td> 
    <td>50</td>
  </tr>

  <tr>
    <td>Eve</td>
    <td>Jackson</td> 
    <td>35</td>
  </tr>
</table><jupyter_text>and render it as a HTML table.<jupyter_code>%render --as HTML
res<jupyter_output><empty_output><jupyter_text>Currently `%render` only renders `stdout` (of `stream` messages, default) and `text` (`text/plain` of `display_data` messages) contents, and you should probably use `%capture raw` to check the type of output before you `%render`. The `%render` magic accepts any renderer that is defined in the `IPython.display` module. The following cell lists all renderers,<jupyter_code>import IPython.display
import inspect

print('Options of magic %render')
for key in IPython.display.__dict__.keys():
    cls = getattr(IPython.display, key)
    if inspect.isclass(cls) and issubclass(cls, IPython.display.DisplayObject):
        print('* {}'.format(key))<jupyter_output>Options of magic %render<jupyter_text>and of course a `%render` magic would treat the output as markdown format and display the items as bullet points:<jupyter_code>%render
import IPython.display
import inspect

print('Options of magic %render')
for key in IPython.display.__dict__.keys():
    cls = getattr(IPython.display, key)
    if inspect.isclass(cls) and issubclass(cls, IPython.display.DisplayObject):
        print('* {}'.format(key))<jupyter_output><empty_output><jupyter_text>The ability to render text output as markdown text alleviatea a problem with the Jupyter notebooks in that its markdown cells cannot contain variables, so you cannot really mix results with their descriptions as easily as what Rmarkdown inline expressions do. However, with the `%render` magic, you can write markdown text as a string in any kernel, and use the `%render` magic to display it. For example, if you have `res` obtained from some analysis<jupyter_code>res <- rnorm(5)<jupyter_output><empty_output><jupyter_text>You can report the result by generating a markdown text programmatically and use the `%render` magic to render it<jupyter_code>%render
cat(paste0('Array of size ', length(res), '\n\n'),
    paste('*', res, '\n'))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11708860759493671, 'fraction_numerical': 0.006931886678722122, 'mean_word_length': 5.528775209050664, 'pattern_counts': {'":': 0, '<': 209, '<?xml version=': 0, '>': 210, 'https://': 4, 'lorem ipsum': 0, 'www.': 3, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '33', 'dir': 'jupyter-structured-clean-dedup', 'id': '459136', 'license': 'MIT', 'n_tokens_mistral': 4328, 'n_tokens_neox': 4162, 'n_words': 1503, 'path': 'src/user_guide/expand_capture_render.ipynb', 'repo_name': 'sosdocs/sos-docs', 'size': '13272'}