<jupyter_start><jupyter_text>Download SmartSeq2 Expression Matrix as an Input to Scanpy Suppose I want to get a SmartSeq2 expression matrix that I can analyze using scanpy. How can I go about finding something like this using the DSS API?<jupyter_code>import hca.dss
client = hca.dss.DSSClient()<jupyter_output><empty_output><jupyter_text>Well, first things first: We're going to need to search for a `.results` file. This file should contain what we need to put together an expression matrix.Here's our ElasticSearch query.<jupyter_code>query = {
    "query": {
        "bool": {
            "must": [
                {
                    "wildcard": {
                        "manifest.files.name": {
                            # We need a *.results file...
                            "value": "*.results"
                        }
                    }
                },
                {
                    "range": {
                        "manifest.version": {
                            # ...and preferably not too old, either.
                            "gte": "2018-07-12T100000.000000Z"
                        }
                    }
                }
            ]
        }
    }
}<jupyter_output><empty_output><jupyter_text>This query looks for bundles that satisfy a *bool*ean condition consisting of two checks, both of which *must* be true. The *match* check looks for the `manifest.files.name` field and returns true if the name of a file listed in a bundle manifest ends with 'results'. The second check, *range*, returns true if the bundle's `manifest.version` has a value greater than or equal to 7/12/18. In short, __this query will find bundles that contain a `.results` file and are newer than July 12, 2018.__ Okay, now let's execute the search. Since the `files` section of the bundle is pretty long, I'll only print the portion containing a results file. If you want, you can always print the entire bundle to get a better picture of where the file is located.<jupyter_code>import json

# Print part of a recent analysis bundle with a results file
bundles = client.post_search(es_query=query, replica='aws', output_format='raw')
the_first_bundle = bundles['results'][0]
bundle_files = the_first_bundle['metadata']['manifest']['files']
for f in bundle_files:
    if f['name'].endswith('.results'):
        results_file_uuid, results_file_version = f['uuid'], f['version']
print(results_file_uuid, results_file_version)<jupyter_output>ec727ae1-d47a-47a3-8c8e-b42d7a0e8cf4 2019-05-18T173116.989870Z<jupyter_text>Okay! It looks like we've found a file uuid we can use. Let's retrieve that file and save it locally.<jupyter_code>results = client.get_file(replica='aws', uuid=results_file_uuid, version=results_file_version)
open('matrix.results', 'w').write(results.decode("utf-8"))<jupyter_output><empty_output><jupyter_text>Here's what our file, `matrix.results`, looks like. I've truncated the output so it doesn't take up too much room.<jupyter_code>print(open('matrix.results', 'r').read()[:852])<jupyter_output>transcript_id	gene_id	length	effective_length	expected_count	TPM	FPKM	IsoPct	posterior_mean_count	posterior_standard_deviation_of_count	pme_TPM	pme_FPKM	IsoPct_from_pme_TPM
ENST00000373020.8	ENSG00000000003.14	2206	2016.44	0.00	0.00	0.00	0.00	0.00	0.00	0.12	0.21	9.99
ENST00000494424.1	ENSG00000000003.14	820	630.44	0.00	0.00	0.00	0.00	0.00	0.00	0.38	0.68	31.95
ENST00000496771.5	ENSG00000000003.14	1025	835.44	0.00	0.00	0.00	0.00	0.00	0.00	0.28	0.51	24.11
ENST00000612152.4	ENSG00000000003.14	3796	3606.44	0.00	0.00	0.00	0.00	0.00	0.00	0.07	0.12	5.59
ENST00000614008.4	ENSG00000000003.14	900	710.44	0.00	0.00	0.00	0.00	0.00	0.00	0.33	0.60	28.36
ENST00000373031.4	ENSG00000000005.5	1339	1149.44	0.00	0.00	0.00	0.00	0.00	0.00	0.21	0.37	23.47
ENST00000485971.1	ENSG00000000005.5	542	352.44	0.00	0.00	0.00	0.00	0.00	0.00	0.67	1.22	76.53
ENST00000371582.8<jupyter_text>For our matrix, however, we might only want _some_ of these values. In my case, suppose I only want the `gene_id` and `TPM` values. We can extract these values easily using Python's `csv` module.<jupyter_code>import csv

# Take the data we want out of the results file and store it into a tsv file

with open('matrix.results', 'r') as infile, open('matrix.tsv', 'w', newline='') as outfile:
    reader = csv.DictReader(infile, delimiter='\t')
    writer = csv.DictWriter(outfile, fieldnames=['gene_id', 'TPM'], delimiter='\t')
    for row in reader:
        writer.writerow({'gene_id': row['gene_id'], 'TPM': row['TPM']})<jupyter_output><empty_output><jupyter_text>Our new file, `matrix.tsv`, looks something like this:<jupyter_code>print(open('matrix.tsv', 'r').read()[:214])<jupyter_output>ENSG00000000003.14	0.00
ENSG00000000003.14	0.00
ENSG00000000003.14	0.00
ENSG00000000003.14	0.00
ENSG00000000003.14	0.00
ENSG00000000005.5	0.00
ENSG00000000005.5	0.00
ENSG00000000419.12	0.00
ENSG00000000419.12	0.00<jupyter_text>Now that we have a file containing what we want, we can transpose it and read it into scanpy.<jupyter_code>import scanpy.api as sc

adata = sc.read_csv(filename='matrix.tsv', delimiter='\t').transpose()<jupyter_output>Observation names are not unique. To make them unique, call `.obs_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.<jupyter_text>But how do we know that everything worked? Let's print our AnnData object (truncating the output again).<jupyter_code>print(adata)
for i in range(0, 153):
    print('{:<6}'.format('{:.1f}'.format(adata.X[i])), end='' if (i + 1) % 17 != 0 else '\n' )<jupyter_output>AnnData object with n_obs × n_vars = 1 × 200468 
0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   
0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   6.5   0.0   0.0   0.0   0.0   0.0   167.1 
0.0   3.2   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   
0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   
0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   
0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   
0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   
0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   
0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0<jupyter_text>And just to make it easier to see the relevant data in our matrix...<jupyter_code>for i in range(len(adata.X)-1):
    if adata.X[i] != 0:
        print(adata.X[i], end=' ')<jupyter_output>6.53 167.05 3.25 2.52 3.04 1.0 7.05 6.08 12.84 10.88 3.11 10.06 8.9 12.26 9.91 1.38 11.71 69.56 89.88 1.45 0.53 10.06 64.21 0.85 39.42 21.5 2.81 21.39 2.46 1.37 0.57 41.7 1.05 6.21 3.64 5.22 11.23 48.57 0.22 6.72 8.65 2.24 0.57 0.86 1.27 1.17 2.48 0.91 9.6 168.78 7.53 434.51 279.39 3.62 1.31 2.09 1.75 130.58 1.08 0.66 7.79 5.89 13.77 5.16 3.22 2.04 0.4 1.67 3.34 2.07 2.22 0.39 11.2 4.03 0.65 2.68 1.93 2.8 36.79 1.35 30.25 2.69 11.25 0.51 22.75 1.92 0.86 0.52 49.27 634.34 650.86 149.68 0.67 24.23 1.02 5.38 10.23 0.38 4.57 11.69 23.09 4.82 4.11 8.05 0.63 10.47 5.71 1.76 1.7 1.3 8.48 0.8 11.97 2.45 11.12 1.39 3.75 31.57 0.46 3.08 2.78 1.91 9.58 11.81 14.14 548.93 1033.31 2.02 11.71 1.48 6.2 1.87 2.8 1.62 2.94 4.99 37.99 0.92 8.03 3.22 65.9 6.83 47.61 2.16 0.44 15.74 6.07 1.0 0.42 2.15 0.88 2.23 5.08 1.11 4.08 4.42 2.3 1.11 19.06 0.62 4.89 1.44 9.8 1.55 1.64 6.58 0.81 1.24 2.85 2.9 17.77 25.15 0.49 4.73 0.04 0.78 2.04 2.26 2.43 0.44 1.34 24.36 2.94 6.62 6.65 21.71 20.38 4.72 0.83 10.44 5.5[...]<jupyter_text>Okay, so we have one AnnData object we can use with scanpy. __But what if we want a second one?__ Let's go through the steps again, this time with a different `.results` file. We can use the same query to get a bunch of analysis bundles, but this time get our `.results` file from the _second_ bundle.<jupyter_code>the_second_bundle = bundles['results'][1]
bundle_files = the_second_bundle['metadata']['manifest']['files']
for f in bundle_files:
    if f['name'].endswith('.results'):
        results_file_uuid, results_file_version = f['uuid'], f['version']<jupyter_output><empty_output><jupyter_text>With the new uuid, we can get the `.results` file itself.<jupyter_code>results2 = client.get_file(replica='aws', uuid=results_file_uuid, version=results_file_version)
open('matrix2.results', 'w').write(results2.decode("utf-8"))<jupyter_output>Waiting 10s before redirect per Retry-After header<jupyter_text>Again, let's take the data we want out of the results file and store it into a tsv file...<jupyter_code>with open('matrix2.results', 'r') as infile, open('matrix2.tsv', 'w', newline='') as outfile:
    reader = csv.DictReader(infile, delimiter='\t')
    writer = csv.DictWriter(outfile, fieldnames=['gene_id', 'TPM'], delimiter='\t')
    for row in reader:
        writer.writerow({'gene_id': row['gene_id'], 'TPM': row['TPM']})<jupyter_output><empty_output><jupyter_text>...and then create another AnnData object & print it.<jupyter_code>adata2 = sc.read_csv( filename='matrix.tsv', delimiter='\t' ).transpose()

print(adata2)
for i in range(0, 153):
    print( '{:<6}'.format('{:.1f}'.format(adata2.X[i])), end='' if (i + 1) % 17 != 0 else '\n' )<jupyter_output>Observation names are not unique. To make them unique, call `.obs_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
{'fraction_non_alphanumeric': 0.13104880581516096, 'fraction_numerical': 0.16926272066458983, 'mean_word_length': 2.744556765163297, 'pattern_counts': {'":': 9, '<': 51, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 16, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '665287', 'license': 'MIT', 'n_tokens_mistral': 4737, 'n_tokens_neox': 3851, 'n_words': 1133, 'path': 'Download Expression Matrix for Scanpy/Download SmartSeq2 Expression Matrix as Input to Scanpy.ipynb', 'repo_name': 'HumanCellAtlas/data-consumer-vignettes', 'size': '9630'}