<jupyter_start><jupyter_text>CORD-19-Research-Data-Set-AtlasThis workbook covers the process of querying data from a db hosted on Atlas.For this code to run, you'll need to create a MongoDB on Atlas named "covid-19" with a collection named "noncomm-subset".  Download dataYou can download the data from:https://pages.semanticscholar.org/coronavirus-researchI'm using the non-commercial use subset (76mb download). To keep the example size manageable, I moved the first 70 records into the data directory of this repository.  Install Mongohttps://docs.mongodb.com/manual/installation/ Mongo ClientYou may want to get familiar with the MongoDB client and CRUD operations before working with python.  Build an Atlas MongoDB from the CORD-19 SubsetBuild a database name "covid-19", and a collection named "noncomm-subset".Add records through (substitute in your username, password, and mongoimport --uri "mongodb+srv://your_username:your_password@your_atlas_uri/covid-19" --collection noncomm-subset --drop --file filename.jsonfor example, to upload one file, mine would be (still redacting password and username)mongoimport --uri "mongodb+srv://username:password@python-mongodb-workshop.unmjr.gcp.mongodb.net/covid-19" --collection noncomm-subset --drop --file 1b58422e266ab9339c919119923229d080f27360.json<jupyter_code>import json
from pymongo import MongoClient
MDB_URL = "mongodb+srv://pymongo_workshop_user:pwd@python-mongo-workshop.jkemn.gcp.mongodb.net/"
client = MongoClient(MDB_URL)
client.list_database_names()
db = client.get_database("covid-19")
db.list_collection_names()
pmc_content = db['noncomm-subset']
for c in db['noncomm-subset'].find().limit(10):
    print(c)
# to print the titles only and suppress the id

for c in db['noncomm-subset'].find({},{ 'metadata.title': 1, '_id': 0 }):
    print(c)
# find one paper by paper_id

for c in db['noncomm-subset'].find({'paper_id': '00a00d0edc750db4a0c299dd1ec0c6871f5a4f24'}):
    print(c)
# query on nested documents
# see: https://docs.mongodb.com/manual/tutorial/query-embedded-documents/

for c in db['noncomm-subset'].find({'metadata.title': 'ACE/ACE2 Ratio and MMP-9 Activity as Potential Biomarkers in Tuberculous Pleural Effusions'}):
    print(c)
# query on an array of embedded documents
# see: https://docs.mongodb.com/manual/tutorial/query-array-of-documents/

for c in db['noncomm-subset'].find({'metadata.authors.first': 'Wen-Yeh'}):
    print(c)<jupyter_output><empty_output><jupyter_text>Query on a text indexTo query on a search phrase or word, you'll need to build a text index on the fields you want to search. For this tutorial, we'll do this with the MongoDB shell. Note - if the field is nested you'll need to put it in qoutation marks when you build the text index.To build the index:```db.pmc_content.createIndex( { "body_text.text": "text" } )```To list all the indexes you have on a collection```db.pmc_content.getIndexes()```To remove the index```db.pmc_content.dropIndex("body_text.text_text")```<jupyter_code>import re
regx = re.compile("pleural", re.IGNORECASE)
for r in db['noncomm-subset'].find({'body_text.text': { '$regex': regx } }):
    print(r)
# query the text index field created on the body_text field
# in the browser use { "body_text.text": "text" } 
for c in db['noncomm-subset'].find({'$text':{'$search':'Pleural'}}):
    print(c)
for a in db['noncomm-subset'].aggregate([{'$group':{'_id':{'$arrayElemAt': ['$metadata.authors.affiliation.location.country', 0]},
                                                    'count':{'$sum': 1}}}]):
    print(a)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12639508928571427, 'fraction_numerical': 0.022879464285714284, 'mean_word_length': 5.005025125628141, 'pattern_counts': {'":': 2, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '296719', 'license': 'MIT', 'n_tokens_mistral': 1234, 'n_tokens_neox': 1129, 'n_words': 340, 'path': 'Covid-19-AllenAI.ipynb', 'repo_name': 'geoffswc/MongoDB-Python-Workshop', 'size': '3584'}