<jupyter_start><jupyter_text>Load all the necessary libraries, these should already be installed. If you need additional libraries, add them to the dependencies.r files and rebuild the image<jupyter_code>library(knitr)
library(plotly)
library(httr)
library(jsonlite)
library(purrr)
library(magrittr)
library(GetoptLong)<jupyter_output><empty_output><jupyter_text>Functions that will eventually be in an R library and maintained alongside the API This is PIC-SURE-HPDS specific code, the same functionality is available through PIC-SURE 2's HPDS resource, but this saves you the PIC-SURE ceremonial wrapping of JSON in more JSON.<jupyter_code>escapeSlashes <- function(field){
    return (gsub("\\", "\\\\", field, fixed=TRUE))
}

wrapInQuotes <- function(value){
    return (paste('"', escapeSlashes(value), '"', sep=''))
}

numericFilter <- function(field, min, max){
    field <- escapeSlashes(field)
    return (qq('"@{field}":{"min":@{min},"max":@{max}}'))
}

categoryFilter <- function(field, categories){
    field <- escapeSlashes(field)
    categories <- paste(map(categories, wrapInQuotes), collapse=', ')
    return (qq('"@{field}":[@{categories}]'))
}


dictionary <- function(query){
    response <- POST("http://pic-sure-hpds-nhanes:8080/PIC-SURE/search/", body=paste('{"query":"',query,'"}', sep=""), content_type_json())
    response
    mergeCategoryValues <- function(entry){
        entry$categoryValues
        entry$categoryValues <- paste(entry$categoryValues, collapse=", ")
        return (entry)        
    }
    
    return(map_df(content(response)$results, mergeCategoryValues))
}    

dictionaryFields <- function(query){
    as.list(dictionary(query)$name)
}

query <- function(selectedFields = NULL, requiredFields = NULL, numericFilters = NULL, categoryFilters = NULL) {
    fields <- function(selectedFields){
        paste(map(selectedFields, wrapInQuotes), collapse=', ')
    }

    query <- qq(
        '{"fields":[@{fields}],
        "requiredFields":[@{requiredFields}],
        "numericFilters":{@{numericFilters}},
        "categoryFilters":{@{categoryFilters}}}', 
          envir = list(
              fields = fields(selectedFields), 
              requiredFields = fields(requiredFields), 
              numericFilters = paste(numericFilters, sep=","), 
              categoryFilters = paste(categoryFilters, sep=",")))
    query
    response <- POST("http://pic-sure-hpds-nhanes:8080/query", body=query, content_type_json())
    content(response)
}

unlock <- function(key){
    query <- qq('{"resourceCredentials":{"key":"@{key}"}}', envir=list(key=key));
    query
    response <- POST("http://pic-sure-hpds-nhanes:8080/PIC-SURE/query", body=query, content_type_json())
    content(response)$resourceStatus
}

getStatus <- function(queryId){
    response <- GET(paste("http://pic-sure-hpds-nhanes:8080/query", queryId, "status", sep="/"))
    content(response)
}

getResult <- function(queryId){
    return(read.csv(paste("http://pic-sure-hpds-nhanes:8080/query", queryId, "result", sep="/")))
}

runQuery <- function(selectedFields = NULL, requiredFields = NULL, numericFilters = NULL, categoryFilters = NULL){
    queryStatus <- query(selectedFields, requiredFields, numericFilters, categoryFilters)
    while(queryStatus$status == "RUNNING" || queryStatus$status == "PENDING"){
        Sys.sleep(.1)
        queryStatus <- getStatus(queryStatus$id)
    }
    if(queryStatus$status == "SUCCESS" && queryStatus$numRows > 0){
        print(paste(
            queryStatus$numRows, " rows and ", 
            queryStatus$numColumns, " columns ", 
            "completed in : ", (queryStatus$completedTime - queryStatus$queuedTime)/1000, " seconds", sep=""))
        return (queryStatus)
    }else{
        queryStatus
    }
}

runQueryAndGetResult <- function(selectedFields = NULL, requiredFields = NULL, numericFilters = NULL, categoryFilters = NULL){
    queryStatus <- runQuery(selectedFields, requiredFields, numericFilters, categoryFilters)
    if(queryStatus$status == "SUCCESS" && queryStatus$numRows > 0){
        return (getResult(queryStatus$id))
    }else{
        queryStatus
    }
}

unlock("85ad01907311feb507c73d0a5cbbfff6")<jupyter_output><empty_output><jupyter_text>Let's start by looking at the whole dictionary file Notice that the important data is all here, if it's continuous we get min and max values, if it's categorical we get all the available categories for filtering. We also get a count of observations, at this point this is a patient count only because we haven't dealt with temporal relationships yet.<jupyter_code>dictionary("")<jupyter_output><empty_output><jupyter_text>Let's look at something more interesting... all variables related to "blood"<jupyter_code>dictionary("blood")<jupyter_output><empty_output><jupyter_text>What if we just want one concept, like "Basophils percent"?  We call the dictionaryFields helper function because it finds us the full concept path for our term.<jupyter_code>queryStatus = runQuery(dictionaryFields("Basophils percent"))
getResult(queryStatus$id)<jupyter_output><empty_output><jupyter_text>What about all concepts matching "blood" for all patients age 5-25 who have Basophils percent data<jupyter_code>numFilters <- c(numericFilter("\\demographics\\AGE\\",5,25))
queryStatus = runQuery(selectedFields=dictionaryFields("blood"), 
                     requiredFields=dictionaryFields("Basophils percent"), 
                     numericFilters=numFilters)
getResult(queryStatus$id)<jupyter_output><empty_output><jupyter_text>How about all data for concepts containing the term "blood" but only where patients have a Basophils percent?  Notice we are requiring patients to have all fields that match the search "Basophils percent". If we try this for the search term "blood" it will require all fields that have blood in their path, which matches no patients so we would get no results.<jupyter_code>queryStatus = runQuery(selectedFields=dictionaryFields("blood"), 
                     requiredFields=dictionaryFields("Basophils percent"))

getResult(queryStatus$id)<jupyter_output><empty_output><jupyter_text>OK, so how about all data for concepts matching "blood" for all patients?  Notice here we just pass the result of our dictionary search, but by using the convenient dictionaryFields function that handles escaping our terms for us and building a vector of fields.<jupyter_code>queryStatus = runQuery(dictionaryFields("blood"))
getResult(queryStatus$id)<jupyter_output><empty_output><jupyter_text>This is better, what about only Male patients who are of white or mexican decent and we want to add all lab values?<jupyter_code>numFilters <- c(numericFilter("\\demographics\\AGE\\",5,25))
catFilters <- c(categoryFilter("\\demographics\\RACE\\", c("white", "mexican")))
queryStatus = runQuery(selectedFields=union(dictionaryFields("laboratory"), dictionaryFields("blood")), 
                     requiredFields=dictionaryFields("Basophils percent"), 
                     numericFilters=numFilters, categoryFilters=catFilters)
getResult(queryStatus$id)<jupyter_output><empty_output><jupyter_text>Now if we really just want all the data, we can do something like this<jupyter_code>queryStatus = runQuery(dictionaryFields(""))
getResult(queryStatus$id)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12339085182141879, 'fraction_numerical': 0.0075321829635716245, 'mean_word_length': 4.809864757358791, 'pattern_counts': {'":': 11, '<': 71, '<?xml version=': 0, '>': 43, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '606300', 'license': 'Apache-2.0', 'n_tokens_mistral': 2174, 'n_tokens_neox': 2048, 'n_words': 533, 'path': 'jupyter-notebooks/PIC-SURE-HPDS NHANES.ipynb', 'repo_name': 'hms-dbmi/pic-sure-hpds-nhanes', 'size': '7302'}