<commit_before># Walkthrough for the EhrscapeR package

-   Introduction
-   Data query
    -   Make HTTP request
    -   GET response
    -   Parse response
    -   Convert to data frame
-   Data query - csv flattening
-   Save data 
    -   Make HTTP request
    -   Prepare composition
    -   POST data
    -   Parse response


## Introduction

[EhrScape](https://www.ehrscape.com/index.html) is an open health data platform providing a rich service framework that allows you to store, query and retrieve electronic health records based on the [openEHR](http://www.openehr.org/) specification. The data is available as [JSON](https://en.wikipedia.org/wiki/JSON) via a [REST](https://en.wikipedia.org/wiki/Representational_state_transfer) web service architecture. 

The overall approach is generic â€“ we want to obtain healthcare data by calling a REST API, and then format the returned result set in R to ready the data for more sophisticated analysis.

RESTful applications use HTTP requests to read data (e.g., make queries) and post data (create and/or update). At its most basic, calling a REST API to obtain data involves making a HTTP GET request to a server. If the call succeeds, youâ€™ll have a document that contains the requested data. To store data using REST API, a HTTP POST request needs to be made to the server along with the newly created data. 

This tutorial describes the `EhrscapeR` package, showcasing the use of R for handling data from EhrScape platform using REST API:

* [GET /query](https://www.ehrscape.com/api-explorer.html?api=thinkehr&service=/query&operation=/query&method=get&inline=true): returns the results of the specified AQL query,
* [GET /query/csv](https://www.ehrscape.com/api-explorer.html?api=thinkehr&service=/query&operation=/query/csv&method=get&inline=true): returns the results of the specified AQL query as CSV, 
* [POST /composition](https://www.ehrscape.com/api-explorer.html?api=thinkehr&service=/composition&operation=/composition&method=post&inline=true): creates and commits a contribution consisting of multiple composition creations, updates or deletes.

Please check the [EhrScape API Explorer](https://www.ehrscape.com/api-explorer.html) for a complete list of REST API. 

The basis of this package are functions from the:

- **httr** package: functions for making HTTP requests
- **jsonlite** package: functions for parsing JSON formated data into data frames (R-suitable form)

An important part is parsing the obtained JSON data into R-usable form. The result set in JSON format can be complex in structure with several nested levels which requires flattening of the structure to the data frame.  

The package `EhrscapeR` includes functions: 

* `get_query`: for a given [AQL](http://www.openehr.org/releases/QUERY/latest/docs/AQL/AQL.html) query returns the result set in a data frame. 
* `get_query_csv`: for a given AQL query returns the CSV formated result set in a data frame. 
* `post_composition`: stores new data records (compositions) in EhrScape.

All the examples below can be checked on EhrScape API Explorer.


## Data query

Data is queried from EhrScape using the `get_query` function:


```r
query_data <- get_query(baseURL, credentials = c(user_name, password), aql_query, 
    full_path = FALSE)
```

Input parameters: 

  - `baseURL`: base url address of the REST service, 
  - `credentials`: authentication pair of username and password
  - `aql_query`: AQL query, 
  - `full_path`: logical value indicating the degree of flattening. If `FALSE`, each value (even repetitions on the same path) in the JSON is assigned a unique identifier. If `TRUE`, values are sorted in the outpt data frame according to JSON path and unique sequence within the same path (if there are several repetitions on the same JSON path).
  
The function connects to the server with provided credentials and AQL query. The response in JSON format is parsed to a data frame and returned as the result of the function. 

Example: Query body temperature, systolic and diastolic measurements for all patients


```r
aql_query <- 
  "select
    t/data[at0002|History|]/events[at0003|Any event|]/data[at0001]/items[at0004|Temperature|]/value as Temperature,
    bp/data[at0001|history|]/events[at0006|any event|]/data[at0003]/items[at0004|Systolic|]/value as Systolic,
    bp/data[at0001|history|]/events[at0006|any event|]/data[at0003]/items[at0005|Diastolic|]/value as Diastolic
from EHR e
contains COMPOSITION c
contains (
    OBSERVATION t[openEHR-EHR-OBSERVATION.body_temperature.v1] and
    OBSERVATION bp[openEHR-EHR-OBSERVATION.blood_pressure.v1])
offset 0 limit 100"

baseURL <- "https://rest.ehrscape.com/rest/v1/"
user_name <- "guidemo"
password <- "<PASSWORD>"
```

Result of the above query is: 

![Result of example query](./figures/EhrscapeR-walkthrough_files/figure-markdown_github/query_result.png)


### Make HTTP request

The function begins by preparing the AQL query for construction of the HTTP request. First, the `rm_white` function of the **qdapRegex** package removes from the input `aql_query` all multiple white spaces, white spaces before commas, semicolons, colons or endmarks, white spaces before or after brackets, leading and trailing white spaces. Next, line break and carriage return symbols are replaced with white space by using the `gsub` function. The edited AQL query string is then URL encoded (`query_enc`) and used to construct the HTTP request `URL_address` 


```r
aql_query <- qdapRegex::rm_white(aql_query)
aql_query <- gsub("\r?\n|\r", " ", aql_query)
query_enc <- utils::URLencode(aql_query)
URL_address = paste(baseURL, "query?aql=", query_enc, sep = "")
```

```
##  chr "https://rest.ehrscape.com/rest/v1/query?aql=select%20t/data[at0002"..
```

### GET response

The request `URL_address` is executed using the `GET` function of the **httr** package. The server response is stored as `resp`. Since EhrScape requires authentication, user credentials have to be included to the `GET` function call. Additionally, the `content_type_json` function is used to set the expected content type returned from the server to JSON. 


```r
resp <- httr::GET(URL_address, httr::authenticate(user_name, password), httr::content_type_json())
```

```
## List of 10
## $ url : chr
##    "https://rest.ehrscape.com/rest/v1/query?aql=select%20t/data[at0002%7"..
##    __truncated__
## $ status_code: int 200
## $ headers :List of 7
## ..$ date : chr "Wed, 16 Dec 2015 16:12:19 GMT"
## ..$ server : chr "Jetty(9.2.11.v20150529)"
## ..$ content-language: chr "en-US"
## ..$ content-type : chr "application/json;charset=UTF-8"
## ..$ vary : chr "Accept-Encoding"
## ..$ content-encoding: chr "gzip"
## ..$ content-length : chr "1094"
## ..- attr(*, "class")= chr [1:2] "insensitive" "list"
## $ all_headers:List of 1
## ..$ :List of 3
## .. ..$ status : int 200
## .. ..$ version: chr "HTTP/1.1"
## .. ..$ headers:List of 7
## .. .. ..$ date : chr "Wed, 16 Dec 2015 16:12:19 GMT"
## .. .. ..$ server : chr "Jetty(9.2.11.v20150529)"
## .. .. ..$ content-language: chr "en-US"
## .. .. ..$ content-type : chr "application/json;charset=UTF-8"
## .. .. ..$ vary : chr "Accept-Encoding"
## .. .. ..$ content-encoding: chr "gzip"
## .. .. ..$ content-length : chr "1094"
## .. .. ..- attr(*, "class")= chr [1:2] "insensitive" "list"
## $ cookies :'data.frame': 0 obs. of 7 variables:
## ..$ domain : logi(0)
## ..$ flag : logi(0)
## ..$ path : logi(0)
## ..$ secure : logi(0)
## ..$ expiration:Classes 'POSIXct', 'POSIXt' num(0)
## ..$ name : logi(0)
## ..$ value : logi(0)
## $ content : raw [1:14549] 7b 22 6d 65 ...
## $ date : POSIXct[1:1], format: "2015-12-16 16:12:19"
## $ times : Named num [1:6] 0 0 0.047 1.404 1.482 ...
## ..- attr(*, "names")= chr [1:6] "redirect" "namelookup" "connect"
##    "pretransfer" ...
## $ request :List of 7
## ..$ method : chr "GET"
## ..$ url : chr
##    "https://rest.ehrscape.com/rest/v1/query?aql=select%20t/data[at0002%7"..
##    __truncated__
## ..$ headers : Named chr [1:2] "application/json, text/xml,
##    application/xml, */*" "application/json"
## .. ..- attr(*, "names")= chr [1:2] "Accept" "Content-Type"
## ..$ fields : NULL
## ..$ options :List of 5
## .. ..$ useragent : chr "libcurl/7.43.0 r-curl/0.9.4 httr/1.0.0"
## .. ..$ cainfo : chr "C:/R/Rlibs/httr/cacert.pem"
## .. ..$ httpauth : num 1
## .. ..$ userpwd : chr "<PASSWORD>mo:gui?!<PASSWORD>"
## .. ..$ customrequest: chr "GET"
## ..$ auth_token: NULL
## ..$ output : list()
## .. ..- attr(*, "class")= chr [1:2] "write_memory" "write_function"
## ..- attr(*, "class")= chr "request"
## $ handle :Class 'curl_handle' <externalptr>
## - attr(*, "class")= chr "response"
```

### Parse response

Response `resp` includes data such as the HTTP request, status, headers, cookies, and also the answer to the request in JSON format. This is stored in the content field, which is extracted with the `content` function to a nested list `json`. 


```r
json <- httr::content(resp)
```

```
## List of 4
## $ meta :List of 1
## ..$ href: chr
##    "http://rest.ehrscape.com/rest/v1/query/?aql=select%20t/data%5Bat0002"..
##    __truncated__
## $ aql : chr "select t/data[at0002|History|]/events[at0003|Any
##    event|]/data[at0001]/items[at0004|Temperature|]/value as Temperature,
##    bp/data["| __truncated__
## $ executedAql: chr "select t/data[at0002|History|]/events[at0003|Any
##    event|]/data[at0001]/items[at0004|Temperature|]/value as Temperature,
##    bp/data["| __truncated__
## $ resultSet :List of 56
## ..$ :List of 3
## .. ..$ Temperature:List of 4
## .. .. ..$ @class : chr "DV_QUANTITY"
## .. .. ..$ magnitude: num 37.3
## .. .. ..$ units : chr "°C"
## .. .. ..$ precision: int 1
## .. ..$ Systolic :List of 3
## .. .. ..$ @class : chr "DV_QUANTITY"
## .. .. ..$ magnitude: num 99
## .. .. ..$ units : chr "mm[Hg]"
## .. ..$ Diastolic :List of 3
## .. .. ..$ @class : chr "DV_QUANTITY"
## .. .. ..$ magnitude: num 70
## .. .. ..$ units : chr "mm[Hg]"
## ..$ :List of 3
## .. ..$ Temperature:List of 4
## .. .. ..$ @class : chr "DV_QUANTITY"
## .. .. ..$ magnitude: num 38.4
## .. .. ..$ units : chr "°C"
## .. .. ..$ precision: int 1
## .. ..$ Systolic :List of 3
## .. .. ..$ @class : chr "DV_QUANTITY"
## .. .. ..$ magnitude: num 107
## .. .. ..$ units : chr "mm[Hg]"
## .. ..$ Diastolic :List of 3
## .. .. ..$ @class : chr "DV_QUANTITY"
## .. .. ..$ magnitude: num 58
## .. .. ..$ units : chr "mm[Hg]"
## ..$ :List of 3
## .. ..$ Temperature:List of 4
## .. .. ..$ @class : chr "DV_QUANTITY"
## .. .. ..$ magnitude: num 36.2
## .. .. ..$ units : chr "°C"
## .. .. ..$ precision: int 1
## .. ..$ Systolic :List of 3
## .. .. ..$ @class : chr "DV_QUANTITY"
## .. .. ..$ magnitude: num 101
## .. .. ..$ units : chr "mm[Hg]"
## .. ..$ Diastolic :List of 3
## .. .. ..$ @class : chr "DV_QUANTITY"
## .. .. ..$ magnitude: num 62
## .. .. ..$ units : chr "mm[Hg]"
## ..$ :List of 3
## .. ..$ Temperature:List of 4
## .. .. ..$ @class : chr "DV_QUANTITY"
## .. .. ..$ magnitude: num 36
## .. .. ..$ units : chr "°C"
## .. .. ..$ precision: int 1
## .. ..$ Systolic :List of 3
## .. .. ..$ @class : chr "DV_QUANTITY"
## .. .. ..$ magnitude: num 122
## .. .. ..$ units : chr "mm[Hg]"
## .. ..$ Diastolic :List of 3
## .. .. ..$ @class : chr "DV_QUANTITY"
## .. .. ..$ magnitude: num 83
## .. .. ..$ units : chr "mm[Hg]"
## .. [list output truncated]
```

Sublists contain the used AQL query and URL address as well as the result, which is extracted to the `json_results` named vector.


```r
json_results <- unlist(json[[4]])
```

```
## Named chr [1:560] "DV_QUANTITY" "37.3" "°C" "1" ...
## - attr(*, "names")= chr [1:560] "Temperature.@class"
##    "Temperature.magnitude" "Temperature.units" "Temperature.precision" ...
```

### Convert to data frame

The `json_results` named vector is converted to two-column data frame where the names of the vector are in one column and the values are in the other


```r
json_nav <- as.data.frame(json_results, stringsAsFactors = FALSE)
json_nav$path <- attr(json_results, "names")
```

```
## 'data.frame':	560 obs. of  2 variables:
## $ json_results: chr "DV_QUANTITY" "37.3" "°C" "1" ...
## $ path : chr "Temperature.@class" "Temperature.magnitude"
##    "Temperature.units" "Temperature.precision" ...
```

The names of the unlisted JSON `json_results` represent paths of the values in the nested JSON. And in the next step the unique paths of the JSON content are determined (`unique_path`) and used to determine the number of different records `rec_num` and their index limits (`limits`) in the `json_results` data_frame. 


```r
unique_path <- unique(json_nav$path)
```

```
##  chr [1:10] "Temperature.@class" "Temperature.magnitude" ...
```

```r
runs <- rle(json_nav$path == unique_path[1])
limits <- cumsum(runs$lengths)
```

```
##  int [1:112] 1 10 11 20 21 30 31 40 41 50 ...
```

```r
rec_num <- sum(runs$values)
```

```
##  int 56
```

The number of different records `rec_num` and their start and end limits `limits` are then used to add a new column `ind` to the `json_nav` data frame to assign each line to the corresponding record number


```r
json_nav$ind <- rep(NA, dim(json_nav)[1])
for(i in 1:rec_num) {
  json_nav$ind[limits[2*i - 1] : limits[2*i]] <- i
}
```

```
## 'data.frame':	560 obs. of  3 variables:
## $ json_results: chr "DV_QUANTITY" "37.3" "°C" "1" ...
## $ path : chr "Temperature.@class" "Temperature.magnitude"
##    "Temperature.units" "Temperature.precision" ...
## $ ind : int 1 1 1 1 1 1 1 1 1 1 ...
```

Below temperature data for the first three patients from the `json_nav` is visualized (the first column indicates which rows are displayed).


---------------------------------------------------
 &nbsp;   json_results          path           ind 
-------- -------------- --------------------- -----
 **1**    DV_QUANTITY    Temperature.@class     1  

 **2**        37.3      Temperature.magnitude   1  

 **3**         °C         Temperature.units     1  

 **4**         1        Temperature.precision   1  

 **11**   DV_QUANTITY    Temperature.@class     2  

 **12**       38.4      Temperature.magnitude   2  

 **13**        °C         Temperature.units     2  

 **14**        1        Temperature.precision   2  

 **21**   DV_QUANTITY    Temperature.@class     3  

 **22**       36.2      Temperature.magnitude   3  

 **23**        °C         Temperature.units     3  

 **24**        1        Temperature.precision   3  
---------------------------------------------------


In the final step, the output data frame `out` is prepared according to the value of the `full_path` parameter, which can be either `FALSE` or `TRUE`. 

#### JSON paths without numbering

If `full_path` is set to `FALSE`, the column names of the `out` data frame are equal to the values of `unique_path` vector. Additionally, the `ind` and `seq` columns are included. This way each column represents elements on the same path. The `ind` column is used to annotate each row with the corresponding record number. The `seq` column is used to annotate each row with the number of repetitions in the same JSON path (to distinguish between several different elements in the same JSON path). If each JSON path contains a single element, all values in the `seq` are equal to 1.

The `seq` column is created using the `with` function, which evaluates the `json_nav` data frame to the expression defined by the `ave` function. The `ave` function groups the values in `json_results` according to unique combinations of variables `ind` and `path`. The `out` data frame is cast (by using function `dcast`) from the molten data frame `json_nav` by sorting the values in `json_results` according to the match between `ind` and `seq` on one side and `path` on the other.


```r
json_nav$seq <- with(json_nav, stats::ave(json_results, ind, path, FUN = seq_along))
json_nav$seq <- as.numeric(json_nav$seq)
out <- reshape2::dcast(ind + seq ~ path, data = json_nav, value.var = "json_results")
```

```
## 'data.frame':	56 obs. of  12 variables:
##  $ ind                  : int  1 2 3 4 5 6 7 8 9 10 ...<commit_msg>test table edit in EhrscapeR-walkthrough.md<commit_after># Walkthrough for the EhrscapeR package

-   Introduction
-   Data query
    -   Make HTTP request
    -   GET response
    -   Parse response
    -   Convert to data frame
-   Data query - csv flattening
-   Save data 
    -   Make HTTP request
    -   Prepare composition
    -   POST data
    -   Parse response


## Introduction

[EhrScape](https://www.ehrscape.com/index.html) is an open health data platform providing a rich service framework that allows you to store, query and retrieve electronic health records based on the [openEHR](http://www.openehr.org/) specification. The data is available as [JSON](https://en.wikipedia.org/wiki/JSON) via a [REST](https://en.wikipedia.org/wiki/Representational_state_transfer) web service architecture. 

The overall approach is generic â€“ we want to obtain healthcare data by calling a REST API, and then format the returned result set in R to ready the data for more sophisticated analysis.

RESTful applications use HTTP requests to read data (e.g., make queries) and post data (create and/or update). At its most basic, calling a REST API to obtain data involves making a HTTP GET request to a server. If the call succeeds, youâ€™ll have a document that contains the requested data. To store data using REST API, a HTTP POST request needs to be made to the server along with the newly created data. 

This tutorial describes the `EhrscapeR` package, showcasing the use of R for handling data from EhrScape platform using REST API:

* [GET /query](https://www.ehrscape.com/api-explorer.html?api=thinkehr&service=/query&operation=/query&method=get&inline=true): returns the results of the specified AQL query,
* [GET /query/csv](https://www.ehrscape.com/api-explorer.html?api=thinkehr&service=/query&operation=/query/csv&method=get&inline=true): returns the results of the specified AQL query as CSV, 
* [POST /composition](https://www.ehrscape.com/api-explorer.html?api=thinkehr&service=/composition&operation=/composition&method=post&inline=true): creates and commits a contribution consisting of multiple composition creations, updates or deletes.

Please check the [EhrScape API Explorer](https://www.ehrscape.com/api-explorer.html) for a complete list of REST API. 

The basis of this package are functions from the:

- **httr** package: functions for making HTTP requests
- **jsonlite** package: functions for parsing JSON formated data into data frames (R-suitable form)

An important part is parsing the obtained JSON data into R-usable form. The result set in JSON format can be complex in structure with several nested levels which requires flattening of the structure to the data frame.  

The package `EhrscapeR` includes functions: 

* `get_query`: for a given [AQL](http://www.openehr.org/releases/QUERY/latest/docs/AQL/AQL.html) query returns the result set in a data frame. 
* `get_query_csv`: for a given AQL query returns the CSV formated result set in a data frame. 
* `post_composition`: stores new data records (compositions) in EhrScape.

All the examples below can be checked on EhrScape API Explorer.


## Data query

Data is queried from EhrScape using the `get_query` function:


```r
query_data <- get_query(baseURL, credentials = c(user_name, password), aql_query, 
    full_path = FALSE)
```

Input parameters: 

  - `baseURL`: base url address of the REST service, 
  - `credentials`: authentication pair of username and password
  - `aql_query`: AQL query, 
  - `full_path`: logical value indicating the degree of flattening. If `FALSE`, each value (even repetitions on the same path) in the JSON is assigned a unique identifier. If `TRUE`, values are sorted in the outpt data frame according to JSON path and unique sequence within the same path (if there are several repetitions on the same JSON path).
  
The function connects to the server with provided credentials and AQL query. The response in JSON format is parsed to a data frame and returned as the result of the function. 

Example: Query body temperature, systolic and diastolic measurements for all patients


```r
aql_query <- 
  "select
    t/data[at0002|History|]/events[at0003|Any event|]/data[at0001]/items[at0004|Temperature|]/value as Temperature,
    bp/data[at0001|history|]/events[at0006|any event|]/data[at0003]/items[at0004|Systolic|]/value as Systolic,
    bp/data[at0001|history|]/events[at0006|any event|]/data[at0003]/items[at0005|Diastolic|]/value as Diastolic
from EHR e
contains COMPOSITION c
contains (
    OBSERVATION t[openEHR-EHR-OBSERVATION.body_temperature.v1] and
    OBSERVATION bp[openEHR-EHR-OBSERVATION.blood_pressure.v1])
offset 0 limit 100"

baseURL <- "https://rest.ehrscape.com/rest/v1/"
user_name <- "guidemo"
password <- "<PASSWORD>?!demo123"
```

Result of the above query is: 

![Result of example query](./figures/EhrscapeR-walkthrough_files/figure-markdown_github/query_result.png)


### Make HTTP request

The function begins by preparing the AQL query for construction of the HTTP request. First, the `rm_white` function of the **qdapRegex** package removes from the input `aql_query` all multiple white spaces, white spaces before commas, semicolons, colons or endmarks, white spaces before or after brackets, leading and trailing white spaces. Next, line break and carriage return symbols are replaced with white space by using the `gsub` function. The edited AQL query string is then URL encoded (`query_enc`) and used to construct the HTTP request `URL_address` 


```r
aql_query <- qdapRegex::rm_white(aql_query)
aql_query <- gsub("\r?\n|\r", " ", aql_query)
query_enc <- utils::URLencode(aql_query)
URL_address = paste(baseURL, "query?aql=", query_enc, sep = "")
```

```
##  chr "https://rest.ehrscape.com/rest/v1/query?aql=select%20t/data[at0002"..
```

### GET response

The request `URL_address` is executed using the `GET` function of the **httr** package. The server response is stored as `resp`. Since EhrScape requires authentication, user credentials have to be included to the `GET` function call. Additionally, the `content_type_json` function is used to set the expected content type returned from the server to JSON. 


```r
resp <- httr::GET(URL_address, httr::authenticate(user_name, password), httr::content_type_json())
```

```
## List of 10
## $ url : chr
##    "https://rest.ehrscape.com/rest/v1/query?aql=select%20t/data[at0002%7"..
##    __truncated__
## $ status_code: int 200
## $ headers :List of 7
## ..$ date : chr "Wed, 16 Dec 2015 16:12:19 GMT"
## ..$ server : chr "Jetty(9.2.11.v20150529)"
## ..$ content-language: chr "en-US"
## ..$ content-type : chr "application/json;charset=UTF-8"
## ..$ vary : chr "Accept-Encoding"
## ..$ content-encoding: chr "gzip"
## ..$ content-length : chr "1094"
## ..- attr(*, "class")= chr [1:2] "insensitive" "list"
## $ all_headers:List of 1
## ..$ :List of 3
## .. ..$ status : int 200
## .. ..$ version: chr "HTTP/1.1"
## .. ..$ headers:List of 7
## .. .. ..$ date : chr "Wed, 16 Dec 2015 16:12:19 GMT"
## .. .. ..$ server : chr "Jetty(9.2.11.v20150529)"
## .. .. ..$ content-language: chr "en-US"
## .. .. ..$ content-type : chr "application/json;charset=UTF-8"
## .. .. ..$ vary : chr "Accept-Encoding"
## .. .. ..$ content-encoding: chr "gzip"
## .. .. ..$ content-length : chr "1094"
## .. .. ..- attr(*, "class")= chr [1:2] "insensitive" "list"
## $ cookies :'data.frame': 0 obs. of 7 variables:
## ..$ domain : logi(0)
## ..$ flag : logi(0)
## ..$ path : logi(0)
## ..$ secure : logi(0)
## ..$ expiration:Classes 'POSIXct', 'POSIXt' num(0)
## ..$ name : logi(0)
## ..$ value : logi(0)
## $ content : raw [1:14549] 7b 22 6d 65 ...
## $ date : POSIXct[1:1], format: "2015-12-16 16:12:19"
## $ times : Named num [1:6] 0 0 0.047 1.404 1.482 ...
## ..- attr(*, "names")= chr [1:6] "redirect" "namelookup" "connect"
##    "pretransfer" ...
## $ request :List of 7
## ..$ method : chr "GET"
## ..$ url : chr
##    "https://rest.ehrscape.com/rest/v1/query?aql=select%20t/data[at0002%7"..
##    __truncated__
## ..$ headers : Named chr [1:2] "application/json, text/xml,
##    application/xml, */*" "application/json"
## .. ..- attr(*, "names")= chr [1:2] "Accept" "Content-Type"
## ..$ fields : NULL
## ..$ options :List of 5
## .. ..$ useragent : chr "libcurl/7.43.0 r-curl/0.9.4 httr/1.0.0"
## .. ..$ cainfo : chr "C:/R/Rlibs/httr/cacert.pem"
## .. ..$ httpauth : num 1
## .. ..$ userpwd : chr "<PASSWORD>"
## .. ..$ customrequest: chr "GET"
## ..$ auth_token: NULL
## ..$ output : list()
## .. ..- attr(*, "class")= chr [1:2] "write_memory" "write_function"
## ..- attr(*, "class")= chr "request"
## $ handle :Class 'curl_handle' <externalptr>
## - attr(*, "class")= chr "response"
```

### Parse response

Response `resp` includes data such as the HTTP request, status, headers, cookies, and also the answer to the request in JSON format. This is stored in the content field, which is extracted with the `content` function to a nested list `json`. 


```r
json <- httr::content(resp)
```

```
## List of 4
## $ meta :List of 1
## ..$ href: chr
##    "http://rest.ehrscape.com/rest/v1/query/?aql=select%20t/data%5Bat0002"..
##    __truncated__
## $ aql : chr "select t/data[at0002|History|]/events[at0003|Any
##    event|]/data[at0001]/items[at0004|Temperature|]/value as Temperature,
##    bp/data["| __truncated__
## $ executedAql: chr "select t/data[at0002|History|]/events[at0003|Any
##    event|]/data[at0001]/items[at0004|Temperature|]/value as Temperature,
##    bp/data["| __truncated__
## $ resultSet :List of 56
## ..$ :List of 3
## .. ..$ Temperature:List of 4
## .. .. ..$ @class : chr "DV_QUANTITY"
## .. .. ..$ magnitude: num 37.3
## .. .. ..$ units : chr "°C"
## .. .. ..$ precision: int 1
## .. ..$ Systolic :List of 3
## .. .. ..$ @class : chr "DV_QUANTITY"
## .. .. ..$ magnitude: num 99
## .. .. ..$ units : chr "mm[Hg]"
## .. ..$ Diastolic :List of 3
## .. .. ..$ @class : chr "DV_QUANTITY"
## .. .. ..$ magnitude: num 70
## .. .. ..$ units : chr "mm[Hg]"
## ..$ :List of 3
## .. ..$ Temperature:List of 4
## .. .. ..$ @class : chr "DV_QUANTITY"
## .. .. ..$ magnitude: num 38.4
## .. .. ..$ units : chr "°C"
## .. .. ..$ precision: int 1
## .. ..$ Systolic :List of 3
## .. .. ..$ @class : chr "DV_QUANTITY"
## .. .. ..$ magnitude: num 107
## .. .. ..$ units : chr "mm[Hg]"
## .. ..$ Diastolic :List of 3
## .. .. ..$ @class : chr "DV_QUANTITY"
## .. .. ..$ magnitude: num 58
## .. .. ..$ units : chr "mm[Hg]"
## ..$ :List of 3
## .. ..$ Temperature:List of 4
## .. .. ..$ @class : chr "DV_QUANTITY"
## .. .. ..$ magnitude: num 36.2
## .. .. ..$ units : chr "°C"
## .. .. ..$ precision: int 1
## .. ..$ Systolic :List of 3
## .. .. ..$ @class : chr "DV_QUANTITY"
## .. .. ..$ magnitude: num 101
## .. .. ..$ units : chr "mm[Hg]"
## .. ..$ Diastolic :List of 3
## .. .. ..$ @class : chr "DV_QUANTITY"
## .. .. ..$ magnitude: num 62
## .. .. ..$ units : chr "mm[Hg]"
## ..$ :List of 3
## .. ..$ Temperature:List of 4
## .. .. ..$ @class : chr "DV_QUANTITY"
## .. .. ..$ magnitude: num 36
## .. .. ..$ units : chr "°C"
## .. .. ..$ precision: int 1
## .. ..$ Systolic :List of 3
## .. .. ..$ @class : chr "DV_QUANTITY"
## .. .. ..$ magnitude: num 122
## .. .. ..$ units : chr "mm[Hg]"
## .. ..$ Diastolic :List of 3
## .. .. ..$ @class : chr "DV_QUANTITY"
## .. .. ..$ magnitude: num 83
## .. .. ..$ units : chr "mm[Hg]"
## .. [list output truncated]
```

Sublists contain the used AQL query and URL address as well as the result, which is extracted to the `json_results` named vector.


```r
json_results <- unlist(json[[4]])
```

```
## Named chr [1:560] "DV_QUANTITY" "37.3" "°C" "1" ...
## - attr(*, "names")= chr [1:560] "Temperature.@class"
##    "Temperature.magnitude" "Temperature.units" "Temperature.precision" ...
```

### Convert to data frame

The `json_results` named vector is converted to two-column data frame where the names of the vector are in one column and the values are in the other


```r
json_nav <- as.data.frame(json_results, stringsAsFactors = FALSE)
json_nav$path <- attr(json_results, "names")
```

```
## 'data.frame':	560 obs. of  2 variables:
## $ json_results: chr "DV_QUANTITY" "37.3" "°C" "1" ...
## $ path : chr "Temperature.@class" "Temperature.magnitude"
##    "Temperature.units" "Temperature.precision" ...
```

The names of the unlisted JSON `json_results` represent paths of the values in the nested JSON. And in the next step the unique paths of the JSON content are determined (`unique_path`) and used to determine the number of different records `rec_num` and their index limits (`limits`) in the `json_results` data_frame. 


```r
unique_path <- unique(json_nav$path)
```

```
##  chr [1:10] "Temperature.@class" "Temperature.magnitude" ...
```

```r
runs <- rle(json_nav$path == unique_path[1])
limits <- cumsum(runs$lengths)
```

```
##  int [1:112] 1 10 11 20 21 30 31 40 41 50 ...
```

```r
rec_num <- sum(runs$values)
```

```
##  int 56
```

The number of different records `rec_num` and their start and end limits `limits` are then used to add a new column `ind` to the `json_nav` data frame to assign each line to the corresponding record number


```r
json_nav$ind <- rep(NA, dim(json_nav)[1])
for(i in 1:rec_num) {
  json_nav$ind[limits[2*i - 1] : limits[2*i]] <- i
}
```

```
## 'data.frame':	560 obs. of  3 variables:
## $ json_results: chr "DV_QUANTITY" "37.3" "°C" "1" ...
## $ path : chr "Temperature.@class" "Temperature.magnitude"
##    "Temperature.units" "Temperature.precision" ...
## $ ind : int 1 1 1 1 1 1 1 1 1 1 ...
```

Below temperature data for the first three patients from the `json_nav` is visualized (the first column indicates which rows are displayed).


---------------------------------------------------
|&nbsp; | json_results |       path          | ind | 
|-------|--------------|---------------------|-----|
| **1** | DV_QUANTITY  | Temperature.@class  |  1  |

| **2** |     37.3     |Temperature.magnitude|  1  |

| **3** |      °C      |  Temperature.units  |  1  |

| **4** |      1       |Temperature.precision|  1  |

|**11** | DV_QUANTITY  | Temperature.@class  |  2  |

|**12** |     38.4     |Temperature.magnitude|  2  |

|**13** |      °C      |  Temperature.units  |  2  |

|**14** |      1       |Temperature.precision|  2  |

|**21** | DV_QUANTITY  | Temperature.@class  |  3  |

|**22** |     36.2     |Temperature.magnitude|  3  |

|**23** |      °C      |  Temperature.units  |  3  |

|**24** |      1       |Temperature.precision|  3  |
---------------------------------------------------


In the final step, the output data frame `out` is prepared according to the value of the `full_path` parameter, which can be either `FALSE` or `TRUE`. 

#### JSON paths without numbering

If `full_path` is set to `FALSE`, the column names of the `out` data frame are equal to the values of `unique_path` vector. Additionally, the `ind` and `seq` columns are included. This way each column represents elements on the same path. The `ind` column is used to annotate each row with the corresponding record number. The `seq` column is used to annotate each row with the number of repetitions in the same JSON path (to distinguish between several different elements in the same JSON path). If each JSON path contains a single element, all values in the `seq` are equal to 1.

The `seq` column is created using the `with` function, which evaluates the `json_nav` data frame to the expression defined by the `ave` function. The `ave` function groups the values in `json_results` according to unique combinations of variables `ind` and `path`. The `out` data frame is cast (by using function `dcast`) from the molten data frame `json_nav` by sorting the values in `json_results` according to the match between `ind` and `seq` on one side and `path` on the other.


```r
json_nav$seq <- with(json_nav, stats::ave(json_results, ind, path, FUN = seq_along))
json_nav$seq <- as.numeric(json_nav$seq)
out <- reshape2::dcast(ind + seq ~ path, data = json_nav, value.var = "json_results")
```

```
## 'data.frame':	56 obs. of  12 variables:
##  $ ind                  : int  1 2 3 4 5 6 7 8 9 10 ...
{'fraction_non_alphanumeric': 0.1706832298136646, 'fraction_numerical': 0.02841614906832298, 'mean_word_length': 3.221421080230729, 'pattern_counts': {'":': 0, '<': 54, '<?xml version=': 0, '>': 10, 'https://': 22, 'lorem ipsum': 0, 'www.': 14, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7452294', 'n_tokens_mistral': 11482, 'n_tokens_neox': 10335, 'n_words': 3540}