<commit_before>---
title: Analyzing NHANES data using R
author: <NAME>
output:
  knitrBootstrap::bootstrap_document:
  theme.chooser: TRUE
  highlight.chooser: TRUE
---
#NHANES Data Analysis Example#

#Introduction
This short tutorial assumes that the user has some prior knowledge of R and Rstudio. If you are not familiar with R, there are number of resources in CDC/ATSDR R user group's website to help you get started. Good resources to keep in mind are:

* [Survey Package Homepage](http://r-survey.r-forge.r-project.org/survey/)
* [adfree](http://www.asdfree.com/)

# NHANES Data Analysis
## Install and load required packages
First we will install and load the packages needed to perform this script.
This snippet checks to see if you have the packages needed installed, and
installs any missing packages:

```{r installpacks, eval = TRUE}
pack_list <- c("survey", "downloader", "foreign", "XML", "dplyr", "magrittr",
"ggplot2", "readr", "stringr")

install.packages(pack_list[!pack_list %in% installed.packages()])
```
It is normally not necissary to re-install packages in a given R installation.


Next, we need to load the packages into memory.
```{r libs, eval=FALSE}
library(downloader)
library(foreign)
library(survey)
library(XML)
library(ggplot2)
library(readr)
library(stringr)
library(magrittr)
library(dplyr)
```

## Basic Workflow

### Download raw survey data from [CDC Website](https://wwwn.cdc.gov/Nchs/Nhanes/Search/Nhanes_continuous.aspx). Main pages for continious NHANES data are:

* [Demographics Variables and Sample Weights](https://wwwn.cdc.gov/nchs/nhanes/Search/DataPage.aspx?Component=Demographics) (almost always needed for any analysis)
* [Dietary Interview](https://wwwn.cdc.gov/Nchs/Nhanes/Search/DataPage.aspx?Component=Dietary)
* [Labratory Data](https://wwwn.cdc.gov/Nchs/Nhanes/Search/DataPage.aspx?Component=Laboratory)
* [Questionaire Data](https://wwwn.cdc.gov/Nchs/Nhanes/Search/DataPage.aspx?Component=Questionnaire)

These files are stored as SAS XPORT files on CDC's internet site. R can be used to rapidly expedite the downloading of these files.


<div class="alert alert-dismissible alert-warning">
  <button type="button" class="close" data-dismiss="alert">&times;</button>
  <h4>Tip!</h4>
  <p> Using R to download files will help make your analysis easier to reproduce.</p>
</div>


#### Identify Files

```{r id files}
demoDataFiles <- read_lines("https://wwwn.cdc.gov/nchs/nhanes/Search/DataPage.aspx?Component=Demographics") %>%
  XML::htmlParse(.) %>%
  XML::readHTMLTable(.)

knitr::kable(demoDataFiles[[2]])
```

We would like to select all demographics surveys in this dataset and download to our local computer.

First, make a directory:
```{r createDir}
dataDir <- paste0("./data/nhanes/", format(Sys.Date(), "%Y%m%d"))
dir.create(dataDir, showWarnings = FALSE, recursive = TRUE)
```

<div class="alert alert-dismissible alert-warning">
  <button type="button" class="close" data-dismiss="alert">&times;</button>
  <h4>Tip!</h4>
  <p> Why did I create a system date? If you noticed that in the XPT files, there are revision dates. If the data are updated and previous versions of the data are unavailable, then you might not be able to guarentee that your results are reproducible because the data may have changed. Best practice is to maintain an archive of original data, and create a copy of working data that is further manipulated.
</p>
</div>

```{r idFiles}
idFiles <- grep("XPT", demoDataFiles[[2]][,4]) %>%
  demoDataFiles[[2]][.,] %T>%
  knitr::kable(.)
```

#### Download the files

The SAS XPT Files
```{r downloadFiles}
serverFiles <- gsub( " .*$", "", idFile$`Data File`) %>%
  paste0(., ".XPT") %>%
  paste("https://wwwn.cdc.gov/Nchs/Nhanes", idFiles$Years, .,  sep="/")


if(Sys.info()['sysname'] !="Windows"){
  sapply(serverFiles, function(x) download.file(x, destfile = paste(dataDir, basename(x),  sep="/"), method="curl", extra="-k"))
} else {
  sapply(serverFiles, function(x) download.file(x, destfile = paste(dataDir, basename(x),  sep="/"), mode="wb"))
}
```

and the Documentation (we not only read these files, but since they are HTML we can extract the code books from them using R):

```{r Download Docs}
docFiles <- gsub("XPT$", "htm", serverFiles)

if(Sys.info()['sysname'] !="Windows"){
sapply(docFiles, function(x) download.file(x, destfile = paste(dataDir, basename(x),  sep="/"), method="curl", extra="-k"))
}else{<commit_msg>adding more<commit_after>---
title: Analyzing NHANES data using R
author: <NAME>
output:
  knitrBootstrap::bootstrap_document:
  theme.chooser: TRUE
  highlight.chooser: TRUE
---
#NHANES Data Analysis Example#

#Introduction
This short tutorial assumes that the user has some prior knowledge of R and Rstudio. If you are not familiar with R, there are number of resources in CDC/ATSDR R user group's website to help you get started. Good resources to keep in mind are:

* [Survey Package Homepage](http://r-survey.r-forge.r-project.org/survey/)
* [adfree](http://www.asdfree.com/)

# NHANES Data Analysis
## Install and load required packages
First we will install and load the packages needed to perform this script.
This snippet checks to see if you have the packages needed installed, and
installs any missing packages:

```{r installpacks, eval = TRUE}
pack_list <- c("survey", "downloader", "foreign", "XML", "dplyr", "magrittr",
"ggplot2", "readr", "stringr")

install.packages(pack_list[!pack_list %in% installed.packages()])
```
It is normally not necissary to re-install packages in a given R installation.


Next, we need to load the packages into memory.
```{r libs, eval=FALSE}
library(downloader)
library(foreign)
library(survey)
library(XML)
library(ggplot2)
library(readr)
library(stringr)
library(magrittr)
library(dplyr)
```

## Basic Workflow

### Download raw survey data from [CDC Website](https://wwwn.cdc.gov/Nchs/Nhanes/Search/Nhanes_continuous.aspx). Main pages for continious NHANES data are:

* [Demographics Variables and Sample Weights](https://wwwn.cdc.gov/nchs/nhanes/Search/DataPage.aspx?Component=Demographics) (almost always needed for any analysis)
* [Dietary Interview](https://wwwn.cdc.gov/Nchs/Nhanes/Search/DataPage.aspx?Component=Dietary)
* [Labratory Data](https://wwwn.cdc.gov/Nchs/Nhanes/Search/DataPage.aspx?Component=Laboratory)
* [Questionaire Data](https://wwwn.cdc.gov/Nchs/Nhanes/Search/DataPage.aspx?Component=Questionnaire)

These files are stored as SAS XPORT files on CDC's internet site. R can be used to rapidly expedite the downloading of these files.


<div class="alert alert-dismissible alert-warning">
  <button type="button" class="close" data-dismiss="alert">&times;</button>
  <h4>Tip!</h4>
  <p> Using R to download files will help make your analysis easier to reproduce.</p>
</div>


#### Identify Files

```{r id files}
demoDataFiles <- read_lines("https://wwwn.cdc.gov/nchs/nhanes/Search/DataPage.aspx?Component=Demographics") %>%
  XML::htmlParse(.) %>%
  XML::readHTMLTable(.)

knitr::kable(demoDataFiles[[2]])
```

We would like to select all demographics surveys in this dataset and download to our local computer.

First, make a directory:
```{r createDir}
dataDir <- paste0("./data/nhanes/", format(Sys.Date(), "%Y%m%d"))
dir.create(dataDir, showWarnings = FALSE, recursive = TRUE)
```

<div class="alert alert-dismissible alert-warning">
  <button type="button" class="close" data-dismiss="alert">&times;</button>
  <h4>Tip!</h4>
  <p> Why did I create a system date? If you noticed that in the XPT files, there are revision dates. If the data are updated and previous versions of the data are unavailable, then you might not be able to guarentee that your results are reproducible because the data may have changed. Best practice is to maintain an archive of original data, and create a copy of working data that is further manipulated.
</p>
</div>

```{r idFiles}
idFiles <- grep("XPT", demoDataFiles[[2]][,4]) %>%
  demoDataFiles[[2]][.,] %T>%
  knitr::kable(.)
```

#### Download the files

Download the SAS XPT Files
```{r downloadFiles}
serverFiles <- gsub( " .*$", "", idFiles[,"Data File"]) %>%
  paste0(., ".XPT") %>%
  paste("https://wwwn.cdc.gov/Nchs/Nhanes", idFiles$Year, .,  sep="/")

destfile = sapply(serverFiles, function(x) paste(dataDir, basename(x),  sep="/"))

serverFiles <- serverFiles[!file.exists(destfile)]
destfiles <- destfiles[!file.exists(destfile)]


if(Sys.info()['sysname'] !="Windows"){
  sapply(serverFiles, function(x) download.file(x, destfile = paste(dataDir, basename(x),  sep="/"), method="curl", extra="-k"))
} else {
  sapply(serverFiles, function(x) download.file(x, destfile = paste(dataDir, basename(x),  sep="/"), mode="wb"))
}
```

and the Documentation (we not only read these files, but since they are HTML we can extract the code books from them using R):

```{r Download Docs}
docFiles <- gsub("XPT$", "htm", serverFiles)

if(Sys.info()['sysname'] !="Windows"){
sapply(docFiles, function(x) download.file(x, destfile = paste(dataDir, basename(x),  sep="/"), method="curl", extra="-k"))
}else{
{'fraction_non_alphanumeric': 0.148656122110386, 'fraction_numerical': 0.002654573609114036, 'mean_word_length': 5.287899860917942, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 49, 'https://': 14, 'lorem ipsum': 0, 'www.': 16, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 25, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1259215', 'n_tokens_mistral': 3028, 'n_tokens_neox': 2884, 'n_words': 919}