<commit_before>---
layout: post
title: "Reading National Sample Survey Data using R"
category: R, r-project
comments: true
---

I came across a nice talk by Sumandro (Riju) Chattopadhyay introducing National Sample Survey (NSS) data ([link here](https://www.youtube.com/watch?v=DLs9eEGJzdo)). NSS data is distributed as fixed-width files, correctly referred to as the "Jurassic way" of data distribution by Riju.

Standard softwares can import fixed width format (fwf) files easily. In STATA, you can use infix or infile (used with a dictionary) commands to import. To read about ways to import NSS data in STATA, read this blog post by [<NAME>](https://zakku78.wordpress.com/2009/02/19/nsso-unit-level-data/).

In my first series of blogs, we will learn about ways to import, manage, visualize and analyze NSS data using R. To read fwf files in R, functions like read.fwf from base R can be used. 

If we have a sample fwf file containg the following three rows of data and the layout as given below:

![Sample fixed-width format file](/images/sample_fwf.png)

VariableName | ColumnStart | ColumnEnd     
--- | --- | ---
Serial Number | 1 | 2             
MonthlyIncome | 3 | 6             
Address       | 7 | 36  

We can import this file using the following R commands.

```r
sample <- read.fwf("sample_fwf.txt", widths = c(2, 4, 30), 
  col.names = c("sNo", "monthlyIncome", "address"), header = FALSE)
sample # print the sample file
```

```
  sNo monthlyIncome                        address
1  10          8724 D-741, Baird Lane, Gole Market
2  11           831       45/12, Dwarka, New Delhi
3  12            NA B5/45, Ergos Appartments Delhi
```

A possible way out to save you from the manual labor of typing out the column names and corresponding widths of large NSS data is to prepare a cleaner version of the layout file which can be imported using R. The clean layout file (screen-shot provided below) should contain the item description, byte length of the item, a short item description to be provided as column names and corresponding column classes as well, if needed.

![Cleaned Layout File](/images/layout_clean.png)<commit_msg>Update 2015-08-31-nsso-data-import.md<commit_after>---
layout: post
title: "Reading National Sample Survey Data using R"
category: R, r-project
comments: true
---

I came across a nice talk by Sumandro (<NAME> introducing National Sample Survey (NSS) data ([link here](https://www.youtube.com/watch?v=DLs9eEGJzdo)). NSS data is distributed as fixed-width files, correctly referred to as the "Jurassic way" of data distribution by Riju.

Standard softwares can import fixed width format (fwf) files easily. In STATA, you can use infix or infile (used with a dictionary) commands to import. To read about ways to import NSS data in STATA, read this blog post by [<NAME>](https://zakku78.wordpress.com/2009/02/19/nsso-unit-level-data/).

In my first series of blogs, we will learn about ways to import, manage, visualize and analyze NSS data using R. To read fwf files in R, functions like read.fwf from base R can be used. 

If we have a sample fwf file containg the following three rows of data and the layout as given below:

![Sample fixed-width format file](/images/sample_fwf.png)

| VariableName | ColumnStart | ColumnEnd | 
| --- | --- | --- |
| Serial Number | 1 | 2 |             
| MonthlyIncome | 3 | 6 |             
| Address       | 7 | 36 |  

We can import this file using the following R commands.

```r
sample <- read.fwf("sample_fwf.txt", widths = c(2, 4, 30), 
  col.names = c("sNo", "monthlyIncome", "address"), header = FALSE)
sample # print the sample file
```

```
  sNo monthlyIncome                        address
1  10          8724 D-741, Baird Lane, Gole Market
2  11           831       45/12, Dwarka, New Delhi
3  12            NA B5/45, Ergos Appartments Delhi
```

A possible way out to save you from the manual labor of typing out the column names and corresponding widths of large NSS data is to prepare a cleaner version of the layout file which can be imported using R. The clean layout file (screen-shot provided below) should contain the item description, byte length of the item, a short item description to be provided as column names and corresponding column classes as well, if needed.

![Cleaned Layout File](/images/layout_clean.png)
{'fraction_non_alphanumeric': 0.08346820809248555, 'fraction_numerical': 0.024046242774566472, 'mean_word_length': 3.295928500496524, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 6, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1350250', 'n_tokens_mistral': 1316, 'n_tokens_neox': 1204, 'n_words': 595}