<commit_before>+++
author = "cristina"
categories = ["datamining"]
date = "2017-04-24T21:13:39+02:00"
description = "In this post you will be given the first directions for creating an application to use the github API and will analyse the programming languages popularities in your community from github data"
image = "FriendsLanguagesUsage.png"
mainclass = "datamining"
tags = ["github","mining","community","API"]
title = "Mining your github community with R - Analysing languages popularity"
+++

In this post we are exploring our github community considering our friends (or followers) in that social network. We will first give some directions for creating an application and starting to use the github API. Then we will extract the information we need to perform our analysis and get what programming languages do our friends use and what is the most popular and the less popular language. Let's explore the languages popularity in your community!

# 1.  Create the app, install packages, authenticate

## 1.1 App registration 

First of all, as we want to access public data from Github, we need to register an app with authentication to do that. This will allow us to have no limit of calls. 
So go to [github developer program page](https://developer.github.com/program/ "github developer program page") and click on __Register now__. Then select an account and in __Personal settings__ tabs select __Authorized applications__. If you already have an authorized aplication to access gitHub API, you must see it here. Otherwise if you don't have any registered app,then go to __OAuth applications__ and click on __Register a new application__. 
There you have to register your new application, giving it a name, a description... Fill the fields and in __Callback URL__ enter [localhost:1410](http://localhost:1410/ "http://localhost:1410/") which is the URL that github will return once the app is authenticated. 
Now you have registered your app, the __Client ID__ and __Client Secret__ for your app will be generated. You can see them clicking on __OAuth applications -> Your App__. Remember to keep them secret and safe!

Wow! you're now a member of the developer program. The next thing we will do is installing the R packages we are going to use to work with gitHub from R.

## 1.2 Packages installation

We will use the version 3 of GitHub API to access data [github v3 API](https://developer.github.com/v3/ "github v3 API"). As all the data from the API will be received in JSON format, we are going to use the function `fromJSON` which will let us use the API url directly, and will parse the JSON data returned to a data frame format. That function is on package `jsonlite`, so we must install it if you don't have it installed:

```r
install.packages('jsonlite')
```

If you have not installed `stringr` package, install it as we just do with `jsonlite`. We are using it to make some common operations on strings. Install `ggplot2` if you have not, too. We must use it to make the graphics.

For connecting to the GitHub app, we will need to install `rgithub` package. We do it directly from github source code. Note that it have a dependency on package `devtools`:

```r
require(devtools)
install_github("cscheid/rgithub")
```

After installing, load the packages:

```r
# load libraries
library(github)
library(jsonlite)
library(stringr)
library(ggplot2)
```

## 1.3 Authenticate the access

We will need to authenticate the access, and we will do it through the function `interactive.login` from the rgithub package, passing to it our __ID__ and __secret__. My recommendation to you is to put that lines in a separate file and not share it with anyone or anywhere.
Then just source the file when you need to authenticate, or just execute its lines. 

```r
# github app autentication
clientID<-"your_client_id_goes_here"
clientSecret<-"your_secret_goes_here"
context<-interactive.login(clientID,clientSecret)
```

# 2. Get your friends info

Now you're authenticated, lets get your followers on GitHub. The function `get.my.followers` will give us the people following us and some info about them:

```r
# get your followers
myFollowers<-get.my.followers(context)
```
we can easily check how many people is following us using `length` function:

```r
# get number of my followers
numFollowing<-length(myFollowers$content)
```

Now we have our followers, lets create a dataframe to store all the info returned. First, we extract each content line in _myFollowers_ list, and append them in a _dataset_ variable using `rbind` function:

```r
# create a dataset with your followers
dataset<-unlist(myFollowers$content[[1]])

for(i in 2:length(myFollowers$content)){
  dataset<-rbind(dataset,unlist(myFollowers$content[[i]]))
}
```

Then we make it a data frame, naming the columns as they were originally named in _myFollowers$content_ list and save it as a csv for later use:

```r
# create a data frame and save it for later use
dataset<-unname(dataset)
dataset<-as.data.frame(dataset)
colnames(dataset)<-c(names(myFollowers$content[[1]]))
write.csv(dataset,file="CrisFollowers.csv")
```

Obviously you can save it with whatever name you like. But don't forget the extension.


# 3. Get our friends' repositories information and create a new dataset

As you sure have seen, in the last information we extracted, there's info about our followers name, id, avatar, type, and some url of interest. However, that info doesn't includes our friends repositories info as repo name, repo language, repo lines of code...
Thats info is fundamental to make our analysis, so we must get it. If you have took a look to the latest extracted data, you must have realized that there's a column named __repos_url__ which tells us that the url to get the repos from a user is __https://api.github.com/users/user/repos__ . For example, to get my repos information we should call __https://api.github.com/users/CritinaHG/repos__ , and we will get the data in JSON format from the API.

So we are getting that data for each of our followers, by reading the already created dataset and getting our followers names, composing their repo url and parsing the returned data from the API using the `fromJSON` function:
 
```r
# read latest created csv 
myFriends<-read.csv("CrisFollowers.csv")

# extract the names
unname<-as.character(myFriends$login)

# extract data from friends' public repositories
compdata<- NULL

for(i in 1:nrow(myFriends)){
  data<-fromJSON(paste0("https://api.github.com/users/",str_trim(unname[i],side = "both"),"/repos?clientID&clientSecret"))
  if(length(data)!=0){
    data<-data[,-4]
    compdata<-rbind(compdata,data)
  }
}

# write data for reuse
write.csv(compdata,"UsersWithRepoInfo.csv")
```

In __clientID__ and __clientSecret__ you should put your ID and secret generated at the begining of the post. It is not necesary but this will let us avoid request limitations. We remove the 4º column because it has redundant data. Whith `rbind` function we are appending the new data obtained to the existing one.

# 4. Lets do some data processing

Lets read (if you have not read it) the dataset : `activeFriends<-read.csv("UsersWithRepoInfo.csv")` We are going to perform some transformations over it to make data more readable for the analysis in R. 
First, as the data timezone is GMT we need to set the timezone parameter. Let's build a function to do that an apply it to every date column:

```r
# make date format supported by R
date.format<-function(datestring){
  date<-as.POSIXct(datestring,format="%Y-%m-%dT%H:%M:%SZ",tz="GMT")
}

# update dates with new format
activeFriends$created_at<-date.format(activeFriends$created_at)
activeFriends$updated_at<-date.format(activeFriends$updated_at)
activeFriends$pushed_at<-date.format(activeFriends$pushed_at)
```

Feel free to explore the dataset. You will have noticed that there're some very interesting columns for our analysis, and others that are not so. Next we are doing is selecting the most interesting columns for our analysis:


```r
# selecting just the interesting cols
activeFriends<-activeFriends[,c("id","name","full_name","private","description","fork","created_at","updated_at","pushed_at","homepage","size","stargazers_count","watchers_count","language",                             "has_issues","has_downloads","forks_count","open_issues_count","forks","open_issues","watchers")]
```

Now binarize the columns which have True and False values:

```r
activeFriends$private<-as.integer(activeFriends$private)
activeFriends$has_issues<-as.integer(activeFriends$has_issues)
activeFriends$fork<-as.integer(activeFriends$fork)
activeFriends$has_downloads<-as.integer(activeFriends$has_downloads)
```

Finally, in the `full_name` column we must just get the user name, because the repos names are already included in `name` column. it can be achieved by expliting each item by the slash, and getting the first:

```r
# Getting the username
activeFriends$full_name<-unlist(lapply(strsplit(as.character(activeFriends$full_name),split = '/',fixed = TRUE),function(x) (x[1])))<commit_msg>changing timezone to Madrid<commit_after>+++
author = "cristina"
categories = ["datamining"]
date = "2017-04-24T21:13:39+02:00"
description = "In this post you will be given the first directions for creating an application to use the github API and will analyse the programming languages popularities in your community from github data"
image = "FriendsLanguagesUsage.png"
mainclass = "datamining"
tags = ["github","mining","community","API"]
title = "Mining your github community with R - Analysing languages popularity"
+++

In this post we are exploring our github community considering our friends (or followers) in that social network. We will first give some directions for creating an application and starting to use the github API. Then we will extract the information we need to perform our analysis and get what programming languages do our friends use and what is the most popular and the less popular language. Let's explore the languages popularity in your community!

# 1.  Create the app, install packages, authenticate

## 1.1 App registration 

First of all, as we want to access public data from Github, we need to register an app with authentication to do that. This will allow us to have no limit of calls. 
So go to [github developer program page](https://developer.github.com/program/ "github developer program page") and click on __Register now__. Then select an account and in __Personal settings__ tabs select __Authorized applications__. If you already have an authorized aplication to access gitHub API, you must see it here. Otherwise if you don't have any registered app,then go to __OAuth applications__ and click on __Register a new application__. 
There you have to register your new application, giving it a name, a description... Fill the fields and in __Callback URL__ enter [localhost:1410](http://localhost:1410/ "http://localhost:1410/") which is the URL that github will return once the app is authenticated. 
Now you have registered your app, the __Client ID__ and __Client Secret__ for your app will be generated. You can see them clicking on __OAuth applications -> Your App__. Remember to keep them secret and safe!

Wow! you're now a member of the developer program. The next thing we will do is installing the R packages we are going to use to work with gitHub from R.

## 1.2 Packages installation

We will use the version 3 of GitHub API to access data [github v3 API](https://developer.github.com/v3/ "github v3 API"). As all the data from the API will be received in JSON format, we are going to use the function `fromJSON` which will let us use the API url directly, and will parse the JSON data returned to a data frame format. That function is on package `jsonlite`, so we must install it if you don't have it installed:

```r
install.packages('jsonlite')
```

If you have not installed `stringr` package, install it as we just do with `jsonlite`. We are using it to make some common operations on strings. Install `ggplot2` if you have not, too. We must use it to make the graphics.

For connecting to the GitHub app, we will need to install `rgithub` package. We do it directly from github source code. Note that it have a dependency on package `devtools`:

```r
require(devtools)
install_github("cscheid/rgithub")
```

After installing, load the packages:

```r
# load libraries
library(github)
library(jsonlite)
library(stringr)
library(ggplot2)
```

## 1.3 Authenticate the access

We will need to authenticate the access, and we will do it through the function `interactive.login` from the rgithub package, passing to it our __ID__ and __secret__. My recommendation to you is to put that lines in a separate file and not share it with anyone or anywhere.
Then just source the file when you need to authenticate, or just execute its lines. 

```r
# github app autentication
clientID<-"your_client_id_goes_here"
clientSecret<-"your_secret_goes_here"
context<-interactive.login(clientID,clientSecret)
```

# 2. Get your friends info

Now you're authenticated, lets get your followers on GitHub. The function `get.my.followers` will give us the people following us and some info about them:

```r
# get your followers
myFollowers<-get.my.followers(context)
```
we can easily check how many people is following us using `length` function:

```r
# get number of my followers
numFollowing<-length(myFollowers$content)
```

Now we have our followers, lets create a dataframe to store all the info returned. First, we extract each content line in _myFollowers_ list, and append them in a _dataset_ variable using `rbind` function:

```r
# create a dataset with your followers
dataset<-unlist(myFollowers$content[[1]])

for(i in 2:length(myFollowers$content)){
  dataset<-rbind(dataset,unlist(myFollowers$content[[i]]))
}
```

Then we make it a data frame, name the columns as they were originally named in _myFollowers$content_ list and save it as a csv for later use:

```r
# create a data frame and save it for later use
dataset<-unname(dataset)
dataset<-as.data.frame(dataset)
colnames(dataset)<-c(names(myFollowers$content[[1]]))
write.csv(dataset,file="CrisFollowers.csv")
```

Obviously you can save it with whatever name you like. But don't forget the extension.


# 3. Get our friends' repositories information and create a new dataset

As you sure have seen, in the last information we extracted, there's info about our followers name, id, avatar, type, and some url of interest. However, that info doesn't includes our friends repositories info as repo name, repo language, repo lines of code...
Thats info is fundamental to make our analysis, so we must get it. If you have took a look to the latest extracted data, you must have realized that there's a column named __repos_url__ which tells us that the url to get the repos from a user is __https://api.github.com/users/user/repos__ . For example, to get my repos information we should call __https://api.github.com/users/CritinaHG/repos__ , and we will get the data in JSON format from the API.

So we are getting that data for each of our followers, by reading the already created dataset and getting our followers names, composing their repo url and parsing the returned data from the API using the `fromJSON` function:
 
```r
# read latest created csv 
myFriends<-read.csv("CrisFollowers.csv")

# extract the names
unname<-as.character(myFriends$login)

# extract data from friends' public repositories
compdata<- NULL

for(i in 1:nrow(myFriends)){
  data<-fromJSON(paste0("https://api.github.com/users/",str_trim(unname[i],side = "both"),"/repos?clientID&clientSecret"))
  if(length(data)!=0){
    data<-data[,-4]
    compdata<-rbind(compdata,data)
  }
}

# write data for reuse
write.csv(compdata,"UsersWithRepoInfo.csv")
```

In __clientID__ and __clientSecret__ you should put your ID and secret generated at the begining of the post. It is not necesary but this will let us avoid request limitations. We remove the 4º column because it has redundant data. Whith `rbind` function we are appending the new data obtained to the existing one.

# 4. Lets do some data processing

Lets read (if you have not read it) the dataset : `activeFriends<-read.csv("UsersWithRepoInfo.csv")` We are going to perform some transformations over it to make data more readable for the analysis in R. 

First, as the data timezone is UTC+2 (also Madrid timezone) we need to set the timezone parameter. Let's build a function to do that an apply it to every date column:

```r
# make date format supported by R
date.format<-function(datestring){
  date<-as.POSIXct(datestring,format="%Y-%m-%d %H:%M:%S",tz="Europe/Madrid", usetz=TRUE)
}

# update dates with new format
activeFriends$created_at<-date.format(activeFriends$created_at)
activeFriends$updated_at<-date.format(activeFriends$updated_at)
activeFriends$pushed_at<-date.format(activeFriends$pushed_at)
```

Feel free to explore the dataset. You will have noticed that there're some very interesting columns for our analysis, and others that are not so. Next we are doing is selecting the most interesting columns for our analysis:


```r
# selecting just the interesting cols
activeFriends<-activeFriends[,c("id","name","full_name","private","description","fork","created_at","updated_at","pushed_at","homepage","size","stargazers_count","watchers_count","language",                             "has_issues","has_downloads","forks_count","open_issues_count","forks","open_issues","watchers")]
```

Now binarize the columns which have True and False values:

```r
activeFriends$private<-as.integer(activeFriends$private)
activeFriends$has_issues<-as.integer(activeFriends$has_issues)
activeFriends$fork<-as.integer(activeFriends$fork)
activeFriends$has_downloads<-as.integer(activeFriends$has_downloads)
```

Finally, in the `full_name` column we must just get the user name, because the repos names are already included in `name` column. it can be achieved by expliting each item by the slash, and getting the first:

```r
# Getting the username
activeFriends$full_name<-unlist(lapply(strsplit(as.character(activeFriends$full_name),split = '/',fixed = TRUE),function(x) (x[1])))
{'fraction_non_alphanumeric': 0.08260559125611029, 'fraction_numerical': 0.006096556269566651, 'mean_word_length': 5.087596121698429, 'pattern_counts': {'":': 0, '<': 59, '<?xml version=': 0, '>': 5, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4110194', 'n_tokens_mistral': 5211, 'n_tokens_neox': 4882, 'n_words': 2357}