<commit_before>\chapter{Data Collection}

\section{Initial Research Question and Data Collection}

The original intent of this research project was to identify differences in commit behavior between paid and volunteer contributors of open source projects. In order to classify contributors as paid or volunteer, a script was written which collected data from various public online profiles of ASF contributors active in 2015 and computed several heuristics;  the intent was to use some of this data features 
to classify individuals as paid or volunteer. 
 However, two issues made it difficult to identify the differences between the two classes:
\begin{enumerate}
	\item \label{manypaid} For a given ASF project and time period, the proportion of active contributors who are paid to work on the project can be very high, sometimes nearing 100\%.
	\item \label{novolunteers} A profile can contain certain markers that are highly correlated with paid contributors, but there are no known markers that are highly correlated with volunteer contributors.
\end{enumerate}
Issue \#\ref{manypaid} was \emph{prima facie} quite surprising---one might expect there to be a somewhat even mix of paid and volunteer contributors, but upon manually examining contributors to a couple projects, it was discovered that these projects were almost exclusively developed by employees of a few companies during the time period under observation. With such skewed samples, it became difficult to write a classifier sensitive enough to be useful.
Issue \#\ref{novolunteers} was an unforeseen limitation of using profile information to identify social group affiliation. Ultimately, there was no reliable way to differentiate between a volunteer and a paid contributor when his/her data does not reveal any obvious link between his/her contributions and his/her employer.
Since creating this classifier proved infeasible, the data was re-purposed to do a social network analysis of ASF contributions instead. In particular, the following facets of the existing dataset were useful for this purpose:
\begin{itemize}
	\item The dataset associates data from multiple accounts/profiles to one contributor, improving the reliability of finding a given contributor’s employer.
	\item Since the contributor’s employer was one of the data points which the algorithm used, a fair amount of employer data was already available.
\end{itemize}

\section{Data Sources}
When researching open source software, there are a multitude of possible software repositories and organizations to base the experiment on. For example, some prior work studied SourceForge projects, while others studied GNOME. \pd{Reword this bit to first include some motivation, viz: For our work, we needed to reliably link user identities across different
user functions. We needed this because...}
We chose to study Apache Software Foundation projects because they have a consistent organization structure with respect to how developers collaborate on Git, Github, and JIRA. This structure made it simpler to write a script that can reliably link user identities across these service.
To find the employer of a given Git committer, the script attempts to collect data from the Git account, and any associated JIRA and/or Github accounts. It also utilizes the Google Custom Search API to find links to LinkedIn profiles, which can then be inspected manually. Each set of values is stored in a separate entry in a PostgreSQL database, to maximize the amount of information available for finding a contributor's employer.
The following information is mined from Git, Github, and JIRA accounts:
\begin{itemize}
	\item Username
	\item Email address
	\item Display name
\end{itemize}
The following additional information is mined from Git accounts:
\begin{itemize}
	\item Commit count per project
\end{itemize}
The following additional information is mined from Github accounts:
\begin{itemize}
	\item Location
	\item Company name
	\item List of organizations
\end{itemize}
Furthermore, to work around the rate limit of the Github API, the script utilized a GHTorrent.org database dump as an offline cache of Github data.

\pd{Add a section here between 2.2 and 2.3, entitled "Schema of Mined Data" explaining the schema of all your data, viz., all relations,  including accountprojects}

\section{Data Collection Constraints}<commit_msg>Thesis: Remove Windows-1252 encoded characters<commit_after>\chapter{Data Collection}

\section{Initial Research Question and Data Collection}

The original intent of this research project was to identify differences in commit behavior between paid and volunteer contributors of open source projects. In order to classify contributors as paid or volunteer, a script was written which collected data from various public online profiles of ASF contributors active in 2015 and computed several heuristics;  the intent was to use some of this data features 
to classify individuals as paid or volunteer. 
 However, two issues made it difficult to identify the differences between the two classes:
\begin{enumerate}
	\item \label{manypaid} For a given ASF project and time period, the proportion of active contributors who are paid to work on the project can be very high, sometimes nearing 100\%.
	\item \label{novolunteers} A profile can contain certain markers that are highly correlated with paid contributors, but there are no known markers that are highly correlated with volunteer contributors.
\end{enumerate}
Issue \#\ref{manypaid} was \emph{prima facie} quite surprising---one might expect there to be a somewhat even mix of paid and volunteer contributors, but upon manually examining contributors to a couple projects, it was discovered that these projects were almost exclusively developed by employees of a few companies during the time period under observation. With such skewed samples, it became difficult to write a classifier sensitive enough to be useful.
Issue \#\ref{novolunteers} was an unforeseen limitation of using profile information to identify social group affiliation. Ultimately, there was no reliable way to differentiate between a volunteer and a paid contributor when his/her data does not reveal any obvious link between his/her contributions and his/her employer.
Since creating this classifier proved infeasible, the data was re-purposed to do a social network analysis of ASF contributions instead. In particular, the following facets of the existing dataset were useful for this purpose:
\begin{itemize}
	\item The dataset associates data from multiple accounts/profiles to one contributor, improving the reliability of finding a given contributor's employer.
	\item Since the contributor's employer was one of the data points which the algorithm used, a fair amount of employer data was already available.
\end{itemize}

\section{Data Sources}
When researching open source software, there are a multitude of possible software repositories and organizations to base the experiment on. For example, some prior work studied SourceForge projects, while others studied GNOME. \pd{Reword this bit to first include some motivation, viz: For our work, we needed to reliably link user identities across different
user functions. We needed this because...}
We chose to study Apache Software Foundation projects because they have a consistent organization structure with respect to how developers collaborate on Git, Github, and JIRA. This structure made it simpler to write a script that can reliably link user identities across these service.
To find the employer of a given Git committer, the script attempts to collect data from the Git account, and any associated JIRA and/or Github accounts. It also utilizes the Google Custom Search API to find links to LinkedIn profiles, which can then be inspected manually. Each set of values is stored in a separate entry in a PostgreSQL database, to maximize the amount of information available for finding a contributor's employer.
The following information is mined from Git, Github, and JIRA accounts:
\begin{itemize}
	\item Username
	\item Email address
	\item Display name
\end{itemize}
The following additional information is mined from Git accounts:
\begin{itemize}
	\item Commit count per project
\end{itemize}
The following additional information is mined from Github accounts:
\begin{itemize}
	\item Location
	\item Company name
	\item List of organizations
\end{itemize}
Furthermore, to work around the rate limit of the Github API, the script utilized a GHTorrent.org database dump as an offline cache of Github data.

\pd{Add a section here between 2.2 and 2.3, entitled "Schema of Mined Data" explaining the schema of all your data, viz., all relations,  including accountprojects}

\section{Data Collection Constraints}
{'fraction_non_alphanumeric': 0.03737891737891738, 'fraction_numerical': 0.002962962962962963, 'mean_word_length': 5.0357634112792296, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1636638', 'n_tokens_mistral': 1990, 'n_tokens_neox': 1892, 'n_words': 1269}