<commit_before>
\section{Discussion}
\label{sec:discussion}


%%% Local Variables: 
%%% mode: latex
%%% TeX-master: "main"
%%% End: <commit_msg>added discussion of king<commit_after>
\section{Discussion}
\label{sec:discussion}
We faced a number of issues over the course of our experiment. During our reverse DNS crawl received a number of reports of port scanning. As such, it was difficult to test our crawler during development, as Amazon would occasionally limit our available bandwidth. Initially it was difficult to manage a large number of Planet Lab nodes. Occasionally nodes crashed or reported extremely high latencies because of very poor networking conditions (a Berkeley node was a repeat offender). Our development efforts were often hampered by the very old software stack on some of these systems. Furthermore, we received notices from Columbia University because of SSH tunnels to node (we are still investigating why this was the case). We had to discard measurement results from 4/13-4/19 because MySQL decreased resolution of recorded time deltas from milliseconds to seconds.

%%% Local Variables: 
%%% mode: latex
%%% TeX-master: "main"
%%% End: 
{'fraction_non_alphanumeric': 0.06506849315068493, 'fraction_numerical': 0.005136986301369863, 'mean_word_length': 5.12041884816754, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7454414', 'n_tokens_mistral': 301, 'n_tokens_neox': 270, 'n_words': 160}