<commit_before><NAME> and <NAME> and <NAME>
Data Mining Final Project Proposal

As <NAME> thinks that the HITS algorithm is the coolest thing ever (and Andy and Conrad also think it's pretty cool), we plan to implement HITS.

DAN TALK ABOUT WHAT YOU DISCUSSED WITH DAVE

While that is all right, we would like to do something cooler, like make a search engine. We don't expect to make a really good search engine, maybe not even one with a real crawler. If making a useful crawler proves too hard, we'll come up with one that crawls/searchs a (hopefully fairly large) set of pages that we will download in advance.

To do this, we will have to:
- implement HITS.
- Maybe write some sort of crawler (that respects robots.txt!) (or find a decent one or just download pages in advance).
- Come up with some way to index pages, possibly using title tags, meta tags, information from links, whatever.
- Come up with some way to search pages. This part will probably not be very good, as we don't have much (any?) background in natural language processing. We could restrict it to Boolean searches or structured queries, which would probably make our lives easier.

We will need to read up on some of these things [Wikipedia](http://en.wikipedia.org/wiki/Web_search_engine) seems like a good starting place.

This is super ambitious, but we hope it will be super fun.<commit_msg>updated proposal<commit_after><NAME> and <NAME> and <NAME>
Data Mining Final Project Proposal
==========================================

As <NAME> thinks that the HITS algorithm is the coolest thing ever (and Andy and Conrad also think it's pretty cool), we plan to implement HITS.

While that is all right, we would like to do something cooler, like make a search engine. We don't expect to make a really good search engine, maybe not even one with a real crawler. If making a useful crawler proves too hard, we'll come up with one that crawls/searchs a (hopefully fairly large) set of pages that we will download in advance.

To do this, we will have to:
- implement HITS.
- Maybe write some sort of crawler (that respects robots.txt!) (or find a decent one or just download pages in advance).
- Come up with some way to index pages, possibly using title tags, meta tags, information from links, whatever.
- Come up with some way to search pages. This part will probably not be very good, as we don't have much (any?) background in natural language processing. We could restrict it to Boolean searches or structured queries, which would probably make our lives easier.

We will need to read up on some of these things [Wikipedia](http://en.wikipedia.org/wiki/Web_search_engine) seems like a good starting place.

This is super ambitious, but we hope it will be super fun.

If we realize that making a search engine is too hard or if we have extra time, we could write a report comparing the results of HITS to PageRank. Also, in addition to authority, we could also calculate the hubbiness for the web pages. We would talk about our results, and maybe we could make a graph showing the pages and links between them as well as their authority and hubbiness.
{'fraction_non_alphanumeric': 0.0603448275862069, 'fraction_numerical': 0.0, 'mean_word_length': 4.301184433164129, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2709738', 'n_tokens_mistral': 788, 'n_tokens_neox': 756, 'n_words': 519}