<jupyter_start><jupyter_text>![Hacker News](https://s3.amazonaws.com/dq-content/354/hacker_news.jpg)This project works with a data set of submissions ot popular technology site [Hacker News](https://news.ycombinator.com/) Brief Introduction of Hacker News[Hacker News](https://news.ycombinator.com/) is a site started by the startup incubator [Y Combinator](https://www.ycombinator.com/), where user-submited stories (known as "post") are voted and commented upon, similar to reddit. Hacker News is extremely popular in technology and startup circles, adn posts that make it to the top of Hacker News' listings can get hundrends of thousands of visitors as a result. DatasetHacker news data set can be found [here](https://www.kaggle.com/hacker-news/hacker-news-posts)> Note: the data set has been reduced from almost `300, 000` rows to approximately `20, 000` rows by removing all submissions that did not receive any comments, and then randomly sampling from the remaining submissions.  Dataset Description of the columnsBelow are the descriptions of the columns:* `id`: The unique identifier from Hacker News for the post* `title`: The title of the post* `url`: The URL that the post links to, if the post has a URL* `num_points`: The number of points the post acquired, calculated as the `total number of upvotes` minus the `total number of downvotes`* `num_comments`: The number of comments that were made on the post* `author`: The username of the person who submitted the post* `created_at`: The date and time at which the post was submitted Dataset headHere are the first few rows of the data set:|id|title|url|num_points|num_comments|author|created_at||---|---|---|---|---|---|---|||||||||Interest is on the posts whose titles begin with either `Ask HN` or `Show HN`. Users submit `Ask HN` post to ask the Hacker News community a specific question. Below are a couple of examples:```Ask HN: How to imporve my personaly website?Ask HN: Am I the only one outraged by Twitter shutting down share counts?Ask HN: Aby recent changes to CSS that broke mobile?```Likewise, users submit `Show HN` posts to show the Hacker News community a project, product, or just generally something interesting. Below are a couple of examples:```Show HN: Wio Link ESP8266 Based Web of Things Hardware Development Platform'Show HN: Something pointless I madeShow HN: Shanhu.io, a programming playground powered by e8vm```The two types of posts will be compared to determine the following:* Do `Ask HN` or `Show HN` receive more comments on average?* Do posts created at a certain time receive more comments on average?  1. Importing libraries needed and reading the dataset into list of lists<jupyter_code>import csv
hn = list(csv.reader(open('hacker_news.csv')))
print(hn[:5])<jupyter_output>[['id', 'title', 'url', 'num_points', 'num_comments', 'author', 'created_at'], ['12224879', 'Interactive Dynamic Video', 'http://www.interactivedynamicvideo.com/', '386', '52', 'ne0phyte', '8/4/2016 11:52'], ['10975351', 'How to Use Open Source and Shut the Fuck Up at the Same Time', 'http://hueniverse.com/2016/01/26/how-to-use-open-source-and-shut-the-fuck-up-at-the-same-time/', '39', '10', 'josep2', '1/26/2016 19:30'], ['11964716', "Florida DJs May Face Felony for April Fools' Water Joke", 'http://www.thewire.com/entertainment/2013/04/florida-djs-april-fools-water-joke/63798/', '2', '1', 'vezycash', '6/23/2016 22:20'], ['11919867', 'Technology ventures: From Idea to Enterprise', 'https://www.amazon.com/Technology-Ventures-Enterprise-Thomas-Byers/dp/0073523429', '3', '1', 'hswarna', '6/17/2016 0:01']]<jupyter_text>> **NOTE:** The first list in the inner lists contains the column headers, and the lists after contain the data for one row. In order to analyze the data, the row containing the column headers needs to be removed<jupyter_code>headers = hn[0]
hn = hn[1:]
print(headers,hn[:5])<jupyter_output>['id', 'title', 'url', 'num_points', 'num_comments', 'author', 'created_at'] [['12224879', 'Interactive Dynamic Video', 'http://www.interactivedynamicvideo.com/', '386', '52', 'ne0phyte', '8/4/2016 11:52'], ['10975351', 'How to Use Open Source and Shut the Fuck Up at the Same Time', 'http://hueniverse.com/2016/01/26/how-to-use-open-source-and-shut-the-fuck-up-at-the-same-time/', '39', '10', 'josep2', '1/26/2016 19:30'], ['11964716', "Florida DJs May Face Felony for April Fools' Water Joke", 'http://www.thewire.com/entertainment/2013/04/florida-djs-april-fools-water-joke/63798/', '2', '1', 'vezycash', '6/23/2016 22:20'], ['11919867', 'Technology ventures: From Idea to Enterprise', 'https://www.amazon.com/Technology-Ventures-Enterprise-Thomas-Byers/dp/0073523429', '3', '1', 'hswarna', '6/17/2016 0:01'], ['10301696', 'Note by Note: The Making of Steinway L1037 (2007)', 'http://www.nytimes.com/2007/11/07/movies/07stein.html?_r=0', '8', '2', 'walterbell', '9/30/2015 4:12']]<jupyter_text>Data filteringSince the concern is on the post titles begining with `Ask HN` or `Show HN`, new lists of lists containing just the data for those titles needs to be created.In order to find the posts that begin with either `Ask HN` or `Show HN`, pythons' string method `startswith` will be used.**Example**: Given a string object, say, `string1`, we can check if it starts with, say, `dq` by inspecting the output of the object `string1.startswith('dq')`. If `string1` starts with `dq`, it will return `True`, otherwise it will return `False````pythonprint('dataquest'.startswith('Data'))print('dataquest'.startswith('data'))``````pythonFalseTrue```In the example above, the first print call gives `False` because `dataquest` does not start with `Data`. The second print call prints `True` because `dataquest` does start with `data`. Capitalization matters.  Extracting `Ask HN` and `Show HN` Posts<jupyter_code>ask_posts = []
show_posts = []
other_posts = []
for row in hn:
    title = row[1]
    if title.lower().startswith('ask hn'):
        ask_posts.append(row)
    elif title.lower().startswith('show hn'):
        show_posts.append(row)
    else:
        other_posts.append(row)
print('Number of posts in ask_posts:',len(ask_posts))
print('Number of posts in show_posts:', len(show_posts))
print('Number of posts in other_posts', len(other_posts))<jupyter_output>Number of posts in ask_posts: 1744
Number of posts in show_posts: 1162
Number of posts in other_posts 17194<jupyter_text>Below are the first five rows in the `ask_posts` list of lists:<jupyter_code>print(ask_posts[:5])<jupyter_output>[['12296411', 'Ask HN: How to improve my personal website?', '', '2', '6', 'ahmedbaracat', '8/16/2016 9:55'], ['10610020', 'Ask HN: Am I the only one outraged by Twitter shutting down share counts?', '', '28', '29', 'tkfx', '11/22/2015 13:43'], ['11610310', 'Ask HN: Aby recent changes to CSS that broke mobile?', '', '1', '1', 'polskibus', '5/2/2016 10:14'], ['12210105', 'Ask HN: Looking for Employee #3 How do I do it?', '', '1', '3', 'sph130', '8/2/2016 14:20'], ['10394168', 'Ask HN: Someone offered to buy my browser extension from me. What now?', '', '28', '17', 'roykolak', '10/15/2015 16:38']]<jupyter_text>Below are the five rows in the `show_posts` list of lists<jupyter_code>print(show_posts[:5])<jupyter_output>[['10627194', 'Show HN: Wio Link  ESP8266 Based Web of Things Hardware Development Platform', 'https://iot.seeed.cc', '26', '22', 'kfihihc', '11/25/2015 14:03'], ['10646440', 'Show HN: Something pointless I made', 'http://dn.ht/picklecat/', '747', '102', 'dhotson', '11/29/2015 22:46'], ['11590768', 'Show HN: Shanhu.io, a programming playground powered by e8vm', 'https://shanhu.io', '1', '1', 'h8liu', '4/28/2016 18:05'], ['12178806', 'Show HN: Webscope  Easy way for web developers to communicate with Clients', 'http://webscopeapp.com', '3', '3', 'fastbrick', '7/28/2016 7:11'], ['10872799', 'Show HN: GeoScreenshot  Easily test Geo-IP based web pages', 'https://www.geoscreenshot.com/', '1', '9', 'kpsychwave', '1/9/2016 20:45']]<jupyter_text>4. Calculating the Average Number of Comments for `Ask HN` and `Show HN` postsDetermining if `ask_posts` or `show_posts` receive more comments on average<jupyter_code>total_ask_comments = 0
for row in ask_posts:
    num_comments = int(row[4])
    total_ask_comments += num_comments
avg_ask_comments = total_ask_comments/len(ask_posts)
print(avg_ask_comments)
total_show_comments = 0
for row in show_posts:
    num_comments = int(row[4])
    total_show_comments += num_comments
avg_show_comments = total_show_comments/len(show_posts)
print(avg_show_comments)<jupyter_output>10.31669535283993<jupyter_text>Based on the sample, on average `ask_posts` receive more comments (`14`) than `show_posts` (`10`). This analysis will focus of `ask_posts` since it receives more comments.  5. Finding the Amount of `ask_posts` and Coments by Hour CreatedDetermining if ask posts created at a certain `time` are more likely to attract comments. The following steps will be used to perform this analysis:1. Calculate the ***amount of ask posts*** created in each hour of the day, alon with the number of comments received2. Calculate the ***average number of comments*** ask posts receive by hour created.<jupyter_code>import datetime as dt
result_list = []
for row in ask_posts:
    created_at = row[6]
    num_comments = int(row[4])
    result_list.append([created_at,num_comments])
counts_by_hour = {}
comments_by_hour= {}
for row in result_list:
    date_created_at=dt.datetime.strptime(row[0], "%m/%d/%Y %H:%M")
    hour_created_at = dt.datetime.strftime(date_created_at, '%H')
    if hour_created_at not in counts_by_hour:
        counts_by_hour[hour_created_at]=1
        comments_by_hour[hour_created_at]=row[1]
    else:
        comments_by_hour[hour_created_at]+=row[1]
        counts_by_hour[hour_created_at]+=1
comments_by_hour
counts_by_hour<jupyter_output><empty_output><jupyter_text>6. Calculating the Average Number of Comments for `Ask HN` posts by HourSince:1. `counts_by_hour`: contains the number of ask posts created during each hour of the day2. `comments_by_hour`: contains the correstponding number of comments ask posts created at each hour received,the two dictionaries will be used to calculate the ***average number of comments*** for posts created during each hour of the day.Creating a list of lists containing the hours during which posts were created and the average number of comments those posts received<jupyter_code>avg_by_hour = []
for hour in comments_by_hour:
    avg_by_hour.append([hour, comments_by_hour[hour]/counts_by_hour[hour]])
avg_by_hour<jupyter_output><empty_output><jupyter_text>7. Sorting and Printing Values from a List of Listsalthough `avg_by_hour` contains the results needed, this formart makes it hart to identify the hours with the highest values. The list needs to be sorted and identifying five highest values in a format that's easier to read.<jupyter_code>swap_avg_by_hour = []
for row in avg_by_hour:
    swap_avg_by_hour.append([row[1], row[0]])
print(swap_avg_by_hour)
sorted_swap = sorted(swap_avg_by_hour, reverse=True)
sorted_swap
print('Top 5 Hours for Ask Posts Comments')
for row in sorted_swap[:5]:
    print(
        "{} {:.2f} average comments per post".format(
            dt.datetime.strptime(row[1], '%H').strftime('%H:%M'),
            row[0]
        )
    )<jupyter_output>Top 5 Hours for Ask Posts Comments
15:00 38.59 average comments per post
02:00 23.81 average comments per post
20:00 21.52 average comments per post
16:00 16.80 average comments per post
21:00 16.01 average comments per post
{'fraction_non_alphanumeric': 0.12784041630529056, 'fraction_numerical': 0.05507372072853426, 'mean_word_length': 5.287350054525627, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 32, 'https://': 10, 'lorem ipsum': 0, 'www.': 10, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '518825', 'license': 'MIT', 'n_tokens_mistral': 4123, 'n_tokens_neox': 3626, 'n_words': 1322, 'path': 'Basics.ipynb', 'repo_name': 'VictorOmondi1997/ExploringHackerNewsPosts', 'size': '11530'}