<jupyter_start><jupyter_text>The problem statement I am interested in is: **predicting the publisher's name from a given title**. For approaching this problem, first I am going to need a dataset consisting of article/post titles with their sources mentioned. The dataset I am going to use is already there as a BigQuery public dataset ([link](https://console.cloud.google.com/bigquery?p=bigquery-public-data&d=hacker_news&page=dataset)). But the dataset needs to be shaped a bit aligning to my needs. This dataset contains all stories and comments from Hacker News from its launch in 2006 to present. Each story contains a story ID, the author that made the post, when it was written, and the number of points the story received. The problem can be modeled as a text classification problem essentially. The steps I would be following:- Derive a small subset of the dataset from BiqQuery as the original dataset is moderately large (402 MB)- Preprocess the dataset initially - Get the data ready for feeding to a sequence model    - Tokenize the titles    - Pad the sequences    - Create embedding matrix w.r.t pretrained embeddings- Build, train and evaluate the modelLet's begin.**Note**: It is advisable to use Google Colab. To be able to use BigQuery you need to get some setup. Follow the instructions [here](https://colab.research.google.com/notebooks/bigquery.ipynb).<jupyter_code># Authenticate yourself with the Google creds
from google.colab import auth
auth.authenticate_user()
print('Authenticated')
# Set your Project ID
import os

PROJECT = 'fast-ai-exploration'
os.environ['PROJECT'] = PROJECT<jupyter_output><empty_output><jupyter_text>I can now run a BQ query to get you a sense of the dataset.<jupyter_code>%%bigquery --project $PROJECT data_preview
SELECT
  url, title, score
FROM
  `bigquery-public-data.hacker_news.stories`
WHERE
  LENGTH(title) > 10
  AND score > 10
  AND LENGTH(url) > 0
LIMIT 10
data_preview.head()<jupyter_output><empty_output><jupyter_text>The problem in the current data is in place of `url` I need the source of the URL. For example, `https://github.com/Groundworkstech/Submicron` should appear as `github`. I would also want to rename the `url` column to `source`. But first, let me figure out the distribution in the titles belonging to several sources.<jupyter_code>%%bigquery --project $PROJECT source_num_articles
SELECT
  ARRAY_REVERSE(SPLIT(REGEXP_EXTRACT(url, '.*://(.[^/]+)/'), '.'))[OFFSET(1)] AS source,
  COUNT(title) AS num_articles
FROM
  `bigquery-public-data.hacker_news.stories`
WHERE
  REGEXP_CONTAINS(REGEXP_EXTRACT(url, '.*://(.[^/]+)/'), '.com$')
  AND LENGTH(title) > 10
GROUP BY
  source
ORDER BY num_articles DESC
source_num_articles.head()<jupyter_output><empty_output><jupyter_text>Not major, but a slight class imbalance is there. I am going to exclude the titles belonging to `youtube`.<jupyter_code>%%bigquery --project $PROJECT full_data
SELECT source, LOWER(REGEXP_REPLACE(title, '[^a-zA-Z0-9 $.-]', ' ')) AS title FROM
  (SELECT
    ARRAY_REVERSE(SPLIT(REGEXP_EXTRACT(url, '.*://(.[^/]+)/'), '.'))[OFFSET(1)] AS source,
    title
  FROM
    `bigquery-public-data.hacker_news.stories`
  WHERE
    REGEXP_CONTAINS(REGEXP_EXTRACT(url, '.*://(.[^/]+)/'), '.com$')
    AND LENGTH(title) > 10
  )
WHERE (source = 'github' OR source = 'nytimes' OR source = 'techcrunch' or source = 'blogspot')
full_data.head()<jupyter_output><empty_output><jupyter_text>The data is gathered now and as the next step, I would like perform a basic round of EDA. I would start by running a `.shape`.<jupyter_code># Number of (rows, coulumns)
full_data.shape
# Class distribution
full_data.source.value_counts()
# Missing value inspection
full_data.isna().sum()
# How is the length of the titles distributed?
full_data['title'].apply(len).describe()
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

plt.style.use('ggplot')
%matplotlib inline
text_lens = full_data['title'].apply(len).values
plt.figure(figsize=(10,8))
sns.set()
g = sns.distplot(text_lens, kde=False, hist_kws={'rwidth':1})
g.set_xlabel('Title length')
g.set_ylabel('Count')
plt.show()<jupyter_output><empty_output><jupyter_text>I would now like to perform some manual inspections so to figure out - how many titles fall above the minimum title length (11)- how many titles have the maximum length (138)<jupyter_code>(text_lens <= 11).sum()
(text_lens == 138).sum()
full_data[text_lens <= 11]['title'].head()
full_data[text_lens == 138][['title', 'source']]<jupyter_output><empty_output><jupyter_text>Since, there is only one title with the title length of 138, I would remove it to avoid any skew.<jupyter_code>full_data = full_data[text_lens < 138].reset_index(drop=True)
full_data.shape<jupyter_output><empty_output><jupyter_text>Data splits with `pandas` `sample()`<jupyter_code># 80% for train
train = full_data.sample(frac=0.8)
full_data.drop(train.index, axis=0, inplace=True)

# 10% for validation
valid = full_data.sample(frac=0.5)
full_data.drop(valid.index, axis=0, inplace=True)

# 10% for test
test = full_data

train.shape, valid.shape, test.shape<jupyter_output><empty_output><jupyter_text>Check the class distributions in the splits<jupyter_code>train.source.value_counts()
valid.source.value_counts()
test.source.value_counts()
!mkdir data<jupyter_output><empty_output><jupyter_text>Serialize the datasets for later use<jupyter_code>train.to_csv('data/train.csv')
valid.to_csv('data/valid.csv')
test.to_csv('data/test.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11633393829401088, 'fraction_numerical': 0.010707803992740472, 'mean_word_length': 5.623798076923077, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 46, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '63456', 'license': 'Apache-2.0', 'n_tokens_mistral': 1860, 'n_tokens_neox': 1772, 'n_words': 564, 'path': "Predicting publisher's name from an article with TF 2.0/Data_Collection_and_basic_EDA.ipynb", 'repo_name': 'joe733/TF-2.0-Hacks', 'size': '5510'}