<jupyter_start><jupyter_text>Core API QuickstartBy [<NAME>](leonyin.org) for [SMaPP NYU](https://wp.nyu.edu/smapp/)[urlExpander](https://github.com/SMAPPNYU/urlExpander) is a Python package for quickly and thoroughly expanding URLs.You can download the software using pip:<jupyter_code>import os
import urlexpander
from runtimestamp.runtimestamp import runtimestamp
runtimestamp('QuickStart User')
print(f"This notebook is using urlExpander v{urlexpander.__version__}")<jupyter_output>Updated 2021-11-05 16:35:16.957041
By QuickStart User
Using Python 3.8.10
On Linux-5.4.0-89-generic-x86_64-with-glibc2.29
This notebook is using urlExpander v0.0.38<jupyter_text>Here is a toy example of some URLs taken from Congressional Twitter accounts:<jupyter_code>urls = [
    'https://trib.al/xXI5ruM',
    'http://bit.ly/1Sv81cj',
    'https://www.youtube.com/watch?v=8NwKcfXvGl4',
    'https://t.co/zNU1eHhQRn',
]<jupyter_output><empty_output><jupyter_text>We can use the `expand` function (see the code) to unshorten any link:<jupyter_code>urlexpander.expand(urls[0])<jupyter_output><empty_output><jupyter_text>It also works on any list of URLs.<jupyter_code>urlexpander.expand(urls)<jupyter_output><empty_output><jupyter_text>To save compute time, we can skip links that don't need to be expanded.The `is_short` function takes any url and checks if the domain is from a known list of link shorteners<jupyter_code>print(f"{urls[1]} returns:")
urlexpander.url_utils.is_short(urls[1])<jupyter_output>http://bit.ly/1Sv81cj returns:<jupyter_text>bit.ly is probably the best known link shortener, Youtube.com however is not a link shortener!<jupyter_code>print(f"{urls[2]} returns:")
urlexpander.url_utils.is_short(urls[2])<jupyter_output>https://www.youtube.com/watch?v=8NwKcfXvGl4 returns:<jupyter_text>urlExpander takes advantage of a list of known domains that offer link shortening services.<jupyter_code>known_shorteners = urlexpander.constants.all_short_domains.copy()
print(len(known_shorteners))<jupyter_output>86<jupyter_text>You can make modifications or use your own `list_of_domains` as an argument for the `is_short` function.<jupyter_code>known_shorteners += ['youtube.com']
print(f"Now {urls[2]} returns:")
urlexpander.url_utils.is_short(urls[2], list_of_domains=known_shorteners) # this is the default<jupyter_output>Now https://www.youtube.com/watch?v=8NwKcfXvGl4 returns:<jupyter_text>Now we can shorten our workload:<jupyter_code># filter only domains that need to be shortenened
urls_to_shorten = [link for link in urls if urlexpander.url_utils.is_short(link)]
urls_to_shorten<jupyter_output><empty_output><jupyter_text>urlExpander's `multithread_expand()` does heavy lifting to quickly and thoroughly expand a list of links:<jupyter_code>expanded_urls = urlexpander.expand(urls_to_shorten)
expanded_urls<jupyter_output><empty_output><jupyter_text>Note that URLs that resolve to defunct pages, still return the domain name -- followed by the type of error surrounded by two underscores IE `http://www.billshusterforcongress.com/__CONNECTIONPOOL_ERROR__`.Instead of filtering the inputs before running the `expand` function, you can assign a filter using the `filter_function` argument.Filter functions can be any boolean function that operates on a string. Below is an example function that filters for t.co links:<jupyter_code>def custom_filter(url):
    '''This function returns True if the url is a shortened Twitter URL'''
    if urlexpander.url_utils.get_domain(url) == 't.co':
        return True
    else:
        return False
resolved_links = urlexpander.expand(urls, 
                                    filter_function=custom_filter, 
                                    verbose=1)
resolved_links<jupyter_output>There are 1 URLs to expand<jupyter_text>Although filtering within the `expand` function is convenient, you will see changes in performance time.<jupyter_code>resolved_links = urlexpander.expand(urls,  
                                    filter_function=urlexpander.url_utils.is_short,
                                    verbose=1)
resolved_links<jupyter_output>There are 3 URLs to expand<jupyter_text>But that is a toy example, let's see how this fairs with a larger dataset.This package comes with a [sampled dataset](https://github.com/SMAPPNYU/urlExpander/blob/master/urlexpander/core/datasets.pyL8-L29) of links extracted from Twitter accounts from the 115th Congress. If you work with Twitter data you'll be glad to know there is a function `urlexpander.tweet_utils.get_link` for creating a similar dataset from Tweets.<jupyter_code>df_congress = urlexpander.datasets.load_congress_twitter_links(nrows=100)

print(f'The dataset has {len(df_congress)} rows')
df_congress.tail(2)
shortened_urls = df_congress[df_congress.link_domain.apply(urlexpander.url_utils.is_short)].tweet_id.nunique()
all_urls = df_congress.tweet_id.nunique()
print(f"About {(shortened_urls / all_urls)*100}% of the links are short!")<jupyter_output>About 38.0% of the links are short!<jupyter_text>The performance of the next script is dependent on your internet connection:<jupyter_code># !curl -s https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py | python -<jupyter_output><empty_output><jupyter_text>Let's see how long it takes to expand these links.This is where the optional parameters for `expand` shine.We can created multiple threads for requests (using `n_workers`), cache results into a json file (`cache_file`), and chunk the input into smaller pieces (using `chunksize`). Why does this last part matter? Something I noticed when expanding links in mass is that performance degrades over time. Chunking the input prevents this from happening (not sure why though)!<jupyter_code>resolved_links = urlexpander.expand(df_congress['link_url_long'], 
                                    chunksize=1280,
                                    n_workers=8, 
                                    cache_file=os.path.join('output', 'temp_cache.json'), 
                                    verbose=1,
                                    filter_function=urlexpander.url_utils.is_short)<jupyter_output>There are 38 URLs to expand<jupyter_text>At SMaPP, the process of link expansion has been a burden on our research.We hope that this software helps you overcome similar obstacles!<jupyter_code>df_congress['expanded_url'] = resolved_links
df_congress['resolved_domain'] = df_congress['expanded_url'].apply(urlexpander.url_utils.get_domain)
df_congress.tail(2)<jupyter_output><empty_output><jupyter_text>Here are the top 25 shared domains from this sampled Congress dataset:<jupyter_code>df_congress.resolved_domain.value_counts().head(25)<jupyter_output><empty_output><jupyter_text>Bonus Round!You can count number of `resolved_domain`s for each `user_id ` using `count_matrix()`.You can even choose which domains are counted by modifying the `domain_list` arg:<jupyter_code>count_matrix = urlexpander.tweet_utils.count_matrix(df_congress,
                                                    user_col='user_id', 
                                                    domain_col='resolved_domain', 
                                                    unique_count_col='tweet_id',
                                                    domain_list=['youtube.com','facebook.com', 'google.com', 'twitter.com'])

count_matrix.tail(3)<jupyter_output><empty_output><jupyter_text>One of the domain lists you might be interested in are US national media outlets -`datasets.load_us_national_media_outlets()` compiled by <NAME> (Forthcoming).<jupyter_code>urlexpander.datasets.load_us_national_media_outlets()[:5]<jupyter_output><empty_output><jupyter_text>We also built a one-size-fits-all scraper that returns the title, description, and/or paragraphs from any given URL.<jupyter_code>url = urls[0]
example = urlexpander.expand_with_content(url)
html = example['resolved_text']
urlexpander.html_utils.search_webpage_title(html)
urlexpander.html_utils.search_webpage_description(html)
urlexpander.html_utils.search_webpage_meta(url, html)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08995298193516456, 'fraction_numerical': 0.012249443207126948, 'mean_word_length': 4.672280701754386, 'pattern_counts': {'":': 0, '<': 74, '<?xml version=': 0, '>': 74, 'https://': 9, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 18, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '334077', 'license': 'MIT', 'n_tokens_mistral': 2613, 'n_tokens_neox': 2513, 'n_words': 627, 'path': 'examples/core_api.ipynb', 'repo_name': 'wlmwng/urlExpander', 'size': '8090'}