<jupyter_start><jupyter_text>Refactoring Python Code --- <NAME>* Twitter/Github: [@littlepea](https://twitter.com/littlepea12)* Email: [<EMAIL>](mailto:<EMAIL>)  Scope* What is refactoring?* Why should we refactor?* When to refactor?* Refactoring examples (how to refactor?)  What is Refactoring? > "Refactoring is a disciplined technique for restructuring an existing body of code, altering its internal structure without changing its external behavior." -- © <NAME>  Or in other words:> Refactoring is a process of improving your code without writing any new functionality.  Why should we refactor? * Make the code easier to read and maintain* Reduce complexity* Improve performance* Make the code more reusable and flexible* Pay off technical debt* Keep the "cost of change" low  When to refactor? * Ideally: as you code...* Otherwise:  * When you find yourself duplicating functionality  * When working with legacy code  * When adding features  * When fixing bugs   The boy-scout rule:  > "always leave the code behind in a better state than you found it." -- © <NAME>  How to refactor?  Types of Refactoring:* Improve form (names, functions size, nesting)* Improve style (PEP8/PyLint, pythonic code)* Reduce duplication (DRY) and complexity (KISS)* Apply design patterns (such as "Ports and Adapters")* Apply SOLID principles (such as "Single Responsibility Principle")* Improve code structure (modularity and composability)  Refactoring examplesWe're going to refactor a sample console movie reviews aggregation utility to demonstrate some examples of all kinds of refactoring.We'll start with a working application that aggregates reviews from Twitter and IMDB using the movie name as input and we'll apply one-by-one different refartoring patters to make it more readable and maintainable python code.  Code "Before Refactoring" [movie_reviews.py](https://github.com/littlepea/python-refactoring-talk/blob/master/before/movie_reviews.py)<jupyter_code>"""Movie Reviews.

Usage:  movie_reviews.py <title>
        movie_reviews.py (-h | --help)
        movie_reviews.py --version

Arguments:
  <title>         Movie title

Options:
  -h --help     Show this screen.
  --version     Show version.
"""

from docopt import docopt
from TwitterSearch import *
from dateutil import parser
from imdbpie import Imdb
import logging
import os

def main(title):
    reviews = []

    # Search tweets
    ts = TwitterSearch(
        consumer_key = os.environ.get('TWITTER_CONSUMER_KEY'),
        consumer_secret = os.environ.get('TWITTER_CONSUMER_SECRET'),
        access_token = os.environ.get('TWITTER_ACCESS_TOKEN'),
        access_token_secret = os.environ.get('TWITTER_TOKEN_SECRET')
    )
    try:
        ts.connect()

        tso = TwitterSearchOrder() # create a TwitterSearchOrder object
        tso.setKeywords(['#' + title + 'Movie']) # let's define all words we would like to have a look for
        tso.setLanguage('en') # we want to see German tweets only
        tso.setIncludeEntities(False) # and don't give us all those entity information

        # add tweets to reviews list
        results = ts.getSearchResults(tso)

    except TwitterSearchException as e: # take care of all those ugly errors if there are some
        logging.exception(str(e))
        ts.cleanUp()
    else:
        for offset in range(results.getSize()):
            if offset > 9:
                break
            tweet = results.getTweetByIndex(offset)
            reviews.append({
                'author': tweet.getUserName(),
                'summary': tweet.getText(),
                'text': tweet.getText(),
                'date': parser.parse(tweet.getCreatedDate(), ignoretz=True),
                'source': 'Twitter'
            })
    finally:
        ts.disconnect()

    # Search Imdb
    imdb = Imdb()
    try:
        response = imdb.search_for_title(title)[0]
        title_id = response['imdb_id']
        response = imdb.get_title_reviews(title_id, max_results=10)
    except IndexError as e:
        logging.exception(str(e))
    else:
        for review in response:
            reviews.append({
                'author': review.username,
                'summary': review.summary,
                'text': review.text,
                'date': parser.parse(review.date, ignoretz=True),
                'source': 'IMDB'
            })

    # Sort reviews by date
    reviews.sort(cmp=_cmprev)

    # Print reviews
    for review in reviews:
        print('(%s) @%s: %s [Source: %s]' % ( review['date'].strftime('%Y-%m-%d'), review['author'], review['summary'], review['source'] ) )

def _cmprev(r1, r2):
    if r1['date']>r2['date']:
        return -1
    elif r1['date']<r2['date']:
        return 1
    else:
        return 0<jupyter_output><empty_output><jupyter_text>Improve Style PEP8/PyLint:* Imports rules/order* Whitespace rules  Imports: BEFORE<jupyter_code>from docopt import docopt
from TwitterSearch import *
from dateutil import parser
from imdbpie import Imdb
import logging
import os<jupyter_output><empty_output><jupyter_text>Problems:* 3rd-party packages before the standard library* Wildcard imports  Imports: AFTER[commit 1fa2de3](https://github.com/littlepea/python-refactoring-talk/pull/1/commits/1fa2de3)<jupyter_code>import logging
import os

from docopt import docopt
from dateutil import parser
from imdbpie import Imdb
from TwitterSearch import TwitterSearch, TwitterSearchOrder, TwitterSearchException<jupyter_output><empty_output><jupyter_text>Improve Names* Explicit better than implicit* Names should reflect the purpose  Sorting function: BEFORE<jupyter_code>def _cmprev(r1, r2):
    if r1['date']>r2['date']:
        return -1
    elif r1['date']<r2['date']:
        return 1
    else:
        return 0<jupyter_output><empty_output><jupyter_text>Sorting function: AFTER[commit 80fc3d6](https://github.com/littlepea/python-refactoring-talk/pull/1/commits/80fc3d6)<jupyter_code>def _sort_by_date_desc(first, second):
    first_date = first['date']
    second_date = second['date']

    if first_date > second_date:
        return -1
    elif first_date < second_date:
        return 1

    return 0<jupyter_output><empty_output><jupyter_text>Create Adapters for 3rd-party code  Benefits:* Decouple the dependencies on the application boundary* Only expose the functionality you need* Expose 3rd-party functionality via a pythonic API* Easy to mock/test* Easy to swap  Twitter Search: BEFORE<jupyter_code>def main(title):
    reviews = []

    # Search tweets
    ts = TwitterSearch(
        consumer_key = os.environ.get('TWITTER_CONSUMER_KEY'),
        consumer_secret = os.environ.get('TWITTER_CONSUMER_SECRET'),
        access_token = os.environ.get('TWITTER_ACCESS_TOKEN'),
        access_token_secret = os.environ.get('TWITTER_TOKEN_SECRET')
    )
    try:
        ts.connect()

        tso = TwitterSearchOrder()  # create a TwitterSearchOrder object
        tso.setKeywords(['#' + title + 'Movie'])  # let's define all words we would like to have a look for
        tso.setLanguage('en')  # we want to see German tweets only
        tso.setIncludeEntities(False)  # and don't give us all those entity information

        # add tweets to reviews list
        results = ts.getSearchResults(tso)

    except TwitterSearchException as e:  # take care of all those ugly errors if there are some
        logging.exception(str(e))
        ts.cleanUp()
    else:
        for offset in range(results.getSize()):
            if offset > 9:
                break
            tweet = results.getTweetByIndex(offset)
            reviews.append({
                'author': tweet.getUserName(),
                'summary': tweet.getText(),
                'text': tweet.getText(),
                'date': parser.parse(tweet.getCreatedDate(), ignoretz=True),
                'source': 'Twitter'
            })
    finally:
        ts.disconnect()<jupyter_output><empty_output><jupyter_text>TwitterReviews adapter[commit 3156496](https://github.com/littlepea/python-refactoring-talk/pull/1/commits/3156496)<jupyter_code>import logging
import os
from dateutil import parser

from TwitterSearch import TwitterSearch, TwitterSearchOrder, TwitterSearchException


class TwitterReviews(object):
    def __init__(self, movie, limit=10, language='en'):
        self.movie = movie
        self.limit = 10
        self.language = language
        self.client = TwitterSearch(
            consumer_key=os.environ.get('TWITTER_CONSUMER_KEY'),
            consumer_secret=os.environ.get('TWITTER_CONSUMER_SECRET'),
            access_token=os.environ.get('TWITTER_ACCESS_TOKEN'),
            access_token_secret=os.environ.get('TWITTER_TOKEN_SECRET')
        )

    def __enter__(self):
        self.client.connect()
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        if exc_type == TwitterSearchException:
            logging.exception(str(exc_val))
            self.client.cleanUp()
        self.client.disconnect()

    @property
    def reviews(self):
        return Reviews(self._get_results(), limit=self.limit)

    def _prepare_request(self):
        tso = TwitterSearchOrder()
        tso.setKeywords(self._get_keywords())
        tso.setLanguage(self.language)
        tso.setIncludeEntities(False)
        return tso

    def _get_keywords(self):
        return ['#' + self.movie + 'Movie']

    def _get_results(self):
        request = self._prepare_request()
        return self.client.getSearchResults(request)


class Reviews(object):
    def __init__(self, tweets, limit=10):
        self.limit = limit
        self.tweets = tweets

    def __len__(self):
        size = self.tweets.getSize()
        return size if size < self.limit else self.limit

    def __getitem__(self, item):
        if item >= len(self):
            raise IndexError
        tweet = self.tweets.getTweetByIndex(item)
        return Review(tweet)


class Review(object):
    def __init__(self, review):
        self.review = review

    @property
    def author(self):
        return self.review.getUserName()

    @property
    def summary(self):
        return self.review.getText()

    @property
    def text(self):
        return self.review.getText()

    @property
    def date(self):
        return parser.parse(self.review.getCreatedDate(), ignoretz=True)

    @property
    def source(self):
        return 'Twitter'<jupyter_output><empty_output><jupyter_text>Twitter Search: AFTER[commit d0a40a4](https://github.com/littlepea/python-refactoring-talk/pull/1/commits/d0a40a4)<jupyter_code>def main(title, year=None):
    reviews = []
    
    with TwitterReviews(title) as reviews_backend:
        for review in reviews_backend.reviews:
            reviews.append(review)<jupyter_output><empty_output><jupyter_text>Remove Duplication (DRY)* Duplicated code should be extracted into separate functions, methods and classes* But the purpose is reusability, not just "not allowing any duplication"  IMDB Reviews: BEFORE<jupyter_code>def main(title, year=None):
    reviews = []
    
    with TwitterReviews(title) as reviews_backend:
        for review in reviews_backend.reviews:
            reviews.append(review)
            
    # Search Imdb
    imdb = Imdb()
    try:
        response = imdb.search_for_title(title)[0]
        title_id = response['imdb_id']
        response = imdb.get_title_reviews(title_id, max_results=10)
    except IndexError as e:
        logging.exception(str(e))
    else:
        for review in response:
            reviews.append({
                'author': review.username,
                'summary': review.summary,
                'text': review.text,
                'date': parser.parse(review.date, ignoretz=True),
                'source': 'IMDB'
            })<jupyter_output><empty_output><jupyter_text>IMDB Reviews adapter[commit aefc54e](https://github.com/littlepea/python-refactoring-talk/pull/1/commits/aefc54e)<jupyter_code>import logging

from dateutil import parser
from imdbpie import Imdb


class IMDBReviews(object):
    def __init__(self, movie, limit=10, language='en'):
        self.movie = movie
        self.limit = limit
        self.language = language
        self.client = Imdb()

    def __enter__(self):
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        if exc_val:
            logging.exception(str(exc_val))

    @property
    def reviews(self):
        return [Review(result) for result in self._get_results()]

    def _get_results(self):
        try:
            response = self.client.search_for_title(self.movie)[0]
            title_id = response['imdb_id']
            response = self.client.get_title_reviews(title_id, max_results=10)
            return response
        except IndexError as e:
            logging.exception(str(e))
            return []


class Review(object):
    def __init__(self, review):
        self.review = review

    @property
    def author(self):
        return self.review.username

    @property
    def summary(self):
        return self.review.summary

    @property
    def text(self):
        return self.review.text

    @property
    def date(self):
        return parser.parse(self.review.date, ignoretz=True)

    @property
    def source(self):
        return 'IMDB'<jupyter_output><empty_output><jupyter_text>IMDB Reviews: AFTER[commit 23888d9](https://github.com/littlepea/python-refactoring-talk/pull/1/commits/23888d9)<jupyter_code>def main(title):
    reviews = []
    
    for backend_class in (TwitterReviews, IMDBReviews):
        with backend_class(title) as reviews_backend:
            for review in reviews_backend.reviews:
                reviews.append(review)<jupyter_output><empty_output><jupyter_text>BaseReview class[commit 4a314cd](https://github.com/littlepea/python-refactoring-talk/pull/1/commits/4a314cd)<jupyter_code>class BaseReview(object):
    def __init__(self, review):
        self.review = review

    @property
    def author(self):
        raise NotImplementedError

    @property
    def summary(self):
        raise NotImplementedError

    @property
    def text(self):
        raise NotImplementedError

    @property
    def date(self):
        raise NotImplementedError

    @property
    def source(self):
        raise NotImplementedError

    def display(self):
        print '(%s) @%s: %s [Source: %s]' % (
            self.date.strftime('%Y-%m-%d'),
            self.author,
            self.summary,
            self.source)<jupyter_output><empty_output><jupyter_text>Adding more backends* Now that we've refactored using new abstractions we can easily add more review sources* Let's add NY Times movie reviews to the mix[commit 801f26c](https://github.com/littlepea/python-refactoring-talk/pull/1/commits/801f26c)<jupyter_code>import logging
import os

import requests
from dateutil import parser

BaseReview = object  # ignore this line, it's just for Jupyter notebook


class NYTimesReviews(object):
    def __init__(self, movie, limit=10, language='en'):
        self.movie = movie
        self.limit = 10
        self.language = language
        self.url = 'https://api.nytimes.com/svc/movies/v2/reviews/search.json'
        self.api_key = os.environ.get('NY_TIMES_API_KEY')

    def __enter__(self):
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        if exc_val:
            logging.exception(str(exc_val))

    @property
    def reviews(self):
        return [Review(result) for result in self._get_results()]

    def _get_results(self):
        response = requests.get(self.url, self._prepare_request_data())
        return response.json()['results']

    def _prepare_request_data(self):
        data = {
            'query': self.movie,
            'api-key': self.api_key
        }
        return data


class Review(BaseReview):
    @property
    def author(self):
        return self.review['byline']

    @property
    def summary(self):
        return self.review['headline']

    @property
    def text(self):
        return self.review['summary_short']

    @property
    def date(self):
        return parser.parse(self.review['date_updated'], ignoretz=True)

    @property
    def source(self):
        return 'NYTimes'<jupyter_output><empty_output><jupyter_text>NY Times Reviews: BEFORE<jupyter_code>def main(title):
    reviews = []

    for backend_class in (TwitterReviews, IMDBReviews):
        with backend_class(title) as reviews_backend:
            for review in reviews_backend.reviews:
                reviews.append(review)

    # Search NYTimes
    url = "https://api.nytimes.com/svc/movies/v2/reviews/search.json"
    data = {
        'query': title,
        'api-key': os.environ.get('NY_TIMES_API_KEY')
    }
    response = requests.get(url, data)
    count = 0
    for review in response.json()['results']:
        if count > 9:
            break
        reviews.append({
            'author': review['byline'],
            'summary': review['headline'],
            'text': review['summary_short'],
            'date': parser.parse(review['date_updated'], ignoretz=True),
            'source': 'NYTimes'
        })
        count += 1<jupyter_output><empty_output><jupyter_text>NY Times Reviews: AFTER[commit 49079b6](https://github.com/littlepea/python-refactoring-talk/pull/1/commits/49079b6)<jupyter_code>def main(title):
    reviews = []

    for backend_class in (TwitterReviews, IMDBReviews, NYTimesReviews):
        with backend_class(title) as reviews_backend:
            for review in reviews_backend.reviews:
                reviews.append(review)<jupyter_output><empty_output><jupyter_text>Fix sorting* `cmp` functions are deprecated in Python 3 and are hard to read* Let's use `key` function instead* And separate getting sorted reviews into a service function  Sorting: BEFORE<jupyter_code>def _cmprev(r1, r2):
    if r1['date']>r2['date']:
        return -1
    elif r1['date']<r2['date']:
        return 1
    else:
        return 0<jupyter_output><empty_output><jupyter_text>Sorting and Services: AFTER[commit fa75a8f](https://github.com/littlepea/python-refactoring-talk/pull/1/commits/fa75a8f)<jupyter_code>def get_reviews(title):
    reviews = []
    for backend_class in (TwitterReviews, IMDBReviews, NYTimesReviews):
        with backend_class(title) as reviews_backend:
            for review in reviews_backend.reviews:
                reviews.append(review)

    return reversed(sorted(reviews, key=_get_review_date))


def _get_review_date(review):
    return review.date<jupyter_output><empty_output><jupyter_text>Final: BEFORE[movie_reviews.py](https://github.com/littlepea/python-refactoring-talk/blob/master/before/movie_reviews.py)<jupyter_code>"""Movie Reviews.

Usage:  movie_reviews.py <title>
        movie_reviews.py (-h | --help)
        movie_reviews.py --version

Arguments:
  <title>         Movie title

Options:
  -h --help     Show this screen.
  --version     Show version.
"""

from docopt import docopt
from TwitterSearch import *
from dateutil import parser
from imdbpie import Imdb
import logging
import os

def main(title):
    reviews = []

    # Search tweets
    ts = TwitterSearch(
        consumer_key = os.environ.get('TWITTER_CONSUMER_KEY'),
        consumer_secret = os.environ.get('TWITTER_CONSUMER_SECRET'),
        access_token = os.environ.get('TWITTER_ACCESS_TOKEN'),
        access_token_secret = os.environ.get('TWITTER_TOKEN_SECRET')
    )
    try:
        ts.connect()

        tso = TwitterSearchOrder() # create a TwitterSearchOrder object
        tso.setKeywords(['#' + title + 'Movie']) # let's define all words we would like to have a look for
        tso.setLanguage('en') # we want to see German tweets only
        tso.setIncludeEntities(False) # and don't give us all those entity information

        # add tweets to reviews list
        results = ts.getSearchResults(tso)

    except TwitterSearchException as e: # take care of all those ugly errors if there are some
        logging.exception(str(e))
        ts.cleanUp()
    else:
        for offset in range(results.getSize()):
            if offset > 9:
                break
            tweet = results.getTweetByIndex(offset)
            reviews.append({
                'author': tweet.getUserName(),
                'summary': tweet.getText(),
                'text': tweet.getText(),
                'date': parser.parse(tweet.getCreatedDate(), ignoretz=True),
                'source': 'Twitter'
            })
    finally:
        ts.disconnect()

    # Search Imdb
    imdb = Imdb()
    try:
        response = imdb.search_for_title(title)[0]
        title_id = response['imdb_id']
        response = imdb.get_title_reviews(title_id, max_results=10)
    except IndexError as e:
        logging.exception(str(e))
    else:
        for review in response:
            reviews.append({
                'author': review.username,
                'summary': review.summary,
                'text': review.text,
                'date': parser.parse(review.date, ignoretz=True),
                'source': 'IMDB'
            })

    # Search NYTimes
    url = "https://api.nytimes.com/svc/movies/v2/reviews/search.json"
    data = {
        'query': title,
        'api-key': os.environ.get('NY_TIMES_API_KEY')
    }
    response = requests.get(url, data)
    count = 0
    for review in response.json()['results']:
        if count > 9:
            break
        reviews.append({
            'author': review['byline'],
            'summary': review['headline'],
            'text': review['summary_short'],
            'date': parser.parse(review['date_updated'], ignoretz=True),
            'source': 'NYTimes'
        })
        count += 1

    # Sort reviews by date
    reviews.sort(cmp=_cmprev)

    # Print reviews
    for review in reviews:
        print('(%s) @%s: %s [Source: %s]' % ( review['date'].strftime('%Y-%m-%d'), review['author'], review['summary'], review['source'] ) )

def _cmprev(r1, r2):
    if r1['date']>r2['date']:
        return -1
    elif r1['date']<r2['date']:
        return 1
    else:
        return 0<jupyter_output><empty_output><jupyter_text>Final `main()` program: AFTER[movie_reviews.py](https://github.com/littlepea/python-refactoring-talk/blob/develop/after/movie_reviews.py)<jupyter_code>def main(title):
    for review in get_reviews(title):
        review.display()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1068442934418168, 'fraction_numerical': 0.00809155527739282, 'mean_word_length': 2.87292243767313, 'pattern_counts': {'":': 0, '<': 92, '<?xml version=': 0, '>': 100, 'https://': 17, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '514082', 'license': 'MIT', 'n_tokens_mistral': 6976, 'n_tokens_neox': 6700, 'n_words': 1628, 'path': 'refactoring.ipynb', 'repo_name': 'littlepea/python-refactoring-talk', 'size': '22427'}