<jupyter_start><jupyter_text>ghtop redux> Things we learned by refactoring a CLI tool with fastcore, rich, and ghapi.- comments: true- toc: true- layout: post- hide: true- categories: [ghapi, fastacore, python]- image: images/fastpages_posts/codespaces/codespaces.png- author: "<NAME> & <NAME>"- permalink: /ghtop<jupyter_code>#hide
from collections import Counter
from matplotlib import pyplot as plt
%matplotlib inline
from fastcore.all import *
from ghapi.all import *<jupyter_output><empty_output><jupyter_text>[1]: https://github.com/nat/ghtop[2]: https://github.com/fastai/ghapi IntroductionWe recently refactored the CLI tool [ghtop][1], created by the CEO of GitHub, [<NAME>](https://github.com/nat).  Nat even described our refactor as a “[tour de force](https://github.com/nat/ghtop/pull/13issuecomment-743786158)”.  This post describes what we learned along the way. MotivationRecently, we released [ghapi](https://github.blog/2020-12-18-learn-about-ghapi-a-new-third-party-python-client-for-the-github-api/), a new python client for the GitHub API.  ghapi provides unparalleled ease of access to the GitHub api, as well as utilities for interacting with GitHub Actions.  Part of our motivation for creating ghapi was to accelerate the development of build, testing and deployment tools that help us in maintaining fastai projects.  We recently started using GitHub Actions to perform a wide variety of tasks automatically like: unit and integration [tests](https://github.com/fastai/nbdev/blob/master/.github/workflows/main.yml), deploying [documentation](https://github.com/fastai/fastcore/blob/master/.github/workflows/docs.yml), building [Docker containers](https://github.com/fastai/docker-containers) and [Conda packages](https://github.com/fastai/fastconda), sharing [releases on Twitter](https://github.com/fastai/tweetrel), and much more.  This automation is key to maintaining the vast open source fastai ecosystem with very few maintainers.Since [ghapi][2] is central to so many of these tasks, we wanted to stress-test its efficacy against other projects.  That’s when we found [ghtop][1]. This tool allows you to stream all the public events happening on GitHub to a CLI dashboard.  We thought it would be a fun learning experience to refactor this code base with various fastai tools such as [ghapi][2] and [fastcore](https://github.com/fastai/fastcore), but also try out new libraries like [rich](https://github.com/willmcgugan/rich).  Features we added to our toolsWhile exploring ghtop, we added several features to various fastai tools that we found to be generally useful. `ghapi` AuthenticationWe added the function [github_auth_device](https://ghapi.fast.ai/auth.htmlgithub_auth_device) which allows users to authenticate their api client with GitHub interactively with a browser.  When we call this function we get the following prompt: ```pygithub_auth_device()```<jupyter_code>#hide-input

# This particular output is fake because it is meant to be the result an interactive session.
# This is for the purposes of formatting for the blog article.
print("""
First copy your one-time code: 276E-C910
Then visit https://github.com/login/device in your browser, and paste the code when prompted.
Shall we try to open the link for you? [y/n]
""")<jupyter_output>First copy your one-time code: 276E-C910
Then visit https://github.com/login/device in your browser, and paste the code when prompted.
Shall we try to open the link for you? [y/n]<jupyter_text>The browser opens a window that looks like this:    ![](ghtop_images/token.png)The function then returns an authenticated token which you can use for various tasks.  While this is not the only way to create a token, this is a user friendly way to create a token, especially for those who are not as familiar with GitHub.  `ghapi` Events As a result of our explorations with ghtop, we added an [event module](https://ghapi.fast.ai/event.html) to ghapi.  This is useful for retrieving and inspecting sample events.  Inspecting sample events is important as it allows you to prototype GitHub Actions workflows locally.  You can sample real events with `load_sample_events`:<jupyter_code>from ghapi.event import load_sample_events
evts = load_sample_events()<jupyter_output><empty_output><jupyter_text>Individual events are formatted as markdown lists to be human readable in Jupyter:<jupyter_code>print(evts[0])<jupyter_output>- id: 14517925737
- type: PushEvent
- actor: 
  - id: 17030246
  - login: BeckhamL
  - display_login: BeckhamL
  - gravatar_id: 
  - url: https://api.github.com/users/BeckhamL
  - avatar_url: https://avatars.githubusercontent.com/u/17030246?
- repo: 
  - id: 154349747
  - name: BeckhamL/leetcode
  - url: https://api.github.com/repos/BeckhamL/leetcode
- payload: 
  - push_id: 6194986903
  - size: 1
  - distinct_size: 1
  - ref: refs/heads/master
  - head: 2055b0fcf22f1c3543e38b60199f6882266d32a5
  - before: cb16921949c969b5153a0c23ce8fe516d2c8d773
  - commits: 
    - 
      - sha: 2055b0fcf22f1c3543e38b60199f6882266d32a5
      - author: 
        - email: <EMAIL>
        - name: <NAME>
      - message: Create detectCapital.ts
      - distinct: True
      - url: https://api.github.com/repos/BeckhamL/leetcode/commits/2055b0fcf22f1c3543e38b60199f6882266d32a5
- public: True
- created_at: 2020-12-13T21:32:34Z<jupyter_text>You can also inspect the json data in an event, which are accessible as attributes:<jupyter_code>evts[0].type<jupyter_output><empty_output><jupyter_text>For example, here is the frequency of all `full_types` in the sample:<jupyter_code>x,y = zip(*Counter([o.full_type for o in evts]).most_common())
plt.figure(figsize=(8, 6))
plt.barh(x[::-1],y[::-1]);<jupyter_output><empty_output><jupyter_text>We can fetch public events in parallel with [GhApi.list_events_parallel](https://ghapi.fast.ai/event.htmlGhApi.list_events_parallel).  This allows you to fetch<jupyter_code>%time
api = GhApi()
evts = api.list_events_parallel()
len(evts)<jupyter_output>CPU times: user 1 µs, sys: 0 ns, total: 1 µs
Wall time: 3.1 µs
{'fraction_non_alphanumeric': 0.09623015873015874, 'fraction_numerical': 0.037037037037037035, 'mean_word_length': 4.844444444444444, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 32, 'https://': 21, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '589924', 'license': 'Apache-2.0', 'n_tokens_mistral': 2077, 'n_tokens_neox': 1874, 'n_words': 641, 'path': '_notebooks/2020-01-29-ghtop.ipynb', 'repo_name': 'dorianalden/fastpages', 'size': '6091'}