<jupyter_start><jupyter_text>Building your first performance monitor With <NAME>Data Scientist at CyberaSupport DevOps Engineer, iReceptor project PyLadies Berlin April 6 2021   Introductions   What does the performance monitor I will show you look like? https://github.com/lfunderburk/Performance-Monitor  How I do it Part I: Perform queries on a regular schedule1. Identify an API (this talk will focus on using REST API's, but can be extended to other kinds of API's)2. Write Python script `perform_queries.py` to perform a query, track how long it takes and save results in a CSV file 3. Use Bash to create a wrapper script for `perform_queries.py` - `automate_queries.sh` (advantage when you want to perform queries on multiple API's)4. Use Crontab to set up a regular schedule for `automate_queries.sh`  How I do it Part II: Use Python + Bash + GitHub + GitHub pages to visualize and share results5. Write a Python script to parse the CSV results and generate plots `plot_results.py`6. Create a new GitHub repository, clone the repository on your local computer7. Write a wrapper script for `plot_results.py`, - `publish_results.sh` that pushes plots generated as HTML files to repository9. Activate GitHub's pages10. Add Bash plotting script to Crontab on a regular schedule  Why I do it this wayThis is a process I have been curating over the course of two years through experience supporting DevOps tasks. Team wanted to identify timeouts in their database and prevent them. The performance monitor I built for them allows them to see when query times are close to the cutoff time, and that rebooting service cuts time in half. This performance monitor won't assess underlying causes for timeouts, and should only be used as a part of a systematic approach to assess how long different queries take.<jupyter_code># Example
%run -i performance_monitor.py 
# An example output file 
sample_file = "./logs/_pT_2021-03-10_20-24-35_Query_Times_carbonintensity_intensity.csv"
pd.read_csv(sample_file)<jupyter_output><empty_output><jupyter_text>What about Bash?There are a few things we need to consider:1. Depending on how often you want the results, you might need to set up a virtual machine (I used Oracle VM VirtualBox in Windows to set up a Linux VM, on 24/7)2. You will also need to configure crontab to ensure you provide the path to Bash and other environment variables, set up a ssh key so crontab can push to GitHub without needing your credentials. 3. Setting up a virtualenv for Python ensures same dependencies and minimizes errors. 4. Wrapper scripts are not necessary, but can greatly increase automation, especially when you scale up to several REST API's, and multiple queries within each of them.   Configuring CrontabOn Linux, type in the command line:    crontab -e    Get a few environment basics right before anything else:    * * * * * env > /tmp/env.output    The above means that every minute, every hour, every day this command will run. You only need it once, you can comment it out afterwards. `CTRL + SHIFT + O` and `ENTER` to save, then `CTRL + SHIFT + X` to exit. Read the content via `cat /tmp/env.output`.  Configuring CrontabTell it where to find Bash    SHELL=/bin/bash Tell it the path to working directory    PWD=/home/lauragf YOUR USERNAME INSTEADTell it any other paths your scripts depend on. Note Anaconda for Python    PATH=/usr/local/sbin:/usr/local/bin:    /usr/sbin:/user/bin:/bin:/home/lauragf/anaconda3/bin  Setting up your scriptLet's go take a look at our Bash scripts....Once they are ready to go, ensure they can be executed (I like giving everyone executing permissions, but this depends entirely on you)...    chmod a+x     Test the script until you are sure it does what you want    bash uk_emissions.sh  Regular schedule through crontab     0 */2 * * * /PATH/TO/BASH/SCRIPT/uk_emissions.sh >     /PATH/TO/BASH/SCRIPT/uk_emissions.out 2>&1     The above means that the script is to be executed during minute 0, every 2 hours, every day. File descriptor 1 is the standard output (stdout).File descriptor 2 is the standard error (stderr)` 2>&1 ` means "redirect the stderr to the same place we are redirecting the stdout"  Plotting results<jupyter_code># Need to visualize
%run -i plot_performance.py "./logs/" "intensity"<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.057116322266078476, 'fraction_numerical': 0.010912468075226375, 'mean_word_length': 4.338289962825279, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 14, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '103988', 'license': 'MIT', 'n_tokens_mistral': 1206, 'n_tokens_neox': 1119, 'n_words': 618, 'path': 'scripts/Slideshow.ipynb', 'repo_name': 'moryama171/Performance-Monitor', 'size': '4320'}