<commit_before>Scripts to load test RSR, that just probably work on my machine.


Files:

1. run-load-test.sh: to replay the production load against a server
1. compare-one-page.sh: compares output and performance of two servers for a given url
1. compare-all-pages.sh: compares a bunch of urls between two rsr servers, using the previous script
    1. Right now it compares rsr prod against rsr1 training environment
    1. Get the last prod logs: `gcloud logging read "logName=projects/akvo-lumen/logs/rsr-nginx AND resource.labels.cluster_name="production" AND NOT "Prometheus" AND NOT "robots" AND \" 200 \" AND NOT \"/my-rsr/main\" AND NOT \"/my-rsr/styles\"" --limit 2000 --format="list" | grep textPayload | cut -f2- -d: | cut -b2- > prod.logs.txt`
    1. Start the rsr1 training env (or modify the compara-one-page.sh script)
    1. Run the script
    1. Urls with different content will be written to `different.txt`
        1. You can manually check what is the difference by using the `compare-one-page.sh $url diff`
    1. Check rsr1 logs to see if there are new errors
    1. Script does not compare HTTP error codes, which it should
1. debug-performance/90-finish.conf: Log configuration that spits the DB requests made by Django.
1. debug-performance/request.sh: Makes a request against a local docker-compose env and stores the DB calls in a file
1. debug-performance/stats-db-calls.sh: basic stats for the previous DB calls file
1. debug-performance/diff-stats.sh: basic diff DB stats from two DB calls file
1. debug-performance/advanced-stats.clj: A more advance Clojure script that shows where the time is spent, not just the number of queries

<commit_msg>[#2208] Modify the load-test README to fetch the correct logs<commit_after>Scripts to load test RSR, that just probably work on my machine.


Files:

1. run-load-test.sh: to replay the production load against a server
1. compare-one-page.sh: compares output and performance of two servers for a given url
1. compare-all-pages.sh: compares a bunch of urls between two rsr servers, using the previous script
    1. Right now it compares rsr prod against rsr1 training environment
    1. Get the last prod logs: `gcloud logging read "logName=projects/akvo-lumen/logs/stdout AND resource.labels.cluster_name="production" AND resource.labels.container_name="rsr-nginx" AND NOT "Prometheus" AND NOT "robots" AND \" 200 \" AND NOT \"/my-rsr/main\" AND NOT \"/my-rsr/styles\"" --limit 2000 --format="list" | grep textPayload | cut -f2- -d: | cut -b2- > prod.logs.txt`
    1. Start the rsr1 training env (or modify the compara-one-page.sh script)
    1. Run the script
    1. Urls with different content will be written to `different.txt`
        1. You can manually check what is the difference by using the `compare-one-page.sh $url diff`
    1. Check rsr1 logs to see if there are new errors
    1. Script does not compare HTTP error codes, which it should
1. debug-performance/90-finish.conf: Log configuration that spits the DB requests made by Django.
1. debug-performance/request.sh: Makes a request against a local docker-compose env and stores the DB calls in a file
1. debug-performance/stats-db-calls.sh: basic stats for the previous DB calls file
1. debug-performance/diff-stats.sh: basic diff DB stats from two DB calls file
1. debug-performance/advanced-stats.clj: A more advance Clojure script that shows where the time is spent, not just the number of queries

{'fraction_non_alphanumeric': 0.08103398199244845, 'fraction_numerical': 0.01858844031367993, 'mean_word_length': 3.654054054054054, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3921835', 'n_tokens_mistral': 1038, 'n_tokens_neox': 982, 'n_words': 475}