<commit_before>vm-analyze
==========

This repository contains a set of scripts for tracing the virtual memory behavior of Linux applications, analyzing this trace data, and generating plots and tables that summarize the virtual memory behavior. To facilitate this analysis, this repository includes scripts for automatically executing a set of Linux applications; these automatic execution scripts may be more widely useful than the virtual memory analysis scripts.

This was my first substantial Python project, so the code may be a bit messy. Because it was developed as I learned the features and conventions of the language, in several places I have imported modules poorly, used inconsistent capitalization, have used tuples when I should have used objects, and so on; I will try to correct these things going forward.

Guide to source code:
* run_apps.py: main script to automatically execute the applications and capture traces of virtual memory behavior.
* generate_plots.py: main script to analyze the output from run_apps.py and generate plots and tables.
* analyze_trace.py: superseded by generate_plots.py.
* analyze/: modules used for analyzing trace data.
* app_scripts/: scripts that control the execution of each application.
* conf/: modules that control the execution of the top-level scripts.
    * applist.py: specifies which applications run_apps.py should execute.
    * PlotList.py: specifies which plots generate_plots.py should generate.
    * system_conf.py: specifies directories and other data that may differ across machines.
* plotting/: modules related to plot generation.
* trace/: modules for executing applications and capturing trace data.
* util/: various utility modules and scripts.

Before using the code in this repository, the following dependencies must be satisfied:
* The [pjh/pyutils](https://github.com/pjh/pyutils) repository must be checked out somewhere and a symlink to pyutils/pjh_utils.py must be created in the util subdirectory of this repository.
* To trace virtual memory activity, my patched Linux 3.9.4 kernel must be running: ...
* ...

So far, the code in this repository has only been run on Ubuntu 12.04 and 13.04 systems. The code is intended to be run with Python 3.3 or greater; some features that are not present in 3.2 are used.
<commit_msg>Update README: pointer to linux-stable repo<commit_after>vm-analyze
==========

This repository contains a set of scripts for tracing the virtual memory behavior of Linux applications, analyzing this trace data, and generating plots and tables that summarize the virtual memory behavior. To facilitate this analysis, this repository includes scripts for automatically executing a set of Linux applications; these automatic execution scripts may be more widely useful than the virtual memory analysis scripts.

This was my first substantial Python project, so the code may be a bit messy. Because it was developed as I learned the features and conventions of the language, in several places I have imported modules poorly, used inconsistent capitalization, used tuples when I should have used objects, used tabs instead of spaces, and so on; I will try to correct these things going forward.

Guide to source code:
* run_apps.py: main script to automatically execute the applications and capture traces of virtual memory behavior.
* generate_plots.py: main script to analyze the output from run_apps.py and generate plots and tables.
* analyze_trace.py: superseded by generate_plots.py.
* analyze/: modules used for analyzing trace data.
* app_scripts/: scripts that control the execution of each application.
* conf/: modules that control the execution of the top-level scripts.
    * applist.py: specifies which applications run_apps.py should execute.
    * PlotList.py: specifies which plots generate_plots.py should generate.
    * system_conf.py: specifies directories and other data that may differ across machines.
* plotting/: modules related to plot generation.
* trace/: modules for executing applications and capturing trace data.
* util/: various utility modules and scripts.

Before using the code in this repository, the following dependencies must be satisfied:
* The [pjh/pyutils](https://github.com/pjh/pyutils) repository must be checked out somewhere and a symlink to pyutils/pjh_utils.py must be created in the util subdirectory of this repository.
* To trace virtual memory activity, my patched Linux 3.9.4 kernel must be running: checkout the vm-analyze branch of [pjh/linux-stable](https://github.com/pjh/linux-stable/tree/vm-analyze), build it and install it.

So far, the code in this repository has only been run on Ubuntu 12.04 and 13.04 systems. The code is intended to be run with Python 3.3 or greater; some features that are not present in 3.2 are used.

{'fraction_non_alphanumeric': 0.052774301209845643, 'fraction_numerical': 0.006257822277847309, 'mean_word_length': 4.627934272300469, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3032810', 'n_tokens_mistral': 1162, 'n_tokens_neox': 1094, 'n_words': 662}