<commit_before>{{cookiecutter.project_slug}}
=============================

{{cookiecutter.project_short_description}}

Project Organization
--------------------

    .
    |-- AUTHORS.md              Author information.
    |-- LICENSE                 [Optional] license.
    |-- Makefile                Provide targets for development automatation.
    |-- README.md               This file.
    |-- config                  Config files, not under version control.
    |   |-- make.help
    |   |-- make.sync
    |   `-- ...
    |-- notes                   Notes and documentations.
    |-- lib                     Extenal code/binary files.
    |-- data                    Data folders.
    |   |-- current             Symlink, points to the latest folder.
    |   |-- data_yyyy_mm_dd     Data files with date stamp.
    |   |-- ...
    |   `-- raw                 Folder with raw data.
    |-- log                     Log files.
    |   |-- current
    |   |-- log_yyyy_mm_dd
    |   `-- ...
    |-- reports                 Reports, polished figures for publication.
    |   |-- current
    |   |-- reports_yyyy_mm_dd
    |   `-- ...
    |-- results                 Results.
    |   |-- current
    |   |-- results_yyyy_mm_dd
    |   `-- ...
    `-- src                     Scripts/software created for this project.

Instructions
------------

A `Makefile` file is provided for development automation. The targets are
in two files: `config/make.help` has the basic utilities, `config/make.sync`
contains targets to sync with remote server (you need to fill in the server
info).

When the project is created, a git repo is initialized, but not committed.

The first step is `make stamp` to create date-stamped folders in `reports`,
`results`, `data` and `log` directories. There will be a `current` symlink
points to these folders, so your code should only consider `current`.

`data` directory has a `raw` folder, contains data from outside the project,
they should not be touched, ever.

Some intermediate data may look like results also, that's OK. As long as you
know where to find them.<commit_msg>[README]: Update folder structure in README.md<commit_after>{{cookiecutter.project_slug}}
=============================

{{cookiecutter.project_short_description}}

Project Organization
--------------------

    .
    |-- AUTHORS.md              Author information.
    |-- LICENSE                 [Optional] license.
    |-- Makefile                Provide targets for development automatation.
    |-- README.md               This file.
    |-- config                  Config files, not under version control.
    |   |-- make.help
    |   |-- make.sync
    |   `-- ...
    |-- lib                     Extenal code/binary files.
    |-- data                    Data folders.
    |   `-- raw                 Folder with raw data.
    |-- log                     Log files.
    |   |-- current
    |   |-- log_yyyy_mm_dd
    |   `-- ...
    |-- reports                 Reports, notes, figures for publication.
    |   |-- current
    |   |-- reports_yyyy_mm_dd
    |   `-- ...
    |-- results                 Results.
    |   |-- current
    |   |-- results_yyyy_mm_dd
    |   `-- ...
    `-- src                     Scripts/software created for this project.

Rationale and Assumptions
-------------------------

1. Data state should only depend on analytic scripts.
1. Storage is relatively cheap. Because no-reproducible method is perfect, we
   inevitably creaet some artifacts during result presentation. It is highly
   desirable to make a record of these results so that we can trace the history
   of the project.
1. A scripting language, such as python and R, is used.

Instructions
------------

A `Makefile` file is provided for development automation. The targets are
in two files: `config/make.help` has the basic utilities, `config/make.sync`
contains targets to sync with remote server (you need to fill in the server
info).

When the project is created, a git repo is initialized, but not committed.

The first step is `make stamp` to create date-stamped folders in `reports`,
`results` and `log` directories. There will be a `current` symlink points to
these folders, so your code should only consider `current`.

`data` directory has a `raw` folder, contains data from outside the project.
They should not be touched, ever.

Intermediate data should be kept in `data` and not date-stamped. This practice
1) saves storage space and 2) avoids unnecessary common data re-building.
{'fraction_non_alphanumeric': 0.12327854286983563, 'fraction_numerical': 0.0011106175033318525, 'mean_word_length': 1.8214285714285714, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4655761', 'n_tokens_mistral': 1209, 'n_tokens_neox': 1159, 'n_words': 452}