<commit_before># Developer Documentation

## Debugging

Moonbase pipes all errors to `stderr`. These are not (yet) stored in any sort of log file, so if you want to capture/record error messages, I suggest using *nix pipes. Debug messages are not special in any respect, they are mostly just `print` statements thrown in at important breakpoints. TODO: add a proper output interface, eventually.

## Programming

The core of Moonbase was written by [<NAME>](https://github.com/jacksarick). This was his first project in Ruby, coming from a primarily functional background. His code reflects this. Very often, there are multiple callback statements chained together in a series. Lists, hashes, and the functions that support them are relied on heavily. Though it is one way to program an application, it is not the only way. If you feel more comfortable writing in a more Object-Oriented fashion, that is totally OK. The only catch is that it must be compatible. As long as a function works, and doesn't bother other functions, it is good to go. This means that global variables are a big no-no. Exactly how the functions work, though, is totally up to the person writing them.

## Writing Scripts

The `scripting` directory is language agnostic. This means you can write a script in whatever language you choose. In the original version of Moonbase, they were all Rust binaries. All that backend is doing is running the script with *nix execution. As long as the file is runnable (this means that compiled languages must be pre-compiled and scripts need shebangs), Moonbase can handle it no problem.

## Application Structure

General breakdown of how the application is structed. It's quite a bit more tangled than this, as a lot of things depend on each other and the like. I've left out `utility`, as it is used by all files.

```
start
├── http-server OR https-server
│          │
└── app <──┘
   │
   ├── backend
   │   ├── composer
   │   │   └── http-lib
   │   ├── http-lib
   │   └── USER SCRIPTS
   │
   └── frontend
       └── composer
           └── http-lib
```

The server setup can be confusing at first. `start` picks ONE server, then passes it to `app`. `app` requires a server to run, but it is really just calling one function from that server, so it doesn't really care which one it gets. So `start` calls the dependancy, but `app` is the one that actually puts it to use.

## Custom Servers
On a related note, Moonbase can theoretically run on a different server structure. `start` just calls a server with `new()` and `start()`. `app` just uses sockets. If you needed it to run over a different protocol, as long as it could handle sockets, Moonbase wouldn't notice. This is how it runs the same over HTTP and HTTPS, though I've never come across another protocol I'd want to implement. Note: this is all hypothetical. Just because you can doesn't mean you should.<commit_msg>Dependencies and spelling<commit_after># Developer Documentation

## Debugging

Moonbase pipes all errors to `stderr`. These are not (yet) stored in any sort of log file, so if you want to capture/record error messages, I suggest using *nix pipes. Debug messages are not special in any respect, they are mostly just `print` statements thrown in at important breakpoints. TODO: add a proper output interface, eventually.

## Programming

The core of Moonbase was written by [<NAME>](https://github.com/jacksarick). This was his first project in Ruby, coming from a primarily functional background. His code reflects this. Very often, there are multiple callback statements chained together in a series. Lists, hashes, and the functions that support them are relied on heavily. Though it is one way to program an application, it is not the only way. If you feel more comfortable writing in a more Object-Oriented fashion, that is totally OK. The only catch is that it must be compatible. As long as a function works, and doesn't bother other functions, it is good to go. This means that global variables are a big no-no. Exactly how the functions work, though, is totally up to the person writing them.

## Dependencies
Moonbase (for now) requires no gems. That means that it can be downloaded, and then run entirely offline. There are no updates from external sources, which is a blessing and a curse. As long as your ruby is up to date, Moonbase will run. It also means that everything in Moonbase is needed by Moonbase, no extra fluff from an unused package, and that if anything breaks, the exact cause of the breakage can be easily found. The downside is that it sometimes requires a lot of extra work.

## Writing Scripts

The `scripting` directory is language agnostic. This means you can write a script in whatever language you choose. In the original version of Moonbase, they were all Rust binaries. All that backend is doing is running the script with *nix execution. As long as the file is runnable (this means that compiled languages must be pre-compiled and scripts need shebangs), Moonbase can handle it no problem.

## Application Structure

General breakdown of how the application is structured. It's quite a bit more tangled than this, as a lot of things depend on each other and the like. I've left out `utility`, as it is used by all files.

```
start
├── http-server OR https-server
│          │
└── app <──┘
   │
   ├── backend
   │   ├── composer
   │   │   └── http-lib
   │   ├── http-lib
   │   └── USER SCRIPTS
   │
   └── frontend
       └── composer
           └── http-lib
```

The server setup can be confusing at first. `start` picks ONE server, then passes it to `app`. `app` requires a server to run, but it is really just calling one function from that server, so it doesn't really care which one it gets. So `start` calls the dependency, but `app` is the one that actually puts it to use.

## Custom Servers
On a related note, Moonbase can theoretically run on a different server structure. `start` just calls a server with `new()` and `start()`. `app` just uses sockets. If you needed it to run over a different protocol, as long as it could handle sockets, Moonbase wouldn't notice. This is how it runs the same over HTTP and HTTPS, though I've never come across another protocol I'd want to implement. Note: this is all hypothetical. Just because you can doesn't mean you should.
{'fraction_non_alphanumeric': 0.062391132224861444, 'fraction_numerical': 0.0, 'mean_word_length': 3.869699306090979, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '147812', 'n_tokens_mistral': 1682, 'n_tokens_neox': 1561, 'n_words': 1037}