<commit_before>gist-like minimal server
========================

requirements:

* (R1) I want to open the main page and have the ability to add a new gist
* (R2) I want to open the main page and see all snippets
* (R3) I want to be able to search for snippets
* (R4) I want to integrate documentation in between the code snippets
* (R5) I want to have versioning on my snippets

R1 - The add page
-----------------

Need to add:

* a title
* some keywords
* (*opt*) force select the language
* (*opt*) path
* the snippet

```
     __________________________________________________________ 
    |X|________________________________________________________|
    |<|>|O| http://example.com/new___________________________ ||
    |   ___________________________________________________    |
    |  | Add a new snippet                      |_COMMIT_| |   |
    |  |                   ______________________________  |   |
    |  | Title:           |_My snippet___________________| |   |
    |  |                   ______________________________  |   |
    |  | Keywords:        |_C++,integers,addition________| |   |
    |  |                   ______________________________  |   |
    |  | Language:        |_<autodetect>_______________|v| |   |
    |  |                   ______________________________  |   |
    |  | Path:            |_/General C/Basic/____________| |   |
    |  |                                                   |   |
    |  | Code:                                             |   |
    |  |  _______________________________________________  |   |
    |  | |int main(int, char* argv[]) {                  | |   |
    |  | |    int x = 2 + 2;                             | |   |
    |  | |    return x;                                  | |   |
    |  | |}                                              | |   |
    |  | |                                               | |   |
    |  | |_______________________________________________| |   |
    |  |___________________________________________________|   |
    |__________________________________________________________|
```

R2 - The directory
------------------

Will contain a windows 1.0 type directory view. Clicking on a folder shall
expand its contents. Clicking on a listing shall open the listing.

The current path shall be displayed so that the user knows where he's at.

There shall be buttons to navigate to the parent directory or to the root directory.

There shall be the possibility of returning to manually input the path.

### R2.1 - The snippet listing

Essentially opens a read-only version of the add new snippet dialog with
the COMMIT button replaced with an edit button. When clicking the edit
button, the add snippet dialog is repurposed for the edition.

The listing dialog will make use of `highlight.js` for the code listing.

The listing dialog shall have the possibility of deleting the current listing.

R3 - The search engine
----------------------

The search functionality is split between two:

1. The internal snippet search
2. The documentation search

*2.* will be ignored for now as it's part of *R4* which we won't support
until later.

*1.* shall return results prioritised as follows:

* the snippet(s) whose title(s) match the query
* partial title matches
* full keyword hits
* partial keyword hits
* code matches

The search shall bear the posibility of excluding hits by path (i.e.
the hits need to be in or in a child of the specified path)

R4 - The google integration
---------------------------

**Deferred to V4** because of complexity.

R5 - The version control
------------------------

**Deferred to V2** because of it's a nice-to-have and not a core requirement.

XX - The main page
------------------

The main page will, by default, show the snippet directory with the
following additions:

* An "add new listing" button that goes to the add new listing dialog
  The path component shall be initialized to the currently navigated path.
* The search bar

XX - The database
-----------------

A listing is a tuple consisting of:

* a unique id (ID)
* the title (TITLE)
* the path (PATH)
* a list of keywords (KEYWORDS)
* a language (LANGUAGE)
* a blob (the code) (CODE)

We need to optimize the following queries:

* query by PATH (1)
* query by ID (2)
* query by title,keyword,code
  - title (3)
  - keyword (4)
  - code (5[^dagger])

[^dagger]: only if some advanced option is enabled.

### Schema

*as a side note: the multimarkdown implementation I use breaks if there's a blank line in the middle of it (for some reason)*

``` Entry: {
    ID: uint, primary key
    TITLE: string
    PATH: string
    LANGUAGE: string, nullable, default null
    CODE: blob
}
```

``` KeywordLink: {
    NAME: string, primary key
    EntryID: uint, primary key
}
```<commit_msg>update schema to include a foreign key<commit_after>gist-like minimal server
========================

requirements:

* (R1) I want to open the main page and have the ability to add a new gist
* (R2) I want to open the main page and see all snippets
* (R3) I want to be able to search for snippets
* (R4) I want to integrate documentation in between the code snippets
* (R5) I want to have versioning on my snippets

R1 - The add page
-----------------

Need to add:

* a title
* some keywords
* (*opt*) force select the language
* (*opt*) path
* the snippet

```
     __________________________________________________________ 
    |X|________________________________________________________|
    |<|>|O| http://example.com/new___________________________ ||
    |   ___________________________________________________    |
    |  | Add a new snippet                      |_COMMIT_| |   |
    |  |                   ______________________________  |   |
    |  | Title:           |_My snippet___________________| |   |
    |  |                   ______________________________  |   |
    |  | Keywords:        |_C++,integers,addition________| |   |
    |  |                   ______________________________  |   |
    |  | Language:        |_<autodetect>_______________|v| |   |
    |  |                   ______________________________  |   |
    |  | Path:            |_/General C/Basic/____________| |   |
    |  |                                                   |   |
    |  | Code:                                             |   |
    |  |  _______________________________________________  |   |
    |  | |int main(int, char* argv[]) {                  | |   |
    |  | |    int x = 2 + 2;                             | |   |
    |  | |    return x;                                  | |   |
    |  | |}                                              | |   |
    |  | |                                               | |   |
    |  | |_______________________________________________| |   |
    |  |___________________________________________________|   |
    |__________________________________________________________|
```

R2 - The directory
------------------

Will contain a windows 1.0 type directory view. Clicking on a folder shall
expand its contents. Clicking on a listing shall open the listing.

The current path shall be displayed so that the user knows where he's at.

There shall be buttons to navigate to the parent directory or to the root directory.

There shall be the possibility of returning to manually input the path.

### R2.1 - The snippet listing

Essentially opens a read-only version of the add new snippet dialog with
the COMMIT button replaced with an edit button. When clicking the edit
button, the add snippet dialog is repurposed for the edition.

The listing dialog will make use of `highlight.js` for the code listing.

The listing dialog shall have the possibility of deleting the current listing.

R3 - The search engine
----------------------

The search functionality is split between two:

1. The internal snippet search
2. The documentation search

*2.* will be ignored for now as it's part of *R4* which we won't support
until later.

*1.* shall return results prioritised as follows:

* the snippet(s) whose title(s) match the query
* partial title matches
* full keyword hits
* partial keyword hits
* code matches

The search shall bear the posibility of excluding hits by path (i.e.
the hits need to be in or in a child of the specified path)

R4 - The google integration
---------------------------

**Deferred to V4** because of complexity.

R5 - The version control
------------------------

**Deferred to V2** because of it's a nice-to-have and not a core requirement.

XX - The main page
------------------

The main page will, by default, show the snippet directory with the
following additions:

* An "add new listing" button that goes to the add new listing dialog
  The path component shall be initialized to the currently navigated path.
* The search bar

XX - The database
-----------------

A listing is a tuple consisting of:

* a unique id (ID)
* the title (TITLE)
* the path (PATH)
* a list of keywords (KEYWORDS)
* a language (LANGUAGE)
* a blob (the code) (CODE)

We need to optimize the following queries:

* query by PATH (1)
* query by ID (2)
* query by title,keyword,code
  - title (3)
  - keyword (4)
  - code (5[^dagger])

[^dagger]: only if some advanced option is enabled.

### Schema

*as a side note: the multimarkdown implementation I use breaks if there's a blank line in the middle of it (for some reason)*

``` Entry: {
    ID: uint, primary key
    TITLE: string
    PATH: string
    LANGUAGE: string, nullable, default null
    CODE: blob
}
```

``` KeywordLink: {
    NAME: string, primary key
    EntryID: uint, primary key, foreign key Entry.ID
}
```
{'fraction_non_alphanumeric': 0.1076811142292901, 'fraction_numerical': 0.0058206007691508155, 'mean_word_length': 2.1998669770535417, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 20, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5051072', 'n_tokens_mistral': 2497, 'n_tokens_neox': 2366, 'n_words': 1041}