<commit_before>[![Build Status](https://travis-ci.org/kingsdigitallab/dprr-django.svg?branch=develop)](https://travis-ci.org/kingsdigitallab/dprr-django)

### Requirements

Python/Django requirements can be found in the `requirements.txt` file.

### Project Structure 

The main application is `promrep`. 

#### Faceting 

Given that the desired output is a list of Persons, we need to Group the results by person. To do that we use a common field to all Assertion index objects, the `person_id`. We’re also using different indexes to reflect the different Assertion types. The queryset is defined in the `promrep/forms.py` file: 

```python
queryset = GroupedSearchQuerySet().models(
    PostAssertion,
    StatusAssertion,
    RelationshipAssertion).group_by('person_id')
```

##### Grouping Facets

Given that we wish to search on the Assertions but list Persons we opted to use the SOLR Grouping/Field collapsing feature: 
* See http://wiki.apache.org/solr/FieldCollapsing for the Solr feature

This is currently unsupported by Haystack. We're using an experimental backend for Haystack 2.0 adapted from the following code:
* See https://gist.github.com/3750774 for the current version of this code

The backend code can be found on `promrep/solr_backends/solr_backend_field_collapsing`. 

##### Adding a new facet

Add the new facet to the `search_index.py` file, following Haystack’s documentation. 

There are 4 types of index classes: 
* GenericIndex 
* PostAssertionIndex
* StatusAssertionIndex
* RelationshipAssertionIndex

After adding the new field to the search_indexes file (as well as the prepare method, if needed), you need to:
* rebuild the SOLR schema xml file (overwriting the `schema.xml` in the solr config folder)
* restart Tomcat
* rebuild the index (using the management command `python manage.py rebuild_index`)

To add the facet to the frontendend, add the facet field to either the `facet_fields` or the `autocomplete facets` lists in the `PromrepFacetedSearchView` (`promrep/views.py`). These lists will be used by the `.facet()` function. These are also used in the template to control how to display the facet.

##### Hierarchical lists

Some authority lists are hierarchical -- see the Office and Province models, for instance. These were developed using the `django-mptt` library: 

https://github.com/django-mptt/django-mptt 

In order to display them correctly all hierarchical model objects need to exist in the template. This is currently done in the `views.py` file, where each model is added to the context (`get_context_data`). In the template we use MPTT `recursetree` tag to recurse the tree and display each node.  

#### Scripts and data 

There are various scripts in the `promrep/scripts` folder. Most were written to be used only once to load data in the staging database. The data itself (usually `csv` files created by the project partners) is not stored on the server but, for provenance reasons, all scripts refer to the name/version of the data file thet was loaded into the database.

<commit_msg>added info about the management scripts to the Readme file<commit_after>[![Build Status](https://travis-ci.org/kingsdigitallab/dprr-django.svg?branch=develop)](https://travis-ci.org/kingsdigitallab/dprr-django)

### Requirements

Python/Django requirements can be found in the `requirements.txt` file.

### Project Structure 

The main application is `promrep`. 

#### Faceting 

Given that the desired output is a list of Persons, we need to Group the results by person. To do that we use a common field to all Assertion index objects, the `person_id`. We’re also using different indexes to reflect the different Assertion types. The queryset is defined in the `promrep/forms.py` file: 

```python
queryset = GroupedSearchQuerySet().models(
    PostAssertion,
    StatusAssertion,
    RelationshipAssertion).group_by('person_id')
```

##### Grouping Facets

Given that we wish to search on the Assertions but list Persons we opted to use the SOLR Grouping/Field collapsing feature: 
* See http://wiki.apache.org/solr/FieldCollapsing for the Solr feature

This is currently unsupported by Haystack. We're using an experimental backend for Haystack 2.0 adapted from the following code:
* See https://gist.github.com/3750774 for the current version of this code

The backend code can be found on `promrep/solr_backends/solr_backend_field_collapsing`. 

##### Adding a new facet

Add the new facet to the `search_index.py` file, following Haystack’s documentation. 

There are 4 types of index classes: 
* GenericIndex 
* PostAssertionIndex
* StatusAssertionIndex
* RelationshipAssertionIndex

After adding the new field to the search_indexes file (as well as the prepare method, if needed), you need to:
* rebuild the SOLR schema xml file (overwriting the `schema.xml` in the solr config folder)
* restart Tomcat
* rebuild the index (using the management command `python manage.py rebuild_index`)

To add the facet to the frontendend, add the facet field to either the `facet_fields` or the `autocomplete facets` lists in the `PromrepFacetedSearchView` (`promrep/views.py`). These lists will be used by the `.facet()` function. These are also used in the template to control how to display the facet.

##### Hierarchical lists

Some authority lists are hierarchical -- see the Office and Province models, for instance. These were developed using the `django-mptt` library: 

https://github.com/django-mptt/django-mptt 

In order to display them correctly all hierarchical model objects need to exist in the template. This is currently done in the `views.py` file, where each model is added to the context (`get_context_data`). In the template we use MPTT `recursetree` tag to recurse the tree and display each node.  

#### Scripts and data 

There are various scripts in the `promrep/scripts` folder. Most were written to be used only once to load data in the staging database. The data itself (usually `csv` files created by the project partners) is not stored on the server but, for provenance reasons, all scripts refer to the name/version of the data file that was loaded into the database.

##### Management Scripts

* `add_highest_office`: script that adds the `highest_office` field to all persons on the database. The script checks if the `highest_office_edited` field is False and, if so, overwrites it.

{'fraction_non_alphanumeric': 0.07120402389185791, 'fraction_numerical': 0.0031436655139893115, 'mean_word_length': 4.68125, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '34555', 'n_tokens_mistral': 1819, 'n_tokens_neox': 1684, 'n_words': 832}