<commit_before># chatbot
a naive chatbot that sometimes misdiagnoses.

This retrieval-based prototype bot uses publicly available health information and a set of NLP techniques to generate indicative diagnosis and NHS leaflet for that diagnosis. The result is not to be treated as medical advice.

![alt text](screenshots/example_cold.png "example common cold")

## Installation
Run `./install.sh`. This will create a virtual environment and install all components required within.

## Running chatbot
Within virtualenv venv (`source venv/bin/activate`), you can launch separate service:

`python3 -m chatbot.services.ask`

`python3 -m chatbot.services.symptoms`

For the first time running either service, it will take longer than usual as it needs to download, process and cache web data. Afterwards, it takes around 5 minutes to launch `ask` and instantly for `symptoms`.

## API endpoints
Following endpoints are available to consume:

`POST /chatbot/api/v1/ask`

It accepts payload as simple as `{"questions": "your questions or description of symptoms"}`


`GET /chatbot/api/v1/symptoms`
`GET /chatbot/api/v1/symptoms/<string:symptom_name>`

To list all leaflets or leaflet for the chosen symptom.

## Further ideas and development
Using vector representation of words (e.g. Word2Vec) rather than sparse matrix to extract proximity of semantic meanings in embedded space;

Explore LSTM as text classification alternative to Naive Bayes
<commit_msg>amend README<commit_after># chatbot
a naive chatbot that sometimes misdiagnoses.

This retrieval-based prototype bot uses publicly available health information and a set of NLP techniques to generate indicative diagnosis and NHS leaflet for that diagnosis. The result is not to be treated as medical advice.

![alt text](screenshots/example_cold.png "example common cold")

## Installation
Run `./install.sh`. This will create a virtual environment and install all components required within.

## Running chatbot
Within virtualenv venv (`source venv/bin/activate`), you can launch separate service:

`python3 -m chatbot.services.ask`

`python3 -m chatbot.services.symptoms`

For the first time running either service, it will take longer than usual as it needs to download, process and cache web data. Afterwards, it takes around 5 minutes to launch `ask` and instantly for `symptoms`.

## API endpoints
Following endpoints are available to consume:

`POST /chatbot/api/v1/ask`

It accepts payload as simple as `{"questions": "your questions or description of symptoms"}`


`GET /chatbot/api/v1/symptoms`
`GET /chatbot/api/v1/symptoms/<string:symptom_name>`

To list all leaflets or leaflet for the chosen symptom.

## Further development ideas
Using word vector rather than sparse matrix to extract semantic proximity in embedded space;

Explore Recurrent Neural Network (e.g. LSTM) to move from retrieval-based model to generative;

Explore supervised Latent Dirichlet Allocation, ConvNet as text classification alternative to Naive Bayes.

{'fraction_non_alphanumeric': 0.06608811748998665, 'fraction_numerical': 0.0043391188251001335, 'mean_word_length': 4.982035928143713, 'pattern_counts': {'":': 2, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '549394', 'n_tokens_mistral': 828, 'n_tokens_neox': 774, 'n_words': 385}