<commit_before># idear
[![][jetbrains-team-svg]][jetbrains-team-page]
[![][travis-status-svg]][travis-build-status]
[![][teamcity-status-svg]][teamcity-build-status]
[![][plugin-repo-svg]][plugin-repo-page]
[![][plugin-download-svg]][plugin-repo-page]

A general purpose [voice user interface](https://en.wikipedia.org/wiki/Voice_user_interface) for the IntelliJ Platform, inspired by [<NAME>](https://www.youtube.com/watch?v=8SkdfdXWYaI). Possible use cases: visually impaired and [RSI](https://en.wikipedia.org/wiki/Repetitive_strain_injury) users. For background information, check out [this presentation](https://speakerdeck.com/breandan/programming-java-by-voice).

## Speech Recognition

ASR is supported by [CMU Sphinx](https://github.com/cmusphinx/sphinx4/) and [Amazon Lex](https://aws.amazon.com/lex/). All recognition is offline by default.

### Speech-to-Text

Whether Lex manages to resolve and fulfill an intent or not, it will still return the recognised utterance in text ([unless it did not hear anything at all](https://github.com/OpenASR/idear/issues/40)).
[`LexASR`](https://github.com/OpenASR/idear/blob/master/src/main/java/org/openasr/idear/asr/awslex/LexASR.kt) and
[`CMUSphinxASR`](https://github.com/OpenASR/idear/blob/master/src/main/java/org/openasr/idear/asr/cmusphinx/CMUSphinxASR.kt)
provide a method `waitForUtterance()` which blocks until the speech to text service returns a string. 

### NLP - Text to Action

If Lex _does_ manage to resolve and fulfill (to the point where it delegates to client-side fulfillment) an intent by 
invoking a Lamba function then [`LexRecognizer`](https://github.com/OpenASR/idear/blob/master/src/main/java/org/openasr/idear/asr/awslex/LexRecogniser.kt)
notifies a `NlpResultListener` that the request has been fulfilled or failed etc.

[`NlpProvider`](https://github.com/OpenASR/idear/blob/master/src/main/java/org/openasr/idear/nlp/NlpProvider.kt)
defines a method `processUtterance()` which takes a string utterance and context. 
[`LexNlp`](https://github.com/OpenASR/idear/blob/master/src/main/java/org/openasr/idear/nlp/lex/LexNlp.kt)
implements `NlpProvider` and notifies the `NlpResultListener`. 

## Text-to-Speech

TTS is supported by [MaryTTS](https://github.com/marytts/marytts) and [Amazon Polly](https://aws.amazon.com/polly/). Speech synthesis is offline by default. 

## Roadmap

Idear is currently a work in progress. These are some of the features we have implemented and are currently working on:

### Activation

- [ ] User presses button or activates voice control by saying something, “Okay __, help me.”
- [ ] “Hello <system user>, welcome to the handsfree audio development interface for IntelliJ IDEA.”
- [ ] “There are a number of commands you can use, for example ‘Open settings’, ‘Find action’, ‘Open file’...”

### Visually Impaired Mode

- [ ] Action reader. When user enables a flag, any selecting menu options or actions read back to user. 
- [ ] Status updates. User says, “Run application”. Plugin responds, “building project”, “compiling application”, “running project”.
- [ ] Text selection. Plugin reads back selected region (rapidly).
- [X] User says, "Where am I?". Plugin responds, "You are inside method X, on line Y".

### Interactive Features

- [ ] User says, “open Analyze”. Plugin responds, “Would you like to ‘Inspect Code’, ‘Code Cleanup’...”
- [ ] User says, “open tip of the day”. Plugin responds, “Did you know that... <tip of the day contents>”<commit_msg>added SST & TTS tasks to README<commit_after># idear
[![][jetbrains-team-svg]][jetbrains-team-page]
[![][travis-status-svg]][travis-build-status]
[![][teamcity-status-svg]][teamcity-build-status]
[![][plugin-repo-svg]][plugin-repo-page]
[![][plugin-download-svg]][plugin-repo-page]

A general purpose [voice user interface](https://en.wikipedia.org/wiki/Voice_user_interface) for the IntelliJ Platform, inspired by [<NAME>](https://www.youtube.com/watch?v=8SkdfdXWYaI). Possible use cases: visually impaired and [RSI](https://en.wikipedia.org/wiki/Repetitive_strain_injury) users. For background information, check out [this presentation](https://speakerdeck.com/breandan/programming-java-by-voice).

## Speech Recognition

ASR is supported by [CMU Sphinx](https://github.com/cmusphinx/sphinx4/) and [Amazon Lex](https://aws.amazon.com/lex/). All recognition is offline by default.

### Speech-to-Text

Whether Lex manages to resolve and fulfill an intent or not, it will still return the recognised utterance in text ([unless it did not hear anything at all](https://github.com/OpenASR/idear/issues/40)).
[`LexASR`](https://github.com/OpenASR/idear/blob/master/src/main/java/org/openasr/idear/asr/awslex/LexASR.kt) and
[`CMUSphinxASR`](https://github.com/OpenASR/idear/blob/master/src/main/java/org/openasr/idear/asr/cmusphinx/CMUSphinxASR.kt)
provide a method `waitForUtterance()` which blocks until the speech to text service returns a string.

If you'd prefer [Mozilla's DeepSpeech](https://github.com/mozilla/deepspeech), or [Picovoice](https://picovoice.ai/#offline-voice-ai), feel free to [help out](https://github.com/OpenASR/idear/issues/52). 

### NLP - Text to Action

If Lex _does_ manage to resolve and fulfill (to the point where it delegates to client-side fulfillment) an intent by 
invoking a Lamba function then [`LexRecognizer`](https://github.com/OpenASR/idear/blob/master/src/main/java/org/openasr/idear/asr/awslex/LexRecogniser.kt)
notifies a `NlpResultListener` that the request has been fulfilled or failed etc.

[`NlpProvider`](https://github.com/OpenASR/idear/blob/master/src/main/java/org/openasr/idear/nlp/NlpProvider.kt)
defines a method `processUtterance()` which takes a string utterance and context. 
[`LexNlp`](https://github.com/OpenASR/idear/blob/master/src/main/java/org/openasr/idear/nlp/lex/LexNlp.kt)
implements `NlpProvider` and notifies the `NlpResultListener`. 

## Text-to-Speech

TTS is supported by [MaryTTS](https://github.com/marytts/marytts) and [Amazon Polly](https://aws.amazon.com/polly/). Speech synthesis is offline by default. 

If you'd prefer [Mozilla's deep-learning TTS](https://github.com/mozilla/tts), feel free to [help out](https://github.com/OpenASR/idear/issues/53). 

## Roadmap

Idear is currently a work in progress. These are some of the features we have implemented and are currently working on:

### Activation

- [ ] User presses button or activates voice control by saying something, “Okay __, help me.”
- [ ] “Hello <system user>, welcome to the handsfree audio development interface for IntelliJ IDEA.”
- [ ] “There are a number of commands you can use, for example ‘Open settings’, ‘Find action’, ‘Open file’...”

### Visually Impaired Mode

- [ ] Action reader. When user enables a flag, any selecting menu options or actions read back to user. 
- [ ] Status updates. User says, “Run application”. Plugin responds, “building project”, “compiling application”, “running project”.
- [ ] Text selection. Plugin reads back selected region (rapidly).
- [X] User says, "Where am I?". Plugin responds, "You are inside method X, on line Y".

### Interactive Features

- [ ] User says, “open Analyze”. Plugin responds, “Would you like to ‘Inspect Code’, ‘Code Cleanup’...”
- [ ] User says, “open tip of the day”. Plugin responds, “Did you know that... <tip of the day contents>”
{'fraction_non_alphanumeric': 0.13653136531365315, 'fraction_numerical': 0.0016400164001640015, 'mean_word_length': 4.78041074249605, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 33, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1706465', 'n_tokens_mistral': 2445, 'n_tokens_neox': 2306, 'n_words': 693}