<commit_before>regex4dummies: A Simple Way To Find Patterns
============================================

.. image:: https://travis-ci.org/DarkmatterVale/regex4dummies.svg?branch=master
    :target: https://travis-ci.org/DarkmatterVale/regex4dummies

Simple pattern finding in strings and natural language processing. Checkout regex4dummies' website at https://darkmattervale.github.io/regex4dummies/


Features
----------

- Automatic pattern detection ( semantic and literal )
- Multiple parsers ( implementations of nltk, pattern, and nlpnet )
- Keyword searching to find specific phrases within text
- Topic analysis & Important information extraction
- Tokenizer and sentence dependency identifier
- Phrase extraction ( noun, verb, prepositional )


Roadmap
----------

Some features I plan to implement in the future:

- Machine Learning. This will allow the parsers to learn multiple grammatical “styles” and be able to successfully parse a much wider selection of strings
- Additional parsers
- Phrase extractor. This can include verb phrases, noun phrases, prepositional phrases, etc
- String comparison function. This will return a value that will determine how similar two strings are

If you have feature requests, feel free to add an issue to the Github issue tracker. All contributions and requests are appreciated!


Usage
-------

regex4dummies is very easy to use. Simply import the library, get some strings, and compare them!

.. code-block:: Python

  from regex4dummies import regex4dummies
  from regex4dummies import Toolkit

  # Creating strings
  strings = [ "This is the first test string.", "This is the second test string." ]

  regex = regex4dummies()

  # Identifying literal patterns in strings
  print regex.compare_strings( parser='default', pattern_detection="literal", text=strings )

  # Identifying semantic patterns in strings using the nltk parser
  print regex.compare_strings( parser='nltk', pattern_detection="semantic", text=strings )

Above is regex4dummies in its simplest form. It allows for additional features as well, including:

.. code-block:: Python

  # Display the version of regex4dummies you are using
  print regex.__version__

  # To use the other parsers, replace the above line of code with either of the following:
  # print regex.compare_strings( parser='pattern', pattern_detection="semantic", text=strings )
  # print regex.compare_strings( parser='nlpnet', pattern_detection="semantic", text=strings )

  # To call all of the parsers, replace the above line of code with the following:
  # print regex.compare_strings( parser='', pattern_detection="semantic", text=strings )

  # To get the topics of the strings, call the get_topics function
  print regex.get_topics( text=strings )

  # Printing pattern information
  pattern_information = regex.get_sentence_information()
    for objects in pattern_information:
        print "[ Pattern ]             : " + objects.pattern
        print "[ Subject ]             : " + objects.subject
        print "[ Verb ]                : " + objects.verb
        print "[ Object ]              : " + objects.object[0]
        print "[ Prep Phrases ]        : " + str( objects.prepositional_phrases )
        print "[ Reliability Score ]   : " + str( objects.reliability_score )
        print "[ Applicability Score ] : " + str( objects.applicability_score )
        print ""

A newly released set of features include a tokenizer and dependency finder function. To use them, simply give a string as the first parameter, and the name of the parser you would like to use as the second for both functions.

.. code-block:: Python

  # Testing the toolkit functions
  tool_tester = Toolkit()

  # Testing the tokenizer functions
  print tool_tester.tokenize( text="This is a test string.", parser="" )

  # Testing the dependency functions
  print tool_tester.find_dependencies( text="This is a test string.", parser="pattern" )

Other features included are demonstrated below.

.. code-block:: Python

  # Testing the information extraction functions
  regex.extract_important_information( text=[ "This is a test string." ] )

  # Testing the ability to extract phrases
  print "Noun Phrases: " + str( tool_tester.extract_noun_phrases( text="This is a test string." ) )
  print "Verb Phrases(Pattern): " + str( tool_tester.extract_verb_phrases( text="This is a test string.", parser="pattern" ) )
  print "Verb Phrases(Nlpnet): " + str( tool_tester.extract_verb_phrases( text="This is a test string.", parser="nlpnet" ) )
  print "Prepositional Phrases: " + str( tool_tester.extract_prepositional_phrases( text="This is a test string in the house." ) )

<commit_msg>Last minute updates to DESCRIPTION<commit_after>regex4dummies: A Simple Way To Find Patterns
============================================

.. image:: https://travis-ci.org/DarkmatterVale/regex4dummies.svg?branch=master
    :target: https://travis-ci.org/DarkmatterVale/regex4dummies

Simple pattern finding in strings and natural language processing. Checkout regex4dummies' website at https://darkmattervale.github.io/regex4dummies/


Features
----------

- Automatic pattern detection ( semantic and literal )
- Multiple parsers ( implementations of nltk, pattern, and nlpnet )
- Keyword searching to find specific phrases within text
- Topic analysis & Important information extraction
- Tokenizer and sentence dependency identifier
- Phrase extraction ( noun, verb, prepositional )
- String comparison


Roadmap
----------

Some features I plan to implement in the future:

- Machine Learning. This will allow the parsers to learn multiple grammatical “styles” and be able to successfully parse a much wider selection of strings
- Additional parsers

If you have feature requests, feel free to add an issue to the Github issue tracker. All contributions and requests are appreciated!


Usage
-------

regex4dummies is very easy to use. Simply import the library, get some strings, and compare them!

.. code-block:: Python

  from regex4dummies import regex4dummies
  from regex4dummies import Toolkit

  # Creating strings
  strings = [ "This is the first test string.", "This is the second test string." ]

  regex = regex4dummies()

  # Identifying literal patterns in strings
  print regex.compare_strings( parser='default', pattern_detection="literal", text=strings )

  # Identifying semantic patterns in strings using the nltk parser
  print regex.compare_strings( parser='nltk', pattern_detection="semantic", text=strings )

Above is regex4dummies in its simplest form. It allows for additional features as well, including:

.. code-block:: Python

  # Display the version of regex4dummies you are using
  print regex.__version__

  # To use the other parsers, replace the above line of code with either of the following:
  # print regex.compare_strings( parser='pattern', pattern_detection="semantic", text=strings )
  # print regex.compare_strings( parser='nlpnet', pattern_detection="semantic", text=strings )

  # To call all of the parsers, replace the above line of code with the following:
  # print regex.compare_strings( parser='', pattern_detection="semantic", text=strings )

  # To get the topics of the strings, call the get_topics function
  print regex.get_topics( text=strings )

  # Printing pattern information
  pattern_information = regex.get_sentence_information()
    for objects in pattern_information:
        print "[ Pattern ]             : " + objects.pattern
        print "[ Subject ]             : " + objects.subject
        print "[ Verb ]                : " + objects.verb
        print "[ Object ]              : " + objects.object[0]
        print "[ Prep Phrases ]        : " + str( objects.prepositional_phrases )
        print "[ Reliability Score ]   : " + str( objects.reliability_score )
        print "[ Applicability Score ] : " + str( objects.applicability_score )
        print ""

A newly released set of features include a tokenizer and dependency finder function. To use them, simply give a string as the first parameter, and the name of the parser you would like to use as the second for both functions.

.. code-block:: Python

  # Testing the toolkit functions
  tool_tester = Toolkit()

  # Testing the tokenizer functions
  print tool_tester.tokenize( text="This is a test string.", parser="" )

  # Testing the dependency functions
  print tool_tester.find_dependencies( text="This is a test string.", parser="pattern" )

Other features included are demonstrated below.

.. code-block:: Python

  # Testing the information extraction functions
  regex.extract_important_information( text=[ "This is a test string." ] )

  # Testing the ability to extract phrases
  print "Noun Phrases: " + str( tool_tester.extract_noun_phrases( text="This is a test string." ) )
  print "Verb Phrases(Pattern): " + str( tool_tester.extract_verb_phrases( text="This is a test string.", parser="pattern" ) )
  print "Verb Phrases(Nlpnet): " + str( tool_tester.extract_verb_phrases( text="This is a test string.", parser="nlpnet" ) )
  print "Prepositional Phrases: " + str( tool_tester.extract_prepositional_phrases( text="This is a test string in the house." ) )

  print "String comparison: " + str( tool_tester.compare_strings( String1="This is a test string.", String2="This is a test string." ) )


{'fraction_non_alphanumeric': 0.09422622880446448, 'fraction_numerical': 0.0030049366816913502, 'mean_word_length': 3.9359110169491527, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2918979', 'n_tokens_mistral': 2571, 'n_tokens_neox': 2447, 'n_words': 963}