<jupyter_start><jupyter_text>Comprehend Advanced Demo - performance test  This notebook shows how to use Comprehend Sentiment API in batch mode and the use of python multithreading<jupyter_code>! pip install -r source/requirements.txt<jupyter_output><empty_output><jupyter_text>Download public movies dataset<jupyter_code>! wget http://www.cs.cornell.edu/people/pabo/movie-review-data/rt-polaritydata.tar.gz
! tar -xf "rt-polaritydata.tar.gz" 
! ls rt-polaritydata<jupyter_output><empty_output><jupyter_text>Explore the dataset<jupyter_code>print("Example of negative sentences\n--\n")
! head rt-polaritydata/rt-polarity.neg
print("\n---Total sentences sentences--\n")
! wc -l  rt-polaritydata/rt-polarity.neg


print("Example of positive sentences\n--\n")
! head rt-polaritydata/rt-polarity.pos
print("\n---Total sentences sentences--\n")
! wc -l  rt-polaritydata/rt-polarity.pos

! head -100 rt-polaritydata/rt-polarity.pos > rt-polaritydata/rt-polarity.pos_100.txt
import sys
sys.path.append("./source")
from source.demo import run_demo, MODE_BATCH<jupyter_output><empty_output><jupyter_text>Test the performance and the impact of increasing the number of threads.  On a single thread, roughly takes 1min 30 sec on a i7 processor<jupyter_code>%%timeit -n1 -r1

run_demo(MODE_BATCH, "rt-polaritydata/rt-polarity.pos", outfile = "/tmp/sentiment.csv", max_threads=1, encoding="latin")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12305516265912306, 'fraction_numerical': 0.009193776520509194, 'mean_word_length': 5.643192488262911, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 18, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '439774', 'license': 'MIT', 'n_tokens_mistral': 510, 'n_tokens_neox': 494, 'n_words': 106, 'path': 'Demo_Advanced.ipynb', 'repo_name': 'elangovana/awscomprehend-sentiment-demo', 'size': '1414'}