<jupyter_start><jupyter_text>Extracting the overall response rate (ORR)  This is an advanced notebook. If you are new to IRuta Notebooks, then please start with the Notebook: `Notebook [IRuta] - Get started.ipynb` This notebook will show you how **batch processing** works. In batch processing, Ruta rules are applied to a collection of documents. In this example IRuta Notebook, the documents are 100 PubMed abstracts. The goal is to extract the overall response rate (ORR). ORR is defined as the proportion of patients who have a partial or complete response to therapy. The results should be collected in a CSV-Table.  Step 1: Extracting sentences, ORR-indicators and numbersWe are reading the files from an input directory. This means that all text files from that directory are processed. For each file, the script:1. Extracts sentences2. Annotates mentionings of the overall response rate (plus synonyms) and find numeric values3. Collect sentences with overall response rate indicator in a TableWe write the intermediate results into a new directory by setting the output directory.<jupyter_code>%inputDir input/pubmed_abstracts/txt100/
%outputDir output/orr_stage1/
%displayMode CSV
%csvConfig SentenceWithORRInd

// Detect sentence ends. Avoid interpreting a dot in a number "5.6" as sentence end.
DECLARE SentEnd, Sentence;
BLOCK(Sentence) Document{}{
    PERIOD{-> SentEnd};
    NUM se:SentEnd{-> UNMARK(se)} NUM;
    se:SentEnd{-> UNMARK(se)} SW;
    SW se:SentEnd{-> UNMARK(se)} NUM _{-PARTOF(PERIOD)};
    ANY+{-PARTOF(Sentence),-PARTOF(SentEnd)-> Sentence};
}

// Find mentionings of overall response rate, treat brackets correctly, detect numbers
DECLARE ORRInd, Value;
DECLARE ORR (Annotation keyword, Value value, Sentence sentence);
WORDLIST orrList = 'wordlists/orr_indicators.txt';
MARKFAST(ORRInd, orrList);
i1:ORRInd{->i1.end=p2.end, UNMARK(i2)} p1:"(" i2:ORRInd p2:")";
(NUM{-PARTOF(Value)} (PERIOD NUM)? SPECIAL.ct=="%"){-> Value};
n:NUM SPECIAL.ct=="%"? SPECIAL.ct=="-" v:@Value{-> v.begin = n.begin};

// Find sentence that contain an overall response rate indicator
DECLARE SentenceWithORRInd;
(ANY ANY @Sentence{CONTAINS(ORRInd)} ANY ANY){-> SentenceWithORRInd};

// Highlight relevant entities within the table.
COLOR(ORRInd, "lightgreen");
COLOR(Value, "pink");<jupyter_output>Processed 100/100 files. (took 2s)
205 rows created.<jupyter_text>Step 2: Collecting ORR with their values in a csv table<jupyter_code>%inputDir output/orr_stage1
%outputDir output/orr_stage2
%csvConfig ORR keyword value sentence
%saveCSV output/orr_result_table.csv
s:Sentence->{
    (i:ORRInd v:Value){-> CREATE(ORR, "keyword"=i, "value"=v, "sentence"=s)};
    (v:Value i:@ORRInd){-> CREATE(ORR, "keyword"=i, "value"=v, "sentence"=s)};
    (i:ORRInd SW v:Value){-> CREATE(ORR, "keyword"=i, "value"=v, "sentence"=s)};
    };<jupyter_output>Processed 100/100 files. (took 2s)
78 rows created.<jupyter_text>Additional Error Analysis: Sentences with ORR indicator, but without matchIt is easy to spot False Positives (sentences where an ORR value has been found) in the above table. On the other hand, False Negatives are not possible to detect with the above table alone. For that reason, it is useful to highlight similar key words (e.g. "response rate", "RR", "rate of response". This is done in the cell below.<jupyter_code>%inputDir output/orr_stage2
%outputDir output/orr_stage3
%csvConfig SentenceWithIndButNoORR

DECLARE Ind;
DECLARE SentenceWithIndButNoORR;

"RR" -> Ind;
"response rate" -> Ind;
"rate of response" -> Ind;
i:Ind{PARTOF(ORRInd) -> UNMARK(i)};
Sentence{CONTAINS(Ind),-CONTAINS(ORR)->SentenceWithIndButNoORR};

COLOR(Ind, "red");<jupyter_output>Processed 100/100 files. (took 1s)
68 rows created.
{'fraction_non_alphanumeric': 0.11194630872483222, 'fraction_numerical': 0.014228187919463087, 'mean_word_length': 5.491289198606272, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '447447', 'license': 'Apache-2.0', 'n_tokens_mistral': 1259, 'n_tokens_neox': 1179, 'n_words': 429, 'path': 'notebooks/introduction/Notebook [IRuta] - Batch Processing for detecting Overall Response Rates.ipynb', 'repo_name': 'averbis/IRuta', 'size': '3725'}