<jupyter_start><jupyter_text>DocumentationWelcome to `lingcorpora`'s documentation. In order to get started, see [Installation](Installation) and then proceed to [Quickstart](Quickstart). For parameter description, see [Making queries](Making-queries). To obtain description of our `Target` and `Result` objects, go to [Working with results](Working-with-results).* [Installation](Installation)* [Quickstart](Quickstart)* [Making queries](Making-queries)  * [lingcorpora.Corpus](lingcorpora.Corpus)  * [Corpora](Corpora)* [Working with results](Working-with-results)  * [lingcorpora.Target](lingcorpora.Target)  * [lingcorpora.Result](lingcorpora.Result) InstallationTo install `lingcorpora`, type in the terminal:```pip install lingcorpora```lingcorpora has dependencies on:* [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/)* [lxml](http://lxml.de)* [requests](http://docs.python-requests.org/en/master/)* [tqdm](https://github.com/tqdm/tqdm)`lingcorpora` will install these packages during installation. If you have troubles with installing dependencies, check Installation section in the documentation of the according packages. QuickstartTo search in any corpus, you need to create an instance of `Corpus` object and use method `search`:<jupyter_code>from lingcorpora import Corpus
rus_corp = Corpus('rus')
rus_results = rus_corp.search('мешок', n_results = 10)
rus_results<jupyter_output>"мешок": 100%|██████████| 10/10 [00:01<00:00,  8.10docs/s]<jupyter_text>All results are stored in the `Result` object. `Result` is the list of `Target` objects. To print the data, you can use the following code:<jupyter_code>for result in rus_results:
    for i,target in enumerate(result):
        print(i+1, target.text)<jupyter_output>1  [lafet, nick]   «Считают, что если они кому-нибудь отсыпят мешок золота, то можно всех пригласить? 
2  [Hamlet, nick]   ещё добрые преподы, которым просто некогда, и которые за мешок с пивом лихо ставили не самые плохие оценки всей группе ^ Но, но, но… 
3  ))) И мешок для сменки Немо, и рюкзак, и пеналы-ручки-тетрадки. 
4  Можешь использовать мешок многократно, простирывая после каждой варки с мылом, но только не в стиральном порошке. 
5  И этот мешок ― как скатерть-самобранка; в нём всё прибывает орехов. 
6  Наверняка пойдёт на дно, как мешок с песком!" 
7   Капитан-шеф-повар побежал было на нос, чтобы увести Эле-Фантика Но как раз в этот миг налетел девятый вал и, подстегиваемый ураганом, рухнул, как девятиэтажный дом, точно на слонёнка - ослепил, оглушил, закрутил в водовороте и снёс за борт, как большой мешок с песком. 
8   8. 39. Древнегреческий учёный Аристотель для доказательства невесомости воздуха взвешивал пустой кожаный мешок и тот же мешок, наполненный воздухом. 
9   8. [...]<jupyter_text>Output can be transformed to `kwic` format:<jupyter_code>for result in rus_results:
    for i,target in enumerate(result):
        print(i+1,'\t'.join(target.kwic(left=5,right=5)))<jupyter_output>1 что если они кому-нибудь отсыпят	мешок	золота, то можно всех пригласить?
2 просто некогда, и которые за	мешок	с пивом лихо ставили не
3 ))) И	мешок	для сменки Немо, и рюкзак,
4 Можешь использовать	мешок	многократно, простирывая после каждой варки
5 И этот	мешок	― как скатерть-самобранка; в нём
6 Наверняка пойдёт на дно, как	мешок	с песком!"
7 снёс за борт, как большой	мешок	с песком.
8 невесомости воздуха взвешивал пустой кожаный	мешок	и тот же мешок, наполненный
9 кожаный мешок и тот же	мешок	, наполненный воздухом.
10 со стороны воздуха на раздутый	мешок	.<jupyter_text>You can search by multiple queries:<jupyter_code>rus_results = rus_corp.search(['сумка','мешок'], n_results = 10)
for result in rus_results:
    for i,target in enumerate(result):
        print(i+1,target.text)
    print()<jupyter_output>"сумка": 100%|██████████| 10/10 [00:05<00:00,  1.77docs/s]
"мешок": 100%|██████████| 10/10 [00:06<00:00,  1.54docs/s]<jupyter_text>[To the top](Documentation) Making queries  lingcorpora.Corpuslingcorpora.Corpus(language, sleep_time=1, sleep_each=5)The object of this class should be instantiated for each corpus. Search is conducted via `search` method. Attributes* **language**: str: in most cases, language [ISO 639-3 code](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) for the corpus with combined codes for parallel corpora. List of available corpora with corresponding codes:| Code         | Corpus                                                                        ||--------------|-------------------------------------------------------------------------------|| bam          | [Corpus Bambara de Reference](Corpus-Bambara-de-Reference)                   || emk          | [Maninka Automatically Parsed corpus](Maninka-Automatically-Parsed-corpus)   || rus          | [National Corpus of Russian Language](National-Corpus-of-Russian-Language)   || rus_parallel | [Parallel subcorpus of National Corpus of Russian Language](Parallel-subcorpus-of-National-Corpus-of-Russian-Language) || zho          | [Center of Chinese Linguistics corpus](Center-of-Chinese-Linguistics-corpus) |* **sleep_time**: int, optional, default 1: the length of pause between requests to the corpus (in seconds). It is required to avoid blocking and corpus breakdown.* **sleep_each**: int, optional, default 5: the number of requests after which a pause is required.* **doc**: str: documentation for chosen corpus (after instance creation).* **results**: list: list of all `Result` objects, each returned by `search` method.* **failed**: list: list of `Result` objects where nothing was found. lingcorpora.Corpus.search(query, \*args, \*\*kwargs)This is a search function that queries the corpus and returns the results. ArgumentsHere we provide a comprehensive list of arguments, although not all arguments are applicable to all corpora and the default values depend on a corpus. For specific description, go to [Corpora](Corpora) subsection.Two arguments are universal:* **query**: str or list[str]: query or queries* **n_results**: int, optional, default 100: number of results wantedOther depend on the corpus:* **kwic**: boolean, optional, default True: `kwic` format (True) or a sentence (False)* **n_left**: int, optional: number of words / symbols (corpus-specific) in the left context* **n_right**: int, optional: number of words / symbols (corpus-specific) in the right context* **query_language**: str: for parallel corpora, language of the query* **subcorpus**: str, optional: subcorpus to search in* **get_analysis**: boolean, optional, default False: whether to download grammatical information if the corpus is annotated* **writing_system**: str, optional: writing system of results  Output* **Result**: list: resuls for queries in current search`search` returns the result of current query and adds it to `results` attribute of corresponding `Corpus` object. In the following example, `first_result` and `second_result` contain results of two queries (the second consists of two `Result` objects), whereas `bam_corp.results` contains all results obtained from the corpus.<jupyter_code>bam_corp = Corpus('bam')

first_result = bam_corp.search('jamana', n_results = 10)
print(first_result)
first_result.clear()
second_result = bam_corp.search(['kuma','fɔra'], n_results = 10)
print(second_result)
bam_corp.results<jupyter_output><empty_output><jupyter_text>lingcorpora.Corpus.retry_failed()This function redoes queries that returned empty results. ArgumentsThere are no arguments, failed queries are taken from `failed` attribute of the `Corpus` instance. OutputList of `Result` objects for which the second try was successful. It also appends successful results to the `results` attribute of the `Corpus` instance and rewrites `failed` attribute with queries that remained unsuccessful.[To the top](Documentation) Corpora [Corpus Bambara de Reference](http://maslinsky.spb.ru/bonito/run.cgi/first_form) Relevant parameters* **query**: str or list[str]: query or queries (currently only exact search by word or phrase is available)* **n_results**: int, default 100: number of results wanted* **kwic**: boolean, default True: `kwic` format (True) or a sentence (False)* **get_analysis**: boolean, default False: whether to collect grammatical tags for target word or not (available only for `corbama-net-non-tonal` subcorpus)* **subcorpus**: str, default `corbama-net-non-tonal`: subcorpus. Available options: `corbama-net-non-tonal`, `corbama-net-tonal`, `corbama-brut` Example of use<jupyter_code>corp = Corpus('bam')
results = corp.search('kɔ́nɔ', n_results = 10, subcorpus='corbama-net-tonal')
for result in results:
    for i,target in enumerate(result):
        print(i+1,target.text)<jupyter_output>"kɔ́nɔ": 100%|██████████| 10/10 [00:00<00:00, 18.33docs/s]<jupyter_text>[To the top](Documentation) [Maninka Automatically Parsed corpus](http://maslinsky.spb.ru/emk/run.cgi/first_form) Relevant parameters* **query**: str or list[str]: query or queries (currently only exact search by word or phrase is available)* **n_results**: int, default 100: number of results wanted* **kwic**: boolean, default True: `kwic` format (True) or a sentence (False)* **subcorpus**: str, default `cormani-brut-lat`: subcorpus. Available options: `cormani-brut-lat`, `corbama-brut-nko`* **writing_system**: str, default `None`: writing system for examples. Available options: `nko`, `latin`. Bug: only `latin` for `corbama-brut-nko` subcorpus. Example of use<jupyter_code>corp = Corpus('emk')
results = corp.search('tuma bɛɛ', n_results = 10, writing_system='nko', kwic=False)
for result in results:
    for i,target in enumerate(result):
        print(i+1,target.text)<jupyter_output>"tuma bɛɛ": 100%|██████████| 10/10 [00:00<00:00, 39.06docs/s]<jupyter_text>[To the top](Documentation) [National Corpus of Russian Language](http://ruscorpora.ru) Relevant parameters* **query**: str or list[str]: query or queries (currently only exact search by word or phrase is available)* **n_results**: int, default 100: number of results wanted* **kwic**: boolean, default True: `kwic` format (True) or a sentence (False)* **get_analysis**: boolean, default False: whether to collect grammatical tags for target word or not* **subcorpus**: str, default `main`: subcorpus. Available options:`main`, `syntax`, `paper`, `regional`, `school`, `dialect`, `poetic`, `spoken`, `accent`, `murco`, `multiparc`, `old_rus`, `birchbark`, `mid_rus`, `orthlib`. Example of use<jupyter_code>corp = Corpus('rus')
results = corp.search('сердце', n_results = 10, subcorpus='poetic')
for result in results:
    for i, target in enumerate(result):
        print(i+1, target.text)<jupyter_output>"сердце": 100%|██████████| 10/10 [00:04<00:00,  2.11docs/s]<jupyter_text>[To the top](Documentation) [Parallel subcorpus of National Corpus of Russian Language](http://ruscorpora.ru/search-para-multi.html) Relevant parameters* **query**: str or list[str]: query or queries (currently only exact search by word or phrase is available)* **query_language**: str: language of the query* **n_results**: int, default 100: number of results wanted* **kwic**: boolean, default True: `kwic` format (True) or a sentence (False)* **get_analysis**: boolean, default False: whether to collect grammatical tags for target word or not* **subcorpus**: str, default `rus` (search by all target languages): subcorpus. Available options: `rus`, `eng`, `bel`, `bul`, `bua`, `esp`, `ita`,`zho`, `lav`, `ger`, `pol`, `ukr`, `fra`, `sve`, `est`. Example of use<jupyter_code>corp = Corpus('rus_parallel')
results = corp.search('авось', query_language='rus', n_results = 10)
for result in results:
    for i,target in enumerate(result):
        print(i+1,target.transl.strip(),'\n',target.text.strip())<jupyter_output>"авось": 100%|██████████| 10/10 [00:04<00:00,  2.28docs/s]<jupyter_text>[To the top](Documentation) [Center of Chinese Linguistics corpus](http://ccl.pku.edu.cn:8080/ccl_corpus/index.jsp)  Relevant parameters* **query**: str or list[str]: query or queries (currently only exact search by word or phrase is available)* **n_results**: int, default 100: number of results wanted* **subcorpus**: str, default `xiandai`: subcorpus. Available options: `xiandai` (modern Chinese) or `dugai` (ancient Chinese).* **n_left**, **n_right**: int, default 30: context lenght (in symbols) Example of use<jupyter_code>corp = Corpus('zho')
results = corp.search('能力#3大', n_results = 10, n_left=15, n_right=15)
for result in results:
    for i,target in enumerate(result):
        print(i+1,target.text)<jupyter_output>"能力#3大": 100%|██████████| 10/10 [00:01<00:00,  6.03docs/s]<jupyter_text>[To the top](Documentation) Working with results lingcorpora.ResultThe object of this class contains all results found. `Result` object is iterable and supports indexing. Attributes* **results**: list[`Target`]: list of results* **N**: int: number of results* **lang**: str: corpus language* **query**: str: search query* **params**: dict: all other parameters of the search<jupyter_code>corp = Corpus('emk')
results = corp.search('tuma', n_results = 10, kwic=False)[0]
results<jupyter_output>"tuma": 100%|██████████| 10/10 [00:00<00:00, 39.82docs/s]<jupyter_text>lingcorpora.Result.export_csv(filename=None, header=True, sep=';')Writes results into a `csv` file considering `kwic` parameter of the search. Arguments* **filename**: str, optional, default `None`: name of the file. If `None`, filename is *lang*_*query*_results.csv with omission of disallowed filename symbols* **header**: boolean, optional, default True: whether to include a header in the table* **sep**: str, optional, default `;`: cell separator in the csv OutputNone, writes results into the file. lingcorpora.Result.clear()Overwrites the `results` attribute to empty list.<jupyter_code>print(results.results)
results.clear()
print(results.results)<jupyter_output>[Target(tuma, ), Target(tuma, ), Target(tuma, ), Target(tuma, ), Target(tuma, ), Target(tuma, ), Target(tuma, ), Target(tuma, ), Target(tuma, ), Target(tuma, )]
[]<jupyter_text>[To the top](Documentation) lingcorpora.Target`Target` contains one item from the result list. Attributes* **text**: str: full example (sentence or `kwic`)* **idxs**: idxs: tuple (l, r): target indices in **text** -> text[l:r]* **meta**: str: sentence / document metadata, e.g. document name, author, etc. (if exists)* **tags**: dict: target tags* **transl**: str: text translation (only for parallel corpora)* **lang**: str: translation language (only for parallel corpora)<jupyter_code>rus_corp = Corpus('rus')
rus_results = rus_corp.search('одеяло', n_results = 10, get_analysis=True)[0]
first_hit = rus_results[0]
first_hit
for k,v in vars(first_hit).items():
    print(k,v)<jupyter_output>text  Я, например, для внучки настегала своими руками лоскутное одеяло, зная, что оно будет её оберегать, давать ей энергию. 
idxs (59, 65)
meta Народный костюм: архаика или современность? // «Народное творчество», 2004
analysis {'lex': ['одеяло'], 'gramm': ['S', 'inan', 'n', 'sg', 'acc', 'disamb'], 'sem': ['r:concr', 't:tool:bedding'], 'flags': ['animred', 'bcomma', 'bmark', 'casered', 'genderred', 'numred']}
gr_tags None
transl None
lang None<jupyter_text>lingcorpora.Target.kwic(left, right, level='word')This function makes `kwic` format for an item for further usage and csv output. Attributes* **left**: int: length of left context* **right**: int: length of right context* **level**: str, optional, default `word`: counting context length by tokens (`word`) or by characters (`char`) Outputtuple: (left context, target, right context)<jupyter_code>print(first_hit.kwic(left=5,right=5))
print(first_hit.kwic(left=30,right=30, level='char'))<jupyter_output>('внучки настегала своими руками лоскутное', 'одеяло', ', зная, что оно будет её')
('егала своими руками лоскутное ', 'одеяло', ', зная, что оно будет её обере')
{'fraction_non_alphanumeric': 0.1388888888888889, 'fraction_numerical': 0.017676767676767676, 'mean_word_length': 5.545867768595041, 'pattern_counts': {'":': 9, '<': 53, '<?xml version=': 0, '>': 45, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '207914', 'license': 'MIT', 'n_tokens_mistral': 5437, 'n_tokens_neox': 5248, 'n_words': 1697, 'path': 'docs/docs.ipynb', 'repo_name': 'OneAdder/lingcorpora.py', 'size': '15840'}