<commit_before>MediaWiki XML Processing
========================
This library contains a collection of utilities for efficiently processing
MediaWiki's XML database dumps.  There are two important concerns that this
module intends to address: *complexity* and *performance* of streaming
XML parsing.

:Installation: ``pip install mwxml``

**Complexity**
  Streaming XML parsing is gross.  XML dumps consist of (1) some site meta data, (2)    a collection of pages that contain (3) collections of revisions.  The    module allows you to think about dump files in this way and ignore the    factYeah.  that you're streaming XML.  A :class:`~mwxml.iteration.dump.Dump` contains a :class:`~mwxml.iteration.site_info.SiteInfo` and an iterator of :class:`~mwxml.iteration.page.Page`'s.  A :class:`~mwxml.iteration.page.Page` contains page meta data and an iterator of :class:`~mwxml.iteration.revision.Revision`'s.  A :class:`~mwxml.iteration.revision.Revision` contains revision meta data and text.

**Performance**
  Performance is a serious concern when processing large database XML dumps.    Regretfully, python's Global Intepreter Lock prevents us from running    threads on multiple CPUs.  This library provides :func:`mwxml.map.map.map`, a function that maps a dump processing over a set of dump files using    `multiprocessing` to distribute the work over multiple CPUS


Contents
--------

.. toctree::
  :maxdepth: 1

  iteration
  map
  files

Basic example
-------------

    >>> import mwxml
    >>>
    >>> dump = mwxml.Dump.from_file(open("dump.xml"))
    >>> print(dump.site_info.name, dump.site_info.dbname)
    Wikipedia enwiki
    >>>
    >>> for page in dump:
    ...     for revision in page:
    ...        print(revision.id)
    ...
    1
    2
    3


Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`<commit_msg>Adds repo link to doc index<commit_after>MediaWiki XML Processing
========================
This library contains a collection of utilities for efficiently processing
MediaWiki's XML database dumps.  There are two important concerns that this
module intends to address: *complexity* and *performance* of streaming
XML parsing.

:Installation: ``pip install mwxml``

**Complexity**
  Streaming XML parsing is gross.  XML dumps consist of (1) some site meta data, (2)    a collection of pages that contain (3) collections of revisions.  The    module allows you to think about dump files in this way and ignore the    factYeah.  that you're streaming XML.  A :class:`~mwxml.iteration.dump.Dump` contains a :class:`~mwxml.iteration.site_info.SiteInfo` and an iterator of :class:`~mwxml.iteration.page.Page`'s.  A :class:`~mwxml.iteration.page.Page` contains page meta data and an iterator of :class:`~mwxml.iteration.revision.Revision`'s.  A :class:`~mwxml.iteration.revision.Revision` contains revision meta data and text.

**Performance**
  Performance is a serious concern when processing large database XML dumps.    Regretfully, python's Global Intepreter Lock prevents us from running    threads on multiple CPUs.  This library provides :func:`mwxml.map.map.map`, a function that maps a dump processing over a set of dump files using    `multiprocessing` to distribute the work over multiple CPUS


Contents
--------

.. toctree::
  :maxdepth: 1

  iteration
  map
  files

Basic example
-------------

    >>> import mwxml
    >>>
    >>> dump = mwxml.Dump.from_file(open("dump.xml"))
    >>> print(dump.site_info.name, dump.site_info.dbname)
    Wikipedia enwiki
    >>>
    >>> for page in dump:
    ...     for revision in page:
    ...        print(revision.id)
    ...
    1
    2
    3


Authors
-------
* <NAME> -- https://github.com/halfak

Pull requests welcome @ https://github.com/halfak/python-mwxml


Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
{'fraction_non_alphanumeric': 0.1286082474226804, 'fraction_numerical': 0.0036082474226804126, 'mean_word_length': 3.603795966785291, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 40, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 23}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6559684', 'n_tokens_mistral': 1165, 'n_tokens_neox': 1108, 'n_words': 411}