<commit_before>Help on package smart_open:

NAME
    smart_open

DESCRIPTION
    Utilities for streaming to/from several file-like data storages: S3 / HDFS / local
    filesystem / compressed files, and many more, using a simple, Pythonic API.
    
    The streaming makes heavy use of generators and pipes, to avoid loading
    full file contents into memory, allowing work with arbitrarily large files.
    
    The main functions are:
    
    * `open()`, which opens the given file for reading/writing
    * `parse_uri()`
    * `s3_iter_bucket()`, which goes over all keys in an S3 bucket in parallel
    * `register_compressor()`, which registers callbacks for transparent compressor handling

PACKAGE CONTENTS
    bytebuffer
    compression
    concurrency
    constants
    doctools
    gcs
    hdfs
    http
    local_file
    s3
    smart_open_lib
    ssh
    tests (package)
    transport
    utils
    version
    webhdfs

FUNCTIONS
    open(uri, mode='r', buffering=-1, encoding=None, errors=None, newline=None, closefd=True, opener=None, ignore_ext=False, transport_params=None)
        Open the URI object, returning a file-like object.
        
        The URI is usually a string in a variety of formats.
        For a full list of examples, see the :func:`parse_uri` function.
        
        The URI may also be one of:
        
        - an instance of the pathlib.Path class
        - a stream (anything that implements io.IOBase-like functionality)
        
        Parameters
        ----------
        uri: str or object
            The object to open.
        mode: str, optional
            Mimicks built-in open parameter of the same name.
        buffering: int, optional
            Mimicks built-in open parameter of the same name.
        encoding: str, optional
            Mimicks built-in open parameter of the same name.
        errors: str, optional
            Mimicks built-in open parameter of the same name.
        newline: str, optional
            Mimicks built-in open parameter of the same name.
        closefd: boolean, optional
            Mimicks built-in open parameter of the same name.  Ignored.
        opener: object, optional
            Mimicks built-in open parameter of the same name.  Ignored.
        ignore_ext: boolean, optional
            Disable transparent compression/decompression based on the file extension.
        transport_params: dict, optional
            Additional parameters for the transport layer (see notes below).
        
        Returns
        -------
        A file-like object.
        
        Notes
        -----
        smart_open has several implementations for its transport layer (e.g. S3, HTTP).
        Each transport layer has a different set of keyword arguments for overriding
        default behavior.  If you specify a keyword argument that is *not* supported
        by the transport layer being used, smart_open will ignore that argument and
        log a warning message.
        
        smart_open supports the following transport mechanisms:
        
        file (smart_open/local_file.py)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        Implements the transport for the file:// schema.
        
        gs (smart_open/gcs.py)
        ~~~~~~~~~~~~~~~~~~~~~~
        Implements file-like objects for reading and writing to/from GCS.
        
        buffer_size: int, optional
            The buffer size to use when performing I/O. For reading only.
        min_part_size: int, optional
            The minimum part size for multipart uploads.  For writing only.
        client: google.cloud.storage.Client, optional
            The GCS client to use when working with google-cloud-storage.
        
        hdfs (smart_open/hdfs.py)
        ~~~~~~~~~~~~~~~~~~~~~~~~~
        Implements reading and writing to/from HDFS.
        
        http (smart_open/http.py)
        ~~~~~~~~~~~~~~~~~~~~~~~~~
        Implements file-like objects for reading from http.
        
        kerberos: boolean, optional
            If True, will attempt to use the local Kerberos credentials
        user: str, optional
            The username for authenticating over HTTP
        password: str, optional
            The password for authenticating over HTTP
        headers: dict, optional
            Any headers to send in the request. If ``None``, the default headers are sent:
            ``{'Accept-Encoding': 'identity'}``. To use no headers at all,
            set this variable to an empty dict, ``{}``.
        
        s3 (smart_open/s3.py)
        ~~~~~~~~~~~~~~~~~~~~~
        Implements file-like objects for reading and writing from/to AWS S3.
        
        buffer_size: int, optional
            The buffer size to use when performing I/O.
        min_part_size: int, optional
            The minimum part size for multipart uploads.  For writing only.
        session: object, optional
            The S3 session to use when working with boto3.
        resource_kwargs: dict, optional
            Keyword arguments to use when accessing the S3 resource for reading or writing.
        multipart_upload_kwargs: dict, optional
            Additional parameters to pass to boto3's initiate_multipart_upload function.
            For writing only.
        singlepart_upload_kwargs: dict, optional
            Additional parameters to pass to boto3's S3.Object.put function when using single
            part upload.
            For writing only.
        multipart_upload: bool, optional
            Default: `True`
            If set to `True`, will use multipart upload for writing to S3. If set
            to `False`, S3 upload will use the S3 Single-Part Upload API, which
            is more ideal for small file sizes.
            For writing only.
        version_id: str, optional
            Version of the object, used when reading object.
            If None, will fetch the most recent version.
        object_kwargs: dict, optional
            Additional parameters to pass to boto3's object.get function.
            Used during reading only.
        
        scp (smart_open/ssh.py)
        ~~~~~~~~~~~~~~~~~~~~~~~
        Implements I/O streams over SSH.
        
        mode: str, optional
            The mode to use for opening the file.
        host: str, optional
            The hostname of the remote machine.  May not be None.
        user: str, optional
            The username to use to login to the remote machine.
            If None, defaults to the name of the current user.
        password: str, optional
            The password to use to login to the remote machine.
        port: int, optional
            The port to connect to.
        transport_params: dict, optional
            Any additional settings to be passed to paramiko.SSHClient.connect
        
        webhdfs (smart_open/webhdfs.py)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        Implements reading and writing to/from WebHDFS.
        
        min_part_size: int, optional
            For writing only.
        
        Examples
        --------
        
        >>> from smart_open import open
        >>>
        >>> # stream lines from an S3 object
        >>> for line in open('s3://commoncrawl/robots.txt'):
        ...    print(repr(line))
        ...    break
        'User-Agent: *\n'
        
        >>> # stream from/to compressed files, with transparent (de)compression:
        >>> for line in open('smart_open/tests/test_data/1984.txt.gz', encoding='utf-8'):
        ...    print(repr(line))
        'It was a bright cold day in April, and the clocks were striking thirteen.\n'
        '<NAME>, his chin nuzzled into his breast in an effort to escape the vile\n'
        'wind, slipped quickly through the glass doors of Victory Mansions, though not\n'
        'quickly enough to prevent a swirl of gritty dust from entering along with him.\n'
        
        >>> # can use context managers too:
        >>> with open('smart_open/tests/test_data/1984.txt.gz') as fin:
        ...    with open('smart_open/tests/test_data/1984.txt.bz2', 'w') as fout:
        ...        for line in fin:
        ...           fout.write(line)
        
        >>> # can use any IOBase operations, like seek
        >>> with open('s3://commoncrawl/robots.txt', 'rb') as fin:
        ...     for line in fin:
        ...         print(repr(line.decode('utf-8')))
        ...         break
        ...     offset = fin.seek(0)  # seek to the beginning
        ...     print(fin.read(4))
        'User-Agent: *\n'
        b'User'
        
        >>> # stream from HTTP
        >>> for line in open('http://example.com/index.html'):
        ...     print(repr(line))
        ...     break
        
        This function also supports transparent compression and decompression 
        using the following codecs:
        
        * .bz2
        * .gz
        
        The function depends on the file extension to determine the appropriate codec.
        
        
        See Also
        --------
        - `Standard library reference <https://docs.python.org/3.7/library/functions.html#open>`__
        - `smart_open README.rst
          <https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst>`__
    
    parse_uri(uri_as_string)
        Parse the given URI from a string.
        
        Parameters
        ----------
        uri_as_string: str
            The URI to parse.
        
        Returns
        -------
        collections.namedtuple
            The parsed URI.
        
        Notes
        -----
        Supported URI schemes are:
        
        * file
        * gs
        * hdfs
        * http
        * s3
        * scp
        * webhdfs
        
        Valid URI examples::
        
        * ./local/path/file
        * ~/local/path/file
        * local/path/file
        * ./local/path/file.gz
        * file:///home/user/file
        * file:///home/user/file.bz2
        * hdfs:///path/file
        * hdfs://path/file
        * s3://my_bucket/my_key
        * s3://my_key:my_secret@my_bucket/my_key
        * s3://my_key:my_secret@my_server:my_port@my_bucket/my_key
        * ssh://username@host/path/file
        * ssh://username@host//path/file
        * scp://username@host/path/file
        * sftp://username@host/path/file
        * webhdfs://host:port/path/file
    
    register_compressor(ext, callback)
        Register a callback for transparently decompressing files with a specific extension.
        
        Parameters
        ----------
        ext: str
            The extension.  Must include the leading period, e.g. ``.gz``.
        callback: callable
            The callback.  It must accept two position arguments, file_obj and mode.
            This function will be called when ``smart_open`` is opening a file with
            the specified extension.
        
        Examples
        --------
        
        Instruct smart_open to use the `lzma` module whenever opening a file
        with a .xz extension (see README.rst for the complete example showing I/O):
        
        >>> def _handle_xz(file_obj, mode):
        ...     import lzma
        ...     return lzma.LZMAFile(filename=file_obj, mode=mode, format=lzma.FORMAT_XZ)
        >>>
        >>> register_compressor('.xz', _handle_xz)
    
    s3_iter_bucket = iter_bucket(bucket_name, prefix='', accept_key=None, key_limit=None, workers=16, retries=3, **session_kwargs)
        Iterate and download all S3 objects under `s3://bucket_name/prefix`.
        
        Parameters
        ----------
        bucket_name: str
            The name of the bucket.
        prefix: str, optional
            Limits the iteration to keys starting wit the prefix.
        accept_key: callable, optional
            This is a function that accepts a key name (unicode string) and
            returns True/False, signalling whether the given key should be downloaded.
            The default behavior is to accept all keys.
        key_limit: int, optional
            If specified, the iterator will stop after yielding this many results.
        workers: int, optional
            The number of subprocesses to use.
        retries: int, optional
            The number of time to retry a failed download.
        session_kwargs: dict, optional
            Keyword arguments to pass when creating a new session.
            For a list of available names and values, see:
            https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html#boto3.session.Session
        
        
        Yields
        ------
        str
            The full key name (does not include the bucket name).
        bytes
            The full contents of the key.
        
        Notes
        -----
        The keys are processed in parallel, using `workers` processes (default: 16),
        to speed up downloads greatly. If multiprocessing is not available, thus
        _MULTIPROCESSING is False, this parameter will be ignored.
        
        Examples
        --------
        
          >>> # get all JSON files under "mybucket/foo/"
          >>> for key, content in iter_bucket(
          ...         bucket_name, prefix='foo/',
          ...         accept_key=lambda key: key.endswith('.json')):
          ...     print key, len(content)
        
          >>> # limit to 10k files, using 32 parallel workers (default is 16)
          >>> for key, content in iter_bucket(bucket_name, key_limit=10000, workers=32):
          ...     print key, len(content)
    
    smart_open(uri, mode='rb', **kw)

DATA
    __all__ = ['open', 'parse_uri', 'register_compressor', 's3_iter_bucket...

VERSION
    2.0.0

FILE
    /home/misha/git/smart_open/smart_open/__init__.py

<commit_msg>updated help.txt for version 2.1.0<commit_after>Help on package smart_open:

NAME
    smart_open

DESCRIPTION
    Utilities for streaming to/from several file-like data storages: S3 / HDFS / local
    filesystem / compressed files, and many more, using a simple, Pythonic API.
    
    The streaming makes heavy use of generators and pipes, to avoid loading
    full file contents into memory, allowing work with arbitrarily large files.
    
    The main functions are:
    
    * `open()`, which opens the given file for reading/writing
    * `parse_uri()`
    * `s3_iter_bucket()`, which goes over all keys in an S3 bucket in parallel
    * `register_compressor()`, which registers callbacks for transparent compressor handling

PACKAGE CONTENTS
    azure
    bytebuffer
    compression
    concurrency
    constants
    doctools
    gcs
    hdfs
    http
    local_file
    s3
    smart_open_lib
    ssh
    tests (package)
    transport
    utils
    version
    webhdfs

FUNCTIONS
    open(uri, mode='r', buffering=-1, encoding=None, errors=None, newline=None, closefd=True, opener=None, ignore_ext=False, transport_params=None)
        Open the URI object, returning a file-like object.
        
        The URI is usually a string in a variety of formats.
        For a full list of examples, see the :func:`parse_uri` function.
        
        The URI may also be one of:
        
        - an instance of the pathlib.Path class
        - a stream (anything that implements io.IOBase-like functionality)
        
        Parameters
        ----------
        uri: str or object
            The object to open.
        mode: str, optional
            Mimicks built-in open parameter of the same name.
        buffering: int, optional
            Mimicks built-in open parameter of the same name.
        encoding: str, optional
            Mimicks built-in open parameter of the same name.
        errors: str, optional
            Mimicks built-in open parameter of the same name.
        newline: str, optional
            Mimicks built-in open parameter of the same name.
        closefd: boolean, optional
            Mimicks built-in open parameter of the same name.  Ignored.
        opener: object, optional
            Mimicks built-in open parameter of the same name.  Ignored.
        ignore_ext: boolean, optional
            Disable transparent compression/decompression based on the file extension.
        transport_params: dict, optional
            Additional parameters for the transport layer (see notes below).
        
        Returns
        -------
        A file-like object.
        
        Notes
        -----
        smart_open has several implementations for its transport layer (e.g. S3, HTTP).
        Each transport layer has a different set of keyword arguments for overriding
        default behavior.  If you specify a keyword argument that is *not* supported
        by the transport layer being used, smart_open will ignore that argument and
        log a warning message.
        
        smart_open supports the following transport mechanisms:
        
        azure (smart_open/azure.py)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~
        Implements file-like objects for reading and writing to/from Azure Blob Storage.
        
        buffer_size: int, optional
            The buffer size to use when performing I/O. For reading only.
        min_part_size: int, optional
            The minimum part size for multipart uploads.  For writing only.
        
        file (smart_open/local_file.py)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        Implements the transport for the file:// schema.
        
        gs (smart_open/gcs.py)
        ~~~~~~~~~~~~~~~~~~~~~~
        Implements file-like objects for reading and writing to/from GCS.
        
        buffer_size: int, optional
            The buffer size to use when performing I/O. For reading only.
        min_part_size: int, optional
            The minimum part size for multipart uploads.  For writing only.
        client: google.cloud.storage.Client, optional
            The GCS client to use when working with google-cloud-storage.
        
        hdfs (smart_open/hdfs.py)
        ~~~~~~~~~~~~~~~~~~~~~~~~~
        Implements reading and writing to/from HDFS.
        
        http (smart_open/http.py)
        ~~~~~~~~~~~~~~~~~~~~~~~~~
        Implements file-like objects for reading from http.
        
        kerberos: boolean, optional
            If True, will attempt to use the local Kerberos credentials
        user: str, optional
            The username for authenticating over HTTP
        password: str, optional
            The password for authenticating over HTTP
        headers: dict, optional
            Any headers to send in the request. If ``None``, the default headers are sent:
            ``{'Accept-Encoding': 'identity'}``. To use no headers at all,
            set this variable to an empty dict, ``{}``.
        
        s3 (smart_open/s3.py)
        ~~~~~~~~~~~~~~~~~~~~~
        Implements file-like objects for reading and writing from/to AWS S3.
        
        buffer_size: int, optional
            The buffer size to use when performing I/O.
        min_part_size: int, optional
            The minimum part size for multipart uploads.  For writing only.
        session: object, optional
            The S3 session to use when working with boto3.
        resource_kwargs: dict, optional
            Keyword arguments to use when accessing the S3 resource for reading or writing.
        multipart_upload_kwargs: dict, optional
            Additional parameters to pass to boto3's initiate_multipart_upload function.
            For writing only.
        singlepart_upload_kwargs: dict, optional
            Additional parameters to pass to boto3's S3.Object.put function when using single
            part upload.
            For writing only.
        multipart_upload: bool, optional
            Default: `True`
            If set to `True`, will use multipart upload for writing to S3. If set
            to `False`, S3 upload will use the S3 Single-Part Upload API, which
            is more ideal for small file sizes.
            For writing only.
        version_id: str, optional
            Version of the object, used when reading object.
            If None, will fetch the most recent version.
        object_kwargs: dict, optional
            Additional parameters to pass to boto3's object.get function.
            Used during reading only.
        
        scp (smart_open/ssh.py)
        ~~~~~~~~~~~~~~~~~~~~~~~
        Implements I/O streams over SSH.
        
        mode: str, optional
            The mode to use for opening the file.
        host: str, optional
            The hostname of the remote machine.  May not be None.
        user: str, optional
            The username to use to login to the remote machine.
            If None, defaults to the name of the current user.
        password: str, optional
            The password to use to login to the remote machine.
        port: int, optional
            The port to connect to.
        transport_params: dict, optional
            Any additional settings to be passed to paramiko.SSHClient.connect
        
        webhdfs (smart_open/webhdfs.py)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        Implements reading and writing to/from WebHDFS.
        
        min_part_size: int, optional
            For writing only.
        
        Examples
        --------
        
        >>> from smart_open import open
        >>>
        >>> # stream lines from an S3 object
        >>> for line in open('s3://commoncrawl/robots.txt'):
        ...    print(repr(line))
        ...    break
        'User-Agent: *\n'
        
        >>> # stream from/to compressed files, with transparent (de)compression:
        >>> for line in open('smart_open/tests/test_data/1984.txt.gz', encoding='utf-8'):
        ...    print(repr(line))
        'It was a bright cold day in April, and the clocks were striking thirteen.\n'
        '<NAME>, his chin nuzzled into his breast in an effort to escape the vile\n'
        'wind, slipped quickly through the glass doors of Victory Mansions, though not\n'
        'quickly enough to prevent a swirl of gritty dust from entering along with him.\n'
        
        >>> # can use context managers too:
        >>> with open('smart_open/tests/test_data/1984.txt.gz') as fin:
        ...    with open('smart_open/tests/test_data/1984.txt.bz2', 'w') as fout:
        ...        for line in fin:
        ...           fout.write(line)
        
        >>> # can use any IOBase operations, like seek
        >>> with open('s3://commoncrawl/robots.txt', 'rb') as fin:
        ...     for line in fin:
        ...         print(repr(line.decode('utf-8')))
        ...         break
        ...     offset = fin.seek(0)  # seek to the beginning
        ...     print(fin.read(4))
        'User-Agent: *\n'
        b'User'
        
        >>> # stream from HTTP
        >>> for line in open('http://example.com/index.html'):
        ...     print(repr(line))
        ...     break
        
        This function also supports transparent compression and decompression 
        using the following codecs:
        
        * .bz2
        * .gz
        
        The function depends on the file extension to determine the appropriate codec.
        
        
        See Also
        --------
        - `Standard library reference <https://docs.python.org/3.7/library/functions.html#open>`__
        - `smart_open README.rst
          <https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst>`__
    
    parse_uri(uri_as_string)
        Parse the given URI from a string.
        
        Parameters
        ----------
        uri_as_string: str
            The URI to parse.
        
        Returns
        -------
        collections.namedtuple
            The parsed URI.
        
        Notes
        -----
        Supported URI schemes are:
        
        * azure
        * file
        * gs
        * hdfs
        * http
        * s3
        * scp
        * webhdfs
        
        Valid URI examples::
        
        * ./local/path/file
        * ~/local/path/file
        * local/path/file
        * ./local/path/file.gz
        * file:///home/user/file
        * file:///home/user/file.bz2
        * hdfs:///path/file
        * hdfs://path/file
        * s3://my_bucket/my_key
        * s3://my_key:my_secret@my_bucket/my_key
        * s3://my_key:my_secret@my_server:my_port@my_bucket/my_key
        * ssh://username@host/path/file
        * ssh://username@host//path/file
        * scp://username@host/path/file
        * sftp://username@host/path/file
        * webhdfs://host:port/path/file
    
    register_compressor(ext, callback)
        Register a callback for transparently decompressing files with a specific extension.
        
        Parameters
        ----------
        ext: str
            The extension.  Must include the leading period, e.g. ``.gz``.
        callback: callable
            The callback.  It must accept two position arguments, file_obj and mode.
            This function will be called when ``smart_open`` is opening a file with
            the specified extension.
        
        Examples
        --------
        
        Instruct smart_open to use the `lzma` module whenever opening a file
        with a .xz extension (see README.rst for the complete example showing I/O):
        
        >>> def _handle_xz(file_obj, mode):
        ...     import lzma
        ...     return lzma.LZMAFile(filename=file_obj, mode=mode, format=lzma.FORMAT_XZ)
        >>>
        >>> register_compressor('.xz', _handle_xz)
    
    s3_iter_bucket = iter_bucket(bucket_name, prefix='', accept_key=None, key_limit=None, workers=16, retries=3, **session_kwargs)
        Iterate and download all S3 objects under `s3://bucket_name/prefix`.
        
        Parameters
        ----------
        bucket_name: str
            The name of the bucket.
        prefix: str, optional
            Limits the iteration to keys starting wit the prefix.
        accept_key: callable, optional
            This is a function that accepts a key name (unicode string) and
            returns True/False, signalling whether the given key should be downloaded.
            The default behavior is to accept all keys.
        key_limit: int, optional
            If specified, the iterator will stop after yielding this many results.
        workers: int, optional
            The number of subprocesses to use.
        retries: int, optional
            The number of time to retry a failed download.
        session_kwargs: dict, optional
            Keyword arguments to pass when creating a new session.
            For a list of available names and values, see:
            https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html#boto3.session.Session
        
        
        Yields
        ------
        str
            The full key name (does not include the bucket name).
        bytes
            The full contents of the key.
        
        Notes
        -----
        The keys are processed in parallel, using `workers` processes (default: 16),
        to speed up downloads greatly. If multiprocessing is not available, thus
        _MULTIPROCESSING is False, this parameter will be ignored.
        
        Examples
        --------
        
          >>> # get all JSON files under "mybucket/foo/"
          >>> for key, content in iter_bucket(
          ...         bucket_name, prefix='foo/',
          ...         accept_key=lambda key: key.endswith('.json')):
          ...     print key, len(content)
        
          >>> # limit to 10k files, using 32 parallel workers (default is 16)
          >>> for key, content in iter_bucket(bucket_name, key_limit=10000, workers=32):
          ...     print key, len(content)
    
    smart_open(uri, mode='rb', **kw)

DATA
    __all__ = ['open', 'parse_uri', 'register_compressor', 's3_iter_bucket...

VERSION
    2.1.0

FILE
    /Users/misha/git/smart_open/smart_open/__init__.py


{'fraction_non_alphanumeric': 0.09064841033728807, 'fraction_numerical': 0.005484067529302125, 'mean_word_length': 1.7808232831655537, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 123, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6373817', 'n_tokens_mistral': 7313, 'n_tokens_neox': 7132, 'n_words': 2636}