<commit_before>cloud-pubsub-logging-python
===========================

    Logging handlers for publishing the logs to Cloud Pub/Sub.

|pypi| |build| |coverage|

You can use the pubsub_logging.PubsubHandler or pubsub_logging.AsyncPubsubHandler to publish the logs to `Cloud Pub/Sub`_. You can use this module with `the standard Python logging module`_.

.. _Cloud Pub/Sub: https://cloud.google.com/pubsub/docs/
.. _the standard Python logging module: https://docs.python.org/2/library/logging.html

Supported version
-----------------

Python 2.7 and Python 3.4 are supported.

Installation
------------

::

    $ pip install pubsub-logging

How to use
----------

Here is an example configuration file.

::

    [loggers]
    keys=root

    [handlers]
    keys=pubsubHandler

    [formatters]
    keys=simpleFormatter

    [logger_root]
    level=NOTSET
    handlers=pubsubHandler

    [handler_pubsubHandler]
    class=pubsub_logging.PubsubHandler
    level=DEBUG
    formatter=simpleFormatter
    args=('projects/tmatsuo-pubsub-sample/topics/log-topic', 1000)

    [formatter_simpleFormatter]
    format=%(asctime)s - %(name)s - %(levelname)s - %(message)s

How to use this config file.

.. code:: python

    logging.config.fileConfig(os.path.join('examples', 'logging.conf'))
    logger = logging.getLogger('root')
    logger.info('This message will be buffered')
    logger.critical('This message will be flushed.')
    

Here is a dynamic usage example.

.. code:: python

    pubsub_handler = PubsubHandler(topic=topic)
    pubsub_handler.setFormatter(
        logging.Formatter(
            '%(asctime)s - %(name)s - %(levelname)s - %(message)s'))

    logger = logging.getLogger('root')
    logger.setLevel(logging.DEBUG)
    logger.addHandler(pubsub_handler)
    logger.info('This message will be buffered')
    logger.critical('This message will be flushed.')

The logs are buffered by default, and when the buffer is full, or when the message has a log level higher or equal to flush_level, the buffered logs will be flushed and sent to Cloud Pub/Sub.

Authentication
--------------

The module uses the `Application Default Credentials`_. You can configure the authentication as follows.

.. _Application Default Credentials: https://developers.google.com/accounts/docs/application-default-credentials

Authentication on App Engine
----------------------------

It should work out of the box. If you're getting an authorization error, please make sure that your App Engine service account has an `Editor` or greater permission on your Cloud project.

Authentication on Google Compute Engine
---------------------------------------

When creating a new instance, please add the Cloud Pub/Sub scope `https://www.googleapis.com/auth/pubsub` to the service account of the instance.

Authentication anywhere else
----------------------------

As `the documentation suggests`_, create a new service account and download its JSON key file, then set the environment variable `GOOGLE_APPLICATION_CREDENTIALS` pointing to the JSON key file. Please note that this service account must have `Editor` or greater permissions on your Cloud project.

.. _the documentation suggests: https://developers.google.com/accounts/docs/application-default-credentials#whentouse


.. |build| image:: https://travis-ci.org/GoogleCloudPlatform/cloud-pubsub-logging-python.svg?branch=master
   :target: https://travis-ci.org/GoogleCloudPlatform/cloud-pubsub-logging-python
.. |pypi| image:: https://img.shields.io/pypi/v/pubsub-logging.svg
   :target: https://pypi.python.org/pypi/pubsub-logging<commit_msg>Updated the README with a usage example of AsyncPubsubHandler.<commit_after>cloud-pubsub-logging-python
===========================

    Logging handlers for publishing the logs to Cloud Pub/Sub.

|pypi| |build| |coverage|

You can use the pubsub_logging.PubsubHandler or pubsub_logging.AsyncPubsubHandler to publish the logs to `Cloud Pub/Sub`_. You can use this module with `the standard Python logging module`_.

.. _Cloud Pub/Sub: https://cloud.google.com/pubsub/docs/
.. _the standard Python logging module: https://docs.python.org/2/library/logging.html

Supported version
-----------------

Python 2.7 and Python 3.4 are supported.

Installation
------------

::

    $ pip install pubsub-logging

How to use
----------

Here is an example configuration file.

::

    [loggers]
    keys=root

    [handlers]
    keys=asyncPubsubHandler

    [formatters]
    keys=simpleFormatter

    [logger_root]
    level=NOTSET
    handlers=asyncPubsubHandler

    [handler_asyncPubsubHandler]
    class=pubsub_logging.AsyncPubsubHandler
    level=DEBUG
    formatter=simpleFormatter
    # Replace {project-name} and {topic-name} with actual ones.
    # The second argument indicates number of workers.
    args=('projects/{project-name}/topics/{topic-name}', 10)

    [formatter_simpleFormatter]
    format=%(asctime)s - %(name)s - %(levelname)s - %(message)s

How to use this config file.

.. code:: python

    logging.config.fileConfig(os.path.join('examples', 'async.conf'))
    logger = logging.getLogger('root')
    logger.info('My first message.')

Here is a dynamic usage example.

.. code:: python

    pubsub_handler = AsyncPubsubHandler(topic=topic)
    pubsub_handler.setFormatter(
        logging.Formatter(
            '%(asctime)s - %(name)s - %(levelname)s - %(message)s'))

    logger = logging.getLogger('root')
    logger.setLevel(logging.DEBUG)
    logger.addHandler(pubsub_handler)
    logger.info('My first message.')

The logs are kept in a buffer first, then moved to the process safe queue, and then the background child processes automatically pick up and send them to Cloud Pub/Sub. The flush call blocks until all of the logs are sent to Cloud Pub/Sub.

Authentication
--------------

The module uses the `Application Default Credentials`_. You can configure the authentication as follows.

.. _Application Default Credentials: https://developers.google.com/accounts/docs/application-default-credentials

Authentication on App Engine
----------------------------

It should work out of the box. If you're getting an authorization error, please make sure that your App Engine service account has an `Editor` or greater permission on your Cloud project.

Authentication on Google Compute Engine
---------------------------------------

When creating a new instance, please add the Cloud Pub/Sub scope `https://www.googleapis.com/auth/pubsub` to the service account of the instance.

Authentication anywhere else
----------------------------

As `the documentation suggests`_, create a new service account and download its JSON key file, then set the environment variable `GOOGLE_APPLICATION_CREDENTIALS` pointing to the JSON key file. Please note that this service account must have `Editor` or greater permissions on your Cloud project.

.. _the documentation suggests: https://developers.google.com/accounts/docs/application-default-credentials#whentouse


.. |build| image:: https://travis-ci.org/GoogleCloudPlatform/cloud-pubsub-logging-python.svg?branch=master
   :target: https://travis-ci.org/GoogleCloudPlatform/cloud-pubsub-logging-python
.. |pypi| image:: https://img.shields.io/pypi/v/pubsub-logging.svg
   :target: https://pypi.python.org/pypi/pubsub-logging
{'fraction_non_alphanumeric': 0.13628877298929454, 'fraction_numerical': 0.0021959923140269007, 'mean_word_length': 3.368705035971223, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 18, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6142559', 'n_tokens_mistral': 2043, 'n_tokens_neox': 1961, 'n_words': 647}