<jupyter_start><jupyter_text>tqdm[![PyPI-Versions](https://img.shields.io/pypi/pyversions/tqdm.svg?logo=python&logoColor=white)](https://pypi.org/project/tqdm)|[![PyPI-Status](https://img.shields.io/pypi/v/tqdm.svg)](https://pypi.org/project/tqdm)|[![Conda-Forge-Status](https://img.shields.io/conda/v/conda-forge/tqdm.svg?label=conda-forge&logo=conda-forge)](https://anaconda.org/conda-forge/tqdm)|[![Docker](https://img.shields.io/badge/docker-pull-blue.svg?logo=docker)](https://hub.docker.com/r/tqdm/tqdm)|[![Snapcraft](https://img.shields.io/badge/snap-install-82BEA0.svg?logo=snapcraft)](https://snapcraft.io/tqdm)-|-|-|-|-[![Build-Status](https://img.shields.io/travis/tqdm/tqdm/master.svg?logo=travis)](https://travis-ci.org/tqdm/tqdm)|[![Coverage-Status](https://coveralls.io/repos/tqdm/tqdm/badge.svg?branch=master)](https://coveralls.io/github/tqdm/tqdm)|[![Branch-Coverage-Status](https://codecov.io/gh/tqdm/tqdm/branch/master/graph/badge.svg)](https://codecov.io/gh/tqdm/tqdm)|[![Codacy-Grade](https://api.codacy.com/project/badge/Grade/3f965571598f44549c7818f29cdcf177)](https://www.codacy.com/app/tqdm/tqdm?utm_source=github.com&amp;utm_medium=referral&amp;utm_content=tqdm/tqdm&amp;utm_campaign=Badge_Grade)|[![Libraries-Rank](https://img.shields.io/librariesio/sourcerank/pypi/tqdm.svg?logo=koding&logoColor=white)](https://libraries.io/pypi/tqdm)|[![PyPI-Downloads](https://img.shields.io/pypi/dm/tqdm.svg?label=pypi%20downloads&logo=python&logoColor=white)](https://pypi.org/project/tqdm)-|-|-|-|-|-[![DOI](https://img.shields.io/badge/DOI-10.21105/joss.01277-green.svg)](https://doi.org/10.21105/joss.01277)|[![DOI-code](https://img.shields.io/badge/DOI-10.5281/zenodo.595120-blue.svg)](https://doi.org/10.5281/zenodo.595120)|[![LICENCE](https://img.shields.io/pypi/l/tqdm.svg)](https://raw.githubusercontent.com/tqdm/tqdm/master/LICENCE)|[![OpenHub-Status](https://www.openhub.net/p/tqdm/widgets/project_thin_badge?format=gif)](https://www.openhub.net/p/tqdm?ref=Thin+badge)|[![README-Hits](https://caspersci.uk.to/cgi-bin/hits.cgi?q=tqdm&style=social&r=https://github.com/tqdm/tqdm&l=https://caspersci.uk.to/images/tqdm.png&f=https://raw.githubusercontent.com/tqdm/tqdm/master/images/logo.gif)](https://caspersci.uk.to/cgi-bin/hits.cgi?q=tqdm&a=plot&r=https://github.com/tqdm/tqdm&l=https://caspersci.uk.to/images/tqdm.png&f=https://raw.githubusercontent.com/tqdm/tqdm/master/images/logo.gif&style=social)-|-|-|-|-`tqdm` means "progress" in Arabic (taqadum, تقدّم) and is anabbreviation for "I love you so much" in Spanish (te quiero demasiado).Instantly make your loops show a smart progress meter - just wrap anyiterable with `tqdm(iterable)`, and you're done!<jupyter_code>from tqdm import tqdm
for i in tqdm(range(10000)):
    pass<jupyter_output>100%|██████████| 10000/10000 [00:00<00:00, 1541060.37it/s]<jupyter_text>`trange(N)` can be also used as a convenient shortcut for`tqdm(xrange(N))`.<jupyter_code>from tqdm import trange
for i in trange(10000, unit_scale=True, desc="hello", unit="epoch"):
    pass<jupyter_output>hello: 100%|██████████| 10.0k/10.0k [00:00<00:00, 1.56Mepoch/s]<jupyter_text>![Screenshot](https://raw.githubusercontent.com/tqdm/tqdm/master/images/tqdm.gif)It can also be executed as a module with pipes:<jupyter_code>! seq 9999999 | tqdm --bytes | wc -l<jupyter_output>75.2MB [00:00, 217MB/s]
9999999<jupyter_text>```sh7z a -bd -r backup.7z docs/ | grep Compressing | \    tqdm --total $(find docs/ -type f | wc -l) --unit files >> backup.log100%|███████████████████████████████▉| 8014/8014 [01:37<00:00, 82.29files/s]``` Overhead is low -- about 60ns per iteration (80ns with `tqdm_gui`), andis unit tested against performance regression. By comparison, thewell-established[ProgressBar](https://github.com/niltonvolpato/python-progressbar) hasan 800ns/iter overhead.In addition to its low overhead, `tqdm` uses smart algorithms to predictthe remaining time and to skip unnecessary iteration displays, whichallows for a negligible overhead in most cases.`tqdm` works on any platform (Linux, Windows, Mac, FreeBSD, NetBSD,Solaris/SunOS), in any console or in a GUI, and is also friendly withIPython/Jupyter notebooks.`tqdm` does not require any dependencies (not even `curses`!), justPython and an environment supporting `carriage return \r` and`line feed \n` control characters.--- Usage`tqdm` is very versatile and can be used in a number of ways.The three main ones are given below. Iterable-basedWrap `tqdm()` around any iterable:<jupyter_code>from tqdm import tqdm
import time
text = ""
for char in tqdm(["a", "b", "c", "d"]):
    time.sleep(0.25)
    text = text + char<jupyter_output>100%|██████████| 4/4 [00:01<00:00,  3.98it/s]<jupyter_text>`trange(i)` is a special optimised instance of `tqdm(range(i))`:<jupyter_code>from tqdm import trange

for i in trange(100):
    time.sleep(0.01)<jupyter_output>100%|██████████| 100/100 [00:01<00:00, 96.76it/s]<jupyter_text>Instantiation outside of the loop allows for manual control over `tqdm()`:<jupyter_code>pbar = tqdm(["a", "b", "c", "d"])
for char in pbar:
    time.sleep(0.25)
    pbar.set_description("Processing %s" % char)<jupyter_output>Processing d: 100%|██████████| 4/4 [00:01<00:00,  3.90it/s]<jupyter_text>ManualManual control on `tqdm()` updates by using a `with` statement:<jupyter_code>with tqdm(total=100) as pbar:
    for i in range(10):
        time.sleep(0.1)
        pbar.update(10)<jupyter_output>100%|██████████| 100/100 [00:01<00:00, 97.81it/s]<jupyter_text>If the optional variable `total` (or an iterable with `len()`) isprovided, predictive stats are displayed.`with` is also optional (you can just assign `tqdm()` to a variable,but in this case don't forget to `del` or `close()` at the end:<jupyter_code>pbar = tqdm(total=100)
for i in range(10):
    time.sleep(0.1)
    pbar.update(10)
pbar.close()<jupyter_output>100%|██████████| 100/100 [00:01<00:00, 97.85it/s]<jupyter_text>Module Perhaps the most wonderful use of ``tqdm`` is in a script or on the commandline. Simply inserting ``tqdm`` (or ``python3 -m tqdm``) between pipes will passthrough all ``stdin`` to ``stdout`` while printing progress to ``stderr``.The example below demonstrated counting the number of lines in all Python filesin the current directory, with timing information included.<jupyter_code>! time find . -name '*.py' -type f -exec cat \{} \; | wc -l
! time find . -name '*.py' -type f -exec cat \{} \; | tqdm | wc -l<jupyter_output>4043186it [00:05, 595767.35it/s]4.09user 1.23system 0:05.88elapsed 90%CPU (0avgtext+0avgdata 3092maxresident)k
20128inputs+0outputs (0major+904021minor)pagefaults 0swaps
4104300it [00:05, 698847.16it/s]
4104300<jupyter_text>Note that the usual arguments for `tqdm` can also be specified.<jupyter_code>! find . -name '*.py' -type f -exec cat \{} \; | tqdm --unit loc --unit_scale --total 4104300 >> /dev/null<jupyter_output>100%|█████████████████████████████████████| 4.10M/4.10M [00:05<00:00, 724kloc/s]<jupyter_text>Backing up a large directory? ```sh$ 7z a -bd -r backup.7z docs/ | grep Compressing |    tqdm --total $(find docs/ -type f | wc -l) --unit files >> backup.log100%|███████████████████████████████▉| 8014/8014 [01:37<00:00, 82.29files/s]```  Documentation<jupyter_code>tqdm?
! tqdm --help<jupyter_output>Extra CLI Options
        -----------------
        name  : type, optional
             TODO: find out why this is needed.
        delim  : chr, optional
            Delimiting character [default: '\n']. Use '\0' for null.
            N.B.: on Windows systems, Python converts '\n' to '\r\n'.
        buf_size  : int, optional
            String buffer size in bytes [default: 256]
            used when `delim` is specified.
        bytes  : bool, optional
            If true, will count bytes, ignore `delim`, and default
            `unit_scale` to True, `unit_divisor` to 1024, and `unit` to 'B'.
        manpath  : str, optional
            Directory in which to install tqdm man pages.
        log  : str, optional
            CRITICAL|FATAL|ERROR|WARN(ING)|[default: 'INFO']|DEBUG|NOTSET.<jupyter_text>Examples and Advance Usage- See the [examples](https://github.com/tqdm/tqdm/tree/master/examples)  folder;- import the module and run `help()`;- consult the [wiki](https://github.com/tqdm/tqdm/wiki)    - this has an      [excellent article](https://github.com/tqdm/tqdm/wiki/How-to-make-a-great-Progress-Bar)      on how to make a **great** progressbar, or- run this file! Description and additional statsCustom information can be displayed and updated dynamically on `tqdm` barswith the `desc` and `postfix` arguments:<jupyter_code>from tqdm import trange
from random import random, randint
from time import sleep

with trange(10) as t:
    for i in t:
        # Description will be displayed on the left
        t.set_description('GEN %i' % i)
        # Postfix will be displayed on the right,
        # formatted automatically based on argument's datatype
        t.set_postfix(loss=random(), gen=randint(1,999), str='h',
                      lst=[1, 2])
        sleep(0.1)

with tqdm(total=10, bar_format="{postfix[0]} {postfix[1][value]:>8.2g}",
          postfix=["Batch", dict(value=0)]) as t:
    for i in range(10):
        sleep(0.1)
        t.postfix[1]["value"] = i / 2
        t.update()<jupyter_output>GEN 9: 100%|██████████| 10/10 [00:01<00:00,  8.87it/s, gen=980, loss=0.762, lst=[1, 2], str=h]
Batch        4<jupyter_text>Points to remember when using ``{postfix[...]}`` in the ``bar_format`` string:- ``postfix`` also needs to be passed as an initial argument in a compatible  format, and- ``postfix`` will be auto-converted to a string if it is a ``dict``-like  object. To prevent this behaviour, insert an extra item into the dictionary  where the key is not a string.  Nested progress bars`tqdm` supports nested progress bars. Here's an example:<jupyter_code>from tqdm.auto import trange
from time import sleep

for i in trange(4, desc='1st loop'):
    for j in trange(5, desc='2nd loop'):
        for k in trange(50, desc='3nd loop', leave=False):
            sleep(0.01)<jupyter_output><empty_output><jupyter_text>On Windows [colorama](https://github.com/tartley/colorama) will be used ifavailable to keep nested bars on their respective lines.For manual control over positioning (e.g. for multi-threaded use),you may specify `position=n` where `n=0` for the outermost bar,`n=1` for the next, and so on:<jupyter_code>from time import sleep
from tqdm import trange, tqdm
from multiprocessing import Pool, freeze_support, RLock

L = list(range(9))

def progresser(n):
    interval = 0.001 / (n + 2)
    total = 5000
    text = "#{}, est. {:<04.2}s".format(n, interval * total)
    for i in trange(total, desc=text, position=n):
        sleep(interval)

if __name__ == '__main__':
    freeze_support()  # for Windows support
    p = Pool(len(L),
             # again, for Windows support
             initializer=tqdm.set_lock, initargs=(RLock(),))
    p.map(progresser, L)
    print("\n" * (len(L) - 2))<jupyter_output><empty_output><jupyter_text>Hooks and callbacks`tqdm` can easily support callbacks/hooks and manual updates.Here's an example with `urllib`:**urllib.urlretrieve documentation**> [...]> If present, the hook function will be called once> on establishment of the network connection and once after each block read> thereafter. The hook will be passed three arguments; a count of blocks> transferred so far, a block size in bytes, and the total size of the file.> [...]<jupyter_code>import urllib, os
from tqdm import tqdm

class TqdmUpTo(tqdm):
    """Provides `update_to(n)` which uses `tqdm.update(delta_n)`."""
    def update_to(self, b=1, bsize=1, tsize=None):
        """
        b  : int, optional
            Number of blocks transferred so far [default: 1].
        bsize  : int, optional
            Size of each block (in tqdm units) [default: 1].
        tsize  : int, optional
            Total size (in tqdm units). If [default: None] remains unchanged.
        """
        if tsize is not None:
            self.total = tsize
        self.update(b * bsize - self.n)  # will also set self.n = b * bsize

eg_link = "https://caspersci.uk.to/matryoshka.zip"
with TqdmUpTo(unit='B', unit_scale=True, miniters=1,
              desc=eg_link.split('/')[-1]) as t:  # all optional kwargs
    urllib.urlretrieve(eg_link, filename=os.devnull,
                       reporthook=t.update_to, data=None)<jupyter_output>matryoshka.zip: 262kB [00:00, 2.26MB/s]<jupyter_text>Inspired by [twine242](https://github.com/pypa/twine/pull/242).Functional alternative in[examples/tqdm_wget.py](https://github.com/tqdm/tqdm/blob/master/examples/tqdm_wget.py).It is recommend to use `miniters=1` whenever there is potentiallylarge differences in iteration speed (e.g. downloading a file overa patchy connection).  Pandas IntegrationDue to popular demand we've added support for `pandas` -- here's an examplefor `DataFrame.progress_apply` and `DataFrameGroupBy.progress_apply`:<jupyter_code>import pandas as pd
import numpy as np
from tqdm import tqdm

df = pd.DataFrame(np.random.randint(0, 100, (100000, 6)))

# Register `pandas.progress_apply` and `pandas.Series.map_apply` with `tqdm`
# (can use `tqdm_gui`, `tqdm_notebook`, optional kwargs, etc.)
tqdm.pandas(desc="my bar!")

# Now you can use `progress_apply` instead of `apply`
# and `progress_map` instead of `map`
df.progress_apply(lambda x: x**2)
# can also groupby:
# df.groupby(0).progress_apply(lambda x: x**2)<jupyter_output>my bar!: 100%|██████████| 6/6 [00:00<00:00, 193.96it/s]<jupyter_text>In case you're interested in how this works (and how to modify it for yourown callbacks), see the[examples](https://github.com/tqdm/tqdm/tree/master/examples)folder or import the module and run `help()`. IPython/Jupyter IntegrationIPython/Jupyter is supported via the `tqdm_notebook` submodule:<jupyter_code>from tqdm import tnrange, tqdm_notebook
from time import sleep

for i in tnrange(3, desc='1st loop'):
    for j in tqdm_notebook(range(100), desc='2nd loop'):
        sleep(0.01)<jupyter_output><empty_output><jupyter_text>In addition to `tqdm` features, the submodule provides a native Jupyterwidget (compatible with IPython v1-v4 and Jupyter), fully working nested barsand colour hints (blue: normal, green: completed, red: error/interrupt,light blue: no ETA); as demonstrated below.![Screenshot-Jupyter3](https://raw.githubusercontent.com/tqdm/tqdm/master/images/tqdm-jupyter-3.gif)It is also possible to let `tqdm` automatically choose betweenconsole or notebook versions by using the `autonotebook` submodule:<jupyter_code>from tqdm.autonotebook import tqdm
tqdm.pandas()<jupyter_output><empty_output><jupyter_text>Note that this will issue a `TqdmExperimentalWarning` if run in a notebooksince it is not meant to be possible to distinguish between `jupyter notebook`and `jupyter console`. Use `auto` instead of `autonotebook` to suppressthis warning. Writing messagesSince `tqdm` uses a simple printing mechanism to display progress bars,you should not write any message in the terminal using `print()` whilea progressbar is open.To write messages in the terminal without any collision with `tqdm` bardisplay, a `.write()` method is provided:<jupyter_code>from tqdm import tqdm, trange
from time import sleep

bar = trange(10)
for i in bar:
    # Print using tqdm class method .write()
    sleep(0.1)
    if not (i % 3):
        tqdm.write("Done task %i" % i)
    # Can also use bar.write()<jupyter_output><empty_output><jupyter_text>By default, this will print to standard output `sys.stdout`. but you canspecify any file-like object using the `file` argument. For example, thiscan be used to redirect the messages writing to a log file or class.---  Do your own experiments here 👇Try `tqdm` youself by adding your code below and running your own experiments.<jupyter_code>import tqdm

# your code here
tqdm.<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1523516096579477, 'fraction_numerical': 0.03822937625754527, 'mean_word_length': 4.461881868131868, 'pattern_counts': {'":': 0, '<': 80, '<?xml version=': 0, '>': 80, 'https://': 49, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '450399', 'license': 'MIT', 'n_tokens_mistral': 6290, 'n_tokens_neox': 5689, 'n_words': 1429, 'path': 'DEMO.ipynb', 'repo_name': 'flying-sheep/tqdm', 'size': '15904'}