<jupyter_start><jupyter_text>tqdm[![Py-Versions](https://img.shields.io/pypi/pyversions/tqdm.svg?logo=python&logoColor=white)](https://pypi.org/project/tqdm)|[![Versions](https://img.shields.io/pypi/v/tqdm.svg)](https://tqdm.github.io/releases)|[![Conda-Forge-Status](https://img.shields.io/conda/v/conda-forge/tqdm.svg?label=conda-forge&logo=conda-forge)](https://anaconda.org/conda-forge/tqdm)|[![Docker](https://img.shields.io/badge/docker-pull-blue.svg?logo=docker&logoColor=white)](https://hub.docker.com/r/tqdm/tqdm)|[![Snapcraft](https://img.shields.io/badge/snap-install-82BEA0.svg?logo=snapcraft)](https://snapcraft.io/tqdm)-|-|-|-|-[![Build-Status](https://img.shields.io/github/workflow/status/tqdm/tqdm/Test/master?logo=GitHub)](https://github.com/tqdm/tqdm/actions?query=workflow%3ATest)|[![Coverage-Status](https://img.shields.io/coveralls/github/tqdm/tqdm/master?logo=coveralls)](https://coveralls.io/github/tqdm/tqdm)|[![Branch-Coverage-Status](https://codecov.io/gh/tqdm/tqdm/branch/master/graph/badge.svg)](https://codecov.io/gh/tqdm/tqdm)|[![Codacy-Grade](https://app.codacy.com/project/badge/Grade/3f965571598f44549c7818f29cdcf177)](https://www.codacy.com/gh/tqdm/tqdm/dashboard)|[![Libraries-Rank](https://img.shields.io/librariesio/sourcerank/pypi/tqdm.svg?logo=koding&logoColor=white)](https://libraries.io/pypi/tqdm)|[![PyPI-Downloads](https://img.shields.io/pypi/dm/tqdm.svg?label=pypi%20downloads&logo=PyPI&logoColor=white)](https://pepy.tech/project/tqdm)-|-|-|-|-|-[![DOI](https://img.shields.io/badge/DOI-10.5281/zenodo.595120-blue.svg)](https://doi.org/10.5281/zenodo.595120)|[![LICENCE](https://img.shields.io/pypi/l/tqdm.svg)](https://raw.githubusercontent.com/tqdm/tqdm/master/LICENCE)|[![OpenHub-Status](https://www.openhub.net/p/tqdm/widgets/project_thin_badge?format=gif)](https://www.openhub.net/p/tqdm?ref=Thin+badge)|[![binder-demo](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/tqdm/tqdm/master?filepath=DEMO.ipynb)|[![awesome-python](https://awesome.re/mentioned-badge.svg)](https://github.com/vinta/awesome-python)-|-|-|-|-`tqdm` derives from the Arabic word *taqaddum* (تقدّم) which can mean"progress," and is an abbreviation for "I love you so much" in Spanish(*te quiero demasiado*).Instantly make your loops show a smart progress meter - just wrap anyiterable with `tqdm(iterable)`, and you're done!<jupyter_code>from tqdm import tqdm
for i in tqdm(range(10000)):
    pass<jupyter_output><empty_output><jupyter_text>`trange(N)` can be also used as a convenient shortcut for`tqdm(xrange(N))`.<jupyter_code>from tqdm import trange
for i in trange(10000, unit_scale=True, desc="hello", unit="epoch"):
    pass<jupyter_output><empty_output><jupyter_text>![Screenshot](https://img.tqdm.ml/tqdm.gif)|[![Video](https://img.tqdm.ml/video.jpg)](https://tqdm.github.io/video) [![Slides](https://img.tqdm.ml/slides.jpg)](https://tqdm.github.io/PyData2019/slides.html)-|-It can also be executed as a module with pipes:<jupyter_code>! seq 9999999 | tqdm --bytes | wc -l<jupyter_output><empty_output><jupyter_text>```shtar -zcf - docs/ | tqdm --bytes --total `du -sb docs/ | cut -f1` > backup.tgz 44%|██████████████▊                   | 153M/352M [00:14<00:18, 11.0MB/s]``` Overhead is low -- about 60ns per iteration (80ns with `tqdm.gui`), andis unit tested against performance regression. By comparison, thewell-established[ProgressBar](https://github.com/niltonvolpato/python-progressbar) hasan 800ns/iter overhead.In addition to its low overhead, `tqdm` uses smart algorithms to predictthe remaining time and to skip unnecessary iteration displays, whichallows for a negligible overhead in most cases.`tqdm` works on any platform (Linux, Windows, Mac, FreeBSD, NetBSD,Solaris/SunOS), in any console or in a GUI, and is also friendly withIPython/Jupyter notebooks.`tqdm` does not require any dependencies (not even `curses`!), justPython and an environment supporting `carriage return \r` and`line feed \n` control characters.--- Usage`tqdm` is very versatile and can be used in a number of ways.The three main ones are given below. Iterable-basedWrap `tqdm()` around any iterable:<jupyter_code>from tqdm import tqdm
from time import sleep
text = ""
for char in tqdm(["a", "b", "c", "d"]):
    sleep(0.25)
    text = text + char<jupyter_output><empty_output><jupyter_text>`trange(i)` is a special optimised instance of `tqdm(range(i))`:<jupyter_code>from tqdm import trange

for i in trange(100):
    sleep(0.01)<jupyter_output><empty_output><jupyter_text>Instantiation outside of the loop allows for manual control over `tqdm()`:<jupyter_code>pbar = tqdm(["a", "b", "c", "d"])
for char in pbar:
    sleep(0.25)
    pbar.set_description("Processing %s" % char)<jupyter_output><empty_output><jupyter_text>ManualManual control of `tqdm()` updates using a `with` statement:<jupyter_code>with tqdm(total=100) as pbar:
    for i in range(10):
        sleep(0.1)
        pbar.update(10)<jupyter_output><empty_output><jupyter_text>If the optional variable `total` (or an iterable with `len()`) isprovided, predictive stats are displayed.`with` is also optional (you can just assign `tqdm()` to a variable,but in this case don't forget to `del` or `close()` at the end:<jupyter_code>pbar = tqdm(total=100)
for i in range(10):
    sleep(0.1)
    pbar.update(10)
pbar.close()<jupyter_output><empty_output><jupyter_text>Module Perhaps the most wonderful use of `tqdm` is in a script or on thecommand line. Simply inserting `tqdm` (or `python -m tqdm`) betweenpipes will pass through all `stdin` to `stdout` while printing progressto `stderr`.The example below demonstrated counting the number of lines in allPython files in the current directory, with timing information included.<jupyter_code>! time find . -name '*.py' -type f -exec cat \{} \; | wc -l
! time find . -name '*.py' -type f -exec cat \{} \; | tqdm | wc -l<jupyter_output><empty_output><jupyter_text>Note that the usual arguments for `tqdm` can also be specified.<jupyter_code>! find . -name '*.py' -type f -exec cat \{} \; | tqdm --unit loc --unit_scale --total 4104300 >> /dev/null<jupyter_output><empty_output><jupyter_text>Backing up a large directory?```shtar -zcf - docs/ | tqdm --bytes --total `du -sb docs/ | cut -f1` > backup.tgz 44%|██████████████▊                   | 153M/352M [00:14<00:18, 11.0MB/s]```This can be beautified further:```shBYTES="$(du -sb docs/ | cut -f1)"tar -cf - docs/ \  | tqdm --bytes --total "$BYTES" --desc Processing | gzip \  | tqdm --bytes --total "$BYTES" --desc Compressed --position 1 \  > ~/backup.tgzProcessing: 100%|██████████████████████| 352M/352M [00:14<00:00, 30.2MB/s]Compressed:  42%|█████████▎            | 148M/352M [00:14<00:19, 10.9MB/s]```Or done on a file level using 7-zip:```sh7z a -bd -r backup.7z docs/ | grep Compressing \  | tqdm --total $(find docs/ -type f | wc -l) --unit files \  | grep -v Compressing100%|██████████████████████████▉| 15327/15327 [01:00<00:00, 712.96files/s]```  Documentation<jupyter_code>tqdm?
! tqdm --help<jupyter_output><empty_output><jupyter_text>Examples and Advance Usage- See the [examples](https://github.com/tqdm/tqdm/tree/master/examples)  folder;- import the module and run `help()`;- consult the [wiki](https://github.com/tqdm/tqdm/wiki)    - this has an      [excellent article](https://github.com/tqdm/tqdm/wiki/How-to-make-a-great-Progress-Bar)      on how to make a **great** progressbar;- check out the [slides from PyData London](https://tqdm.github.io/PyData2019/slides.html), or- run this file! Description and additional statsCustom information can be displayed and updated dynamically on `tqdm` barswith the `desc` and `postfix` arguments:<jupyter_code>from tqdm import tqdm, trange
from random import random, randint
from time import sleep

with trange(10) as t:
    for i in t:
        # Description will be displayed on the left
        t.set_description('GEN %i' % i)
        # Postfix will be displayed on the right,
        # formatted automatically based on argument's datatype
        t.set_postfix(loss=random(), gen=randint(1,999), str='h',
                      lst=[1, 2])
        sleep(0.1)

with tqdm(total=10, bar_format="{postfix[0]} {postfix[1][value]:>8.2g}",
          postfix=["Batch", dict(value=0)]) as t:
    for i in range(10):
        sleep(0.1)
        t.postfix[1]["value"] = i / 2
        t.update()<jupyter_output><empty_output><jupyter_text>Points to remember when using `{postfix[...]}` in the `bar_format` string:- `postfix` also needs to be passed as an initial argument in a  compatible format, and- `postfix` will be auto-converted to a string if it is a `dict`-like  object. To prevent this behaviour, insert an extra item into the  dictionary where the key is not a string.Additional `bar_format` parameters may also be defined by overriding`format_dict`, and the bar itself may be modified using `ascii`:<jupyter_code>from tqdm import tqdm
from time import sleep

class TqdmExtraFormat(tqdm):
    """Provides a `total_time` format parameter"""
    @property
    def format_dict(self):
        d = super(TqdmExtraFormat, self).format_dict
        total_time = d["elapsed"] * (d["total"] or 0) / max(d["n"], 1)
        d.update(total_time=self.format_interval(total_time) + " in total")
        return d

for i in TqdmExtraFormat(
      range(9), ascii=" .oO0",
      bar_format="{total_time}: {percentage:.0f}%|{bar}{r_bar}"):
    if i == 4:
        break<jupyter_output><empty_output><jupyter_text>Note that `{bar}` also supports a format specifier `[width][type]`.- `width`    + unspecified (default): automatic to fill `ncols`    + `int >= 0`: fixed width overriding `ncols` logic    + `int < 0`: subtract from the automatic default- `type`    + `a`: ascii (`ascii=True` override)    + `u`: unicode (`ascii=False` override)    + `b`: blank (`ascii="  "` override)This means a fixed bar with right-justified text may be created byusing: `bar_format="{l_bar}{bar:10}|{bar:-10b}right-justified"`  Nested progress bars`tqdm` supports nested progress bars. Here's an example:<jupyter_code>from tqdm.auto import trange
from time import sleep

for i in trange(4, desc='1st loop'):
    for j in trange(5, desc='2nd loop'):
        for k in trange(50, desc='3rd loop', leave=False):
            sleep(0.01)<jupyter_output><empty_output><jupyter_text>On Windows [colorama](https://github.com/tartley/colorama) will be usedif available to keep nested bars on their respective lines.For manual control over positioning (e.g. for multi-processing use),you may specify `position=n` where `n=0` for the outermost bar, `n=1`for the next, and so on. However, it's best to check if tqdm can workwithout manual position first.```pythonfrom time import sleepfrom tqdm import trange, tqdmfrom multiprocessing import Pool, RLock, freeze_supportL = list(range(9))def progresser(n):    interval = 0.001 / (n + 2)    total = 5000    text = "{}, est. {:<04.2}s".format(n, interval * total)    for _ in trange(total, desc=text, position=n):        sleep(interval)if __name__ == '__main__':    freeze_support()   for Windows support    tqdm.set_lock(RLock())   for managing output contention    p = Pool(initializer=tqdm.set_lock, initargs=(tqdm.get_lock(),))    p.map(progresser, L)```Note that in Python 3, `tqdm.write` is thread-safe:```pythonfrom time import sleepfrom tqdm import tqdm, trangefrom concurrent.futures import ThreadPoolExecutorL = list(range(9))def progresser(n):    interval = 0.001 / (n + 2)    total = 5000    text = "{}, est. {:<04.2}s".format(n, interval * total)    for _ in trange(total, desc=text):        sleep(interval).auto    if n == 6:        tqdm.write("n == 6 completed.")        tqdm.write("`tqdm.write()` is thread-safe in py3!")if __name__ == '__main__':    with ThreadPoolExecutor() as p:        p.map(progresser, L)```  Hooks and callbacks`tqdm` can easily support callbacks/hooks and manual updates.Here's an example with `urllib`:**`urllib.urlretrieve` documentation**> [...]> If present, the hook function will be called once> on establishment of the network connection and once after each block read> thereafter. The hook will be passed three arguments; a count of blocks> transferred so far, a block size in bytes, and the total size of the file.> [...]<jupyter_code>import urllib, os
from tqdm import tqdm
urllib = getattr(urllib, 'request', urllib)

class TqdmUpTo(tqdm):
    """Provides `update_to(n)` which uses `tqdm.update(delta_n)`."""
    def update_to(self, b=1, bsize=1, tsize=None):
        """
        b  : int, optional
            Number of blocks transferred so far [default: 1].
        bsize  : int, optional
            Size of each block (in tqdm units) [default: 1].
        tsize  : int, optional
            Total size (in tqdm units). If [default: None] remains unchanged.
        """
        if tsize is not None:
            self.total = tsize
        return self.update(b * bsize - self.n)  # also sets self.n = b * bsize

eg_link = "https://caspersci.uk.to/matryoshka.zip"
with TqdmUpTo(unit='B', unit_scale=True, unit_divisor=1024, miniters=1,
              desc=eg_link.split('/')[-1]) as t:  # all optional kwargs
    urllib.urlretrieve(eg_link, filename=os.devnull,
                       reporthook=t.update_to, data=None)
    t.total = t.n<jupyter_output><empty_output><jupyter_text>Inspired by [twine242](https://github.com/pypa/twine/pull/242).Functional alternative in[examples/tqdm_wget.py](https://github.com/tqdm/tqdm/blob/master/examples/tqdm_wget.py).It is recommend to use `miniters=1` whenever there is potentially largedifferences in iteration speed (e.g. downloading a file over a patchyconnection).**Wrapping read/write methods**To measure throughput through a file-like object's `read` or `write`methods, use `CallbackIOWrapper`:```pythonfrom tqdm import tqdmfrom tqdm.utils import CallbackIOWrapperwith tqdm(total=file_obj.size,          unit='B', unit_scale=True, unit_divisor=1024) as t:    fobj = CallbackIOWrapper(t.update, file_obj, "read")    while True:        chunk = fobj.read(chunk_size)        if not chunk:            break    t.reset()     ... continue to use `t` for something else```Alternatively, use the even simpler `wrapattr` convenience function,which would condense both the `urllib` and `CallbackIOWrapper` examplesdown to:<jupyter_code>import urllib, os
from tqdm import tqdm

eg_link = "https://caspersci.uk.to/matryoshka.zip"
response = getattr(urllib, 'request', urllib).urlopen(eg_link)
with tqdm.wrapattr(open(os.devnull, "wb"), "write",
                   miniters=1, desc=eg_link.split('/')[-1],
                   total=getattr(response, 'length', None)) as fout:
    for chunk in response:
        fout.write(chunk)<jupyter_output><empty_output><jupyter_text>The `requests` equivalent is nearly identical:<jupyter_code>import requests, os
from tqdm import tqdm

eg_link = "https://caspersci.uk.to/matryoshka.zip"
response = requests.get(eg_link, stream=True)
with tqdm.wrapattr(open(os.devnull, "wb"), "write",
                   miniters=1, desc=eg_link.split('/')[-1],
                   total=int(response.headers.get('content-length', 0))) as fout:
    for chunk in response.iter_content(chunk_size=4096):
        fout.write(chunk)<jupyter_output><empty_output><jupyter_text>Pandas IntegrationDue to popular demand we've added support for `pandas` -- here's an examplefor `DataFrame.progress_apply` and `DataFrameGroupBy.progress_apply`:<jupyter_code>import pandas as pd
import numpy as np
from tqdm import tqdm

df = pd.DataFrame(np.random.randint(0, 100, (100000, 6)))

# Register `pandas.progress_apply` and `pandas.Series.map_apply` with `tqdm`
# (can use `tqdm.gui.tqdm`, `tqdm.notebook.tqdm`, optional kwargs, etc.)
tqdm.pandas(desc="my bar!")

# Now you can use `progress_apply` instead of `apply`
# and `progress_map` instead of `map`
df.progress_apply(lambda x: x**2)
# can also groupby:
# df.groupby(0).progress_apply(lambda x: x**2)<jupyter_output><empty_output><jupyter_text>In case you're interested in how this works (and how to modify it foryour own callbacks), see the[examples](https://github.com/tqdm/tqdm/tree/master/examples) folder orimport the module and run `help()`. Keras Integration```pythonA `keras` callback is also available:from tqdm.keras import TqdmCallback...model.fit(..., verbose=0, callbacks=[TqdmCallback()])``` IPython/Jupyter IntegrationIPython/Jupyter is supported via the `tqdm.notebook` submodule:<jupyter_code>from tqdm.notebook import trange, tqdm
from time import sleep

for i in trange(3, desc='1st loop'):
    for j in tqdm(range(100), desc='2nd loop'):
        sleep(0.01)<jupyter_output><empty_output><jupyter_text>In addition to `tqdm` features, the submodule provides a native Jupyterwidget (compatible with IPython v1-v4 and Jupyter), fully working nestedbars and colour hints (blue: normal, green: completed, red:error/interrupt, light blue: no ETA); as demonstrated below.![Screenshot-Jupyter3](https://img.tqdm.ml/jupyter-3.gif)The `notebook` version supports percentage or pixels for overall width(e.g.: `ncols='100%'` or `ncols='480px'`).It is also possible to let `tqdm` automatically choose between consoleor notebook versions by using the `autonotebook` submodule:<jupyter_code>from tqdm.autonotebook import tqdm
tqdm.pandas()<jupyter_output><empty_output><jupyter_text>Note that this will issue a `TqdmExperimentalWarning` if run in anotebook since it is not meant to be possible to distinguish between`jupyter notebook` and `jupyter console`. Use `auto` instead of`autonotebook` to suppress this warning.Note that notebooks will display the bar in the cell where it wascreated. This may be a different cell from the one where it is used. Ifthis is not desired, the creation of the bar must be delayed/moved tothe cell where it is desired to be displayed.Another possibility is to have a single bar (near the top of thenotebook) which is constantly re-used (using `reset()` rather than`close()`). For this reason, the notebook version (unlike the CLIversion) does not automatically call `close()` upon `Exception`.<jupyter_code>from tqdm.notebook import tqdm
pbar = tqdm()
# different cell
iterable = range(100)
pbar.reset(total=len(iterable))  # initialise with new `total`
for i in iterable:
    pbar.update()
pbar.refresh()  # force print final status but don't `close()`<jupyter_output><empty_output><jupyter_text>Writing messagesThis is a work in progress (see[737](https://github.com/tqdm/tqdm/issues/737)).Since `tqdm` uses a simple printing mechanism to display progress bars,you should not write any message in the terminal using `print()` while aprogressbar is open.To write messages in the terminal without any collision with `tqdm` bardisplay, a `.write()` method is provided:<jupyter_code>from tqdm.auto import tqdm, trange
from time import sleep

bar = trange(10)
for i in bar:
    # Print using tqdm class method .write()
    sleep(0.1)
    if not (i % 3):
        tqdm.write("Done task %i" % i)
    # Can also use bar.write()<jupyter_output><empty_output><jupyter_text>By default, this will print to standard output `sys.stdout`. but you canspecify any file-like object using the `file` argument. For example,this can be used to redirect the messages writing to a log file or class.[![README-Hits](https://caspersci.uk.to/cgi-bin/hits.cgi?q=tqdm&style=social&r=https://github.com/tqdm/tqdm&l=https://img.tqdm.ml/favicon.png&f=https://img.tqdm.ml/logo.gif)](https://caspersci.uk.to/cgi-bin/hits.cgi?q=tqdm&a=plot&r=https://github.com/tqdm/tqdm&l=https://img.tqdm.ml/favicon.png&f=https://img.tqdm.ml/logo.gif&style=social)|(Since 19 May 2016)-|-  Do your own experiments here 👇Try `tqdm` youself by adding your code below and running your own experiments.<jupyter_code>import tqdm

# your code here
tqdm.<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14384012142676447, 'fraction_numerical': 0.019984821654439665, 'mean_word_length': 4.589932126696833, 'pattern_counts': {'":': 0, '<': 101, '<?xml version=': 0, '>': 106, 'https://': 59, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '23345', 'license': 'MIT', 'n_tokens_mistral': 7283, 'n_tokens_neox': 6777, 'n_words': 1784, 'path': 'DEMO.ipynb', 'repo_name': 'MatthieuBizien/tqdm', 'size': '19765'}