<commit_before>.. testsetup:: *

    from pytorch_lightning.trainer.trainer import Trainer
    from pytorch_lightning.core.lightning import LightningModule

Learning Rate Finder
--------------------

For training deep neural networks, selecting a good learning rate is essential
for both better performance and faster convergence. Even optimizers such as
`Adam` that are self-adjusting the learning rate can benefit from more optimal
choices.

To reduce the amount of guesswork concerning choosing a good initial learning
rate, a `learning rate finder` can be used. As described in this `paper <https://arxiv.org/abs/1506.01186>`_ 
a learning rate finder does a small run where the learning rate is increased 
after each processed batch and the corresponding loss is logged. The result of 
this is a `lr` vs. `loss` plot that can be used as guidance for choosing a optimal
initial lr. 

Warnings:
- For the moment, this feature only works with models having a single optimizer.
- LR support for DDP is not implemented yet, it is comming soon.

Using Lightnings build-in LR finder
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

In the most basic use case, this feature can be enabled during trainer construction
with ``Trainer(auto_lr_find=True)``. When ``.fit(model)`` is called, the lr finder
will automatically be run before any training is done. The ``lr`` that is found
and used will be written to the console and logged together with all other
hyperparameters of the model.
    
.. testcode::
        
    # default, no automatic learning rate finder
    trainer = Trainer(auto_lr_find=True)

When the ``lr`` or ``learning_rate`` key in hparams exists, this flag sets your learning_rate.
In both cases, if the respective fields are not found, an error will be thrown.
        
.. testcode::

    class LitModel(LightningModule):

        def __init__(self, hparams):
            self.hparams = hparams

        def configure_optimizers(self):
            return Adam(self.parameters(), lr=self.hparams.lr|self.hparams.learning_rate)

    # finds learning rate automatically
    # sets hparams.lr or hparams.learning_rate to that learning rate
    trainer = Trainer(auto_lr_find=True)<commit_msg>Fixed the default value of auto_lr_find in docs (#1854)<commit_after>.. testsetup:: *

    from pytorch_lightning.trainer.trainer import Trainer
    from pytorch_lightning.core.lightning import LightningModule

Learning Rate Finder
--------------------

For training deep neural networks, selecting a good learning rate is essential
for both better performance and faster convergence. Even optimizers such as
`Adam` that are self-adjusting the learning rate can benefit from more optimal
choices.

To reduce the amount of guesswork concerning choosing a good initial learning
rate, a `learning rate finder` can be used. As described in this `paper <https://arxiv.org/abs/1506.01186>`_ 
a learning rate finder does a small run where the learning rate is increased 
after each processed batch and the corresponding loss is logged. The result of 
this is a `lr` vs. `loss` plot that can be used as guidance for choosing a optimal
initial lr. 

Warnings:
- For the moment, this feature only works with models having a single optimizer.
- LR support for DDP is not implemented yet, it is comming soon.

Using Lightning's built-in LR finder
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

In the most basic use case, this feature can be enabled during trainer construction
with ``Trainer(auto_lr_find=True)``. When ``.fit(model)`` is called, the LR finder
will automatically be run before any training is done. The ``lr`` that is found
and used will be written to the console and logged together with all other
hyperparameters of the model.
    
.. testcode::
        
    # default: no automatic learning rate finder
    trainer = Trainer(auto_lr_find=False)

When the ``lr`` or ``learning_rate`` key in hparams exists, this flag sets your learning_rate.
In both cases, if the respective fields are not found, an error will be thrown.
        
.. testcode::

    class LitModel(LightningModule):

        def __init__(self, hparams):
            self.hparams = hparams

        def configure_optimizers(self):
            return Adam(self.parameters(), lr=self.hparams.lr|self.hparams.learning_rate)

    # finds learning rate automatically
    # sets hparams.lr or hparams.learning_rate to that learning rate
    trainer = Trainer(auto_lr_find=True)
{'fraction_non_alphanumeric': 0.0847457627118644, 'fraction_numerical': 0.004971751412429379, 'mean_word_length': 3.7952329360780066, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7216109', 'n_tokens_mistral': 1201, 'n_tokens_neox': 1147, 'n_words': 551}