a d09@sddlmZmZmZmZddlmZmZmZddl Z ddl m Z ddl mZddlmZddlmZddlmZdd lmZdd lmZdd lmZmZGd d d eZGdddZddeeedfeeeeedddddZdS))AnyDictOptionalUnion)Literal NotRequired TypedDictN)BatchSizeFinder)Callback)LearningRateFinder)LightningDataModule) TrainerStatus) _LRFinder)MisconfigurationException)EVAL_DATALOADERSTRAIN_DATALOADERSc@s.eZdZUeeeed<eeeed<dS) _TunerResultlr_findscale_batch_sizeN)__name__ __module__ __qualname__rrr__annotations__intrrg/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/pytorch_lightning/tuner/tuning.pyrs rc@s6eZdZdZdddddZeeefeeefdddd Zd%d e ee e fe e e e e e e e eefe e eefed ed ddZeeddddZd&d e ee dfe e e e e ded eeeeee ed ddZd'd e ee dfe e e e e ded eeeeeee ed" d#d$ZdS)(TunerzTuner class to tune your model. pl.TrainerN)trainerreturncCs ||_dSN)r)selfrrrr__init__%szTuner.__init__) auto_lr_findauto_scale_batch_sizercCs||j_||j_dSr )rr#r$)r!r#r$rrron_trainer_init(szTuner.on_trainer_initfitzpl.LightningModuler&validatetestpredict) modeltrain_dataloadersval_dataloaders dataloaders datamodulescale_batch_size_kwargslr_find_kwargsmethodrc Cs0|pi}|pi}t} |jj||jj} |jjjrP| rPtd|jjjd|jjrt |jjt rv| d|jj|j ||||||fi|| d<|jj r,tj|jj_t |tr|}d}|dus|dur|durtd|jjj||||d| dd|j||||||fi|| d <tj|jj_| S) NzC`trainer.tune()` is currently not supported with `Trainer(strategy=z)`.moderzlYou cannot pass `train_dataloader` or `val_dataloaders` to `trainer.tune()` if datamodule is already passed.)r,r-r/ update_attrTr)rrZstrategyconnectr$Z_accelerator_connectorZis_distributedrZ strategy_name isinstancestr setdefaultrr#r RUNNINGstatestatusr Z_data_connectorZ attach_datarFINISHED) r!r+r,r-r.r/r0r1r2resultZ is_tuningrrr_tune,sN          z Tuner._tune)argskwargsrcOs*tj|jj_d|j_|jj|i|dS)z[`_run` wrapper to set the proper state during tuning, as this can be called multiple times.TN)r r9rr:r;Ztraining_run)r!r?r@rrrrAls z Tuner._runpower batch_sizepl.LightningDataModule) r+r,r-r.r/r2r3steps_per_trialinit_val max_trialsbatch_arg_namerc st|j||||t||| | | dd_g|jj|j_|dkrX|j||||nR|dkrt|jj|||dn6|dkr|jj|||dn|dkr|jj|||dfdd |jjD|j_d |j_ j S) aIteratively try to find the largest batch size for a given model that does not give an out of memory (OOM) error. Args: model: Model to tune. train_dataloaders: A collection of :class:`torch.utils.data.DataLoader` or a :class:`~pytorch_lightning.core.datamodule.LightningDataModule` specifying training samples. In the case of multiple dataloaders, please see this :ref:`section `. val_dataloaders: A :class:`torch.utils.data.DataLoader` or a sequence of them specifying validation samples. dataloaders: A :class:`torch.utils.data.DataLoader` or a sequence of them specifying val/test/predict samples used for running tuner on validation/testing/prediction. datamodule: An instance of :class:`~pytorch_lightning.core.datamodule.LightningDataModule`. method: Method to run tuner on. It can be any of ``("fit", "validate", "test", "predict")``. mode: Search strategy to update the batch size: - ``'power'``: Keep multiplying the batch size by 2, until we get an OOM error. - ``'binsearch'``: Initially keep multiplying by 2 and after encountering an OOM error do a binary search between the last successful batch size and the batch size that failed. steps_per_trial: number of steps to run with a given batch size. Ideally 1 should be enough to test if an OOM error occurs, however in practise a few are needed init_val: initial batch size to start the search with max_trials: max number of increases in batch size done before algorithm is terminated batch_arg_name: name of the attribute that stores the batch size. It is expected that the user has provided a model or datamodule that has a hyperparameter with that name. We will look for this attribute name in the following places - ``model`` - ``model.hparams`` - ``trainer.datamodule`` (the datamodule passed to the tune method) )r3rHrIrJrKTr&r()r/r)r*csg|]}|ur|qSrr.0cbZbatch_size_finderrr z*Tuner.scale_batch_size..F) _check_tuner_configurationrr _early_exit callbacksr&r(r)r*r$Zoptimal_batch_size) r!r+r,r-r.r/r2r3rHrIrJrKrrOrrrs*8zTuner.scale_batch_size:0yE>d exponential@F) r+r,r-r.r/r2min_lrmax_lr num_trainingr3early_stop_thresholdr4rc s|dkrtdt|j||||t||| | | | dd_g|jj|j_|j||||fdd|jjD|j_d|j_jS)aYEnables the user to do a range test of good initial learning rates, to reduce the amount of guesswork in picking a good starting learning rate. Args: model: Model to tune. train_dataloaders: A collection of :class:`torch.utils.data.DataLoader` or a :class:`~pytorch_lightning.core.datamodule.LightningDataModule` specifying training samples. In the case of multiple dataloaders, please see this :ref:`section `. val_dataloaders: A :class:`torch.utils.data.DataLoader` or a sequence of them specifying validation samples. dataloaders: A :class:`torch.utils.data.DataLoader` or a sequence of them specifying val/test/predict samples used for running tuner on validation/testing/prediction. datamodule: An instance of :class:`~pytorch_lightning.core.datamodule.LightningDataModule`. min_lr: minimum learning rate to investigate max_lr: maximum learning rate to investigate num_training: number of learning rates to test mode: Search strategy to update learning rate after each batch: - ``'exponential'``: Increases the learning rate exponentially. - ``'linear'``: Increases the learning rate linearly. early_stop_threshold: Threshold for stopping the search. If the loss at any point is larger than early_stop_threshold*best_loss then the search is stopped. To disable, set to None. update_attr: Whether to update the learning rate attribute or not. Raises: MisconfigurationException: If learning rate/lr in ``model`` or ``model.hparams`` isn't overridden when ``auto_lr_find=True``, or if you are using more than one optimizer. r&z:method='fit' is an invalid configuration to run lr finder.)rZr[Znum_training_stepsr3r]r4Tcsg|]}|ur|qSrrrLZlr_finder_callbackrrrP rQz!Tuner.lr_find..F) rrRrr rSrTr&r#Z optimal_lr) r!r+r,r-r.r/r2rZr[r\r3r]r4rr^rrs"6 z Tuner.lr_find)NNNNNNr&) NNNNr&rBrCrDrErF) NNNNr&rUrVrWrXrYF)rrr__doc__r"rr7boolr%rrr rrrrrr>rArrfloatrrrrrrr"s" @  Urr&rrGr')rr,r-r.r2rcCsd}||vr"td|d|d|dkrD|durdtd|dn |dusT|durdtd|d g}|jD]:}t|tr|jr|d qnt|trn|jrn|d qnt |d krtd |ddnt |dkrtddS)Nr'zmethod z is invalid. Should be one of .r&zIn tuner with method=zs, `dataloaders` argument should be None, please consider setting `train_dataloaders` and `val_dataloaders` instead.zIn tuner with `method`=zt, `train_dataloaders` and `val_dataloaders` arguments should be None, please consider setting `dataloaders` instead.r r rVz&Trainer is already configured with a `rz9` callback.Please remove it if you want to use the Tuner.rDzTrainer is already configured with `LearningRateFinder` and `BatchSizeFinder` callbacks. Please remove them if you want to use the Tuner.) ValueErrorrrTr6r r$appendr r#len)rr,r-r.r2Zsupported_methodsZconfigured_callbacksrNrrrrRs4       rR)NNNr&)typingrrrrZtyping_extensionsrrrZpytorch_lightningplZ-pytorch_lightning.callbacks.batch_size_finderr Z$pytorch_lightning.callbacks.callbackr Z%pytorch_lightning.callbacks.lr_finderr Z!pytorch_lightning.core.datamoduler Z pytorch_lightning.trainer.statesr Z!pytorch_lightning.tuner.lr_finderrZ&pytorch_lightning.utilities.exceptionsrZ!pytorch_lightning.utilities.typesrrrrrRrrrrs2       t