a dF @sddlZddlZddlZddlZddlmZddlmZmZm Z m Z m Z m Z m Z ddlZddlZddlmZddlZddlmZddlmZddlmZddlmZmZdd lmZdd l m!Z!m"Z"m#Z#ej$%d durdd l&m'Z'n dd l'm'Z'ed Z(e re(rddl)m*Z+ddl,m-Z-e.e/Z0dde1dddZ2GdddZ3d0dde4e4e5e1e e4e6e e3d ddZ7de e1efdd d!Z8de5e e4dd"d#d$Z9de e1efdd%d&d'Z:Gd(d)d)eZ;Gd*d+d+eZdS)1N)deepcopy)AnycastDictListOptional TYPE_CHECKINGUnion)RequirementCache)_TORCH_LRSCHEDULER)Callback)MisconfigurationException)lightning_hasattrlightning_setattr)rank_zero_warn) LRSchedulerLRSchedulerConfig STEP_OUTPUTZ ipywidgets)tqdmZ matplotlib)Axes pl.Trainerpl.LightningModule)trainermodelreturncCsdt|jtr0t||js*td|jd|jSd}|D]}t||r8|Sq8td|ddS)Nz`auto_lr_find` was set to zG, however could not find this as a field in `model` or `model.hparams`.)lrZ learning_ratez]When `auto_lr_find=True`, either `model` or `model.hparams` should have one of these fields: z overridden.) isinstance auto_lr_findstrrr )rrZ attr_optionsattrr j/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/pytorch_lightning/tuner/lr_finder.py_determine_lr_attr_name/s     r"c@sleZdZdZeeeeddddZddddd Zde e e d e d d ddZ deee edddZ dS) _LRFindera,LR finder object. This object stores the results of lr_find(). Args: mode: either `linear` or `exponential`, how to increase lr after each step lr_min: lr to start search from lr_max: lr to stop search num_training: number of steps to take between lr_min and lr_max Example:: # Run lr finder lr_finder = trainer.lr_find(model) # Results stored in lr_finder.results # Plot using lr_finder.plot() # Get suggestion lr = lr_finder.suggestion() N)modelr_minlr_max num_trainingrcCs8|dvsJd||_||_||_||_i|_d|_dS)N)linear exponentialz/mode should be either `linear` or `exponential`r)r$r%r&r'results_total_batch_idx)selfr$r%r&r'r r r!__init__]sz_LRFinder.__init__rrrc Csddlm}|jj}t|dkr4tdt|d|d}|jgt|j}t|j|D]\}}||d<||d<qZ||j |j f}|j dkrt |nt |} tt| } |g|j_t| d dd g|j_||j|jd S) zDecorate `trainer.strategy.setup_optimizers` method such that it sets the user's originally specified optimizer together with a new scheduler that takes care of the learning rate search.r)_set_scheduler_opt_idxz(`model.configure_optimizers()` returned z;, but learning rate finder only works with single optimizerrZ initial_lrr(step)intervalZopt_idxN)Z pytorch_lightning.core.optimizerr/strategy optimizerslenr r%Z param_groupszipr&r'r$ _LinearLR_ExponentialLRrrrlr_scheduler_configs) r,rr/r4 optimizerZnew_lrsZ param_groupZnew_lrargs schedulerr r r!_exchange_schedulerhs"     z_LRFinder._exchange_schedulerFrz plt.Figure)suggestshowaxrc Csts tdddlm}|jd}|jd}|durB|\}}n|j}||||jdkrh| d| d| d |r| }|j r|j||j ||j d d d d |r||S)aPlot results from lr_find run Args: suggest: if True, will mark suggested lr to use with a red point show: if True, will show figure ax: Axes object to which the plot is to be drawn. If not provided, a new figure is created. zpTo use the `plot` method, you must have Matplotlib installed. Install it by running `pip install -U matplotlib`.rNrlossr)logz Learning rateZLoss oZred)Z markersizemarkercolor)_MATPLOTLIB_AVAILABLEr matplotlib.pyplotpyplotr*Zsubplotsfigureplotr$Z set_xscaleZ set_xlabelZ set_ylabel suggestion _optimal_idxr?) r,r>r?r@pltlrslossesZfig_r r r!rKs,          z_LRFinder.plotrCr0) skip_beginskip_endrcCspt|jd|| }|t|}t|dkrHtdd|_dSt| }|||_|jd|jS)aThis will propose a suggestion for an initial learning rate based on the point with the steepest negative gradient. Args: skip_begin: how many samples to skip in the beginning; helps to avoid too naive estimates skip_end: how many samples to skip in the end; helps to avoid too optimistic estimates Returns: The suggested initial learning rate to use, or `None` if a suggestion is not possible due to too few loss samples. rAzFailed to compute suggestion for learning rate because there are not enough points. Increase the loop iteration limits or the size of your dataset/dataloader.Nr) nparrayr*isfiniter5rBerrorrMZgradientZargmin)r,rRrSrPZmin_gradr r r!rLs   z_LRFinder.suggestion)FFN)rCr0) __name__ __module__ __qualname____doc__rfloatintr-r=boolrrKrLr r r r!r#Cs  )r#:0yE>r0dr)@F) rrmin_lrmax_lrr'r$early_stop_threshold update_attrrcCsp|jrtddS|r t||}tj|jdtd} |j | } | | |j } t |} t||||jr||jt||||} | |t|| |j || krtd|j d| j|jdj|jdjd|jj| _t|| |jr|j|j | j| _|rL| } | durLt!||| td | |j"#| |j $| d |j_%| S) z9See :meth:`~pytorch_lightning.tuner.tuning.Tuner.lr_find`z>Skipping learning rate finder since `fast_dev_run` is enabled.Nz .lr_find_z.ckptzLR finder stopped early after z steps due to diverging loss.r)rrAzLearning rate set to F)&Z fast_dev_runrr"ospathjoinZdefault_root_diruuiduuid4r3 broadcastZsave_checkpoint global_step__lr_finder_dump_params__lr_finder_reset_paramsZprogress_bar_callbackdisabler#r= _try_loop_runrBinfor*update callbacksrOrPfit_loopZtotal_batch_idxr+__lr_finder_restore_paramsenablerLrZ_checkpoint_connectorrestoreZremove_checkpoint restarting)rrrcrdr'r$rerfZ lr_attr_nameZ ckpt_pathZ start_stepsparamsZ lr_finderrr r r!lr_findsB        "       r{r.c Cs:|jj|jj|jj|j|j|j|jj|j t |j d S)N) r4r9optimizer_frequenciesrtloggersr max_stepslimit_val_batchesloop_state_dict) r3r4r9r|rtr}rrur~rrZ state_dict)rr r r!rns rn)rr'rercCsbddlm}g|j_g|j_d|_t||ddg|_|jdurD|nd|_||j |j _ ||_ dS)Nr) DummyLoggerFr0)progress_bar_refresh_rate) Z pytorch_lightning.loggers.loggerrr3r9r|r _LRCallbackrtloggerrmrur~r)rr'rerr r r!ro's ro)rrzrcCs|d|j_|d|j_|d|j_|d|_|d|_|d|_|d|j_|d|_ |j}| t |d d |_ d |_ dS) Nr4r9r|rrtr}r~rrF)r3r4r9r|rrtr}rur~rload_state_dictrry should_stoprrzZloopr r r!rv7s        rvc@sXeZdZdZdeeeeedddZdd eed d d d Z dd e eed dddZ d S)raSpecial callback used by the learning rate finder. This callback logs the learning rate before each batch and logs the corresponding loss after each batch. Args: num_training: number of iterations done by the learning rate finder early_stop_threshold: threshold for stopping the search. If the loss at any point is larger than ``early_stop_threshold*best_loss`` then the search is stopped. To disable, set to ``None``. progress_bar_refresh_rate: rate to refresh the progress bar for the learning rate finder beta: smoothing value, the loss being logged is a running average of loss values logged until now. ``beta`` controls the forget rate i.e. if ``beta=0`` all past information is ignored. rbr\(\?)r'rerbetacCs:||_||_||_g|_g|_d|_d|_||_d|_dS)Ng) r'rerrPrOavg_loss best_lossr progress_bar)r,r'rerrr r r!r-Wsz_LRCallback.__init__rrN)r pl_modulebatch batch_idxrcCsX|jjd|jdkrdS|jr:|jdur:td|jd|_|j|j dj j ddS)zACalled before each training batch, logs the lr that will be used.r0rNzFinding best initial lr)desctotal) ruraccumulate_grad_batchesrrrr'rOappendr9r<r)r,rrrrr r r!on_train_batch_starths z _LRCallback.on_train_batch_start)rroutputsrrrc Cs|jjd|jdkrdS|jr*|j|jj}|dusBJ|}|j}|j |j d|j ||_ |j d|j |d} |j dur|dkr| |j |j krd|_ |jr|j|j|j |_ | |j ks|dkr| |_ |j| dS)z>Called when the training batch ends, logs the calculated loss.r0rNT)rurrrrsZ running_losslastitemrmrrrerrcloser3rlrPr) r,rrrrrZ loss_tensorZ current_lossZ current_stepZ smoothed_lossr r r!on_train_batch_endts&     z_LRCallback.on_train_batch_end)rbrr) rYrZr[r\r^rr]r-rrrrr r r r!rGs   rcsbeZdZdZd ejjeeedfdd Z e edddZ e e ee efdd d ZZS) r7a<Linearly increases the learning rate between two boundaries over a number of iterations. Args: optimizer: wrapped optimizer. end_lr: the final learning rate. num_iter: the number of iterations over which the test occurs. last_epoch: the index of last epoch. Default: -1. r:end_lrnum_iter last_epochcs||_||_t||dSNrrsuperr-r,r:rrr __class__r r!r-sz_LinearLR.__init__rcsPjd}|jjdkr6fddjD}nddjD}|_|S)Nr0rcsg|]}|j|qSr r.0Zbase_lrrr,r r! z$_LinearLR.get_lr..cSsg|]}|qSr r rr r r!rrrrZbase_lrs_lrr,Z curr_itervalr rr!get_lrs   z_LinearLR.get_lrcCs|jSrrr,r r r!rsz _LinearLR.lr)rrYrZr[r\torchZoptimZ Optimizerr]r^r-rrpropertyr r __classcell__r r rr!r7s   r7csbeZdZdZd ejjeeedfdd Z e edddZ e e ee efdd d ZZS) r8aFExponentially increases the learning rate between two boundaries over a number of iterations. Arguments: optimizer: wrapped optimizer. end_lr: the final learning rate. num_iter: the number of iterations over which the test occurs. last_epoch: the index of last epoch. Default: -1. rrcs||_||_t||dSrrrrr r!r-sz_ExponentialLR.__init__rcsPjd}|jjdkr6fddjD}nddjD}|_|S)Nr0rcsg|]}|j|qSr rrrr r!rrz)_ExponentialLR.get_lr..cSsg|]}|qSr r rr r r!rrrrr rr!rs   z_ExponentialLR.get_lrcCs|jSrrrr r r!rsz_ExponentialLR.lr)rrr r rr!r8s   r8cCs*|j}|t|dd|_|dS)NrF)rurrryrunrr r r!rqsrq)r`r0rar)rbF)? importlibloggingrgrjcopyrtypingrrrrrrr numpyrUrZ lightning_utilities.core.importsr Zpytorch_lightningplZ lightning_fabric.utilities.typesr Zpytorch_lightning.callbacksr Z&pytorch_lightning.utilities.exceptionsr Z#pytorch_lightning.utilities.parsingrrZ%pytorch_lightning.utilities.rank_zerorZ!pytorch_lightning.utilities.typesrrrutil find_specZ tqdm.autorrGrHrIrNZmatplotlib.axesr getLoggerrYrBrr"r#r]r^r_r{rnrorvrr7r8rqr r r r!sd $            LP##