a  d@s4dZddlZddlmZddlZGdddeZdS)a6 Nvidia NovoGrad Optimizer. Original impl by Nvidia from Jasper example: - https://github.com/NVIDIA/DeepLearningExamples/blob/master/PyTorch/SpeechRecognition/Jasper Paper: `Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks` - https://arxiv.org/abs/1905.11286 N) Optimizercs@eZdZdZdfdd Zfd d Zedd d ZZ S) NvNovoGrada( Implements Novograd algorithm. Args: params (iterable): iterable of parameters to optimize or dicts defining parameter groups lr (float, optional): learning rate (default: 1e-3) betas (Tuple[float, float], optional): coefficients used for computing running averages of gradient and its square (default: (0.95, 0.98)) eps (float, optional): term added to the denominator to improve numerical stability (default: 1e-8) weight_decay (float, optional): weight decay (L2 penalty) (default: 0) grad_averaging: gradient averaging amsgrad (boolean, optional): whether to use the AMSGrad variant of this algorithm from the paper `On the Convergence of Adam and Beyond`_ (default: False) MbP?gffffff?g\(\?:0yE>rFc sd|kstd|d|ks,td|d|dkrDdksXntd|dd|dkrpdksntd|dt||||||d }tt|||dS) NgzInvalid learning rate: {}zInvalid epsilon value: {}rg?z%Invalid beta parameter at index 0: {}z%Invalid beta parameter at index 1: {})lrbetaseps weight_decaygrad_averagingamsgrad) ValueErrorformatdictsuperr__init__) selfparamsrr r r r r defaults __class__^/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/timm/optim/nvnovograd.pyr szNvNovoGrad.__init__cs,tt|||jD]}|ddqdS)Nr F)rr __setstate__ param_groups setdefault)rstategrouprrrr1s zNvNovoGrad.__setstate__NcCsd}|dur:t|}Wdn1s00Y|jD]}|dD]}|jdur`qN|j}|jrttd|d}|j|}t|dkrd|d<t||d<t g |dj |d<|rt g |dj |d <|d|d}} |r|d } |d \} } |dd 7<t t |d } | dkrD| | n| | j| d | d |rtj| | | d| |d}n| |d}|||ddkr|j||dd |dr|d | || ||j||d d qNq@|S)zPerforms a single optimization step. Arguments: closure (callable, optional): A closure that reevaluates the model and returns the loss. Nrz#Sparse gradients are not supported.r rstepexp_avg exp_avg_sqmax_exp_avg_sqr r)alpha)outr r r r)torchZ enable_gradrgradZ is_sparse RuntimeErrorrlenZ zeros_likezerostoZdevicesumpowZcopy_Zmul_Zadd_maxsqrtZdiv_)rclosureZlossrpr'r rr r!r"Zbeta1Zbeta2ZnormZdenomrrrr6sP $         zNvNovoGrad.step)rrrrFF)N) __name__ __module__ __qualname____doc__rrr&Zno_gradr __classcell__rrrrr s r)r5r&Ztorch.optim.optimizerrmathrrrrrs