a  d@s,dZddlZddlmZGdddeZdS)aL RMSProp modified to behave like Tensorflow impl Originally cut & paste from PyTorch RMSProp https://github.com/pytorch/pytorch/blob/063946d2b3f3f1e953a2a3b54e0b34f1393de295/torch/optim/rmsprop.py Licensed under BSD-Clause 3 (ish), https://github.com/pytorch/pytorch/blob/master/LICENSE Modifications Copyright 2021 Ross Wightman N) Optimizercs@eZdZdZdfd d Zfd d ZedddZZ S) RMSpropTFaEImplements RMSprop algorithm (TensorFlow style epsilon) NOTE: This is a direct cut-and-paste of PyTorch RMSprop with eps applied before sqrt and a few other modifications to closer match Tensorflow for matching hyper-params. Noteworthy changes include: 1. Epsilon applied inside square-root 2. square_avg initialized to ones 3. LR scaling of update accumulated in momentum buffer Proposed by G. Hinton in his `course `_. The centered version first appears in `Generating Sequences With Recurrent Neural Networks `_. Arguments: params (iterable): iterable of parameters to optimize or dicts defining parameter groups lr (float, optional): learning rate (default: 1e-2) momentum (float, optional): momentum factor (default: 0) alpha (float, optional): smoothing (decay) constant (default: 0.9) eps (float, optional): term added to the denominator to improve numerical stability (default: 1e-10) centered (bool, optional) : if ``True``, compute the centered RMSProp, the gradient is normalized by an estimation of its variance weight_decay (float, optional): weight decay (L2 penalty) (default: 0) decoupled_decay (bool, optional): decoupled weight decay as per https://arxiv.org/abs/1711.05101 lr_in_momentum (bool, optional): learning rate scaling is included in the momentum buffer update as per defaults in Tensorflow {Gz??绽|=rFTc sd|kstd|d|ks,td|d|ksBtd|d|ksXtd|d|ksntd|t|||||||| d} tt||| dS)NrzInvalid learning rate: {}zInvalid epsilon value: {}zInvalid momentum value: {}zInvalid weight_decay value: {}zInvalid alpha value: {})lrmomentumalphaepscentered weight_decaydecoupled_decaylr_in_momentum) ValueErrorformatdictsuperr__init__) selfparamsrr r r r r rrdefaults __class__^/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/timm/optim/rmsprop_tf.pyr0s zRMSpropTF.__init__cs8tt|||jD]}|dd|ddqdS)Nr rr F)rr __setstate__ param_groups setdefault)rstategrouprrrrBs  zRMSpropTF.__setstate__Nc Cs>d}|dur:t|}Wdn1s00Y|jD]}|dD]}|jdur`qN|j}|jrttd|j|}t|dkrd|d<t||d<|ddkrt ||d<|d rt ||d <|d}d |d }|dd 7<|ddkr:|dr(| d |d|dn|j ||dd}|j | d||d|d r|d } | j || |d|j| | dd |d} n| |d} |ddkr |d} |dr| |dj|| |dd| | n*| |d|| |j | |d dqN|j|| |d dqNq@|S)zPerforms a single optimization step. Arguments: closure (callable, optional): A closure that reevaluates the model and returns the loss. Nrz)RMSprop does not support sparse gradientsrstep square_avgr Zmomentum_bufferr grad_avgg?r r rr)r )valuer r)torchZ enable_gradrgradZ is_sparse RuntimeErrorrlenZ ones_likeZ zeros_likeZmul_addZadd_powZaddcmulZsqrt_Zaddcdiv_) rclosureZlossr pr)rr"Zone_minus_alphar#avgbufrrrr!HsR $          zRMSpropTF.step)rrrrrFFT)N) __name__ __module__ __qualname____doc__rrr(Zno_gradr! __classcell__rrrrrs! r)r5r(Z torch.optimrrrrrrs