a  d@s4dZddlZddlZddlmZGdddeZdS)z AdamW Optimizer Impl copied from PyTorch master NOTE: Builtin optim.AdamW is used by the factory, this impl only serves as a Python based reference, will be removed someday N) Optimizercs@eZdZdZdfdd Zfd d Zedd d ZZ S)AdamWaImplements AdamW algorithm. The original Adam algorithm was proposed in `Adam: A Method for Stochastic Optimization`_. The AdamW variant was proposed in `Decoupled Weight Decay Regularization`_. Arguments: params (iterable): iterable of parameters to optimize or dicts defining parameter groups lr (float, optional): learning rate (default: 1e-3) betas (Tuple[float, float], optional): coefficients used for computing running averages of gradient and its square (default: (0.9, 0.999)) eps (float, optional): term added to the denominator to improve numerical stability (default: 1e-8) weight_decay (float, optional): weight decay coefficient (default: 1e-2) amsgrad (boolean, optional): whether to use the AMSGrad variant of this algorithm from the paper `On the Convergence of Adam and Beyond`_ (default: False) .. _Adam\: A Method for Stochastic Optimization: https://arxiv.org/abs/1412.6980 .. _Decoupled Weight Decay Regularization: https://arxiv.org/abs/1711.05101 .. _On the Convergence of Adam and Beyond: https://openreview.net/forum?id=ryQu7f-RZ MbP?g?g+?:0yE>{Gz?Fcsd|kstd|d|ks,td|d|dkrDdksXntd|dd|dkrpdksntd|dt|||||d }tt|||dS) NgzInvalid learning rate: {}zInvalid epsilon value: {}rg?z%Invalid beta parameter at index 0: {}z%Invalid beta parameter at index 1: {})lrbetaseps weight_decayamsgrad) ValueErrorformatdictsuperr__init__)selfparamsr r r r r defaults __class__Y/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/timm/optim/adamw.pyr'szAdamW.__init__cs,tt|||jD]}|ddqdS)Nr F)rr __setstate__ param_groups setdefault)rstategrouprrrr5s zAdamW.__setstate__NcCsd}|dur:t|}Wdn1s00Y|jD]}|dD]}|jdur`qN|jd|d|d|j}|jrtd|d}|j|}t |dkrd|d <t ||d <t ||d <|rt ||d <|d |d }} |r|d } |d \} } |d d7<d| |d } d| |d }|| j |d| d| | j ||d| d|rtj | | | d| t| |d}n| t| |d}|d| }|j||| dqNq@|S)zPerforms a single optimization step. Arguments: closure (callable, optional): A closure that reevaluates the model and returns the loss. Nrrr r zJAdam does not support sparse gradients, please consider SparseAdam insteadr rstepexp_avg exp_avg_sqmax_exp_avg_sqr )alpha)value)outr )torchZ enable_gradrgraddataZmul_Z is_sparse RuntimeErrorrlenZ zeros_likeZadd_Zaddcmul_maxsqrtmathZaddcdiv_)rclosureZlossrpr'r rr r!r"Zbeta1Zbeta2Zbias_correction1Zbias_correction2ZdenomZ step_sizerrrr:sH $      z AdamW.step)rrrrF)N) __name__ __module__ __qualname____doc__rrr&Zno_gradr __classcell__rrrrr s r)r3r-r&Ztorch.optim.optimizerrrrrrrs