a d0@s:ddlZddlmZddlmZmZmZmZmZmZm Z m Z ddl m Z m Z ddlmZddgZGd ddeZd d je e ed e_de ee ee ee ee ee eeeeeeeed ddZe ee ee ee ee eeeeeeeed ddZe ee ee ee ee eeeeeeeed ddZdS)N)Tensor) Optimizer_use_grad_for_differentiable _get_value_stack_if_compiling_default_to_fused_or_foreach_differentiable_doc _maximize_doc _foreach_doc)ListOptional)"_group_tensors_by_device_and_dtypeAdamaxadamaxcsVeZdZddddeeeedfd d Zfd d Zd dZedddZ Z S)rMb`?g?g+?:0yE>rNF)maximizedifferentiableforeachrrc sd|kstd|d|ks,td|d|dkrDdksXntd|dd|dkrpdksntd|dd|kstd |t|||||||d } t|| dS) NzInvalid learning rate: {}zInvalid epsilon value: {}rg?z%Invalid beta parameter at index 0: {}rz%Invalid beta parameter at index 1: {}zInvalid weight_decay value: {})lrbetaseps weight_decayrrr) ValueErrorformatdictsuper__init__) selfparamsrrrrrrrdefaults __class__[/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/torch/optim/adamax.pyr! s(  zAdamax.__init__cst||jD](}|dd|dd|ddqt|j}t|dkoft |dd}|s|D]}t t |d|d<qpdS)NrrFrrstep) r __setstate__ param_groups setdefaultliststatevalueslentorchZ is_tensortensorfloat)r"r.groupZ state_valuesZstep_is_tensorsr%r'r(r*/s     zAdamax.__setstate__c Cs|dD]}|jdurq|||jjr2td||j|j|}t|dkrtd|d<tj|tj d|d<tj|tj d|d<||d||d||dqdS) Nr#z(Adamax does not support sparse gradientsrrr))Z memory_formatexp_avgexp_inf) gradappendZ is_sparse RuntimeErrorr.r0r1r2Z zeros_likeZpreserve_format) r"r4params_with_gradgradsexp_avgsexp_infs state_stepspr.r'r'r( _init_group=s&        zAdamax._init_groupcCsd}|dur:t|}Wdn1s00Y|jD]}g}g}g}g}g}|d\} } |d} |d} |d} |d}|d}|d}|||||||t|||||| | | | | |||d q@|S) zPerforms a single optimization step. Args: closure (Callable, optional): A closure that reevaluates the model and returns the loss. Nrrrrrrr)rbeta1beta2rrrrr)r1Z enable_gradr+rAr)r"closureZlossr4r;r<r=r>r?rBrCrrrrrrr'r'r(r)VsD $  z Adamax.step)rrrrN)N) __name__ __module__ __qualname__r boolr!r*rArr) __classcell__r'r'r%r(r s" " aImplements Adamax algorithm (a variant of Adam based on infinity norm). .. math:: \begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \gamma \text{ (lr)}, \beta_1, \beta_2 \text{ (betas)},\theta_0 \text{ (params)},f(\theta) \text{ (objective)}, \: \lambda \text{ (weight decay)}, \\ &\hspace{13mm} \epsilon \text{ (epsilon)} \\ &\textbf{initialize} : m_0 \leftarrow 0 \text{ ( first moment)}, u_0 \leftarrow 0 \text{ ( infinity norm)} \\[-1.ex] &\rule{110mm}{0.4pt} \\ &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do} \\ &\hspace{5mm}g_t \leftarrow \nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}if \: \lambda \neq 0 \\ &\hspace{10mm} g_t \leftarrow g_t + \lambda \theta_{t-1} \\ &\hspace{5mm}m_t \leftarrow \beta_1 m_{t-1} + (1 - \beta_1) g_t \\ &\hspace{5mm}u_t \leftarrow \mathrm{max}(\beta_2 u_{t-1}, |g_{t}|+\epsilon) \\ &\hspace{5mm}\theta_t \leftarrow \theta_{t-1} - \frac{\gamma m_t}{(1-\beta^t_1) u_t} \\ &\rule{110mm}{0.4pt} \\[-1.ex] &\bf{return} \: \theta_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] \end{aligned} For further details regarding the algorithm we refer to `Adam: A Method for Stochastic Optimization`_. a Args: params (iterable): iterable of parameters to optimize or dicts defining parameter groups lr (float, optional): learning rate (default: 2e-3) betas (Tuple[float, float], optional): coefficients used for computing running averages of gradient and its square eps (float, optional): term added to the denominator to improve numerical stability (default: 1e-8) weight_decay (float, optional): weight decay (L2 penalty) (default: 0) {foreach} {maximize} {differentiable} .. _Adam\: A Method for Stochastic Optimization: https://arxiv.org/abs/1412.6980 rF) r#r<r=r>r?rrrrrBrCrrcCstdd|Dstd|dur4t||dd\} }|rJtjrJtd|r^tjs^t}nt}|||||||| | | | ||d dS) zrFunctional API that performs adamax algorithm computation. See :class:`~torch.optim.Adamax` for details. css|]}t|tjVqdS)N) isinstancer1r).0tr'r'r( zadamax..zPAPI has changed, `state_steps` argument must contain a list of singleton tensorsNF)Z use_fusedz6torch.jit.script not supported with foreach optimizers)rrBrCrrrr)allr:rr1ZjitZ is_scripting_multi_tensor_adamax_single_tensor_adamax)r#r<r=r>r?rrrrrBrCrr_funcr'r'r(rs2) r#r<r=r>r?rrBrCrrrrcCs2t|D]"\} } || }| s"|n| }|| }|| }|| }|d7}| dkr^|j| | d}t| rt| } t|}t|}t|}||j|d|dt||d| | dgd}| stj |dd|dn| tj |dddd|t |}||}| j||| dqdS)NrralphaFkeepdimout)rW)value) enumerateaddr1 is_complex view_as_realZmul_add_cat unsqueezeabs unsqueeze_ZamaxZcopy_rZaddcdiv_)r#r<r=r>r?rrBrCrrrriparamr8r6r7Zstep_tnorm_bufbias_correctionclrr'r'r(rQs0     $rQ) r#r<r=r>r?rBrCrrrrrc sr| r Jdt|dkrdSt|||||g} | D]4\} }}}}| rTt|}dd| D} dd|D}dd|D}dd|D}t|d|dkrtj|| |d }t|tj||dd t||t||D]L\}}t | d| |  dgd}tj|dd ||fd qfd d|D}tfd d|D}t| |||q6dS)Nz#_foreach ops don't support autogradrcSs$g|]}t|rt|n|qSr'r1r\r]rKxr'r'r( 9rNz(_multi_tensor_adamax..cSs$g|]}t|rt|n|qSr'rhrir'r'r(rk:rNcSs$g|]}t|rt|n|qSr'rhrir'r'r(rk;rNcSs$g|]}t|rt|n|qSr'rhrir'r'r(rk<rNrrTFrVcsg|]}dt|qS)r)r)rKr))rBr'r(rkPrNcsg|]}d|qS)r')rKrf)rr'r(rkQrN)r0rr/r1Z _foreach_negZ _foreach_add_Z _foreach_addZ _foreach_mul_zipr_r`rar^rbmaxnewlongrZ_foreach_addcdiv_)r#r<r=r>r?rBrCrrrrrZgrouped_tensorsZgrouped_paramsZ grouped_gradsZgrouped_exp_avgsZgrouped_exp_infsZgrouped_state_stepsr7r8reZbias_correctionsrgr')rBrr(rPs2       rP)NFF)r1rZ optimizerrrrrrr r r typingr r Ztorch.utils._foreach_utilsr__all__rr__doc__rHr3rrQrPr'r'r'r(sn ( {5 8 5