a dd@sddlmZmZddlZddlmZddlmZmZmZm Z m Z m Z m Z m Z mZmZmZddlmZddgZGd ddeZd d jeee e ed e_deeeeeeeeeeeeeeeeeeeeeeeeeeeeedddZeeeeeeeeeeeeeeeeeeeeeeeeedddZeeeeeeeeeeeeeeeeeeeeeeeeedddZeeeeeeeeeeeeeeeeeeeeeeeeeddddZdS))ListOptionalN)Tensor) Optimizer_use_grad_for_differentiable _get_value_stack_if_compiling_dispatch_sqrt_default_to_fused_or_foreach_capturable_doc_differentiable_doc _foreach_doc _fused_doc _maximize_doc)"_group_tensors_by_device_and_dtypeAdamadamcsdeZdZdddddddeeeeeeedfdd Zfd d Zd d ZedddZ Z S)rMbP?g?g+?:0yE>rFNforeachmaximize capturabledifferentiablefusedc s d|kstd|d|ks,td|d|dkrDdksXntd|dd|dkrpdksntd|dd|kstd |t|||||||| | | d } t|| | r| rtd d |_td d|jDstd|rtddS)NzInvalid learning rate: {}zInvalid epsilon value: {}rg?z%Invalid beta parameter at index 0: {}rz%Invalid beta parameter at index 1: {}zInvalid weight_decay value: {}) lrbetaseps weight_decayamsgradrrrrrz)`fused` does not support `differentiable`Tcss,|]$}|dD]}|jo t|VqqdS)paramsN)is_cudatorchZis_floating_point).0Zpgpr(Y/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/torch/optim/adam.py +sz Adam.__init__..zF`fused=True` requires all the params to be CUDA, floating point Tensorz0`fused` and `foreach` cannot be `True` together.) ValueErrorformatdictsuper__init__ RuntimeErrorZ_step_supports_amp_scalingall param_groups) selfr#rrr r!r"rrrrrdefaults __class__r(r)r/s4z Adam.__init__cst||jD]L}|dd|dd|dd|dd|dd|ddqt|j}t|dkot |dd }|s|D]}t t |d |d <qdS) Nr"Frrrrrrstep) r. __setstate__r2 setdefaultliststatevalueslenr%Z is_tensortensorfloat)r3r;groupZ state_valuesZstep_is_tensorsr5r(r)r83s       zAdam.__setstate__c Cs.|dD]}|jdur|||jjr2td||j|j|} t| dkr|dsd|drxtjdtj|j dnt d| d <tj |tj d | d <tj |tj d | d <|d rtj |tj d | d<|| d || d |d r|| d|dr| d j rtd|| d qdS)Nr#zJAdam does not support sparse gradients, please consider SparseAdam insteadrrrr)dtypedevicerr7)Z memory_formatexp_avg exp_avg_sqr"Zmax_exp_avg_sqrzB`requires_grad` is not supported for `step` in differentiable mode)gradappendZ is_sparser0r;r=r%zerosr?rDr>Z zeros_likeZpreserve_formatZ requires_grad) r3r@params_with_gradgradsexp_avgs exp_avg_sqsmax_exp_avg_sqs state_stepsr'r;r(r(r) _init_groupBs2      zAdam._init_groupc Cs|d}|durBt|}Wdn1s80Y|jD]}g}g}g}g}g}g} |d\} } |||||||| t|||||| |d| | |d|d|d|d|d|d |d |d t|d dt|d ddqH|S)zPerforms a single optimization step. Args: closure (Callable, optional): A closure that reevaluates the model and returns the loss. Nrr"rr!r rrrrr grad_scale found_inf) r"beta1beta2rr!r rrrrrrQrR)Z _cuda_graph_capture_health_checkr%Z enable_gradr2rPrgetattr) r3closureZlossr@rJrKrLrMrNrOrSrTr(r(r)r7lsX $     z Adam.step)rrrrF)N) __name__ __module__ __qualname__rboolr/r8rPrr7 __classcell__r(r(r5r)r s% *a Implements Adam algorithm. .. math:: \begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \gamma \text{ (lr)}, \beta_1, \beta_2 \text{ (betas)},\theta_0 \text{ (params)},f(\theta) \text{ (objective)} \\ &\hspace{13mm} \lambda \text{ (weight decay)}, \: \textit{amsgrad}, \:\textit{maximize} \\ &\textbf{initialize} : m_0 \leftarrow 0 \text{ ( first moment)}, v_0\leftarrow 0 \text{ (second moment)},\: \widehat{v_0}^{max}\leftarrow 0\\[-1.ex] &\rule{110mm}{0.4pt} \\ &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do} \\ &\hspace{5mm}\textbf{if} \: \textit{maximize}: \\ &\hspace{10mm}g_t \leftarrow -\nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}\textbf{else} \\ &\hspace{10mm}g_t \leftarrow \nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}\textbf{if} \: \lambda \neq 0 \\ &\hspace{10mm} g_t \leftarrow g_t + \lambda \theta_{t-1} \\ &\hspace{5mm}m_t \leftarrow \beta_1 m_{t-1} + (1 - \beta_1) g_t \\ &\hspace{5mm}v_t \leftarrow \beta_2 v_{t-1} + (1-\beta_2) g^2_t \\ &\hspace{5mm}\widehat{m_t} \leftarrow m_t/\big(1-\beta_1^t \big) \\ &\hspace{5mm}\widehat{v_t} \leftarrow v_t/\big(1-\beta_2^t \big) \\ &\hspace{5mm}\textbf{if} \: amsgrad \\ &\hspace{10mm}\widehat{v_t}^{max} \leftarrow \mathrm{max}(\widehat{v_t}^{max}, \widehat{v_t}) \\ &\hspace{10mm}\theta_t \leftarrow \theta_{t-1} - \gamma \widehat{m_t}/ \big(\sqrt{\widehat{v_t}^{max}} + \epsilon \big) \\ &\hspace{5mm}\textbf{else} \\ &\hspace{10mm}\theta_t \leftarrow \theta_{t-1} - \gamma \widehat{m_t}/ \big(\sqrt{\widehat{v_t}} + \epsilon \big) \\ &\rule{110mm}{0.4pt} \\[-1.ex] &\bf{return} \: \theta_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] \end{aligned} For further details regarding the algorithm we refer to `Adam: A Method for Stochastic Optimization`_. a Args: params (iterable): iterable of parameters to optimize or dicts defining parameter groups lr (float, optional): learning rate (default: 1e-3) betas (Tuple[float, float], optional): coefficients used for computing running averages of gradient and its square (default: (0.9, 0.999)) eps (float, optional): term added to the denominator to improve numerical stability (default: 1e-8) weight_decay (float, optional): weight decay (L2 penalty) (default: 0) amsgrad (bool, optional): whether to use the AMSGrad variant of this algorithm from the paper `On the Convergence of Adam and Beyond`_ (default: False) {foreach} {maximize} {capturable} {differentiable} {fused} .. _Adam\: A Method for Stochastic Optimization: https://arxiv.org/abs/1412.6980 .. _On the Convergence of Adam and Beyond: https://openreview.net/forum?id=ryQu7f-RZ rF)r#rKrLrMrNrOrrrrrQrRr"rSrTrr!r rc Cs| dur"|dur"t||dd\}}| dur.d} |dur:d}tdd|DsTtd|rjtjrjtd| r~tjs~t}n|rtjst}nt}|||||||| | |||||||| | ddS) zmFunctional API that performs Adam algorithm computation. See :class:`~torch.optim.Adam` for details. NF)Z use_fusedcss|]}t|tjVqdSN) isinstancer%r)r&tr(r(r)r* zadam..zPAPI has changed, `state_steps` argument must contain a list of singleton tensorsz6torch.jit.script not supported with foreach optimizers) r"rSrTrr!r rrrrQrR) r r1r0r%ZjitZ is_scripting _fused_adam_multi_tensor_adam_single_tensor_adam)r#rKrLrMrNrOrrrrrQrRr"rSrTrr!r r_funcr(r(r)rsB)r#rKrLrMrNrOrQrRr"rSrTrr!r rrrc CsH|dur|dusJt|D]$\}}|s2||n|| }||}||}||}|rl|jrd|jslJd|d7}| dkr|j|| d}t|rt|}t|}t|}t|}|| j|d| d|| j|| d| d|s|r|}dt | |}dt | |}| |}| }| }|r|rR|| }n||}||t|||| ||| |}n| ||| |}|||qt|}d| |}d| |}| |}t|}|rtj|||||d|| || }n| || }|j||| dqdS)N@If capturable=True, params and state_steps must be CUDA tensors.rralpha)value)out) enumerater$addr% is_complex view_as_realZmul_Zadd_Zaddcmul_ZconjpownegsqrtcloneZcopy_maximumZaddcdiv_rr )r#rKrLrMrNrOrQrRr"rSrTrr!r rrriparamrGrErFZstep_tr7bias_correction1bias_correction2 step_sizeZ step_size_negbias_correction2_sqrtZmax_exp_avg_sqs_idenomr(r(r)rb,sV         rbc !s t|dkrdS|r4tddt||Ds4Jd|durD|dusHJ|rTJdt||||||g}|D]\}}}}}}|rtt|}dd|D}dd|D}d d|D}d d|D}t|d | dkrtj ||| d }t |tj||d d t |t |||d |rTfd d|D}fdd|D}t |d t |d t |t |t|}t|t |t|}|rt||t|}t|t||t|| }t|t ||}n@t|} t| t||t|| }t|t | |}t|||qpfdd|D}fdd|D}tfdd|D}dd|D}|rt||t|}t||t || }n"t|} t| |t | | }t||||qpdS)Nrcss|]\}}|jo|jVqdSr\)r$)r&r'r7r(r(r)r*r_z%_multi_tensor_adam..rez#_foreach ops don't support autogradcSs$g|]}t|rt|n|qSr(r%rlrmr&xr(r(r) r_z&_multi_tensor_adam..cSs$g|]}t|rt|n|qSr(rzr{r(r(r)r}r_cSs$g|]}t|rt|n|qSr(rzr{r(r(r)r}r_cSs$g|]}t|rt|n|qSr(rzr{r(r(r)r}r_rrfcsg|]}t|qSr(r%rnr&r7rSr(r)r}r_csg|]}t|qSr(r~rrTr(r)r}r_csg|]}dt|qSrBrrrr(r)r}r_csg|]}dt|qSrBrrrr(r)r}r_csg|]}|dqS)r(r&Zbc)rr(r)r}r_cSsg|] }t|qSr()r rr(r(r)r}r_)r=r1ziprr<r%Z _foreach_negtuple _foreach_add_Z _foreach_addZ _foreach_mul_Z_foreach_addcmul_ _foreach_sub_Z _foreach_neg_Z _foreach_divZ_foreach_reciprocal_Z _foreach_sqrtZ_foreach_maximum_Z _foreach_div_Z _foreach_mulZ_foreach_addcdiv_r )!r#rKrLrMrNrOrQrRr"rSrTrr!r rrrgrouped_tensors device_params device_gradsdevice_exp_avgsdevice_exp_avg_sqsdevice_max_exp_avg_sqsdevice_state_stepsZparams_rurvrwrxZmax_exp_avg_sq_sqrtZeps_over_step_sizeryZexp_avg_sq_sqrtr()rSrTrr)ras|                           ra)r#rKrLrMrNrOrQrRr"rSrTrr!r rrrreturnc Cs|dur|j|ind}|dur(|j|ind}t||||||g}|D]\}}|||f\}}}}}}d\}}|dur||vr|j|dd||<||}|dur||vr|j|dd||<||}t|dtj|||||||| | | | | |||d|durDt||gt|qDdS)N)NNT)Z non_blockingr) r"rrSrTr!r rrQrR)rDrtor%rZ _fused_adam_rr=)r#rKrLrMrNrOrQrRr"rSrTrr!r rrrZgrad_scale_dictZfound_inf_dictrrDrCrrrrrrZdevice_grad_scaleZdevice_found_infr(r(r)r`sR   r`)NFFNNN)typingrrr%rZ optimizerrrrr r r r r rrrZtorch.utils._foreach_utilsr__all__rr,__doc__rZr?rrbrar`r(r(r(r)s 4 &I E ` u