a d@sddlZddlmZmZmZmZmZmZddlZddlm Z m Z ddl m Z m Z eej eej fZgdZdeeeeeeej dd d Zdeeeeeeej dd d Zdeeeedd ddZdS)N)UnionIterableListDictTupleOptional)Tensorinf)"_group_tensors_by_device_and_dtype_has_foreach_support)clip_grad_norm_clip_grad_normclip_grad_value_@F) parametersmax_norm norm_typeerror_if_nonfiniteforeachreturncst|tjr|g}dd|D}t|}tt|dkrFtdS|djtdd|Dg}tkrfdd|D}t|dkr|dnt t |}ng}| D]l\\} } \}|dus|rt || d r| t|q|rtd | jd q| fd d|Dqtt fd d|D}|rjt||rjtdd||d} tj| dd} | D]\\} } \}|dus|rt || d rt|| | n>|rtd | jd n$| | } |D]}|| qq|S)aDClips gradient norm of an iterable of parameters. The norm is computed over all gradients together, as if they were concatenated into a single vector. Gradients are modified in-place. Args: parameters (Iterable[Tensor] or Tensor): an iterable of Tensors or a single Tensor that will have gradients normalized max_norm (float): max norm of the gradients norm_type (float): type of the used p-norm. Can be ``'inf'`` for infinity norm. error_if_nonfinite (bool): if True, an error is thrown if the total norm of the gradients from :attr:`parameters` is ``nan``, ``inf``, or ``-inf``. Default: False (will switch to True in the future) foreach (bool): use the faster foreach-based implementation. If ``None``, use the foreach implementation for CUDA and CPU native tensors and silently fall back to the slow implementation for other device types. Default: ``None`` Returns: Total norm of the parameter gradients (viewed as a single vector). cSsg|]}|jdur|jqSNgrad.0pra/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/torch/nn/utils/clip_grad.py 'z#clip_grad_norm_..rgcSsg|] }|qSr)detachrgrrrr.rcs"g|]}|qSr)r absmaxtor! first_devicerrr1rNdevice:foreach=True was passed, but can't use the foreach API on  tensorscsg|]}t|qSr)torchnormr!)rrrr;rcsg|]}|qSr)r%)rr.r&rrr=rzThe total norm of order z for gradients from `parameters` is non-finite, so it cannot be clipped. To disable this error and scale the gradients by the non-finite norm anyway, set `error_if_nonfinite=False`gư>g?)r$) isinstancer-rfloatlenZtensorr*r r r$stackitemsr extendZ _foreach_norm RuntimeErrortyper. logical_orisnanisinfclampZ _foreach_mul_r%r Zmul_)rrrrrgrads grouped_gradsZnormsZ total_normr*_Z clip_coefZclip_coef_clampedZclip_coef_clamped_devicer"r)r'rrr sH    &    r cCstjdddt|||||S)zClips gradient norm of an iterable of parameters. .. warning:: This method is now deprecated in favor of :func:`torch.nn.utils.clip_grad_norm_`. z[torch.nn.utils.clip_grad_norm is now deprecated in favor of torch.nn.utils.clip_grad_norm_.) stacklevel)warningswarnr )rrrrrrrrr Ws r )r clip_valuerrcCst|tjr|g}t|}dd|D}t|g}|D]v\\}}\}|dusT|r|t||dr|t|| t||q:|rt d|j dq:|D]}|j j | |dqq:dS)aClips gradient of an iterable of parameters at specified value. Gradients are modified in-place. Args: parameters (Iterable[Tensor] or Tensor): an iterable of Tensors or a single Tensor that will have gradients normalized clip_value (float): maximum allowed value of the gradients. The gradients are clipped in the range :math:`\left[\text{-clip\_value}, \text{clip\_value}\right]` foreach (bool): use the faster foreach-based implementation If ``None``, use the foreach implementation for CUDA and CPU native tensors and silently fall back to the slow implementation for other device types. Default: ``None`` cSsg|]}|jdur|jqSrrrrrrryrz$clip_grad_value_..Nr)r+r,)minr$) r/r-rr0r r3r Z_foreach_clamp_min_Z_foreach_clamp_max_r5r6dataZclamp_)rrBrr;r<r*r=rrrrres r)rFN)rFN)N)r@typingrrrrrrr-rr Ztorch.utils._foreach_utilsr r Z_tensor_or_tensors__all__r0boolr r rrrrrs&  L