a dj@sddlmZmZddlmZddlmZmZmZm Z m Z m Z ddl Z ddl Z ddlZddlmZddgZGd d d ZGd ddeZd d ZGdddZdS)) defaultdictabc)Enum)AnyDictListOptionalTuplecastN)amp_definitely_not_availableOptState GradScalerc@s2eZdZdZejddddZejdddZdS) _MultiDeviceReplicatorz_ Lazily serves copies of a tensor to requested devices. Copies are cached per-device. N) master_tensorreturncCs&|js|jjdksJ||_i|_dS)Nxla)is_cudadevicetypemaster_per_device_tensors)selfrrc/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/torch/cuda/amp/grad_scaler.py__init__sz_MultiDeviceReplicator.__init__rcCs6|j|d}|dur2|jj|ddd}||j|<|S)NT)r non_blockingcopy)rgetrto)rrretvalrrrrs  z_MultiDeviceReplicator.get)__name__ __module__ __qualname____doc__torchTensorrrrrrrr src@seZdZdZdZdZdS)r rr N)r"r#r$READYUNSCALEDSTEPPEDrrrrr #scCs tjidS)N)stagefound_inf_per_device)r r)rrrr_refresh_per_optimizer_state)sr.c@s eZdZUeejed<eejed<eeee e ffed<d=d d Z e ejejfd d d Z ddZddZddZddZddZddZd>ddZddZdd Zd!d"Zd#d$Zd%d&Zd'd(Zd)d*Zd+d,Zd-d.Zd/d0Zd1d2Zd3d4Z d5d6Z!d7d8Z"d9d:Z#d;d<Z$dS)?r_scaleZ_grows_tracker_per_optimizer_states@@?TcCs|rtrtdd|_n||_|jr||dks8Jd|dksHJd||_d|_||_||_||_d|_ d|_ t t |_ dS)NzLtorch.cuda.amp.GradScaler is enabled, but CUDA is not available. Disabling.F?z The growth factor must be > 1.0.z!The backoff factor must be < 1.0.r)r warningswarn_enabled _init_scaler/_growth_factor_backoff_factor_growth_interval_init_growth_tracker_growth_trackerrr.r0)rZ init_scale growth_factorbackoff_factorgrowth_intervalZenabledrrrrqs  zGradScaler.__init__rcCsHd}|jdus Jd|||jdus)rfuncnameZfixrrr_check_scale_growth_trackersz&GradScaler._check_scale_growth_trackercCsF|jdusJdtjd|jtj|d|_tjd|jtj|d|_dS)Nz)_growth_tracker initialized before _scaler dtyper)r>r&fullr9float32r/r=int32)rdevrrr_lazy_init_scale_growth_trackersz*GradScaler._lazy_init_scale_growth_trackercsjs |St|tjrf|js,|jjdks,JjdurB|jjdusPJ|jj |jddSgfdd|S)a2 Multiplies ('scales') a tensor or list of tensors by the scale factor. Returns scaled outputs. If this instance of :class:`GradScaler` is not enabled, outputs are returned unmodified. Args: outputs (Tensor or iterable of Tensors): Outputs to scale. rNTrrcst|tjrv|js"|jjdks"JtdkrbjdurD|jjdusRJ t j|d |jSt|t j rt|}t|ttfrt||S|SntddS)Nrrz2outputs must be a Tensor or an iterable of Tensors) isinstancer&r'rrrlenr/rLappendrrrIterablemaplisttuple ValueError)valiterable apply_scalerstashrrrYs       z%GradScaler.scale..apply_scale) r8rNr&r'rrrr/rLr )routputsrrXrscales    zGradScaler.scalec Cst|}t|}tdd}t|jD]}|dD]t} | jdurHq8|sb| jjtjkrbtd| jj r| jjtjur| j | _| j } n| j} || j | j | q8q,|D]0\} } | D]} t| || || qqWdn1s0Y|jS)NcSsttSN)rrSrrrrz,GradScaler._unscale_grads_..paramsz%Attempting to unscale FP16 gradients.)rrr&Zno_gradZ param_groupsZgradrGfloat16rUZ is_sparseZcoalesceZ_valuesrrPitemsvaluesZ*_amp_foreach_non_finite_check_and_unscale_rr)r optimizer inv_scale found_infZ allow_fp16Zper_device_inv_scaleZper_device_found_infZper_device_and_dtype_gradsgroupparamZ to_unscalerZper_dtype_gradsZgradsrrr_unscale_grads_s.        &zGradScaler._unscale_grads_cCs|js dS|d|jt|}|dtjur:tdn|dtjurPtd|jdus^J|j }t j ddt j|jjd}||||d |d <tj|d<dS) as Divides ("unscales") the optimizer's gradient tensors by the scale factor. :meth:`unscale_` is optional, serving cases where you need to :ref:`modify or inspect gradients` between the backward pass(es) and :meth:`step`. If :meth:`unscale_` is not called explicitly, gradients will be unscaled automatically during :meth:`step`. Simple example, using :meth:`unscale_` to enable clipping of unscaled gradients:: ... scaler.scale(loss).backward() scaler.unscale_(optimizer) torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm) scaler.step(optimizer) scaler.update() Args: optimizer (torch.optim.Optimizer): Optimizer that owns the gradients to be unscaled. .. note:: :meth:`unscale_` does not incur a CPU-GPU sync. .. warning:: :meth:`unscale_` should only be called once per optimizer per :meth:`step` call, and only after all gradients for that optimizer's assigned parameters have been accumulated. Calling :meth:`unscale_` twice for a given optimizer between each :meth:`step` triggers a RuntimeError. .. warning:: :meth:`unscale_` may unscale sparse gradients out of place, replacing the ``.grad`` attribute. Nunscale_r,zMunscale_() has already been called on this optimizer since the last update().z(unscale_() is being called after step().rErFFr-)r8rDr0idr r* RuntimeErrorr+r/doubleZ reciprocalfloatr&rHrIrri)rrdoptimizer_statererfrrrrjs   zGradScaler.unscale_cOs2d}tdd|dDs.|j|i|}|S)Ncss|]}|VqdSr])item).0vrrr !r_z-GradScaler._maybe_opt_step..r-)sumrcstep)rrdrpargskwargsr!rrr_maybe_opt_stepszGradScaler._maybe_opt_stepc s|js|j|i|Sd|vr&td|d|jt|}|dtjurTtdd}t|dr2|j r2|}dt |jj v}|rt d t|d|ind|dtjur|||ttjtfd d |d D}|dtjkrdn|_||_|j|i|}tj|d<|s.|`|`|S|dtjurL||t|d d ksfJd|j||g|Ri|}tj|d<|S)a :meth:`step` carries out the following two operations: 1. Internally invokes ``unscale_(optimizer)`` (unless :meth:`unscale_` was explicitly called for ``optimizer`` earlier in the iteration). As part of the :meth:`unscale_`, gradients are checked for infs/NaNs. 2. If no inf/NaN gradients are found, invokes ``optimizer.step()`` using the unscaled gradients. Otherwise, ``optimizer.step()`` is skipped to avoid corrupting the params. ``*args`` and ``**kwargs`` are forwarded to ``optimizer.step()``. Returns the return value of ``optimizer.step(*args, **kwargs)``. Args: optimizer (torch.optim.Optimizer): Optimizer that applies the gradients. args: Any arguments. kwargs: Any keyword arguments. .. warning:: Closure use is not currently supported. closurez@Closure use is not currently supported if GradScaler is enabled.rvr,z7step() has already been called since the last update().N_step_supports_amp_scalingZ grad_scalerzGradScaler is going to stop passing itself as a keyword argument to the passed optimizer. In the near future GradScaler registers `grad_scale: Tensor` and `found_inf: Tensor` to the passed optimizer and let the optimizer use them directly.csg|]}|jjddqS)T)r)r r)rrtZscalerrr dsz#GradScaler.step..r-rz/No inf checks were recorded for this optimizer.)r8rvrmrDr0rlr r+hasattrr{inspect signature parametersr6r7 FutureWarningupdater)_check_inf_per_device_get_scale_asyncr r&r'rurcr*Z grad_scalerfrjrOry) rrdrwrxrpr!Zkwargs_Zhas_grad_scaler_kwargrfrr}rrv%sR        zGradScaler.stepNcs|js dS|d\}|durt|tr8|j|nLd}t|tjjsRJ|| dksfJ||j dusxJ||j |nzfdd|j D}t|dksJd |d}t|dkrtdt|D]}|||7}qt|||j|j|jtt|_ dS) a Updates the scale factor. If any optimizer steps were skipped the scale is multiplied by ``backoff_factor`` to reduce it. If ``growth_interval`` unskipped iterations occurred consecutively, the scale is multiplied by ``growth_factor`` to increase it. Passing ``new_scale`` sets the new scale value manually. (``new_scale`` is not used directly, it's used to fill GradScaler's internal scale tensor. So if ``new_scale`` was a tensor, later in-place changes to that tensor will not further affect the scale GradScaler uses internally.) Args: new_scale (float or :class:`torch.cuda.FloatTensor`, optional, default=None): New scale factor. .. warning:: :meth:`update` should only be called at the end of the iteration, after ``scaler.step(optimizer)`` has been invoked for all optimizers used this iteration. Nrz[new_scale should be a float or a 1-element torch.cuda.FloatTensor with requires_grad=False.r Fcs.g|]&}|dD]}|jjddqqS)r-TrM)rcr r)rrstaterfr/rrr~sz%GradScaler.update..rz,No inf checks were recorded prior to update.)r8rDrNror/fill_r&ZcudaZ FloatTensorZnumelZ requires_gradZcopy_r0rcrOrangeZ_amp_update_scale_r:r;r<rr.)rZ new_scaler>reasonZ found_infsZfound_inf_combinedirrrr|s6   zGradScaler.updatecCs|jSr]rrrrrrszGradScaler._get_scale_asynccCs*|jr"|jdur|jS|SdSdS)z Returns a Python float containing the current scale, or 1.0 if scaling is disabled. .. warning:: :meth:`get_scale` incurs a CPU-GPU sync. Nr5)r8r/r9rrqrrrr get_scaleszGradScaler.get_scalecCs|jS)zL Returns a Python float containing the scale growth factor. r:rrrrget_growth_factorszGradScaler.get_growth_factorcCs ||_dS)zd Args: new_scale (float): Value to use as the new scale growth factor. NrrZ new_factorrrrset_growth_factorszGradScaler.set_growth_factorcCs|jS)zM Returns a Python float containing the scale backoff factor. r;rrrrget_backoff_factorszGradScaler.get_backoff_factorcCs ||_dS)ze Args: new_scale (float): Value to use as the new scale backoff factor. Nrrrrrset_backoff_factorszGradScaler.set_backoff_factorcCs|jS)zF Returns a Python int containing the growth interval. r<rrrrget_growth_intervalszGradScaler.get_growth_intervalcCs ||_dS)za Args: new_interval (int): Value to use as the new growth interval. Nr)rZ new_intervalrrrset_growth_intervalszGradScaler.set_growth_intervalcCs(|jr |jdur|jS|jSdSdS)Nr)r8r>r=rqrrrr_get_growth_trackerszGradScaler._get_growth_trackercCs|jS)zM Returns a bool indicating whether this instance is enabled. )r8rrrr is_enabledszGradScaler.is_enabledcCs(|jr$||j|j|j|dSiS)a Returns the state of the scaler as a :class:`dict`. It contains five entries: * ``"scale"`` - a Python float containing the current scale * ``"growth_factor"`` - a Python float containing the current growth factor * ``"backoff_factor"`` - a Python float containing the current backoff factor * ``"growth_interval"`` - a Python int containing the current growth interval * ``"_growth_tracker"`` - a Python int containing the number of recent consecutive unskipped steps. If this instance is not enabled, returns an empty dict. .. note:: If you wish to checkpoint the scaler's state after a particular iteration, :meth:`state_dict` should be called after :meth:`update`. )r\r?r@rAr>)r8rr:r;r<rrrrr state_dictszGradScaler.state_dictcCs|js dSt|dkrtd|d|_|jdurB|j|d|d|_|d|_|d|_|d|_ |j dur|j |ddS) z Loads the scaler state. If this instance is disabled, :meth:`load_state_dict` is a no-op. Args: state_dict(dict): scaler state. Should be an object returned from a call to :meth:`state_dict`. NrzeThe source state dict is empty, possibly because it was saved from a disabled instance of GradScaler.r\r?r@rAr>) r8rOrmr9r/rr:r;r<r=r>)rrrrrload_state_dicts        zGradScaler.load_state_dictcCsR|j}|jrNt|jdks&Jd||d<||d<d|d<d|d<|S)NrzpA GradScaler instance may only be pickled at the beginning of an iteration, or at the end after scaler.update().r9r=r/r>)__dict__rr8rOr0rrrrrrr __getstate__'s   zGradScaler.__getstate__cCs|j|dSr])rrrrrr __setstate__5szGradScaler.__setstate__cCsj|d\}}tjddtj|jd}tjddtj|jd}||||d|jt|d<|jt|dS)NrrEr5rFrkTr-)rDr&rHrIrrir0rl)rrdr/_Zdummy_inv_scalerfrrrr8s z GradScaler._check_inf_per_devicecCs|jt|dS)Nr-)r0rl)rrdrrr_found_inf_per_deviceCsz GradScaler._found_inf_per_device)r1r2r3r4T)N)%r"r#r$rr&r'__annotations__rintstrrrr rDrLr\rirjryrvrrrrrrrrrrrrrrrrrrrrrr-sB B ,(4W ;  ) collectionsrrenumrtypingrrrrr r rr6r&commonr __all__rr r.rrrrrs