a d]@sUddlmZmZmZddlZddlmZddlm Z ddl Z ddl Z ddl Z ddl mZmZmZmZddlmmZddlmZddlmZgdZeZeeefed <eZeeefed <ejejjj gZ!Gd d d Z"e"Z#d dZ$ddZ%ddZ&e'dddZ(d'eeje)e)ee)e)fdddZ*dZ+dZ,dZ-dZ.dZ/eded d!d"Z0eded d#d$Z1Gd%d&d&Z2dS)() OrderedDict defaultdictabcN)deepcopy)chain)CallableDictListTuple)RemovableHandle) is_compiling) Optimizer register_optimizer_step_pre_hook!register_optimizer_step_post_hook_global_optimizer_pre_hooks_global_optimizer_post_hooksc@seZdZdZddZdS)_RequiredParameterzCSingleton class representing a required parameter for an Optimizer.cCsdS)Nzselfrr^/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/torch/optim/optimizer.py__repr__sz_RequiredParameter.__repr__N)__name__ __module__ __qualname____doc__rrrrrrsrcsfdd}|S)Nc sNt}z4t|jd|g|Ri|}Wt|n t|0|S)Ndifferentiable)torchZis_grad_enabledZset_grad_enableddefaults)rargskwargsZ prev_gradretfuncrr _use_grads z/_use_grad_for_differentiable.._use_gradr)r#r$rr"r_use_grad_for_differentiables r%cCs tjstr|S|SdSN)rjit is_scriptingr itemxrrr _get_value'sr,cCs"tjstrt|S|SdSr&)rr'r(r stackr*rrr_stack_if_compiling.s r.r*cCs,tjst|tjr|St|SdSr&)rr'r( isinstanceTensorsqrtmathr*rrr_dispatch_sqrt4sr3F)paramsr use_fusedreturncCsHtjs|rdS|o&tdd|D}| o>tdd|D}||fS)N)FFcss2|]*}|dup(t|tvo(|jo(t|VqdSr&)type_foreach_supported_typesis_cudaris_floating_point.0prrr Esz/_default_to_fused_or_foreach..css(|] }|dupt|tvo|jVqdSr&)r7r8r9r;rrrr>Hs)rr'r(all)r4rr5Zfusedforeachrrr_default_to_fused_or_foreach@s rAa+foreach (bool, optional): whether foreach implementation of optimizer is used. If unspecified by the user (so foreach is None), we will try to use foreach over the for-loop implementation on CUDA, since it is usually significantly more performant. (default: None)a(fused (bool, optional): whether the fused implementation (CUDA only) is used. Currently, `torch.float64`, `torch.float32`, `torch.float16`, and `torch.bfloat16` are supported. (default: None) .. note:: The foreach and fused implementations are typically faster than the for-loop, single-tensor implementation. Thus, if the user has not specified BOTH flags (i.e., when foreach = fused = None), we will attempt defaulting to the foreach implementation when the tensors are all on CUDA. For example, if the user specifies True for fused but nothing for foreach, we will run the fused implementation. If the user specifies False for foreach but nothing for fused (or False for fused but nothing for foreach), we will run the for-loop implementation. If the user specifies True for both foreach and fused, we will prioritize fused over foreach, as it is typically faster. We attempt to use the fastest, so the hierarchy goes fused -> foreach -> for-loop. HOWEVER, since the fused implementation is relatively new, we want to give it sufficient bake-in time, so we default to foreach and NOT fused when the user has not specified either flag.acapturable (bool, optional): whether this instance is safe to capture in a CUDA graph. Passing True can impair ungraphed performance, so if you don't intend to graph capture this instance, leave it False (default: False)a]differentiable (bool, optional): whether autograd should occur through the optimizer step in training. Otherwise, the step() function runs in a torch.no_grad() context. Setting to True can impair performance, so leave it False if you don't intend to run autograd through this instance (default: False)zymaximize (bool, optional): maximize the params based on the objective, instead of minimizing (default: False).Nhookr6cCstt}|t|j<|S)aRegister a pre hook common to all optimizers. The hook should have the following signature:: hook(optimizer, args, kwargs) -> None or modified args and kwargs Args: hook (Callable): A user defined hook which is registered on all optimizers. Returns: :class:`torch.utils.hooks.RemoveableHandle`: a handle that can be used to remove the added hook by calling ``handle.remove()`` )hooksr ridrDhandlerrrrts  rcCstt}|t|j<|S)aRegister a post hook common to all optimizers. The hook should have the following signature:: hook(optimizer, args, kwargs) -> None Args: hook (Callable): A user defined hook which is registered on all optimizers. Returns: :class:`torch.utils.hooks.RemoveableHandle`: a handle that can be used to remove the added hook by calling ``handle.remove()`` )rEr rrFrGrrrrs  rc@seZdZdZddZddZddZdd Zd d Zd d Z e ddZ ddZ e dedddZe dedddZddZddZd%edddZd d!Zd"d#Zd$S)&r aXBase class for all optimizers. .. warning:: Parameters need to be specified as collections that have a deterministic ordering that is consistent between runs. Examples of objects that don't satisfy those properties are sets and iterators over values of dictionaries. Args: params (iterable): an iterable of :class:`torch.Tensor` s or :class:`dict` s. Specifies what Tensors should be optimized. defaults: (dict): a dict containing default values of optimization options (used when a parameter group doesn't specify them). cCstjd||_t|_t|_|t|tj rHt dt |t t |_g|_t|}t|dkrttdt|dt sd|ig}|D]}||qd|_dS)Nzpython.optimizerzZparams argument given to the optimizer should be an iterable of Tensors or dicts, but got rz%optimizer got an empty parameter listr4T)rZ_CZ_log_api_usage_oncerr_optimizer_step_pre_hooks_optimizer_step_post_hooks_patch_step_functionr/r0 TypeErrortypenamerdictstate param_groupslistlen ValueErroradd_param_group$_warned_capturable_if_run_uncaptured)rr4rrP param_grouprrr__init__s&      zOptimizer.__init__cCs|j|j|jdS)NrrOrPrXrrrr __getstate__szOptimizer.__getstate__cCsJ|j|d|jvrt|_d|jvr0t|_||jdddS)NrIrJrF)__dict__updaterrIrJrKr setdefault)rrOrrr __setstate__s   zOptimizer.__setstate__cCsp|jjd}t|jD]L\}}|d7}|d|7}t|D] }|dkr@|d|||7}q@q|d7}|S)Nz ( zParameter Group {0} r4z {0}: {1} )) __class__r enumeraterPformatsortedkeys)r format_stringigroupkeyrrrrs zOptimizer.__repr__cCs~tjrztjrztj}|rFtdd|jDsFtd|jj dt |ddsztdd|jDrz|szt dd |_ dS) Ncss|]}|dVqdSZ capturableNrr<rgrrrr>z=Optimizer._cuda_graph_capture_health_check..z;Attempting CUDA graph capture of step() for an instance of z' but param_groups' capturable is False.rUFcss|]}|dVqdSrirrjrrrr>rka This instance was constructed with capturable=True or some of all the param_groups came with capturable=True, but step() is running without CUDA graph capture. If you never intend to graph-capture this instance, capturable=True can impair performance, and you should set capturable=False.T)rZhas_cudaZcudaZ is_availableZis_current_stream_capturingr?rP RuntimeErrorr`rgetattrwarningswarnrU)rZ capturingrrr _cuda_graph_capture_health_checks$  z*Optimizer._cuda_graph_capture_health_checkcCsdS)aEntry point for `torch.profile.profiler`. When python tracing is enabled the profiler will hook into this function at the CPython level to inspect the optimizer's parameters and param groups. It is called it after `step()` since many optimizers lazily initialize state. This is a workaround due to lack of a proper step hook on the optimizer, and will be removed if it exists. Nrrrrr_optimizer_step_codes zOptimizer._optimizer_step_codecstfdd}|S)Nc s|^}}d|jj}tjj|tt |j D]L}||||}|dur:t |t rrt |dkrr|\}}q:td|dq:|i|}|t|j t D]}||||q|WdS1s0YdS)NzOptimizer.step#{}.stepz? must return None or a tuple of (new_args, new_kwargs),but got .)rbr`rrautogradprofilerrecord_functionrrvaluesrIr/tuplerRrlrqrJr) rr r_Z profile_nameZpre_hookresultoutZ post_hookr"rrwrapper s    z,Optimizer.profile_hook_step..wrapper) functoolswraps)r#r|rr"rprofile_hook_stepszOptimizer.profile_hook_stepcCsDd|jj|_t|jjdd}|s@||jj|j_d|jj_dS)Nz Optimizer.zero_grad#{}.zero_gradhookedT)rbr`r_zero_grad_profile_namermsteprr)rrrrrrK#s zOptimizer._patch_step_functionrBrCcCst|j}||j|j<|S)aRegister an optimizer step pre hook which will be called before optimizer step. It should have the following signature:: hook(optimizer, args, kwargs) -> None or modified args and kwargs The ``optimizer`` argument is the optimizer instance being used. If args and kwargs are modified by the pre-hook, then the transformed values are returned as a tuple containing the new_args and new_kwargs. Args: hook (Callable): The user defined hook to be registered. Returns: :class:`torch.utils.hooks.RemoveableHandle`: a handle that can be used to remove the added hook by calling ``handle.remove()`` )rEr rIrFrrDrHrrrregister_step_pre_hook*s  z Optimizer.register_step_pre_hookcCst|j}||j|j<|S)aRegister an optimizer step post hook which will be called after optimizer step. It should have the following signature:: hook(optimizer, args, kwargs) -> None The ``optimizer`` argument is the optimizer instance being used. Args: hook (Callable): The user defined hook to be registered. Returns: :class:`torch.utils.hooks.RemoveableHandle`: a handle that can be used to remove the added hook by calling ``handle.remove()`` )rEr rJrFrrrrregister_step_post_hook@s  z!Optimizer.register_step_post_hookcsLidfddfdd|jD}fdd|jD}||dS) aKReturns the state of the optimizer as a :class:`dict`. It contains two entries: * state - a dict holding current optimization state. Its content differs between optimizer classes. * param_groups - a list containing all parameter groups where each parameter group is a dict rcsbdd|D}fddt|dDfdd|dD|d<t|d7|S)NcSsi|]\}}|dkr||qS)r4rr<kvrrr drkz.pack_group..cs&i|]\}}t|vrt||qSrrF)r<rfr=param_mappingsrrres  r4csg|]}t|qSrrr;rrr grkz.pack_group..)itemsr[rarR)rgpacked)r start_indexrr pack_groupbs "z(Optimizer.state_dict..pack_groupcsg|] }|qSrrr<g)rrrrjrkz(Optimizer.state_dict..cs.i|]&\}}t|tjr$t|n||qSr)r/rr0rFrrrrrlsz(Optimizer.state_dict..rOrP)rPrOr)rrPZ packed_stater)rrrr state_dictTs  zOptimizer.state_dictc s&t|}|j}|d}t|t|kr.tddd|D}dd|D}tddt||Drjtddd ttd d|Dtd d|DD}dfd d tt }|d D]0\}} ||vr||} | | || <q| ||<qddfddt||D} | || dd S)zLoads the optimizer state. Args: state_dict (dict): optimizer state. Should be an object returned from a call to :meth:`state_dict`. rPzrkz,Optimizer.load_state_dict..css|]}t|dVqdSrrrrrrr>rkcss|]\}}||kVqdSr&r)r<Zp_lenZs_lenrrrr>rkz]loaded state dict contains a parameter group that doesn't match the size of optimizer's groupcSsi|]\}}||qSrr)r<Zold_idr=rrrrrkz-Optimizer.load_state_dict..css|]}|dVqdSrrrrrrr>rkcss|]}|dVqdSrrrrrrr>rkNcst|tjr8|dkr4r(|j}|j}|St|trZfdd|DSt|t j rt |fdd|DS|SdS)zBMake a deep copy of value, casting all tensors to device of param.rcs i|]\}}|||dqS))rhrrcastparamrrrrkz;Optimizer.load_state_dict..cast..c3s|]}|VqdSr&r)r<rrrrr>rkz:Optimizer.load_state_dict..cast..N) r/rr0r:todtypedevicerNrcontainer_abcsIterabler7)rvaluerh)r)rrrs     z'Optimizer.load_state_dict..castrOcSs|d|d<|S)Nr4r)rgZ new_grouprrr update_groups z/Optimizer.load_state_dict..update_groupcsg|]\}}||qSrr)r<rZng)rrrrsz-Optimizer.load_state_dict..r)N) rrPrRrSanyzipr from_iterablerrNrr]) rrgroupsZ saved_groupsZ param_lensZ saved_lensZid_maprOrrrrPr)rrrload_state_dictss4  zOptimizer.load_state_dictT) set_to_nonec Cs|jdd}t|ds ||r0tdd}tjj|j |j D]}|dD]r}|j durT|rnd|_ qT|j j dur|j n |j d|r|j jr|j qT||j j|j j|j qTqH|r|D] \}}|D]}t|qqWdn1s0YdS)aSets the gradients of all optimized :class:`torch.Tensor` s to zero. Args: set_to_none (bool): instead of setting to zero, set the grads to None. This will in general have lower memory footprint, and can modestly improve performance. However, it changes certain behaviors. For example: 1. When the user tries to access a gradient and perform manual ops on it, a None attribute or a Tensor full of 0s will behave differently. 2. If the user requests ``zero_grad(set_to_none=True)`` followed by a backward pass, ``.grad``\ s are guaranteed to be None for params that did not receive a gradient. 3. ``torch.optim`` optimizers have a different behavior if the gradient is 0 or None (in one case it does the step with a gradient of 0 and in the other it skips the step altogether). r@FrcSsttSr&)rrQrrrrrkz%Optimizer.zero_grad..r4N)rgethasattrrKrrrtrurvrrPZgradZgrad_fnZdetach_Zrequires_grad_Z is_sparseZzero_rrappendrrwZ_foreach_zero_) rrr@Zper_device_and_dtype_gradsrgr=ryZper_dtype_gradsZgradsrrr zero_grads*            zOptimizer.zero_gradcCstdS)afPerforms a single optimization step (parameter update). Args: closure (Callable): A closure that reevaluates the model and returns the loss. Optional for most optimizers. .. note:: Unless otherwise specified, this function should not modify the ``.grad`` field of the parameters. N)NotImplementedError)rclosurerrrrs zOptimizer.stepcCs\t|tsJd|d}t|tjr2|g|d<n t|trFtdn t||d<|dD]D}t|tjs|tdt||j ddsZ|j sZ|j sZt dqZ|j D]2\}}|tur||vrt d|q|||q|d}t|tt|kr tjd d d t}|jD]}|t|dq|t|dsLt d |j|dS) aAdd a param group to the :class:`Optimizer` s `param_groups`. This can be useful when fine tuning a pre-trained network as frozen layers can be made trainable and added to the :class:`Optimizer` as training progresses. Args: param_group (dict): Specifies what Tensors should be optimized along with group specific optimization options. zparam group must be a dictr4zoptimizer parameters need to be organized in ordered collections, but the ordering of tensors in sets will change between runs. Please use a list instead.z>optimizer can only optimize Tensors, but one of the params is rNz can't optimize a non-leaf TensorzJparameter group didn't specify a value of required optimization parameter zoptimizer contains a parameter group with duplicate parameters; in future, this will cause an error; see github.com/pytorch/pytorch/issues/40967 for more information) stacklevelz7some parameters appear in more than one parameter group)r/rNrr0setrLrQrMrrZis_leafZ retains_gradrSrrequiredr\rRrnrorPr[ isdisjointr)rrVr4rnamedefaultZ param_setrgrrrrTs>          zOptimizer.add_param_groupN)T)rrrrrWrYr]rrprq staticmethodrrKrr rrrrboolrrrTrrrrr s"    @) r )F)3 collectionsrrrrrcopyr itertoolsrrnr}r2typingrrr r Ztorch.utils.hooksutilsrEr Z torch._utilsr __all__rint__annotations__rr0nnZ parameter Parameterr8rrr%r,r.floatr3rrAZ _foreach_docZ _fused_docZ_capturable_docZ_differentiable_docZ _maximize_docrrr rrrrsB