a d4@sUddlZddlZddlmZddlmZmZmZmZm Z m Z ddl Z ddl m Z ddl mZddlmZddlmZgZeeed<eeZGdd d ejZeeed d d ZdS) N)deepcopy)Any CollectionDictListMappingUnion)optim) ShardedTensor)FullyShardedDataParallel__all__c@seZdZdZdeeeeje ffe j e eee fejddddZddZeee fdd d Zde dd d d Zeee fddddZeee fddddZddddZeee fdddZeee fdddZdS)_NamedOptimizera ``_NamedOptimizer`` takes a dict of parameters and exposes ``state_dict`` by parameter key. We replace the original key (number) in an optim to the fully qualifed name (FQN) string. User can initialize the optim as they initialize a PyTorch optim, the only difference is that they also need to pass in the FQN of each parameters. Args: named_parameters (Mapping[str, Union[torch.Tensor, ShardedTensor]]): Mapping from FQN to parameter. optimizer_class (optim.Optimizer): The class of optimizer to instantiate. param_groups (Collection[Mapping[str, Any]]): `param_groups` to pass to optimizer if specified. The key of the inner map needs to be FQNs. Default: None module (nn.Module): the module whose parameters to updated by the optimizer. args: arguments to pass to the optimizer constructor. kwargs: arguments to pass to the optimizer constructor. Example:: >>> # xdoctest: +SKIP("distributed") >>> from torch import optim >>> from torch.distributed.optim import _NamedOptimizer >>> >>> # Define the named optimizer. >>> m = Model(...) >>> named_optim = _NamedOptimizer(m.named_parameters(), optim.SGD) >>> # Forward pass + backward pass. >>> named_optim.step() >>> ... >>> # Call state_dict for the named optimizer returns a FQN state_dict. >>> named_optim.state_dict() Warning: This API is still in development and subject to change. TODO: Add tutorial for _NamedOptimizer. TODO: Add documentation in the docstring for the public attributes like self.param_groups and self.named_parameters. N)named_parametersoptimizer_class param_groupsmodulereturnc Ostjd||_|t||_|dur6|jn|}||g|Ri||_||_ |durrt |j |_ nft ddd|jD}g} |D]8} | dD]*} | |vrtd| d| || qq| |_ |jj|_dS)Nz'torch.distributed.optim._NamedOptimizerzvSince we pass in param_groups, we will use param_groups to initialize the optimizer, not all parameters of the module.cSsi|]\}}||qSr.0keyparamrrp/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/torch/distributed/optim/named_optimizer.py [z,_NamedOptimizer.__init__..paramszExpect param name z% found in param group but is missing.)torchZ_CZ_log_api_usage_oncer_param_groups_checkdictrvalues _optimizerrlistkeysordered_param_keyswarningswarnitems ValueErrorappend) selfrrrrargskwargsZparams_for_optimizer param_to_keyr#grouprrrr__init__>s<    z_NamedOptimizer.__init__cCs|jdur|jD]x}t|ts&Jdd|vs6Jd|d}t|tjrP|g}t|}|D]"}t|tjs\tdt|q\||d<qdS)Nparam group must be a dictrz#param group must contain key paramsz>optimizer can only optimize Tensors, but one of the params is )r isinstancerrTensorr! TypeErrortypename)r) param_grouprrrrrrhs     z#_NamedOptimizer._param_groups_check)rc sj}|d}fdd|dD}g}|D]b}g}|dD]}|j|qDdt|i}|D]\} } | dkrnt| || <qn||q4||dS)z Return the ``state_dict`` of the optimzer. Instead of using number to index parameters, we will use module fully qualifed name (FQN) as the key. rcsi|]\}}j||qSr)r#)rZst_key state_valr)rrrsz._NamedOptimizer.state_dict..stater)r7r)r state_dictr&r(r#sortedr_post_state_dict) r)r8rZ ret_stateZ ret_groupsr- param_keysrZ ret_groupkvrr6rr8ys       z_NamedOptimizer.state_dict)closurercCs|jj|ddS)z Performs a single optimization step. This will call :meth:`torch.optim.Optimizer.step` on the wrapped optimizer. r>N)r step)r)r>rrrr@sz_NamedOptimizer.step)r8rc Cs|j}||}|d}|d}t|dkr8tdt|jD]l\}}||vrZqBt||t||krtdt||d|dt||||D]\}}|||vrtd|d|d|||} t |t rnt | t sJt| } t| } | | kr.z%some parameters are not in the moduleN) r0rrr1r!rr&r'r#r(r add_param_groupr)r)r4rr,rrrrrKs     z_NamedOptimizer.add_param_groupcCsB|jD]&\}}|jr t|}tj||_q |jdddS)z Runs a dummy optimizer step, which allows to initialize optimizer state because we do lazy init for most optimizers. This allows doing in-place loading of optimizer state from a checkpoint. Nr?) rr&Z requires_gradrZ zeros_likeZautogradVariableZgradr@)r)_rtrrr init_states  z_NamedOptimizer.init_statecCs"t|jtrt|j|j|S|SN)r0rFSDPZload_optim_state_dict_pre_hookr r)r8rrrrC-s   z$_NamedOptimizer._pre_load_state_dictcCs"t|jtrt|j|j||SrP)r0rrQZoptim_state_dict_post_hookr rRrrrr:4s z _NamedOptimizer._post_state_dict)NN)N)__name__ __module__ __qualname____doc__rstrrrr1r r OptimizerrrnnModuler.rrr8r@rIrKrOrCr:rrrrr s$. * kr )r;rcCsdt|S)zQ Concatenate all param keys as a unique indentifier for one param group. rA)joinr9)r;rrrrH:srH)loggingr$copyrtypingrrrrrrrZtorch.nnrYr Z'torch.distributed._shard.sharded_tensorr Ztorch.distributed.fsdpr rQr rW__annotations__ getLoggerrSloggerrXr rHrrrrs        )