a d%> @s ddlZddlZddlmZmZddlmZmZmZm Z m Z m Z m Z m Z mZddlmZddlmZgdZedddZGd d d eZejeee eejed d d ZGdddeZejeeeedddZejeee eejedddZejeedddZejeeedddZedddfejeeee e eeje e eejedddZej ej!he_"ej#he_$ej%eee d d!d"d#Z&ejeejd$d%d&Z'ejeejd'd(d)Z(d0ejeee eje ej)eee ejefd+d,d-Z*Gd.d/d/Z+dS)1N)ABCabstractmethod) AnyCallablecastDict GeneratorOptionalSetTupleType) _BatchNorm)always_wrap_policylambda_auto_wrap_policytransformer_auto_wrap_policysize_based_auto_wrap_policy enable_wrapwrapModuleWrapPolicyreturncOsdS)z A simple recursive wrap policy that always returns ``True``. This means that every submodule is wrapped by the wrapper class in :func:`_recursive_wrap`. T)argskwargsrrd/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/torch/distributed/fsdp/wrap.pyrsrc@s.eZdZdZddZeeedddZdS) _FSDPPolicyzv This defines an abstract base class that represents an FSDP policy for constructing ``FlatParameter`` s. cCsdSNrselfrrr__init__*sz_FSDPPolicy.__init__rcCsdSrrrrrrpolicy-sz_FSDPPolicy.policyN) __name__ __module__ __qualname____doc__rpropertyrrr rrrrr"s r)modulerecursenonwrapped_numelmodule_classesrcCs|rdSt|t|S)a This auto wrap policy wraps every module that is an instance of any type in ``module_classes`` as its own FSDP instance. The root module given by ``module`` is always wrapped as an FSDP instance regardless. Since the wrapping proceeds bottom up, each FSDP instance manages the parameters in its subtree excluding any already managed by a child FSDP instance. Args: module (nn.Module): Current module being considered. recurse (bool): If ``False``, then this function must decide whether ``module`` should be wrapped as an FSDP instance or not. If ``True``, then the function is still recursing down the module tree as a part of the DFS. nonwrapped_numel (int): Parameter numel not yet wrapped. module_classes (Set[Type[nn.Module]]): Set of module classes that are wrapped as FSDP instances. Returns: ``True`` if ``recurse=True``, and whether ``module`` should be wrapped if ``recurse=False``. T) isinstancetuple)r&r'r(r)rrr_module_wrap_policy3sr,csJeZdZdZeeejdddZe ddZ e dfdd Z Z S) rz5This is a wrapper around :func:`_module_wrap_policy`.r)cCstjt|d|_t||_dS)Nr-) functoolspartialr,_policystr_module_classes_str)rr)rrrrVs zModuleWrapPolicy.__init__cCs|jSr)r0rrrrr ]szModuleWrapPolicy.policyrcstd|jdS)N())super__repr__r2r __class__rrr6aszModuleWrapPolicy.__repr__)r!r"r#r$r r nnModulerr%r r1r6 __classcell__rrr7rrSs  r)r&r'r( lambda_fnrcCs|rdS||S)aU A convenient auto wrap policy to wrap submodules based on an arbitrary user function. If `lambda_fn(submodule) == True``, the submodule will be wrapped as a `wrapper_cls` unit. Return if a module should be wrapped during auto wrapping. The first three parameters are required by :func:`_recursive_wrap`. Args: module (nn.Module): Current module being considered. recurse (bool): If ``False``, then this function must decide whether ``module`` should be wrapped as an FSDP instance or not. If ``True``, then the function is still recursing down the module tree as a part of the DFS. nonwrapped_numel (int): Parameter numel not yet wrapped. lambda_fn (Callable[[nn.Module], bool]): If this returns ``True``, then this module will be wrapped. Tr)r&r'r(r<rrrresr)r&r'r(transformer_layer_clsrcCst||||S)a- See :func:`_module_wrap_policy`, where ``transformer_layer_cls`` is the same as ``module_classes``. Note that shared parameters must be wrapped in the same FSDP instance, so this auto wrap policy can help wrap shared embeddings into the same FSDP instance for transformer models. )r,)r&r'r(r=rrrrs r)r&r'rcOs|rdSt|tSdS)zQ A policy that wraps ``BatchNorm`` instances in their own FSDP instance. TN)r*r )r&r'rrrrr_wrap_batchnorm_individuallys r>)r&r'r(rcstfdd|DS)zv A policy that wraps ``module`` if any policy in the passed in iterable of ``policies`` returns ``True``. c3s|]}|VqdSrr).0r r&r(r'rr z_or_policy..)any)r&r'r(Zpoliciesrr@r _or_policys rDgחA)r&r'r(min_num_paramsforce_leaf_modulesexclude_wrap_modulesrcCs`|durtjn|}|dur tjn|}|}||k}|rH|oFt|t| S|oZt|t| SdS)a A size-based auto wrap policy. Args: module (nn.Module): Current module being considered. recurse (bool): If ``False``, then this function must decide whether ``module`` should be wrapped as an FSDP instance or not. If ``True``, then the function is still recursing down the module tree as a part of the DFS. nonwrapped_numel (int): Parameter numel not yet wrapped. min_num_params (int): Customizable policy input that controls the size threshold over which a module is ready to be wrapped. This is in units of numel. force_leaf_modules (Set[Type[nn.Module]]): Set of module types to keep as leaves, i.e. their children will never be wrapped. exclude_wrap_modules (Set[Type[nn.Module]]): Set of module types to be excluded in wrapping. Returns: Whether ``module`` should be wrapped. N)rFORCE_LEAF_MODULESEXCLUDE_WRAP_MODULESr*r+)r&r'r(rErFrGZmin_nonwrapped_numelZis_largerrrrs!r)NNN) wrapper_clswrapper_kwargsrcksHid|i|}tfi|dVWdn1s:0YdS)a Context manager to wrap modules using a wrapper. Useful for when you'd like to apply the same configuration arguments to all child modules that you wrap. A particularly important use case is wrapping large layers so that they get sharded (in-place) during initialization, to avoid running out of system memory. Large layers can indicate that they should be sharded via the ``wrap`` annotation and this context manager can provide the exact configuration for these nested instances. Usage:: with enable_wrap(wrapper_cls, **params): # Wraps layer in FSDP by default if within context self.l1 = wrap(torch.nn.Linear(5, 5)) Args: wrapper_cls: Class that `wrap` annotation will `wrap` modules with, such as `FullyShardedDataParallel`. **wrapper_kwargs: Configuration settings that will be passed to all ``wrap`` instances inside the context rJN)_ConfigAutoWrap)rJrKrrrrrsr)r&wrap_overridesrcKs:tjr6tjdusJitj|}t|tjfi|S|S)a Annotate that a module should be wrapped. Annotated modules will only be wrapped if inside of an :func:`enable_wrap` context manager. This allows a module to be initialized both with and without a wrapper without code change. The class that this function wraps the passed in ``nn.Module`` with is the passed in ``wrapper_cls`` argument into ``enable_wrap``. Both ``enable_wrap`` and ``wrap`` can take in kwargs specifying how to construct the ``wrapper_cls`` instance. In the case of duplicate kwargs in ``enable_wrap`` and ``wrap``, the argument passed into ``wrap`` will be respected. Usage:: with enable_wrap(wrapper_cls=FSDP, **fsdp_config): # Wraps layer in FSDP by default if within context self.l1 = wrap(torch.nn.Linear(5, 5)) Args: module (nn.Module): module to wrap (if in :func:`enable_wrap` context) **wrap_overrides: configuration overrides that will take priority over the values provided by the :func:`enable_wrap` context N)rLin_autowrap_contextrJr_wrap)r&rMrrrrsr)r&rJrcKsD|dus Jt|dr4i||j}||fi|S||fi|S)N_wrap_overrides)hasattrrP)r&rJrZ overridesrrrrO3s   rOF)r&auto_wrap_policyrJignored_modulesignored_paramsonly_wrap_childrenrrc s4|dusJd|dus Jd|D]>\}}||vr:q(zt|tt|rPJWq(tydYq(0q(tfdd|D} |dusJ||d| dr,d} |D]D\} }||vrqtf||||d |\} } t || | | | 7} q| | }|s$||d |dr$t ||fi|| fS|| fS|dfS) a Wraps submodules of ``module`` for which ``auto_wrap_policy`` returns ``True`` with ``wrapper_cls``. Args: module (nn.Module): Module to recursively wrap. auto_wrap_policy (Callable): A callable representing a policy that determines which modules to recursively wrap with ``wrapper_cls``. ignored_modules (Set[torch.nn.Module]): Modules to ignore when wrapping. ignored_params (Set[torch.nn.Parameter]): Parameters to ignore when wrapping; these should be the parameters contained in the modules in ``ignored_modules``. Returns: (nn.Module, int): ``module`` after wrapping and the numel recursively wrapped. NzMust specify auto_wrap_policy.zMust specify wrapper_clsc3s|]}|vr|VqdSr)Znumel)r?prTrrrAgsz"_recursive_wrap..T)r&r'r(r)r&rRrJrSrTF) Z named_modulesr*rtype TypeErrorsum parametersZnamed_children_recursive_wrapsetattrrO)r&rRrJrSrTrUr_childr(Ztotal_wrapped_numelnameZ wrapped_childZnum_wrapped_params remainderrrWrr\@sH      r\c@seZdZUdZdZeed<dZee ed<iZ e e e fed<e e e fddd Zee dd d d Zedd ddZdd ddZe e e ddddZdS)rLz Helper class to wrap modules based on default config args via a context manager. See :func:`enable_wrap` for more information. FrNNrJrrcKs ||_dSrrb)rrrrrrsz_ConfigAutoWrap.__init__)rrcCsHtjrtddt_d|vs(Jdtt|dt_|d=|t_dS)Nz]You are already within an autowrap context and we currently do not supported nested autowrap.TrJz9Expected to pass in wrapper_cls arg into _ConfigAutoWrap.)rLrNNotImplementedErrorkeysrrrJrrbrrrenable_autowrap_contexts z'_ConfigAutoWrap.enable_autowrap_contextrcCsdt_dt_it_dS)NF)rLrNrJrrrrrdisable_autowrap_contextsz(_ConfigAutoWrap.disable_autowrap_contextcCs||jdSr)rerrrrr __enter__sz_ConfigAutoWrap.__enter__)exc_typeexc_valexc_tbrcCs |dSr)rf)rrhrirjrrr__exit__sz_ConfigAutoWrap.__exit__)r!r"r#r$rNbool__annotations__rJr rrrr1rr staticmethodrerfrgrkrrrrrLs  rL)F), contextlibr.abcrrtypingrrrrrr r r r Ztorch.nnr9Ztorch.nn.modules.batchnormr __all__rlrrr:intr,rrrr>rDrZ ModuleListZ ModuleDictrIZMultiheadAttentionrHcontextmanagerrrrO Parameterr\rLrrrrs,             7 #% J