a dn@sfddlZddlZddlmZmZddlmZmZddlm Z m Z m Z m Z m Z mZmZmZmZmZmZmZddlZddlmZddlmZddlmmZddlmZddlm Z ddl!m"Z"m#Z#dd l$m%Z%m&Z&dd l'm(Z(m)Z)m*Z*m+Z+m,Z,gd Z-Gd d d eZ.GdddeZ/GdddeZ0GdddeZ1Gdddej2Z3GdddZ4ee4dfZ5dS)N)autoEnum) accumulatechain) AnyDict GeneratorIteratorList NamedTuple no_type_checkOptionalSequenceSetTupleUnion)Tensor)DTensor)_set_fsdp_flattenedHandleTrainingState)_ext_post_unflatten_transform_ext_pre_flatten_transform)_alloc_storage _free_storage_no_dispatch_record_stream _same_storagep_assert) FlatParameterFlatParamHandleFlatParamShardMetadata ParamInfoSharedParamInfoHandleShardingStrategyc@s,eZdZUdZeed<ejed<eed<dS)r!z-Information for an original module parameter. param_namemodule module_nameN__name__ __module__ __qualname____doc__str__annotations__nnModuler0r0j/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/torch/distributed/fsdp/flat_param.pyr!Hs  r!c@sFeZdZUdZeed<ejed<eed<eed<ejed<eed<dS) r"ai Additional information for a shared parameter. For each shared parameter, we designate one module and its parameter variable to be the primary owner, determined as the first one encountered in the parameter walk. These are prefixed with "prim". The primary module and parameter do not have their own :class:`SharedParamInfo` instance. r$r%r&prim_param_name prim_moduleprim_module_nameNr'r0r0r0r1r"Ps    r"c@s\eZdZUdZeedfed<eejdfed<ee dfed<eee e fdfed<dS)r a This holds metadata specific to this rank's shard of the flattened parameter. Attributes: param_names (Tuple[str, ...]): Prefixed parameter names of this rank's shard of the parameters; see :class:`FlatParameter`. param_shapes (Tuple[torch.Size, ...]): Parameter shapes of this rank's shard of the parameters; see :class:`FlatParameter`. param_numels (Tuple[int, ...]): Parameter numels of this rank's shard of the parameters; see :class:`FlatParameter`. param_offsets (Tuple[Tuple[int, int], ...]): [start, end] offsets (in units of numels) giving this rank's part of each flattened original module parameter. .Z param_namesZ param_shapesZ param_numels param_offsetsN) r(r)r*r+rr,r-torchSizeintr0r0r0r1r bs r c@s*eZdZeZeZeZeZeZdS)r#N) r(r)r*rZ FULL_SHARDZ SHARD_GRAD_OPNO_SHARDZ HYBRID_SHARDZ_HYBRID_SHARD_ZERO2r0r0r0r1r#|s r#c @s\eZdZdZeeeeeejee ee ee e ee je ee jdd ddZdS)raE This is the flattened parameter used by :class:`FullyShardedDataParallel`. It is comprised of one or more original parameters, which are flattened and concatenated to construct the flattened parameter. Under the current design, this parameter logically represents both the unsharded and sharded flattened parameter, and its data changes storages dynamically. - In the :class:`FullyShardedDataParallel` constructor, the parameter is initialized as unsharded and then sharded in-place. - At runtime, the parameter is lazily (re)-initialized. The sharded parameter data is saved in ``self._local_shard``, and a new ``Tensor`` ``self._full_param_padded`` is created, which is the all-gather destination and owns the unsharded parameter storage thereafter. (See :meth:`FlatParamHandle.init_flat_param_attributes`.) - Throughout runtime, the parameter data changes storages as needed, e.g. to the sharded flattened parameter, reduced-precision sharded flattened parameter, or the unsharded flattened parameter. Attributes: _unpadded_unsharded_size (torch.Size): Unsharded flattened parameter's size without padding. _padded_unsharded_size (torch.Size): Unsharded flattened parameter's size with padding. This is only set for sharded strategies since they require padding for the all-gather. _sharded_size (torch.Size): Sharded flattened parameter's size with padding. This is also set for ``NO_SHARD``, in which case it is the same as the unsharded sizes. (We omit "padded" because there is no analogous unpadded one.) _param_infos (Tuple[ParamInfo, ...]): Each parameter's parameter info entry; see :class:`ParamInfo`. _numels (Tuple[int, ...]): Each parameter's numel. _shapes (Tuple[torch.Size, ...]): Each parameter's shape. _fqns (Tuple[str, ...]): The original parameters' FQNs prefixed from the owning handle's ``_fully_sharded_module``. The names are guaranteed to be unique within the subtree rooted at that module. _num_params (int): Number of original parameters flattened into this flattened parameter; this is the length of ``_param_infos``, ``_numels``, ``_shapes``, and ``_fqns``. _shared_param_infos (Tuple[SharedParamInfo, ...]): Shared parameter info entries; see :class:`SharedParamInfo`. _param_extensions (Tuple[Optional[Any], ...]): Parameter extensions (i.e. some per-parameter state) used to customize pre-flatten and post-unflatten behavior. This is experimental, and users should not depend on its existence in the future. _modules (Set[nn.Module]): Modules that contain some original parameter that is flattened into the ``FlatParameter``. _shard_param_offsets (List[Tuple[int, int])): [start, end] offsets (in units of numel) giving this rank's part of each flattened original module parameter; for any parameter ``p`` that is not sharded across ranks, this will be [0, ``p.numel()``-1]. _shard_indices (Tuple[int, int]): [start, end] indices (in units of parameters) for this rank's shard of the original model parameters, where the parameters follow the order in which they were originally flattened; this indexes appropriately into any data structure that follows the flattening order (e.g. ``_param_infos``, ``_numels``, etc.). _shard_numel_padded (int): Numel padded for this rank's sharded flattened parameter. _local_shard (Tensor): Sharded flattened parameter with padding if using a sharded strategy. If using ``NO_SHARD``, then this is the unpadded unsharded flattened parameter, and there is no notion of a sharded flattened parameter or padded unsharded flattened parameter. _full_param_padded (Tensor): Unsharded flattened parameter with padding. This is not defined for ``NO_SHARD``. When using mixed precision for parameters, this has the low precision. _full_prec_full_param_padded (Tensor): Full precision unsharded flattened parameter with padding. This is used for unsharding outside of computation when using mixed precision for parameters. This is never defined for ``NO_SHARD``. _post_backward_hook_state (Tuple[AccumulateGrad, RemovableHandle]): Flattened parameter's :class:`AccumulateGrad` object and post-backward hook handle. _mp_shard (Tensor): Low precision sharded flattened parameter with padding. This is only defined when parameter mixed precision is enabled. For ``NO_SHARD``, this is used for computation. _cpu_grad (Tensor): Sharded gradient with padding stored on CPU. This is only defined when offloading parameters is enabled. _saved_grad_shard (Tensor): Sharded gradient with padding from previous iterations for gradient accumulation without :meth:`no_sync`. _params (Optional[List[nn.Parameter]]): The original parameter variables if ``use_orig_params=True`` and ``None`` otherwise. _shared_params (Optional[List[nn.Parameter]]): The original shared parameter variables if ``use_orig_params=True`` and ``None`` otherwise. _tensors (Optional[List[Optional[Tensor]]]): This saves the ``Tensor`` views created in the forward and tracked by autograd when ``use_orig_params=True`` and is ``None`` otherwise. This is to preserve those ``Tensor`` variables for the backward to ensure that the ``FlatParameter`` 's ``AccumulateGrad`` object does not change in which case the post-backward hook does not run. This is relevant for cases like reentrant activation checkpointing. _is_grad_none (Optional[List[bool]]): A mask over the original parameters' gradients indicating if it is logically ``None`` or not if ``use_orig_params=True`` and ``None`` otherwise. This is needed because only some of the parameters may have ``None`` gradient, in which case the ``FlatParameter`` gradient must be non-``None`` and must use zeros to approximate those original ``None`` gradients. This mask informs FSDP to set the original parameter gradients to ``None`` (instead of zeros) as needed. N) param_infosnumelsshapesfqnsshared_param_infosparam_extensionsparams shared_paramsreturnc Cst|t|ksJt|t|ks(Jt|t|ksz/FlatParameter._init_metadata..cSsh|] }|jqSr0rCrEZspir0r0r1rGrHNcSsg|]}dqS)Fr0rE_r0r0r1 #sz0FlatParameter._init_metadata..cSsg|]}dqSNr0rJr0r0r1rL&sF)lenZ _num_paramstuple _param_infos_numels_shapes_fqns_shared_param_infos_param_extensionsunionZ_modules_params_shared_paramsrrrange _is_grad_none_tensorssize_unpadded_unsharded_size_post_backward_called) selfr:r;r<r=r>r?r@rAparamr0r0r1_init_metadatasH             zFlatParameter._init_metadata)r(r)r*r+r r!r8r6r7r,r"rr r. Parameterrar0r0r0r1rsm  rc seZdZdZeejejej e e e ej e ej e eje d fdd Zee ejeje ddddZeeeje ed d d Ze ej e ej dd d dZeddZeeeddddZeeeeeeefdfeeeffdddZeeeeeeefdddZeeeeeeefdddZeeeeejdddZ e!eeefdd d!Z"e#dd"d#Z$e%eddd$d%Z&e dd&d'Z'd(d)Z(d*d+Z)e dd,d-Z*d.d/Z+ejdd0d1Z,eed2d3d4Z-ejdd2d5d6Z.d7d8Z/d9d:Z0ed;d<Z1d=d>Z2d?d@Z3dAdBZ4e5j6dCdDZ7e dEdFdGZ8dHdIZ9dJdKZ:dddLdMZ;edee ejedddTdUZ?e5j6e@ddVdWZAedddXdYZBedddZd[ZCee dd\d]ZDe eeeejee dd^d_d`ZEdadbZFdcddZGdedfZHeIejddgdhZJee didjdkZKe.convert_to_params..r0)rr0r0r1convert_to_paramssz;FlatParamHandle._init_flat_param..convert_to_params)setdiscardrN ValueErrorZ named_modulesZnamed_parametersappendr"typerr|Zis_floating_point requires_gradrr!numelshaperflatten_paramsr{r rr6rr.rbra)r_r@r%rkZ params_setr:r;r<r=r>Zshared_param_memoZparams_to_flattenrAr?r|rZsubmodule_name submoduler$r`r3r4r2 extensionZfqnrr0r0r1rzss           z FlatParamHandle._init_flat_param)r@rrBcCsTt,dd|D}tj|dd}Wdn1s:0Yt||d}|S)a Flattens the parameters in ``params`` into a single :class:`FlatParameter`. This should be the only way used to construct :class:`FlatParameter` s. We expose this factory method for checkpointing (e.g. sharded state dict). The flattened parameter's metadata should only be initialized once (see :meth:`_init_metadata`), but its tensor data may be reloaded. cSs0g|](}t|tjr"|dn|dqS))rr.rbdetachreshape)rEpr0r0r1rLsz2FlatParamHandle.flatten_params..rZdimN)r)r6no_gradcatr)r@rZ flat_paramsZflat_param_datar{r0r0r1rs , zFlatParamHandle.flatten_params)rgrhrBcCsh|du|_|du|_|jr0|js0||_|j|_n|p8|j|_|pD|j|_|jdusVJ|jdusdJdS)aM Precondition: ``self.flat_param`` is set via :meth:`_init_flat_param`. This ensures that this handle's parameters have a single dtype. Postcondition: This sets ``self._fwd_bwd_param_dtype`` and ``self._reduce_dtype``. If ``mp_param_dtype`` or ``mp_reduce_dtype`` is ``None``, then we assume the original parameter dtype. One special case is if ``mp_param_dtype`` is not ``None`` and ``mp_reduce_dtype`` is ``None``, in which case we assume the gradient reduction dtype matches the forward/backward parameter dtype. N)_low_prec_param_dtype_specified _low_prec_reduce_dtype_specified_fwd_bwd_param_dtype _reduce_dtyper})r_rgrhr0r0r1rs     z)FlatParamHandle._init_param_reduce_dtypescCs|j}|js$|dd|dnt|dkd|}t||j |j \}}| |||j }||j dd}||||| dkr| d|jr|dS)a Shards the handle's ``FlatParameter``. In terms of memory, this allocates new memory for the sharded flattened parameter and frees the unsharded flattened parameter's storage. Postcondition: ``self.flat_param`` is the sharded flattened parameter. Shard metadata attributes are set for all sharding strategies. ``process_group``, ``rank``, and ``world_size`` attributes are set if using a sharded strategy. rrz;The `FlatParameter` is not the sole occupant of its storageN)r{uses_sharded_strategy_init_shard_metadatarrZstorage_offset_typed_storager _get_shardrprqset__sizeZ_resize_rt_use_sharded_views)r_r{Z orig_storagesharded_flat_param numel_paddedstartendr0r0r1shard!s&      zFlatParamHandle.shard)rrrrBcCsx|j|j_|j}t|dko(||kd|d|t||kd|d||||\|j_|j_||j_dS)a Initializes shard-related metadata for this rank's shard of the flattened parameter: ``_sharded_size``, ``_shard_param_offsets``, ``_shard_indices``, and ``_shard_numel_padded``. Args: numel_padded (int): Numel padded for this rank's sharded flattened parameter. start (int): Start index in the sharded flattened parameter assigned to this rank. end (int): End index (inclusive) in the sharded flattened parameter assigned to this rank. If this exceeds the sharded flattened parameter's numel, then it is truncated. Precondition: ``self.flat_param`` 's data is the sharded flattened parameter. rzstart: z end: znumel_padded: z sharded_flat_param_numel: N) r{r\ _sharded_sizerr_get_shard_metadata_shard_param_offsets_shard_indicesZ_shard_numel_padded)r_rrrZsharded_flat_param_numelr0r0r1rBs " z$FlatParamHandle._init_shard_metadata.)rrrBc Cs|}g}g}t|D]Z\}\}}||ks||kr6q||krDd} n||} t|||} |||| | fqt|dkrd} t|dksJn0|d|df} t|| d| ddksJt|| fS)a Computes the shard metadata based on ``start`` and ``end``, which give the closed interval of the unsharded flattened parameter specifying the shard. Args: start (int): Start index (in units of numel) of this rank's shard of the flattened parameter. end (int): End index (in units of numel and inclusive) of this rank's shard of the flattened parameter. Return: Tuple[Tuple[Tuple[int, int], ...], Tuple[int, int]]: See ``_shard_param_offsets`` and ``_shard_indices`` in :class:`FlatParameter` 's docstring. r)rrrr)_get_flat_param_offsets enumerateminrrNrO) r_rrZflat_param_offsetsZshard_param_indices_rangeZshard_param_offsetsi param_start param_endZintra_param_startZintra_param_endZshard_param_indicesr0r0r1rgs2  z#FlatParamHandle._get_shard_metadata)tensorrprqrBcCsdt||}t||dkr0|dd}n||}|d|}|dks\Jd||fS)aa Returns the shard of ``tensor`` without any padding for the given ``rank`` and ``world_size`` and the numel to pad for that shard. If ``tensor`` is already flattened or may be viewed in the flattened shape (which is true in the expected usage), then this method does not allocate any new tensor memory. rrz5Chunk's size should be at most the first chunk's size)r6flattenchunkrNZ new_emptyr)rrprqchunksr numel_to_padr0r0r1_get_unpadded_shardsz#FlatParamHandle._get_unpadded_shardcCs:t|||\}}|}|dkr2t|d|g}||fS)a( Returns the shard of ``tensor`` with padding for the given ``rank`` and ``world_size`` and the numel padded for that shard. This method allocates new memory (via :meth:`clone`) since the unsharded ``tensor`` may be deallocated after this method returns. r)rrcloneFpad)rrprqrrrr0r0r1rs zFlatParamHandle._get_shardcCs^t|jdksJ|jt|||\}}|}t|dksJJ|t|d|gS)z Returns the shape of ``tensor`` after sharding including padding. This requires ``tensor`` to have 1D shape and ensures that the returned shape is 1D. rr)rNrrrr\r6r7)rrprqZunpadded_sharded_tensorrZunpadded_sharded_sizer0r0r1_get_sharded_sizesz!FlatParamHandle._get_sharded_size)rBcCsBtt|jj}dg|dd}dd|D}tt||}|S)zReturns [start, end] offsets of each original parameter's flattened data in the unsharded flattened parameter (without padding).rNrcSsg|] }|dqS)rr0)rErr0r0r1rLrHz;FlatParamHandle._get_flat_param_offsets..)listrr{rQzip)r_Zcumulative_sumZstartsZendsr5r0r0r1rs z'FlatParamHandle._get_flat_param_offsetscCst|jdrt|jds Jd|jjd}|jjd}||krNt||dntdd}t|jj||jj||jj||jjddS)z`Returns shard-related metadata specific to this rank's shard of the flattened parameter.rrz'Shard metadata has not been initializedrrN) hasattrr{rslicer rSrRrQr)r_Zshard_param_start_indexZshard_param_end_indexslr0r0r1shard_metadatas"     zFlatParamHandle.shard_metadatacCsH|j}|j|jkr<|js |j|_|js4|js4|j|_|j|_td}|j rft |j|kd|jn | |j|j |_ |j r|j |_ tj|j |d|_|jrtj|j |j|jd|_t|j|jrD|jr|jn|j}||j}tj||j|d|_|j|_t|j|jrDtj||j|jd|_t|jdS)a This initializes some attributes on the handle's ``FlatParameter``. This should be called during lazy initialization since it requires the parameter to be on the compute device if not offloading to CPU and we want to give users the chance to move the parameter appropriately after the FSDP constructor. For each tensor attribute on the ``FlatParameter``, see the unshard and reshard methods in this class for the allocation and free pattern. cpuzWExpects the `FlatParameter` to be on CPU when parameter CPU offloading is enabled, not rd)rdr|N)r{r|r}rrrrr6rdrsr_check_on_compute_devicedata _local_shardZ pin_memory zeros_like _cpu_grad_uses_param_mixed_precision _mp_shardrrrrqzeros_full_param_paddedr\_padded_unsharded_size_full_prec_full_param_padded)r_r{Z cpu_deviceZunsharded_param_dtypeZpadded_unsharded_numelr0r0r1init_flat_param_attributessh         z*FlatParamHandle.init_flat_param_attributescCszd}|jr|}|jr(|js(|s(nB|jrB|jsB|d}n(|jrj|jj |j krj|j |j ddd}| |j|S)a$ Returns: ``False`` if this is a no-op and ``True`` otherwise. Postcondition: ``self.flat_param`` 's data is on the device for communication and is what should be all-gathered. This means that it matches the dtype of the expected unsharded parameter. FTZ non_blocking) rt_writeback_orig_paramsrrs needs_unshardr_force_full_precision_use_low_precision_shardr{rd flat_param_tor)r_retr0r0r1 pre_unshardTs$  zFlatParamHandle.pre_unshardcCsF||j}t|j|j|j|jj|jdd|j|_ dS)z Allocates the low precision shard directly on the compute device and switches to using the low precision sharded flattened parameter. TrN) _check_low_precision_shardr{rrrr\copy_tordrr_r{r0r0r1ros z(FlatParamHandle._use_low_precision_shardcCsJ|s*|jr|n|j}||dS|}||}||dS)a Runs the unshard logic. This includes all-gathering the flattened parameter and switching to using the unsharded flattened parameter. If the handle does not need unsharding, then this only switches to using the unsharded flattened parameter. For ``NO_SHARD``, this is a no-op. If FSDP is in :meth:`summon_full_params` and the handle uses parameter mixed precision, then the parameter is forced to full precision. N)rr _get_padded_unsharded_flat_paramr{_use_unsharded_flat_param"_alloc_padded_unsharded_flat_param_all_gather_flat_param)r_unsharded_flat_parampadded_unsharded_flat_paramr0r0r1unshards    zFlatParamHandle.unshardcCs,|js dS|}||k}| S)zBReturns if the handle's flattened parameter needs to be unsharded.F)rrrrr)r_rZalready_unshardedr0r0r1rs zFlatParamHandle.needs_unshardcCs0||j}|}||t||j|S)a* Allocates the *padded* unsharded flattened parameter. The unpadded unsharded flattened parameter is always a view into the padded one. This padded parameter is saved to a different attribute on the ``FlatParameter`` depending on if we force full precision. )_check_sharded_strategyr{r_check_storage_freedrrr_r{rr0r0r1rs   z2FlatParamHandle._alloc_padded_unsharded_flat_paramcCs@||j}|jr6|j}t|j|jkd|jn|j}|S)z Returns a reference to the padded unsharded flattened parameter depending on the calling context. This should only be called if using a sharded strategy. zExpects full precision but got )rr{rrrr|rrrr0r0r1rs  z0FlatParamHandle._get_padded_unsharded_flat_param)rrBcCsftt|dot|dd|jj}||j}t||kd|d|t|||j|S)z All-gathers the handle's flattened parameter to the destination ``padded_unsharded_flat_param``, and switches to using the all-gathered tensor. rjrqzEExpects a process group and world size to have been set via `shard()`Expects  numel but got ) rrr{rrrqrxall_gather_into_tensorrj)r_rrZexpected_numelr0r0r1rs  z&FlatParamHandle._all_gather_flat_paramcCsj|jj}|d|||j_|jtjk}|jtjk}|j rV|j | oN| dn|rf|j dddS)z Switches to using the *unpadded* unsharded flattened parameter, which is a view into the *padded* unsharded flattened parameter. NrlF) r{r]rviewrrwrFORWARD BACKWARD_PRErtr~)r_runsharded_sizeZ in_forwardZin_pre_backwardr0r0r1rs    z)FlatParamHandle._use_unsharded_flat_paramcCs$|jr|jr|||jdS)zo Runs the post-unshard logic. This includes freeing the low precision shard if needed. N)rr!_free_low_precision_sharded_paramrr{r_r0r0r1 post_unshards zFlatParamHandle.post_unshardcCs,|t|jjtjt|jjdS)z4Frees the low precision sharded flattened parameter.N)rrr{rr6cudacurrent_streamrrr0r0r1rs z1FlatParamHandle._free_low_precision_sharded_paramcCs"|js|dS|j}||tjdtj|jd}|jdu|d<t j ||j d|d|j krtd|_ |dStj|j|jd}|jdur|jt jjkrtd|jdd|_ tj|j|jd}n||j|j|_ |j }t |||j |jj}|d|||_|dS) a Unshards the handle's ``FlatParameter`` 's gradient. If all ranks have ``None`` gradient, then all original parameters will as well. This method performs an all-reduce and an all-gather. The additional all-reduce is tolerable since this method is not meant to be used on the computation critical path. Postcondition: ``_saved_grad_shard`` is defined and contains the value to set ``flat_param.grad`` after gradients are resharded. Nr)r|rdr)groupr[Rank z] Only some but not all ranks have a `None` `FlatParameter` gradient, so FSDP is using zeros to approximate those ranks' sharded gradients being `None`)r_use_unsharded_grad_viewsr{_check_unshardedr6rint32rdgradrxZ all_reducerjrq_saved_grad_shardemptyrry DebugLevelDETAILwarningswarnrpr_check_shardedrr]rr)r_r{Z num_grad_noneZpadded_unsharded_grad sharded_gradrr0r0r1 unshard_gradsD     zFlatParamHandle.unshard_gradcCs4|jr||jsdS|jj|j_t|jddS)Nr)rt_use_sharded_grad_viewsrr{rrdelattrrr0r0r1 reshard_gradLs  zFlatParamHandle.reshard_gradcCs(t|jtjtjfvd|j}|jdur$|j|jksJ|jj |j kr$| |j|jj |j k}t| pp|j d|j d|jj |j|j k}|r|s|jj |_|j}ntt|dd|j}|j j}|jr|j|kr|||_ n,|j}t|j|kd|d|jd|_dS) z Prepares the gradient for the backward computation by saving and clearing any existing sharded gradient in ``.grad`` to enable computing a new unsharded gradient. z:Expects to be in `BACKWARD_PRE` or `IDLE` (if prefetching)Nz&Expects the sharded gradient to be on but got rz7`_cpu_grad` should be defined if the gradient is on CPUzFExpects `.grad` to be the unsharded gradient in `no_sync()` with size z but got size )rrwrrrvr{rr\r]rdrrsrrrrrr|rurr)r_r{Zgrad_offloadedZprev_iter_synced_gradientsrZlocal_shard_dtypeZpadded_unsharded_sizer0r0r1prepare_gradient_for_backwardTsb        z-FlatParamHandle.prepare_gradient_for_backwardcsfdd}j}t|drB|||j|_||nXt|dr|||j|jr|j|_||nt j p|j dt|drt |ddS)z Prepares the gradient for optimizer computation by moving the sharded gradient to the ``.grad`` attribute. csFjrB|jdusJ|jjjkrB|jj|j_jrBdSrM)rurr|rrrrtr)r{rr0r1"cast_grad_to_param_dtype_if_neededs zVFlatParamHandle.prepare_gradient_for_optim..cast_grad_to_param_dtype_if_neededrrzcAll sharded parameters that received a gradient in the post-backward should use `_saved_grad_shard`N) r{rr _check_on_cpurrrrr^rrr)r_rr{r0rr1prepare_gradient_for_optims*           z*FlatParamHandle.prepare_gradient_for_optimc csL|t|j|jjkd|jjd|j||j|j}|}t||kd| t d| zddVWt|j|jjkd|jjd|j| }|d|j|j||n\t|j|jjkd|jjd|j| }|d|j|j||0dS)ao Moves the unpadded unsharded flattened parameter to CPU while in the context and moves it back to the previous device upon exit. For now, this assumes the ``FlatParameter`` is the unpadded unsharded flattened parameter since (1) there is no reason to include the padding in the copy and (2) there is no use case for the sharded flattened parameter. Precondition: ``self.flat_param`` 's data is the unpadded unsharded flattened parameter on the compute device, and the handle uses a sharded strategy. Postcondition: Same as the precondition. z Expects size rzEExpects the unpadded parameter to be a view into the padded parameterrN)rrr{r\r]rrZ _data_ptrrrr6rd_free_unsharded_flat_paramrrrr)r_Zunpadded_storage_ptrZpadded_storage_ptrrr0r0r1to_cpusF  zFlatParamHandle.to_cpu)free_unsharded_flat_paramcCs||r|dS)z Runs the reshard logic. This includes freeing the unsharded flattened parameter if ``free_unsharded_flat_param`` and switching to using the sharded flattened parameter. N)_use_sharded_flat_paramr )r_r r0r0r1reshards zFlatParamHandle.reshardcCs|jr|js|js|dS)aE Runs the post-reshard logic. This includes freeing any memory that can now be freed given that the ``FlatParameter`` points to the full precision sharded flattened parameter. Precondition: ``self.flat_param`` 's data points to the full precision sharded flattened parameter. N)rrrrrr0r0r1 post_reshards zFlatParamHandle.post_reshardcCs@||}||||t|tjt|dS)z Frees the padded unsharded flattened parameter. The tensor to free depends on the calling context since the unshard may have forced full precision, in which case a different tensor is used. N) rr_check_storage_allocatedrrr6rrr)r_rr0r0r1r s   z*FlatParamHandle._free_unsharded_flat_paramcCs|j}|jr.|jj}t|tdkd||j|_|jr||j t j kr|j duol|j ol|j j|jk}|r||n|dS)z2Switches to using the sharded flattened parameter.rz-Expects the local shard to be on CPU but got N)r{rsrrdrr6rrtrrwrrrrrr]rr)r_r{rdZaccumulated_grad_in_no_syncr0r0r1r  s&     z'FlatParamHandle._use_sharded_flat_param)r{rrBcCsh|dur |}t||jkd|jd|dddttj||jdd|j|jD}|S) a Returns unflattened ``Tensor`` views into ``tensor`` if it is not ``None`` or ``flat_param`` otherwise, where the unflattening is based on ``flat_param`` 's metadata. In other words, to get views into the unsharded flattened parameter, pass ``tensor`` as ``None``, but to get views into tensor optimizer state, pass ``tensor`` as the optimizer state tensor. Nrrz numelcss$|]\}}}t|||VqdSrM)rr)rEZ subtensorrZparam_extensionr0r0r1 Usz4FlatParamHandle._get_unflat_views..rr) rrr]rr6splitrQrRrU)r{rviewsr0r0r1_get_unflat_views@s z!FlatParamHandle._get_unflat_views)rmrBcCst||j||j}tt||jjD]8\}\}\}}}t||rRt|||jr|rt |t ur|t ||t |q*|jj|}t |||||_q*|r||t |q*|} |jr<|jtjkr|jjdusJ||jj|<nR|jtjkr<|jjdus J|jj|} t| dud|| _| dus8J| } t ||| |jr*|jtjkr*| |j|<q*t|jjD]\}\}}}} } } t||rt||tt| | d| d| t| | }t| pt|t j d|dt ||jr|r|jj|}t |||||_nR|r@t|t j s2J|||n,t ||||jrr|jtjkrr||j|<qrdS)a Unflattens the unsharded flattened parameter by setting the original module parameter variables to be views into it. Args: as_params (bool): If ``True``, then registers the original parameters as ``nn.Parameter`` s; if ``False``, then registers the original parameters only as ``Tensor`` s. ``False`` should be used during forward/backward computation and when hiding the original parameters from :meth:`nn.Module.named_parameters`. Nz.Expects `Tensor` to have been saved in forwardzModule z is missing parameter z as_params=z type(prim_param)=)rr{rrrrPrrrtrrsetattrr.rbrWrZregister_parameterrwrrr[rr _parametersrTgetattrrrX)r_rmrrrr$r%rKr`Z param_varrr2r3r4 prim_paramZ shared_paramr0r0r1r~_s                     z$FlatParamHandle._use_unsharded_viewsc Cs|jjdurN|jjdusJ|jjdus,Jt|jj|jjD] }d|_q>dS||jj||j|jj}tt||jj D]x\}\}\}}}t t |||jj |dt ||}|j|jks|j|jkr|jdurt||_||j_q||_qt|jjD]\}\}}}} } }t t |||r6|d|n|dt ||}t | | } |j| jjksv|j| jjkr|jdurt||_| j|j_n| j|_qdS)z Unflattens the unsharded flattened parameter's gradient by setting the original module parameter variables' gradients to be views into it. Nz is missingr)r{rrWrXrrrrrrPrrrSrrr|r6 empty_likerrT) r_r`rrrr$r%rKr&r2r3rr0r0r1rs\              z)FlatParamHandle._use_unsharded_grad_viewsc cs6|jddzdVW|jddn|jdd0dS)a1 Assumes the flattened parameter is unsharded. When in the context, unflattens the original parameters as ``nn.Parameter`` views into the flattened parameter, and after the context, restores the original parameters as ``Tensor`` views into the flattened parameter. TrlNF)r~rr0r0r1unflatten_as_paramss z#FlatParamHandle.unflatten_as_paramscCs|js|jdddS||j|jj\}}d}|jjdusBJtt|jj|jjD]\}\}\}}}t |||||ko||ko|jj } | r|jj ||\} } | | d} |j||| |_ || 7}qXt j d|jj|jjdd|_ qX|jjdusJtt|jj|jjD]6\}\}\}}}} }}t |||t|| }||_ q|jtjkr|jjduslJtt|jjD]}d|jj|<q|dS)aF Sets the original module parameter variables' data to be flattened views into the sharded flattened parameter. The views are kept as flattened to simplify the case where a parameter is sharded across ranks. Parameters whose data is not present in the sharded flattened parameter have their data set to a size-0 empty tensor. We do not delete them to ensure to preserve expected behaviors like model printability. Parameters whose data is present must preserve their variables to be passable to an optimizer. TrlNrrF)r|rdr)rr~rr{rrWrrrPrrrr6rr|rdrXrTrrwr BACKWARD_POSTr[rYrN)r_rroffsetrr`r$r%rKin_sharded_flat_paramrrnumel_in_shardr2r3rr0r0r1r sR            z"FlatParamHandle._use_sharded_viewscCs|j}|||j}|durX|jdus,J|jdus:Jt|j|jD] }d|_qHdS|||j\}}d}|jdus~Jt|jD]\}}||ko||ko|j }|rd|j ||\} } | | d} |j dusJ|j rT|j |sT|j s|j |j kr8|jdurt||_|||| |j|j_n|||| |j|_nd|_|| 7}qd|_q|jdus|Jtt|j|jD]L\}\}\} } } } }} t|| }|r|j rt|| }|j|_nd|_qdS)a6 Sets the original module parameter variables' gradients to be flattened views into the sharded flattened parameter's gradient. This is a no-op if there is no gradient. Parameters whose data is not present in the sharded flattened parameter and parameters with ``requires_grad=False`` have their gradients set to ``None``. Since the gradient variables do not need to be preserved, this method does not manipulate existing ``Tensor`` data directly and creates new ``Tensor`` variables instead. Nrr)r{rrrWrXrrrrrrZrrur|r6rrrrrrTrr)r_r{rr`rrrrrrrrrKr2r3rr0r0r1rFs^              z'FlatParamHandle._use_sharded_grad_viewsc Cs|jr||jsdS|j}|j\}}d}|jdus8Jd}tt|j|jD]\}\}\}} } t| |snqN||ko||ko|jj } | sqN|j ||\} } | | d}t | ||u}|pt || }|rt | |}||j|<|r t |g}||||||dd}|jdurF|jdurFt |g}|d|j|||dn|jdur|jsb|jsh|jn|j}|dupt |j| }|r|durt |}t |g}||j||||d||_||7}qNt|jD]4\}\}} } }}} t | |t ||urtdq|S)a Iterates over the original parameters and writes back any parameters that changed storages (due to a non-inplace operator) to the handle's ``FlatParameter``. This method preserves the ``FlatParameter` 's device even if an original parameter's device changes. Raises: RuntimeError: If an original parameter or gradient changes storages but no longer has the expected flattened shape. Returns: ``True`` if some writeback happened, and ``False`` otherwise. FrNrTz/Changing shared parameters is not supported yet)r is_shardedr{rrWrrrPrrrrr6r7_writeback_tensorrrsrrrTNotImplementedError)r_r{rrrZ wrotebackrr`r$r%rKrrrrZ param_changedZneeds_param_writebackexpected_shapeZflat_param_gradZneeds_grad_writebackr2r3r0r0r1rs                   z&FlatParamHandle._writeback_orig_params) src_tensor dst_tensor tensor_indexr"ris_paramrBc Cs6tt|dkd||jtjjkrt|dr6|jnt}|durL|j nd}|dur^|j nd} t d|d|rvdndd |j d |d |d |j d | |dur|j |krtd|rdndd|d|j |dur|||||n6|||||jjdus&Jd|jj|<dS)a# Writes back ``src_tensor`` to ``dst_tensor`` at offset ``offset``, where ``src_tensor`` should have shape ``expected_shape``. ``is_param`` indicates if the tensor is the parameter (if ``True``) or gradient (if ``False``). If ``src_tensor`` is ``None``, then the effect is zeroing instead of copying. ``tensor_index`` gives the index of ``src_tensor`` in the metadata structures. Raises: RuntimeError: If the ``src_tensor`` does not have the expected shape. rz$Expects a 1D expected shape but got rpNrz] rbZGradientz needs writeback in z expected shape=z shape=z expected device=z device=zCannot writeback when the Z parameterZgradientz shape changes Expects rT)rrNryrxrrrrpZget_rankrrdrrrw RuntimeErrorrrZzero_r{rZ) r_r#r$r%r"rr&rpZ src_shapeZ src_devicer0r0r1r sD z!FlatParamHandle._writeback_tensorcCs<|js dS|j}|jdusJtdd|jDr8d|_dS)aZ When ``use_orig_params=True``, sets the underlying ``flat_param.grad`` to ``None`` if *all* of the original parameters' ``.grad`` are ``None``. This is targeting ``optim.zero_grad(set_to_none=True)``, in which case we want to free the gradients as soon after the ``zero_grad()`` call as possible. Ncss|]}|jduVqdSrM)r)rEr`r0r0r1r-rHz9FlatParamHandle._clear_grads_if_needed..)rtr{rWallrrr0r0r1_clear_grads_if_needed!s z&FlatParamHandle._clear_grads_if_neededcCsZ|jjD]\}}}t||rt||q|jjD]$\}}}}}}t||r0t||q0dSrM)r{rPrrrT)r_r$r%rKr0r0r1_deregister_orig_params0s    z'FlatParamHandle._deregister_orig_paramscOsB|jj|i||j_|jr>||jr2|n |jdddS)z.cSsh|] }|jqSr0rCrIr0r0r1rGIrH)r{rPrVrTrr0r0r1 _get_modulesEszFlatParamHandle._get_modules)rrBcCs*t|jdr|jsdS|jj}||kS)z Returns if ``tensor`` is *currently* sharded. For ``NO_SHARD``, we choose to have this always return ``False`` for clarity. rF)rr{rrr\)r_r sharded_sizer0r0r1rLs zFlatParamHandle.is_shardedccs:dd|jjD}t|jj|D]\}}}||fVq dS)NcSs$g|]\}}}}}}t|||qSr0r!rEr$r%r&rKr0r0r1rL[sz:FlatParamHandle.parameter_module_names..)r{rTrrP)r_r>r$rKr&r0r0r1parameter_module_namesZs  z&FlatParamHandle.parameter_module_namesccs,dd|jjDD]\}}}||fVqdS)NcSs$g|]\}}}}}}t|||qSr0r/r0r0r0r1rLlszAFlatParamHandle.shared_parameter_module_names..)r{rT)r_r$rKr&r0r0r1shared_parameter_module_namesks  z-FlatParamHandle.shared_parameter_module_namescCsTg}|jj\}}tt|jjD].}||kr ||kr |jjr ||jj|q |S)z@Returns the FQNs of the parameters present in this rank's shard.)r{rrYrNrSrr)r_Z fqns_in_shardrrrr0r0r1_fqns_in_shardys  zFlatParamHandle._fqns_in_shardcCsX|j}t|dr|j}nrenumrr itertoolsrrtypingrrrr r r r r rrrrr6Ztorch.distributedZ distributedrxZtorch.nnr.Ztorch.nn.functionalZ functionalrrZtorch.distributed._tensorrZ$torch.distributed.fsdp._common_utilsrrZ_fsdp_extensionsrr_utilsrrrrr__all__r!r"r r#rbrrZ _HandlesKeyr0r0r0r1sF8     2d