a d=@sddlZddlmZddlmZddlmZmZmZm Z m Z m Z m Z m Z mZddlZddlmZddlmZddlmZddlmZdd lmZmZmZdd lmZdd lmZdd lm Z dd l!m"Z"ddl#m$Z$m%Z%m&Z&ddl'm(Z(m)Z)m*Z*m+Z+ddl'm,Z-ddl'm.Z.ddl/m0Z0m1Z1ddl2m3Z3ddl4m5Z5e rRddl6m7Z7m8Z8m9Z9m:Z:dZ;Gddde"e%Ze ee?dfdd d!d"Z@ee?d#d$d%ZAdS)&N)contextmanager) timedelta) AnyDict GeneratorListOptionalTupleType TYPE_CHECKINGUnion)Tensor)Module) Optimizer) Accelerator) CheckpointIOClusterEnvironment Precision)default_pg_timeout) FSDPPrecision)_SubprocessScriptLauncher)ParallelStrategy)_BackwardSyncControl_Sharded TBroadcast)_distributed_available-_get_default_process_group_backend_for_device_init_dist_connection_sync_ddp_if_availablegroup)ReduceOp)_TORCH_GREATER_EQUAL_1_12_TORCH_GREATER_EQUAL_1_13)rank_zero_only) reset_seed)BackwardPrefetch CPUOffloadFullyShardedDataParallelMixedPrecision)fsdpfsdp_full_shard_offloadcsDeZdZdZddddddeddddf eeeeej ee ee ee ee eeeeddfededeeeeeeefedd fdd Zeej d d d Zeed d d Zejedddd Zeed ddZeee efd ddZeee d ddZeedd ddZdd ddZdd fdd Z eee!e"eee!fdddZ#edd d!d"Z$e!e!d#d$d%Z%edd d&d'Z&e'e(d d(d)Z)d>e*eeeee+e fe*d+d,d-Z,eedd.d/d0Z-d?e.ee.d2d3d4Z/e0edd5d6d7Z1dd d8d9Z2e d d:d;Z3dd d`__. Defaults have been set and options have been exposed, but may require configuration based on your level of memory/speed efficiency. We suggest having a look at `this tutorial `__ for more information. Arguments: cpu_offload: Enable offloading parameters and gradients to CPU to save GPU memory at the cost of speed. You can also pass a config: ``cpu_offload=CPUOffload(offload_params=True)``. Note that this currently implicitly enables gradient offloading to CPU in order for parameters and gradients to be on same device to work with the optimizer. This API is subject to change. Default: no offoading backward_prefetch: This is an experimental feature that is subject to change in the near future. It allows users to enable two different backward prefetching algorithms to help backward communication and computation overlapping. The pros and cons of each algorithm is explained in the class ``BackwardPrefetch``. mixed_precision: Mixed Precision config. By default, Lightning will enable FP16 if ``precision=16`` or BF16 if ``precision=bf16`` unless a config is passed in. This is only available in PyTorch 1.12 and later. activation_checkpointing: A single layer or a list of layer classes for which you want to enable activation checkpointing. This is typically your transformer block (including attention + feed-forward). Enabling this can free up a significant amount of memory at the cost of speed since activations in these layers need to be recomputed during backpropagation. \**kwargs: Optional keywoard arguments passed to the FSDP context manager which will configure the FSDP class when wrapping modules. Nr'r&r)) acceleratorparallel_devicescluster_environment checkpoint_io precisionprocess_group_backendtimeout cpu_offloadbackward_prefetchmixed_precisionactivation_checkpointingkwargsreturnc  sts tdtj|||||dd|_||_||_t|_| |_ | rRt sRt d| pXg} t | t sj| gn| |_t||_| |_| |_dS)Nz9`FSDPStrategy` is supported from PyTorch v1.12.0 onwards.)r-r.r/r0r1zOActivation checkpointing requires torch >= 1.13.0. HINT: `pip install -U torch`)r"NotImplementedErrorsuper__init__ _num_nodes_process_group_backend_timeout_FSDPBackwardSyncControlZ_backward_sync_control _ddp_kwargsr# ValueError isinstancelist_activation_checkpointing_init_cpu_offloadr4r5r6) selfr-r.r/r0r1r2r3r4r5r6r7r8 __class__i/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/lightning_fabric/strategies/fsdp.pyr=Ys, zFSDPStrategy.__init__)r9cCs|jdusJ|j|jSN)r. local_rankrHrKrKrL root_deviceszFSDPStrategy.root_devicecCs|jSrMr>rOrKrKrL num_nodesszFSDPStrategy.num_nodes)rRr9cCs ||_dSrMrQ)rHrRrKrKrLrRscCs|jdurt|jSdS)Nr)r.lenrOrKrKrL num_processesszFSDPStrategy.num_processescCst|j|j|jdS)N)Z num_replicasrank)dictrRrT global_rankrOrKrKrLdistributed_sampler_kwargssz'FSDPStrategy.distributed_sampler_kwargscCs|jSrM)r?rOrKrKrLr2sz"FSDPStrategy.process_group_backendcCs$|jr |jSt|jtr |jjSdSrM)r6rDr1rmixed_precision_configrOrKrKrLrYs z#FSDPStrategy.mixed_precision_configcCs.|jdusJ|jjs*t|j|j|j|_dSrM)r/Zcreates_processes_externallyrrTrRZ _launcherrOrKrKrL_configure_launchersz FSDPStrategy._configure_launchercs|tdSrM)_setup_distributedr<setup_environmentrOrIrKrLr\szFSDPStrategy.setup_environment)module optimizersr9cCstdt|jddS)NzThe `z` does not support the joint setup of module and optimizer(s). Please do it in this order: Create the model, call `setup_module`, create the optimizer, call `setup_optimizer`.)r;type__name__)rHr]r^rKrKrLsetup_module_and_optimizerssz(FSDPStrategy.setup_module_and_optimizersr(r]r9cszddlmd|jvr8tfdd|Dr8|jd=f||j|j|j|jj d|j}t rv|j rvt ||j d|S)z|Wraps the model into a :class:`~torch.distributed.fsdp.fully_sharded_data_parallel.FullyShardedDataParallel` module.rr(Zauto_wrap_policyc3s|]}t|VqdSrMrD).0modrcrKrL sz,FSDPStrategy.setup_module..)r]r4r5r6 device_id)r]layers) 2torch.distributed.fsdp.fully_sharded_data_parallelr(rBanymodulesr4r5rYrPindexr#rF_setup_activation_checkpointing)rHr]Zwrapped_modulerKrcrL setup_modules"  zFSDPStrategy.setup_module optimizerr9cs^ddlmt|j}|dkr.td|dtfdd|jddDrR|Std d S) aHSet up an optimizer for a model wrapped with FSDP. This setup method doesn't modify the optimizer or wrap the optimizer. The only thing it currently does is verify that the optimizer was created after the model was wrapped with :meth:`setup_module` with a reference to the flattened parameters. r FlatParameterr:zSAn optimizer used with an FSDP model does not support multiple param groups. Found z parameter groups.c3s|]}t|VqdSrMrdreparamrrrKrLrgz/FSDPStrategy.setup_optimizer..paramszThe optimizer does not seem to reference any FSDP parameters. HINT: Make sure to create the optimizer after setting up the model.N)torch.distributed.fsdprsrS param_groupsrCrk)rHrqZ num_groupsrKrrrLsetup_optimizers   zFSDPStrategy.setup_optimizercCsdSrMrK)rHr]rKrKrLmodule_to_deviceszFSDPStrategy.module_to_deviceccshddlm}ddlm}|f||j|j|j|jjd|j dVWdn1sZ0YdS)Nrrc) enable_wrap)Z wrapper_clsr4r5r6rh) rjr(Ztorch.distributed.fsdp.wrapr|r4r5rYrPrmrB)rHr(r|rKrKrLmodule_sharded_contexts  z#FSDPStrategy.module_sharded_contextmean)tensorr reduce_opr9cCst|trt|||d}|S)N)r)rDr r)rHrr rrKrKrL all_reduces zFSDPStrategy.all_reduce)argsr8r9cOs<ts dStjdkr.tjj|jjgdn tjdS)NZnccl)Z device_ids)rtorch distributed get_backendbarrierrPrm)rHrr8rKrKrLrs zFSDPStrategy.barrierr)objsrcr9cCs2|g}|j|krdg}tjj||tjd|dS)Nrr)rWrrZbroadcast_object_list_groupZWORLD)rHrrrKrKrL broadcasts  zFSDPStrategy.broadcast)strategy_registryr9cCs8trtjsdS|jd|dd|jd|ddddS)Nr*zFully Sharded Data Parallel) descriptionr+z.Fully Sharded Data Parallel and CPU OffloadingT)rr4)r"rrZ is_availableregister)clsrrKrKrLregister_strategiessz FSDPStrategy.register_strategiescCsFt||jt_||_|jdus.Jt|j|j|j ddS)N)r3) r%_set_world_ranksrWr$rU_get_process_group_backendr?r/rr@rOrKrKrLr[s  zFSDPStrategy._setup_distributedcCs|jpt|jSrM)r?rrPrOrKrKrLr'sz'FSDPStrategy._get_process_group_backendcCsL|jdurdS|j|j|j|j|j|j|j|jt_ dSrM) r/Zset_global_rankZ node_rankrTrNZset_world_sizerRrWr$rUrOrKrKrLr*s  zFSDPStrategy._set_world_ranks)Nr~)r)6r` __module__ __qualname____doc__rrrrrZdevicerrrstrrr boolr rrr=propertyrPintrRsetterrTrrXr2rYrZr\rr rarorzr{rrr}r r!rrrr classmethodrr[rr __classcell__rKrKrIrLr,6s~$  *  r,r()r]rir9csBddlm}m}m}fdd}tj||jd}||||ddS)Nr)apply_activation_checkpointingcheckpoint_wrapperCheckpointImplcst|tSrM)rDtuple) submodulerirKrL9rvz1_setup_activation_checkpointing..)Zcheckpoint_impl)Zcheckpoint_wrapper_fncheck_fn)Z;torch.distributed.algorithms._checkpoint.checkpoint_wrapperrrr functoolspartialZ NO_REENTRANT)r]rirrrrwrapperrKrrLrn2s rnc@s eZdZeeedddZdS)rArbccsfddlm}t||s4td|jjd|jjd|dVWdn1sX0YdS)zuBlocks gradient synchronization inside the :class:`~torch.distributed.fsdp.FullyShardedDataParallel` wrapper.rrczABlocking backward sync is only possible if the module passed to `zB.no_backward_sync` is wrapped in `FullyShardedDataParallel`. Got: .N)rjr(rD TypeErrorrJr`Zno_sync)rHr]r(rKrKrLno_backward_syncBs   z)_FSDPBackwardSyncControl.no_backward_syncN)r`rrrrrrrKrKrKrLrAAsrAr')r4r9cCs(ddlm}t||r|S|t|dS)Nr)r')Zoffload_params)rxr'rDr)r4r'rKrKrLrGRs rGrpcs,ddlmtfdd|jddDS)Nrrrc3s|]}t|VqdSrMrdrtrrrKrLrg[rvz-_optimizer_has_flat_params..rw)rxrsrkry)rqrKrrrL_optimizer_has_flat_paramsXs r)Br contextlibrdatetimertypingrrrrrr r r r rr Ztorch.nnrZ torch.optimrZlightning_fabric.acceleratorsrZlightning_fabric.pluginsrrrZ5lightning_fabric.plugins.collectives.torch_collectiverZ'lightning_fabric.plugins.precision.fsdprZ7lightning_fabric.strategies.launchers.subprocess_scriptrZ$lightning_fabric.strategies.parallelrZ$lightning_fabric.strategies.strategyrrrZ&lightning_fabric.utilities.distributedrrrrr rr!Z"lightning_fabric.utilities.importsr"r#Z$lightning_fabric.utilities.rank_zeror$Zlightning_fabric.utilities.seedr%rjr&r'r(r)Z _FSDP_ALIASESr,rnrArrGrrKrKrKrLs:  ,            }