a d1@sBddlZddlmZmZddlmZddlmZmZm Z m Z m Z m Z m Z mZmZmZddlZddlmZddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lm Z dd l!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(m)Z)m*Z*edZ+edZ,e-e.Z/GdddeZ0GdddeZ1GdddeZ2dS)N)ABCabstractmethod)contextmanager) AnyDict GeneratorIterableListMappingOptionalTupleTypeVarUnion)Tensor)Module) Optimizer) DataLoader) Accelerator) CheckpointIO)TorchCheckpointIO) Precision) _Launcher)move_data_to_device)_optimizer_to_device)_PATH OptimizableReduceOp TBroadcastTReducec@sZeZdZdZdaeeeeeeddddZe e e j dddZ e e edd d Ze eedd d Ze eedd dZejeddddZe edddZejeeddddZe edddZejeeddddZddddZddddZeedddZeeeeeeefd d!d"Zeed#d$d%Zeed&d'd(Z e edd#d)d*Z!dbe"ee j e"d+d,d-Z#e$eee"e"dd.d/d0Z%e&e"e"d1d2d3Z'e dce$ee"ee$d5d6d7Z(e dde)e$e"fee"ee)e*e+fe)e$e"fd9d:d;Z,e deee+dd<d=d>Z-e dfe.e/e.d@dAdBZ0dgeeedDdEdFZ1dhe2e+e"fe3ee"ddGdHdIZ4ee2e+e)e"e$ffd#dJdKZ5ee2e+e$fd&dLdMZ6e3e2e+e"fdNdOdPZ7ee8e+e"fddQdRdSZ9e)ee:efe8e+e"fddTdUdVZ;e3ddWdXdYZe2e+e"fdd\d]d^Z?e+dd_d`Z@dS)iStrategyzcBase class for all strategies that change the behaviour of the training, validation and test- loop.N) accelerator checkpoint_io precisionreturncCs"||_||_||_d|_d|_dSN) _accelerator_checkpoint_io _precision _launcherZ_backward_sync_control)selfr r!r"r*m/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/lightning_fabric/strategies/strategy.py__init__+s zStrategy.__init__r#cCsdS)zReturns the root device.Nr*r)r*r*r+ root_device7szStrategy.root_devicecCsdS)zcWhether the current process is the rank zero process not only on the local node, but for all nodes.Nr*r.r*r*r+is_global_zero<szStrategy.is_global_zerocCs|jSr$)r(r.r*r*r+launcherAszStrategy.launchercCs|jSr$r%r.r*r*r+r EszStrategy.accelerator)r r#cCs ||_dSr$r2)r)r r*r*r+r IscCs|jdurt|_|jSr$)r&rr.r*r*r+r!Ms zStrategy.checkpoint_io)ior#cCs ||_dSr$)r&)r)r3r*r*r+r!SscCs|jdur|jStSr$)r'rr.r*r*r+r"WszStrategy.precision)r"r#cCs ||_dSr$)r')r)r"r*r*r+r"[scCsdS)z&Attach the launcher based on Strategy.Nr*r.r*r*r+_configure_launcher_szStrategy._configure_launchercCs |jdusJ|j|jdS)zSetup any processes or distributed connections. This must be called by the framework at the beginning of every process, before any distributed communication takes place. N)r Z setup_devicer/r.r*r*r+setup_environmentbszStrategy.setup_environment) dataloaderr#cCs|S)zWraps the dataloader if necessary. Args: dataloader: iterable. Ideally of type: :class:`torch.utils.data.DataLoader` r*)r)r6r*r*r+process_dataloaderkszStrategy.process_dataloader)module optimizersr#cs$|}fdd|D}||fS)zSet up a model and multiple optimizers together. The returned objects are expected to be in the same order they were passed in. The default implementation will call :meth:`setup_module` and :meth:`setup_optimizer` on the inputs. csg|]}|qSr*)setup_optimizer).0 optimizerr.r*r+ |z8Strategy.setup_module_and_optimizers..) setup_module)r)r8r9r*r.r+setup_module_and_optimizersss z$Strategy.setup_module_and_optimizersr8r#cCs|S)zDPerforms setup for the model, e.g., by wrapping it by another class.r*r)r8r*r*r+r?szStrategy.setup_module)r<r#cCs|S)zHPerforms setup for the optimizer, e.g., by wrapping it by another class.r*r)r<r*r*r+r:szStrategy.setup_optimizercCsdS)z&Moves the model to the correct device.Nr*rBr*r*r+module_to_deviceszStrategy.module_to_device)batchdevicer#cCs|p|j}t||S)a%Moves the batch to the correct device. The returned batch is of the same type as the input batch, just having all tensors on the correct device. Args: batch: The batch of samples to move to the correct device device: The target device )r/r)r)rErFr*r*r+batch_to_devices zStrategy.batch_to_device)tensorr8argskwargsr#cOs<|j|||jj||g|Ri||j||dS)z0Forwards backward-calls to the precision plugin.N)r"Z pre_backwardbackwardZ post_backward)r)rHr8rIrJr*r*r+rKszStrategy.backward)r<rJr#cKs|jj|fi|S)zPerforms the actual optimizer step. Args: optimizer: the optimizer performing the step **kwargs: Any extra arguments to ``optimizer.step`` )r"optimizer_step)r)r<rJr*r*r+rLs zStrategy.optimizer_stepF)rHgroup sync_gradsr#cCsdS)aPerform an all_gather on all processes. Args: tensor: the tensor to all_gather group: the process group to gather results from sync_grads: flag that allows users to synchronize gradients for all_gather op Nr*)r)rHrMrNr*r*r+ all_gatherszStrategy.all_gathermean)rHrM reduce_opr#cCsdS)a+Reduces the given tensor (e.g. across GPUs/processes). Args: tensor: the tensor to sync and reduce group: the process group to reduce reduce_op: the reduction operation. Defaults to 'mean'. Can also be a string 'sum' or ReduceOp. Nr*)r)rHrMrQr*r*r+ all_reduceszStrategy.all_reduce)namer#cCsdS)zSynchronizes all processes which blocks processes until the whole group enters this function. Args: name: an optional name to pass into barrier. Nr*)r)rSr*r*r+barrierszStrategy.barrierr)objsrcr#cCsdS)zBroadcasts an object to all processes. Args: obj: the object to broadcast src: source rank Nr*)r)rUrVr*r*r+ broadcastszStrategy.broadcastT)decisionallr#cCs|S)z/Reduce a boolean decision across all processes.r*)r)rXrYr*r*r+reduce_boolean_decisionsz Strategy.reduce_boolean_decision) checkpointfilepathstorage_optionsr#cCs|jr|jj|||ddS)a>Save model/training states as a checkpoint file through state-dump and file-write. Args: checkpoint: dict containing model and trainer state filepath: write-target file's path storage_options: parameter for how to save to storage, passed to ``CheckpointIO`` plugin )r]N)r0r!save_checkpoint)r)r[r\r]r*r*r+r^s zStrategy.save_checkpointcCs|S)zReturns model state.) state_dictrBr*r*r+get_module_state_dictszStrategy.get_module_state_dictcCs,t|dr$||jr |SiS|S)zReturns state of an optimizer. Allows for syncing/collating optimizer state from processes in custom plugins. consolidate_state_dict)hasattrrar0r_rCr*r*r+get_optimizer_states zStrategy.get_optimizer_state)checkpoint_pathr#cCstj|j|Sr$)torchZcudaZ empty_cacher!load_checkpoint)r)rdr*r*r+rfs zStrategy.load_checkpoint)r8r[r#cCs||ddS)Nr_)load_state_dict)r)r8r[r*r*r+load_module_state_dictszStrategy.load_module_state_dict)r9r[r#cCsFt|ts|g}|d}t||D]\}}||t||jq"dS)Noptimizer_states) isinstancerziprgrr/)r)r9r[rir<Z opt_stater*r*r+load_optimizer_state_dicts   z"Strategy.load_optimizer_state_dict)r\r#cCs|jr|j|dS)zpRemove checkpoint filepath from the filesystem. Args: filepath: Path to checkpoint N)r0r!remove_checkpoint)r)r\r*r*r+rm szStrategy.remove_checkpointcCs0|j|jdusJ|j|jdS)zThis method is called to teardown the training process. It is the right place to release memory and free other resources. N)r"teardownr r!r.r*r*r+rns  zStrategy.teardown)strategy_registryr#cCsdSr$r*)clsror*r*r+register_strategiesszStrategy.register_strategiescCsdt|jdS)NzThe `z` does not support setting up the module and optimizer(s) independently. Please call `setup_module_and_optimizers(model, [optimizer, ...])` to jointly set them up.)type__name__r.r*r*r+_err_msg_joint_setup_required#sz&Strategy._err_msg_joint_setup_required)NNN)N)NF)NrP)N)r)T)N)Ars __module__ __qualname____doc__r rrrr,propertyrrerFr/boolr0rr1r setterr!r"r4r5rr7rr rr r@r?r:rDrrGrrKrrLrOrrstrrRrTrintrWrZrrr^r`rcrfr rhrrlrmrn classmethodrqrtr*r*r*r+r(s             rc@s(eZdZdZeeeedddZdS)_BackwardSyncControlaInterface for any :class:`Strategy` that wants to offer a functionality to enable or disable gradient synchronization during/after back-propagation. The most common use-case is gradient accumulation. If a :class:`Strategy` implements this interface, the user can implement their gradient accumulation loop very efficiently by disabling redundant gradient synchronization. rAcCsdS)zBlocks the synchronization of gradients during the backward pass. This is a context manager. It is only effective if it wraps a call to `.backward()`. Nr*rBr*r*r+no_backward_sync2sz%_BackwardSyncControl.no_backward_syncN) rsrurvrwrrrrrr*r*r*r+r~*sr~c@s&eZdZdZeeedddZdS)_ShardedzoMixin-interface for any :class:`Strategy` that wants to expose functionality for sharding model parameters.r-ccs dVdS)aA context manager that goes over the instantiation of an :class:`torch.nn.Module` and handles sharding of parameters on creation. By sharding layers directly on instantiation, one can reduce peak memory usage and initialization time. Nr*r.r*r*r+module_sharded_context?sz_Sharded.module_sharded_contextN)rsrurvrwrrrrr*r*r*r+r;sr)3loggingabcrr contextlibrtypingrrrrr r r r r rrerZtorch.nnrZ torch.optimrZtorch.utils.datarZlightning_fabric.acceleratorsrZ)lightning_fabric.plugins.io.checkpoint_iorZ$lightning_fabric.plugins.io.torch_iorZ"lightning_fabric.plugins.precisionrZ*lightning_fabric.strategies.launchers.baserZ%lightning_fabric.utilities.apply_funcrZ$lightning_fabric.utilities.optimizerrZ lightning_fabric.utilities.typesrrrrr getLoggerrslogrr~rr*r*r*r+s0 0