a dPT@sjddlZddlZddlmZmZddlmZmZmZm Z m Z m Z m Z m Z mZmZddlZddlmZddlmZddlmZddlmZddlZddlmZdd lmZdd lmZdd l m!Z!dd l"m#Z#m$Z$dd l%m&Z&ddl'm(Z(m)Z)ddl*m+Z+ddl,m-Z-ddl.m/Z/ddl0m1Z1ddl2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8edZ9edZ:e;e<Z=GdddeZ>dS)N)ABCabstractmethod) AnyCallableDict GeneratorListMappingOptionalTupleTypeVarUnion)Tensor)Module) Optimizer) DataLoader) CheckpointIO) _Launcher)move_data_to_device)ReduceOp)_optimizer_to_device_optimizers_to_device)_PATH)"_init_optimizers_and_lr_schedulersLightningOptimizer)TorchCheckpointIO)_WrappingCheckpointIO)PrecisionPlugin) TrainerFn)LRSchedulerConfig PredictStep STEP_OUTPUTTestStep TrainingStepValidationStep TBroadcastTReducec@seZdZdZdedeeeeddddZeee ddd Z eeddd d Z e j ddd d d Z eedddZ e j eeddddZ eedddZej eeddddZeeedddZej eeddddZdddddZddddZddd d!Zd"dd#d$d%Zd"dd#d&d'Zddd(d)Zeeeefd*d+d,Zeeeeeeeed-d.d/Zdeeegefee de!feed0d1d2Z"e!eee#e!eefd3d4d5Z$e!e!dd6d7Z%eed*d8d9Z&deee'j(eed;dd?Z+e*ddd@dAZ,ee*e-ddBdCZ.e*de eefeeee e/efe eefdEdFdGZ0e*deeddHdIdJZ1e*de2ee2dKdLdMZ3e*deeee-edOdPdQZ4de-e-e-dSdTdUZ5eddVdWdXZ6eddVdYdZZ7eee!dd[d\Z8e8j ee!dd]d^d\Z8eeddd_d`Z9e:eeefdadbdcZ;eeee?didjdkZ@dddldmZAeeee?didndoZBeeee?didpdqZCeee?didrdsZDe?e?dtdudvZEe?e?dtdwdxZFe?e?dtdydzZGeHeHd{d|d}ZIee-dd~dZJee-dddZKee-dddZLeeefdddZMdeeefe:eeddddZNe:ddddZOePjQeRdddZSddddZTeUeeefddddZVddddZWddddZXddddZYddddZZddddZ[ddddZ\ddddZ]ddddZ^eeddddZ_d"dd#ddZ`edddZaeddddZbdS)StrategyzcBase class for all strategies that change the behaviour of the training, validation and test- loop.Nzpl.accelerators.Accelerator) accelerator checkpoint_ioprecision_pluginreturncCs@||_||_||_d|_d|_d|_g|_i|_g|_g|_ dSN) _accelerator_checkpoint_io_precision_plugin_lightning_module_model _launcher _optimizers_lightning_optimizerslr_scheduler_configsoptimizer_frequencies)selfr(r)r*r8n/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/pytorch_lightning/strategies/strategy.py__init__7szStrategy.__init__)r+cCs|jSr,)r2r7r8r8r9launcherHszStrategy.launchercCs|jSr,r-r;r8r8r9r(LszStrategy.accelerator)r(r+cCs ||_dSr,r=)r7r(r8r8r9r(PscCs0|jdurt|_nt|jtr*t|j_|jSr,)r.r isinstancerr)r;r8r8r9r)Ts     zStrategy.checkpoint_io)ior+cCs ||_dSr,)r.)r7r?r8r8r9r)]scCs|jdur|jStSr,)r/rr;r8r8r9r*aszStrategy.precision_plugin)r*r+cCs ||_dSr,)r/)r7r*r8r8r9r*escCs|jSr,)r3r;r8r8r9 optimizersiszStrategy.optimizers)r@r+cs$|_fddtjD_dS)Ncs i|]\}}|t||qSr8)rZ_to_lightning_optimizer).0idxoptr;r8r9 psz'Strategy.optimizers..)r3 enumerater@r4)r7r@r8r;r9r@ms zpl.LightningModule)modelr+cCs||_||_dS)zTCalled by the accelerator to connect the accelerator and the model with this plugin.N)r0rFr7rFr8r8r9connecttszStrategy.connectcCsdS)z&Attach the launcher based on Strategy.Nr8r;r8r8r9_configure_launcheryszStrategy._configure_launchercCs |jdusJ|j|jdS)zSetup any processes or distributed connections. This is called before the LightningModule/DataModule setup hook which allows the user to access the accelerator environment before setup is complete. N)r(Z setup_device root_devicer;r8r8r9setup_environment|szStrategy.setup_environmentz pl.Trainer)trainerr+cCs:|jjtjkrdS|jdus Jt|j\|_|_|_dS)zCreates optimizers and schedulers. Args: trainer: the Trainer, these optimizers should be connected to N) statefnrZFITTINGlightning_modulerr@r5r6r7rLr8r8r9setup_optimizerss zStrategy.setup_optimizerscCs>|jdusJ|j||||t|j|jdS)z{Setup plugins for the trainer fit and creates optimizers. Args: trainer: the trainer instance N)r(setuprQsetup_precision_pluginrr@rJrPr8r8r9rRs   zStrategy.setupcCs@|jdusJ|j|j|j|j\}}}||_||_||_dS)z1Attaches the precision plugin to the accelerator.N)rFr*rHr@r5)r7rFr@r5r8r8r9rSs zStrategy.setup_precision_plugin) optimizerr+cCs<t|tr|j}t|dr4||jr0|SiS|S)zReturns state of an optimizer. Allows for syncing/collating optimizer state from processes in custom plugins. consolidate_state_dict)r>rZ _optimizerhasattrrUis_global_zero state_dictr7rTr8r8r9optimizer_states   zStrategy.optimizer_state) closure_lossrT optimizer_idxargskwargsr+cOsh|||jdusJ|j||j}|jj||j||g|Ri||j||j}|||S)aForwards backward-calls to the precision plugin. Args: closure_loss: a tensor holding the loss value to backpropagate optimizer: An optional optimizer that gets passed down to the precision plugin's backward optimizer_idx: An optional optimizer index that gets passed down to the precision plugin's backward \*args: Positional arguments that get passed down to the precision plugin's backward, intended as arguments for the actual function that performs the backward, like :meth:`~torch.Tensor.backward`. \**kwargs: Keyword arguments for the same purpose as ``*args``. N) pre_backwardrOr*backward post_backward)r7r[rTr\r]r^r8r8r9r`s " zStrategy.backward)rTopt_idxclosurerFr^r+cKs6|p|j}t|tjsJ|jj|f|||d|S)a~Performs the actual optimizer step. Args: optimizer: the optimizer performing the step opt_idx: index of the current optimizer closure: closure calculating the loss value model: reference to the model, optionally defining optimizer step related hooks \**kwargs: Keyword arguments to to ``optimizer.step`` )rFr\rc)rOr>plZLightningModuler*optimizer_step)r7rTrbrcrFr^r8r8r9res zStrategy.optimizer_step)rFr@r+cs$|}fdd|D}||fS)zSetup a model and multiple optimizers together. The returned objects are expected to be in the same order they were passed in. The default implementation will call :meth:`_setup_model` and :meth:`_setup_optimizer` on the inputs. csg|]}|qSr8)_setup_optimizer)rArTr;r8r9 z8Strategy._setup_model_and_optimizers..) _setup_model)r7rFr@r8r;r9_setup_model_and_optimizerss z$Strategy._setup_model_and_optimizerscCs|S)zDPerforms setup for the model, e.g., by wrapping it by another class.r8rGr8r8r9riszStrategy._setup_modelcCs|S)zHPerforms setup for the optimizer, e.g., by wrapping it by another class.r8rYr8r8r9rfszStrategy._setup_optimizerr)batchdevicedataloader_idxr+cCs2|j}|p|j}|dur(|j|||dSt||S)ayMoves the batch to the correct device. The returned batch is of the same type as the input batch, just having all tensors on the correct device. Args: batch: The batch of samples to move to the correct device device: The target device dataloader_idx: The index of the dataloader to which the batch belongs. N)rlrm)rOrJZ_apply_batch_transfer_handlerr)r7rkrlrmrFr8r8r9batch_to_devices  zStrategy.batch_to_devicecCsdS)zReturns the root device.Nr8r;r8r8r9rJszStrategy.root_devicecCsdS)z&Moves the model to the correct device.Nr8r;r8r8r9model_to_deviceszStrategy.model_to_devicecCsdS)zcWhether the current process is the rank zero process not only on the local node, but for all nodes.Nr8r;r8r8r9rWszStrategy.is_global_zeromean)tensorgroup reduce_opr+cCsdS)a+Reduces the given tensor (e.g. across GPUs/processes). Args: tensor: the tensor to sync and reduce group: the process group to reduce reduce_op: the reduction operation. Defaults to 'mean'. Can also be a string 'sum' or ReduceOp. Nr8)r7rqrrrsr8r8r9reduce"szStrategy.reduce)namer+cCsdS)zSynchronizes all processes which blocks processes until the whole group enters this function. Args: name: an optional name to pass into barrier. Nr8)r7rur8r8r9barrier2szStrategy.barrier)objsrcr+cCsdS)zBroadcasts an object to all processes. Args: obj: the object to broadcast src: source rank Nr8)r7rwrxr8r8r9 broadcast:szStrategy.broadcastF)rqrr sync_gradsr+cCsdS)aPerform an all_gather on all processes. Args: tensor: the tensor to all_gather group: the process group to gather results from sync_grads: flag that allows users to synchronize gradients for all_gather op Nr8)r7rqrrrzr8r8r9 all_gatherCszStrategy.all_gatherT)decisionallr+cCs|S)z/Reduce a boolean decision across all processes.r8)r7r|r}r8r8r9reduce_boolean_decisionMsz Strategy.reduce_boolean_decision)r[r+cCsdS)z.Run before precision plugin executes backward.Nr8r7r[r8r8r9r_QszStrategy.pre_backwardcCsdS)z-Run after precision plugin executes backward.Nr8rr8r8r9raTszStrategy.post_backwardcCs|jdur|jS|jS)z0Returns the potentially wrapped LightningModule.N)r1r0r;r8r8r9rFWszStrategy.model) new_modelr+cCs ||_dSr,)r1)r7rr8r8r9rF\scCs|jS)zrFr# training_stepr7r]r^r8r8r9rss zStrategy.training_stepcCsdSr,r8r;r8r8r9post_training_step|szStrategy.post_training_stepcOsP|j2t|jtsJ|jj|i|WdS1sB0YdS)zThe actual validation step. See :meth:`~pytorch_lightning.core.module.LightningModule.validation_step` for more details N)r*Zval_step_contextr>rFr$validation_steprr8r8r9rs zStrategy.validation_stepcOsP|j2t|jtsJ|jj|i|WdS1sB0YdS)z}The actual test step. See :meth:`~pytorch_lightning.core.module.LightningModule.test_step` for more details N)r*Ztest_step_contextr>rFr" test_steprr8r8r9rs zStrategy.test_stepcOsP|j2t|jtsJ|jj|i|WdS1sB0YdS)zThe actual predict step. See :meth:`~pytorch_lightning.core.module.LightningModule.predict_step` for more details N)r*Zpredict_step_contextr>rFr predict_steprr8r8r9rs zStrategy.predict_step)outputr+cCs|Sr,r8r7rr8r8r9training_step_endszStrategy.training_step_endcCs|Sr,r8rr8r8r9validation_step_endszStrategy.validation_step_endcCs|Sr,r8rr8r8r9 test_step_endszStrategy.test_step_end) dataloaderr+cCs|S)zWraps the dataloader if necessary. Args: dataloader: iterable. Ideally of type: :class:`torch.utils.data.DataLoader` r8)r7rr8r8r9process_dataloaderszStrategy.process_dataloadercCsdS)aOverride to delay restoring from checkpoint till after pre-dispatch. This is useful when the plugin requires all the setup hooks to run before loading checkpoint. Returns: If true, restore checkpoint after pre_dispatch. Fr8r;r8r8r9restore_checkpoint_after_setupsz'Strategy.restore_checkpoint_after_setupcCsdS)zOverride to disable Lightning restoring optimizers/schedulers. This is useful for plugins which manage restoring optimizers/schedulers. Tr8r;r8r8r9lightning_restore_optimizersz$Strategy.lightning_restore_optimizercCsdS)zSave model/training states as a checkpoint file through state-dump and file-write. Args: checkpoint: dict containing model and trainer state filepath: write-target file's path storage_options: parameter for how to save to storage, passed to ``CheckpointIO`` plugin )rN)rWr)save_checkpoint)r7rrrr8r8r9rs zStrategy.save_checkpoint)rr+cCs|jr|j|dS)zpRemove checkpoint filepath from the filesystem. Args: filepath: Path to checkpoint N)rWr)remove_checkpoint)r7rr8r8r9rszStrategy.remove_checkpointccs dVdS)aProvide hook to create modules in a distributed aware context. This is useful for when we'd like to shard the model instantly, which is useful for extremely large models which can save memory and initialization time. Returns: Model parallel context. Nr8r;r8r8r9model_sharded_contextszStrategy.model_sharded_contextcCsjt|jtd|jdur:t|jjd|j |j |j dusRJ|j |j dS)zThis method is called to teardown the training process. It is the right place to release memory and free other resources. cpuNz: moving model to CPU)rr@rrlrOlogZdetail __class____name__rr*teardownr(r)r;r8r8r9rs    zStrategy.teardown)strategy_registryr+cCsdSr,r8)clsrr8r8r9register_strategiesszStrategy.register_strategiescCsdS)zCalled when train begins.Nr8r;r8r8r9on_train_startszStrategy.on_train_startcCsdS)zCalled when validation begins.Nr8r;r8r8r9on_validation_startszStrategy.on_validation_startcCsdS)zCalled when test begins.Nr8r;r8r8r9 on_test_startszStrategy.on_test_startcCsdS)zCalled when predict begins.Nr8r;r8r8r9on_predict_startszStrategy.on_predict_startcCsdS)zCalled when train ends.Nr8r;r8r8r9 on_train_end szStrategy.on_train_endcCsdS)zCalled when validation ends.Nr8r;r8r8r9on_validation_endszStrategy.on_validation_endcCsdS)zCalled when test end.Nr8r;r8r8r9 on_test_endszStrategy.on_test_endcCsdS)zCalled when predict ends.Nr8r;r8r8r9on_predict_endszStrategy.on_predict_end)rk batch_idxr+cCsdS)zCCalled in the training loop before anything happens for that batch.Nr8)r7rkrr8r8r9on_train_batch_startszStrategy.on_train_batch_startcCs|j|dS)zFHook to do something before the training/evaluation/prediction starts.N)r*dispatchrPr8r8r9rszStrategy.dispatchcCstt|}i|d<|S)Nr4)dictvarsr7rMr8r8r9 __getstate__"s zStrategy.__getstate__)rMr+cCs||_|j|_dSr,)__dict__r@rr8r8r9 __setstate__(szStrategy.__setstate__)NNN)N)Nr)Nrp)N)r)NF)T)N)cr __module__ __qualname____doc__r rrr:propertyrr<r(setterr)r*rrr@rHrIrKrQrRrSrstrrrZintrr`rr rrer rjrirfrrlrnrrJroboolrWrrtrvr%ryr{r~r_rarFrOrrr rrr!rrrrrrrrrrrrrrrr contextlibcontextmanagerrrr classmethodrrrrrrrrrrrrrr8r8r8r9r'4s      "  "            r')?rloggingabcrrtypingrrrrrr r r r r rrZtorch.nnrZ torch.optimrZtorch.utils.datarZpytorch_lightningrdZlightning_fabric.pluginsrZ*lightning_fabric.strategies.launchers.baserZlightning_fabric.utilitiesrZ&lightning_fabric.utilities.distributedrZ$lightning_fabric.utilities.optimizerrrZ lightning_fabric.utilities.typesrZ pytorch_lightning.core.optimizerrrZpytorch_lightning.pluginsrZ$pytorch_lightning.plugins.io.wrapperrZ#pytorch_lightning.plugins.precisionrZ pytorch_lightning.trainer.statesrZ!pytorch_lightning.utilities.typesrr r!r"r#r$r%r& getLoggerrrr'r8r8r8r9s20