a d@sddlZddlZddlmZmZddlmZddlmZddl m Z m Z m Z m Z mZmZmZmZmZmZmZmZddlZddlmZddlmZddlmZddlmZdd lmZdd l m!Z!dd l"m#Z#m$Z$m%Z%m&Z&m'Z'dd l(m)Z)dd l*m+Z+ddl,m-Z-ddl.m/Z/m0Z0m1Z1ddl2m3Z3m4Z4m5Z5m6Z6m7Z7ddl8m9Z9m:Z:ddl;mZ>m?Z?ddl@mAZAmBZBmCZCmDZDddlEmFZFddlGmHZHddlImJZJddlKmLZLmMZMmNZNGdddZOePdddZQe dddd ZRdS)!N)contextmanager nullcontext)partial)Path) AnyCallablecastDict GeneratorListMappingOptionaloverloadSequenceTupleUnion)apply_to_collection) is_overridden)rank_zero_warn)Tensor) Optimizer) BatchSampler DataLoaderDistributedSampler RandomSamplerSequentialSampler)Logger) Precision) Accelerator) _Connector _PLUGIN_INPUT_PRECISION_INPUT)DeepSpeedStrategy FSDPStrategySingleDeviceStrategyStrategy XLAStrategy)_Sharded TBroadcast)move_data_to_device)convert_tensors_to_scalarsconvert_to_tensors)_auto_add_worker_init_fn_replace_dunder_methods_update_dataloaderhas_iterable_dataset)DistributedSamplerWrapperseed_everything)PossibleUserWarning)_FabricDataLoader _FabricModule_FabricOptimizerc @seZdZdZdeeeefeeeefeee e ee fe e eee e e feee e e feeee efdd ddZeedd d Zeedd d Zeejdd dZee dddZee dddZee dddZee dddZeedddZee edddZeedddZe e e dddZd d!eje ee d"d#d$Z!dejee"d%d&d'Z#e ee$e%e$d(ffd)d*d+Z&d d d,e'eeee'e e'fd-d.d/Z(de'eee'd0d1d2Z)dd3e*e ee"e dd4d5d6Z+e,e-d7dd8d9Z.e/ejejd:d;d<Z0e/e*e*d:d=d<Z0e/e e d:d>d<Z0eeje*e feeje*e fd:d?d<Z0e e ddd@dAZ1deeddBdCdDZ2dee*e3e e%fee eee*e3e e%fdFdGdHZ4de5e e5dJdKdLZ6e,de"ee-dMdNdOZ7e,e-ddPdQZ8e3ee feee9fddRdSdTZ:eee9fe dUdVdWZ;deedee ee dd^d_d`Z?de@ee fee ddadbdcZAeBdee eee dddedfZCer?r)rZ _connectorr; _strategyr: _acceleratorr> _precision isinstancelist _callbacks_loggers _models_setup_prepare_run_method _is_using_clisetup_environment) selfr:r;r<r=r>r?r@rArO`/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/lightning_fabric/fabric.py__init__Ns&    zFabric.__init__rBcCs|jSN)rDrNrOrOrPr:pszFabric.acceleratorcCs|jSrS)rCrTrOrOrPr;tszFabric.strategycCs|jjS)zwThe current device this process runs on. Use this to create tensors directly on the device if needed. )rCZ root_devicerTrOrOrPdevicexsz Fabric.devicecCst|jddS)zEThe global index of the current process across all devices and nodes. global_rankrgetattrrCrTrOrOrPrVszFabric.global_rankcCst|jddS)zOThe index of the current process among the processes running on the local node. local_rankrrWrTrOrOrPrYszFabric.local_rankcCst|jddS)zThe index of the current node. node_rankrrWrTrOrOrPrZszFabric.node_rankcCst|jddS)zCThe total number of processes running across all devices and nodes. world_sizer8rWrTrOrOrPr[szFabric.world_sizecCs|jjS)zWhether this rank is rank zero.)rCis_global_zerorTrOrOrPr\szFabric.is_global_zerocCs|jS)z%Returns all loggers passed to Fabric.rIrTrOrOrPrAszFabric.loggerscCs |jdS)z[Returns the first logger in the list passed to Fabric, which is considered the main logger.rr]rTrOrOrPloggersz Fabric.logger)argskwargsrBcOsdS)zAll the code inside this run method gets accelerated by Fabric. You can pass arbitrary arguments to this function when overriding it. NrOrNr_r`rOrOrPrunsz Fabric.runT)move_to_device)module optimizersrcrBcs|||}j|}|r2j|t|d}|rNj|t|\}}n j|}t|j|d}| |rxj n t | j fdd|D}j d7_ t|dr|_||_|rt|g|RS|S)aSet up a model and its optimizers for accelerated training. Args: module: A :class:`torch.nn.Module` to set up *optimizers: The optimizer(s) to set up (no optimizers is also possible) move_to_device: If set ``True`` (default), moves the model to the correct device. Set this to ``False`` and alternatively use :meth:`to_device` manually. Returns: The tuple containing wrapped module and the optimizers, in the same order they were passed in. modelreoriginal_modulecsg|]}t|jdqS) optimizerr;r6rC.0rkrTrOrP z Fabric.setup..r8_fabric)_validate_setuprEconvert_module_move_model_to_devicerGrCZsetup_module_and_optimizers setup_moduler5torUnext parametersrJhasattrrqZ_fabric_optimizerstuple)rNrdrcrerirOrTrPsetups(     z Fabric.setup)rdrcrBcCs|||}|j|}|r,|j|gd}|j|}t||j|d}t|jtsr| |rb|j n t | j t |dr||_|jd7_|S)aSet up a model for accelerated training or inference. This is the same as calling ``.setup(model)`` with no optimizers. It is useful for inference or for certain strategies like `FSDP` that require setting up the module before the optimizer can be created and set up. See also :meth:`setup_optimizers`. Args: module: A :class:`torch.nn.Module` to set up move_to_device: If set ``True`` (default), moves the model to the correct device. Set this to ``False`` and alternatively use :meth:`to_device` manually. Returns: The wrapped model. rfrhrqr8)_validate_setup_modulerErsrtrCrur5rFr#rvrUrwrxryrqrJ)rNrdrcrirOrOrPrus     zFabric.setup_module.)rerBcsJ|fdd|D}fdd|D}t|dkrB|dSt|S)aySet up one or more optimizers for accelerated training. Some strategies do not allow setting up model and optimizer independently. For them, you should call ``.setup(model, optimizer, ...)`` instead to jointly set them up. Args: *optimizers: One or more optmizers to set up. Returns: The wrapped optimizer(s). csg|]}j|qSrO)rCZsetup_optimizerrmrTrOrProrpz+Fabric.setup_optimizers..csg|]}t|jdqSrjrlrmrTrOrProrpr8r)_validate_setup_optimizerslenrzrNrerOrTrPsetup_optimizerss zFabric.setup_optimizersreplace_samplerrc) dataloadersrrcrBcs<|fdd|D}t|dkr4|dn|}|S)a]Set up one or multiple dataloaders for accelerated training. If you need different settings for each dataloader, call this method individually for each one. Args: *dataloaders: A single dataloader or a sequence of dataloaders. replace_sampler: If set ``True`` (default), automatically wraps or replaces the sampler on the dataloader(s) for distributed training. If you have a custom sampler defined, set this to this argument to ``False``. move_to_device: If set ``True`` (default), moves the data returned by the dataloader(s) automatically to the correct device. Set this to ``False`` and alternatively use :meth:`to_device` manually on the returned data. Returns: The wrapped dataloaders, in the same order they were passed in. csg|]}j|dqS)r)_setup_dataloader)rn dataloaderrcrrNrOrPro$sz,Fabric.setup_dataloaders..r8r)_validate_setup_dataloadersr~)rNrrcrrOrrPsetup_dataloaderss  zFabric.setup_dataloaders)rrrcrBcCs|j}|r*||r*|j|fi|jj}t||}t||j|j|}|rbt |jt sb|j nd}t ||d}t t|}|S)aSet up a single dataloader for accelerated training. Args: dataloader: The dataloader to accelerate. replace_sampler: If set ``True`` (default), automatically wraps or replaces the sampler on the dataloader for distributed training. If you have a custom sampler defined, set this to this argument to ``False``. move_to_device: If set ``True`` (default), moves the data returned by the dataloader automatically to the correct device. Set this to ``False`` and alternatively use :meth:`to_device` manually on the returned data. Returns: The wrapped dataloader. N)rrU)sampler_requires_distributed_sampler_get_distributed_samplerrCdistributed_sampler_kwargsr.r,rVZprocess_dataloaderrFr&rUr4rr)rNrrrcrrUZlite_dataloaderrOrOrPr+s     zFabric._setup_dataloader)rg)tensorr_rgr`rBcOs||dur|jn|}t|jtr\|durT|jdkr8td|jdkrJtd|jj}n||j_|j j ||g|Ri|dS)aReplaces ``loss.backward()`` in your training loop. Handles precision and automatically for you. Args: tensor: The tensor (loss) to back-propagate gradients from. *args: Optional positional arguments passed to the underlying backward function. model: Optional model instance for plugins that require the model for backward(). **kwargs: Optional named keyword arguments passed to the underlying backward function. Note: When using ``strategy="deepspeed"`` and multiple models were set up, it is required to pass in the model as argument here. NrzJNo models were set up for backward. Did you forget to call `self.setup()`?r8zWhen using multiple models + deepspeed, please provide the model used to perform the optimization: `self.backward(loss, model=model)`) _forward_modulerFrCr"rJ RuntimeError ValueErrorrgZ_deepspeed_enginerEbackward)rNrrgr_r`rdrOrOrPrKs     zFabric.backward)NNNccs4|jdVWdn1s&0YdS)zA context manager to automatically convert operations for the chosen precision. Use this only if the `forward` method of your model does not cover all operations you wish to run with the chosen precision setting. N)rEZforward_contextrTrOrOrPautocastis zFabric.autocast)objrBcCsdSrSrOrNrrOrOrP to_devicesszFabric.to_devicecCsdSrSrOrrOrOrPrwscCsdSrSrOrrOrOrPr{scCs8t|tjr*|j|j|j||St||jdS)aMove a :class:`torch.nn.Module` or a collection of tensors to the current device, if it is not already on that device. Args: obj: An object to move to the device. Can be an instance of :class:`torch.nn.Module`, a tensor, or a (nested) collection of tensors (e.g., a dictionary). Returns: A reference to the object that was moved to the new device. rU) rFnnModulerDZ setup_devicerUrCZmodule_to_devicer)rrOrOrPrs  cOs|jdkrt|i|dS)zPrint something only on the first process. Arguments passed to this method are forwarded to the Python built-in :func:`print` function. rN)rYprintrarOrOrPrs z Fabric.print)namerBcCs|jj|ddS)a Wait for all processes to enter this call. Use this to synchronize all parallel processes, but only if necessary, otherwise the overhead of synchronization will cause your program to slow down. Example:: if self.global_rank == 0: # let process 0 download the dataset dataset.download_files() # let all processes wait before reading the dataset self.barrier() # now all processes can read the files and start training )rN)rCbarrier)rNrrOrOrPrszFabric.barrierF)datagroup sync_gradsrBcCs:|dur |ntjjj}t||jd}t|t|jj ||dS)a;Gather tensors or collections of tensors from multiple processes. Args: data: int, float, tensor of shape (batch, ...), or a (possibly nested) collection thereof. group: the process group to gather results from. Defaults to all processes (world) sync_grads: flag that allows users to synchronize gradients for the all_gather operation Return: A tensor of shape (world_size, batch, ...), or if the input was a collection the output will also be a collection with tensors of this shape. Nr)rr) torchZ distributedrZWORLDr+rUrrrC all_gather)rNrrrrOrOrPrszFabric.all_gatherr)rsrcrBcCs|jj||dS)N)r)rC broadcast)rNrrrOrOrPrszFabric.broadcast)rdenabledrBccst|tstd|r"t|jtr*t}n>|jjdurXtd|jjj dt dt}n|jj |j }|dVWdn1s0YdS)aSkip gradient synchronization during backward to avoid redundant communication overhead. Use this context manager when performing gradient accumulation to speed up training with multiple devices. Example:: # Accumulate gradient 8 batches at a time with self.no_backward_sync(model, enabled=(batch_idx % 8 != 0)): output = model(input) loss = ... self.backward(loss) ... For those strategies that don't support it, a warning is emitted. For single-device strategies, it is a no-op. Both the model's `.forward()` and the `self.backward()` call need to run under this context. Args: module: The module for which to control the gradient synchronization. enabled: Whether the context manager is enabled or not. ``True`` means skip the sync, ``False`` means do not skip. zrYou need to set up the model first before you can call `self.no_backward_sync()`: `model = self.setup(model, ...)`NThe `z` does not support skipping the gradient synchronization. Remove `.no_backward_sync()` from your code or choose a different strategy.category) rFr5 TypeErrorrCr$rZ_backward_sync_controlr __class____name__r3no_backward_syncr)rNrdrcontextrOrOrPrs"  zFabric.no_backward_syncccsHt|jtr>|jdVWdqD1s20YndVdS)aShard the parameters of the model instantly when instantiating the layers. Use this context manager with strategies that support sharding the model parameters to save peak memory usage. Example:: with self.sharded_model(): model = MyModel() The context manager is strategy-agnostic and for the ones that don't do sharding, it is a no-op. N)rFrCr'Zmodule_sharded_contextrTrOrOrP sharded_models  &zFabric.sharded_model)contentfilepathrBcCs|j||dS)adSave checkpoint contents to a file. How and which processes save gets determined by the `strategy`. For example, the `ddp` strategy saves checkpoints only on process 0. Args: content: A dictionary with contents, i.e., the state dict of your model filepath: A path to where the file should be saved N)rCZsave_checkpoint)rNrrrOrOrPsaves z Fabric.save)rrBcCs |j|S)zLoad a checkpoint from a file. How and which processes load gets determined by the `strategy` Args: filepath: A path to where the file is located )rCZload_checkpoint)rNrrOrOrPload sz Fabric.load)functionr_r`rBcOsztrtd|dur*t|js*tdt|j|p6t}|g|}|j j durl|j j j |g|Ri|S||i|S)Nz}This script was launched through the CLI, and processes have already been created. Calling `.launch()` again is not allowed.zThe function passed to `Fabric.launch()` needs to take at least one argument. The launcher will pass in the `Fabric` object so you can use it inside the function.) rLrinspect signaturerxrr_run_with_setup _do_nothingrClauncherlaunch)rNrr_r`rOrOrPrs  z Fabric.launch) hook_namer_r`rBcOsZ|jD]N}t||d}|dur qt|sFtdt|jd|dq||i|qdS)aTrigger the callback methods with the given name and arguments. Not all objects registered via ``Fabric(callbacks=...)`` must implement a method with the given name. The ones that have a matching method name will get called. Args: hook_name: The name of the callback method. *args: Optional positional arguments that get passed down to the callback method. **kwargs: Optional keyword arguments that get passed down to the callback method. Example:: class MyCallback: def on_train_epoch_end(self, results): ... fabric = Fabric(callbacks=[MyCallback()]) fabric.call("on_train_epoch_end", results={...}) NzSkipping the callback `.z` because it is not callable.)rHrXcallablertyper)rNrr_r`callbackmethodrOrOrPcall&s  z Fabric.call)rvaluesteprBcCs|j||i|ddS)aLog a scalar to all loggers that were added to Fabric. Args: name: The name of the metric to log. value: The metric value to collect. If the value is a :class:`torch.Tensor`, it gets detached from the graph automatically. step: Optional step number. Most Logger implementations auto-increment the step value by one with every log call. You can specify your own value here. metricsrN)log_dict)rNrrrrOrOrPlogLs z Fabric.log)rrrBcCs&t|}|jD]}|j||dqdS)aLog multiple scalars at once to all loggers that were added to Fabric. Args: metrics: A dictionary where the key is the name of the metric and the value the scalar to be logged. Any :class:`torch.Tensor` in the dictionary get detached from the graph automatically. step: Optional step number. Most Logger implementations auto-increment this value by one with every log call. You can specify your own value here. rN)r*rIZ log_metrics)rNrrr^rOrOrPrXs  zFabric.log_dict)seedworkersrBcCs|dur d}t||dS)zHelper function to seed everything without explicitly importing Lightning. See :func:`pytorch_lightning.seed_everything` for more details. NTrrr1rrOrOrPr2eszFabric.seed_everything) run_methodr_r`rBcOsFt|j|}|jjdur4|jjj|g|Ri|S||i|SdSrS)rrrCrr)rNrr_r`rOrOrP _run_implqs  zFabric._run_impl) run_functionr_r`rBc Os|j|ttdbtt:||i|WdWdWdS1sh0YWdn1s0YWdn1s0YdS)Ndataset)rCrMrr-rr)rNrr_r`rOrOrPrxs  zFabric._run_with_setup)rgrerBcst|jtfdd|Dr4tdtdt|jtrt | }| |}t | fdd| D|D]*}|j D]}fdd|d D|d <qq|n | |}|S) Nc3s|]}|jkVqdSrSr)rnparam)initial_devicerOrP rpz/Fabric._move_model_to_device..zThe model passed to `Fabric.setup()` has parameters on different devices. Since `move_to_device=True`, all parameters will be moved to the new device. If this is not desired, set `Fabric.setup(..., move_to_device=False)`.rcsi|]\}}||qSrOrO)rnrr)params_on_devicerOrP rpz0Fabric._move_model_to_device..csg|]}||qSrO)get)rnp)mappingrOrProrpz0Fabric._move_model_to_device..params)rwrxrUanyrr3rFrCr&dictZnamed_parametersritemsZ param_groups)rNrgreZparams_before_moverkZ param_grouprO)rrrrPrts        zFabric._move_model_to_device)rrBcCs*t|jddduo(t|jt o(t| S)Nr)rXr;rFrrr/)rNrrOrOrPrs  z$Fabric._requires_distributed_sampler)rr`rBcKs`|dt|jt|dttddt|jttfrNt|j fi|St |jfi|S)NshufflerZPL_GLOBAL_SEEDr) setdefaultrFrrintosgetenvrrrr0)rr`rOrOrPrs zFabric._get_distributed_samplercCs4td|trtrtdt|dt|j|jdS)NrbzOverriding `Fabric.run()` and launching from the CLI is not allowed. Run the script normally, or change your code to directly call `fabric = Fabric(...); fabric.setup(...)` etc.)rr7rLrsetattrrrrbrTrOrOrPrKs zFabric._prepare_run_method)rdrerBcCsRt|trtdtdd|Dr,tdt|jtrNtdt|jddS)Nz9A model should be passed only once to the `setup` method.css|]}t|tVqdSrSrFr6rnoptrOrOrPrrpz)Fabric._validate_setup..z>An optimizer should be passed only once to the `setup` method.rz` requires the model and optimizer(s) to be set up separately. Create and set up the model first through `model = self.setup_model(model)`. Then create the optimizer and set it up: `optimizer = self.setup_optimizer(optimizer)`.) rFr5rrrCr#rrr)rNrdrerOrOrPrrs  zFabric._validate_setup)rdrBcCst|trtddS)Nz@A model should be passed only once to the `setup_module` method.)rFr5r)rNrdrOrOrPr|s zFabric._validate_setup_modulecCsRt|jttfr(tdt|jjd|s4tdtdd|DrNtddS)Nrzc` requires the model and optimizer(s) to be set up jointly through `.setup(model, optimizer, ...)`.z<`setup_optimizers` requires at least one optimizer as input.css|]}t|tVqdSrSrrrOrOrPrrpz4Fabric._validate_setup_optimizers..zIAn optimizer should be passed only once to the `setup_optimizers` method.) rFrCr"r&rrrrrrrOrOrPr}sz!Fabric._validate_setup_optimizers)rrBcCsD|s tdtdd|Dr&tdtdd|Dr@tddS)Nz>`setup_dataloaders` requires at least one dataloader as input.css|]}t|tVqdSrS)rFr4rndlrOrOrPrrpz5Fabric._validate_setup_dataloaders..zJA dataloader should be passed only once to the `setup_dataloaders` method.css|]}t|t VqdSrS)rFrrrOrOrPrrpzGOnly PyTorch DataLoader are currently supported in `setup_dataloaders`.)rrr)rrOrOrPrs z"Fabric._validate_setup_dataloaders)NNNr8r9NNN)T)TT)N)NF)r)T)N)N)N)NN)Pr __module__ __qualname____doc__r rstrrr%r rr!r rrrQpropertyr:r;rrUrVrYrZr[boolr\rAr^rbrrrr{r5rur6rrrrrrrrr rrrrrr rr(rrrrrrrrrrr r staticmethodr2rrrtrrrrKrrrr|r}rrOrOrOrPr73s "  3%    ( -"  $&    r7rRcCstttjddS)NZ LT_CLI_USED0)rrrenvironrrOrOrOrPrLsrL)_rBcGsdSrSrO)rrOrOrPrsr)Srr contextlibrr functoolsrpathlibrtypingrrrr r r r r rrrrrZtorch.nnrZ#lightning_utilities.core.apply_funcrZ"lightning_utilities.core.overridesrZ"lightning_utilities.core.rank_zerorrZ torch.optimrZtorch.utils.datarrrrrZlightning_fabric.loggersrZlightning_fabric.pluginsrZ)lightning_fabric.accelerators.acceleratorrZlightning_fabric.connectorrr r!Zlightning_fabric.strategiesr"r#r$r%r&Z$lightning_fabric.strategies.strategyr'r(Zlightning_fabric.utilitiesr)Z%lightning_fabric.utilities.apply_funcr*r+Zlightning_fabric.utilities.datar,r-r.r/Z&lightning_fabric.utilities.distributedr0Zlightning_fabric.utilities.seedr2Z#lightning_fabric.utilities.warningsr3Zlightning_fabric.wrappersr4r5r6r7rrLrrOrOrOrPsD  8             0