a d@sddlmZmZddlmZddlmZmZmZm Z m Z ddl Z ddl m Z ddl ZddlmZmZddlmZmZddlmZdd lmZdd lmZGd d d eeZdS) )ABCabstractmethod)contextmanager)AnyDict GeneratorListOptionalN)Tensor) CheckpointIOClusterEnvironment)_all_gather_ddp_if_availableReduceOp) LayerSync)PrecisionPlugin)StrategycsteZdZdZd)edeeejeeee ee dfdd Z e e ejddd Ze edd d Ze edd d Ze edddZe edddZe edddZe eeejdddZejeeejddddZe eeefdddZeddddZd*eeeeeddd Zd+eeed"d#d$Ze e!dd%d&Z"ddfd'd( Z#Z$S),ParallelStrategyz8Plugin for training with multiple processes in parallel.Nzpl.accelerators.Accelerator) acceleratorparallel_devicescluster_environment checkpoint_ioprecision_plugincs(tj|||d||_||_d|_dS)N)rrr)super__init__rrZ _layer_sync)selfrrrrr __class__n/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/pytorch_lightning/strategies/parallel.pyr szParallelStrategy.__init__)returncCsdS)zReturn the root device.Nrrrrr root_device-szParallelStrategy.root_devicecCs|jdur|jSdSNr)r global_rankr rrrr#2szParallelStrategy.global_rankcCs|jdur|jSdSr")r local_rankr rrrr$6szParallelStrategy.local_rankcCs|jdur|jSdSr")r node_rankr rrrr%:szParallelStrategy.node_rankcCs|jdur|jSdS)N)r world_sizer rrrr'>szParallelStrategy.world_sizecCs |jdkSr")r#r rrris_global_zeroBszParallelStrategy.is_global_zerocCs|jSNZ_parallel_devicesr rrrrFsz!ParallelStrategy.parallel_devices)rrcCs ||_dSr)r*)rrrrrrJscCs"t|jdurt|jnd|jdS)Nr)Z num_replicasZrank)dictrlenr#r rrrdistributed_sampler_kwargsNsz+ParallelStrategy.distributed_sampler_kwargs)tracercCsdS)z/Function to re-conciliate processes on failure.Nr)rr.rrrreconciliate_processesUsz'ParallelStrategy.reconciliate_processesF)tensorgroup sync_gradsrcCst|||dS)z&Perform a all_gather on all processes.)r1r2)r )rr0r1r2rrr all_gatherXszParallelStrategy.all_gatherT)decisionallrcCsBtjt||jd}|j|tjd}|r6t||jknt|}|S)aReduces a boolean decision over distributed processes. By default is analagous to ``all`` from the standard library, returning ``True`` only if all input decisions evaluate to ``True``. If ``all`` is set to ``False``, it behaves like ``any`` instead. Args: decision: A single input decision. all: Whether to logically emulate ``all`` or ``any``. Defaults to True. Returns: bool: The reduced boolean decision. )device)Z reduce_op) torchr0intr!reducerZSUMboolr')rr4r5rrrreduce_boolean_decision\s z(ParallelStrategy.reduce_boolean_decisionccsNt|jtjjjrD|jdVWdqJ1s80YndVdS)zBlocks ddp sync gradients behaviour on backwards pass. This is useful for skipping sync when accumulating gradients, reducing communication overhead Returns: context manager with sync behaviour off N) isinstancemodelplZ utilitiestypesZDistributedDataParallelZno_syncr rrrblock_backward_syncms &z$ParallelStrategy.block_backward_synccs&|jdusJ|jtdSr))rteardownrr rrrrAzs zParallelStrategy.teardown)NNNNN)NF)T)%__name__ __module__ __qualname____doc__r rr7r6r r rrpropertyrr!r8r#r$r%r'r:r(rsetterrstrrr-r/r r3r;rrr@rA __classcell__rrrrrsL   r)abcrr contextlibrtypingrrrrr r7r Zpytorch_lightningr>Zlightning_fabric.pluginsr r Z&lightning_fabric.utilities.distributedr rZpytorch_lightning.pluginsrZ#pytorch_lightning.plugins.precisionrZ%pytorch_lightning.strategies.strategyrrrrrrs