a d@slddlmZmZddlZddlmZddlmZGdddeZGdddeZGd d d ej j j j Z dS) )ABCabstractmethodN)Tensor)Modulec@s8eZdZdZeeedddZeeedddZdS) LayerSynczxAbstract base class for creating plugins that wrap layers of a model with synchronization logic for multiprocessing.modelreturncCsdS)zJOverride this method to apply synchronization to the layers of this model.Nselfrr r m/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/pytorch_lightning/plugins/layer_sync.pyapplyszLayerSync.applycCsdS)zEOverride this method to undo all modifications made in :meth:`apply`.Nr r r r r revertszLayerSync.revertN)__name__ __module__ __qualname____doc__rrrrr r r r rs rc@s0eZdZdZeedddZeedddZdS)NativeSyncBatchNormzA plugin that wraps all batch normalization layers of a model with synchronization logic for multiprocessing. This plugin has no effect in single-device operation. rcCstjj|S)aAdd global batchnorm for a model spread across multiple GPUs and nodes. Override this method to synchronize batchnorm layers between specific process groups instead of the whole world. Args: model: Reference to the current LightningModule Return: LightningModule with batchnorm layers synchronized within the process groups. )torchnn SyncBatchNormZconvert_sync_batchnormr r r r r*s zNativeSyncBatchNorm.applycCs|}t|tjjjjrt|j|j|j |j |j }|j rnt |j |_ |j|_Wdn1sd0Y|j|_|j|_|j|_t|dr|j|_|D]\}}||||q~|S)aConvert the wrapped batchnorm layers back to regular batchnorm layers. Args: model: Reference to the current LightningModule Return: LightningModule with regular batchnorm layers that will no longer sync across processes. Nqconfig) isinstancerrmodules batchnormr _BatchNormXdZ num_featuresZepsZmomentumZaffineZtrack_running_statsZno_gradZweightZbiasZ running_meanZ running_varZnum_batches_trackedhasattrrZnamed_childrenZ add_moduler)r rZconverted_modulenamechildr r r r8s$  & zNativeSyncBatchNorm.revertN)rrrrrrrr r r r r#src@seZdZeddddZdS)rN)inputr cCsdS)Nr )r r r r r _check_input_dimZsz_BatchNormXd._check_input_dim)rrrrr!r r r r rYsr)abcrrrrZtorch.nnrrrrrrZ _BatchNormrr r r r s    6