a  d@sdZddlZddlmZddlmmZdeeee e e ddd Z dej eeee e d d d Z Gd ddej Zdee e dddZGdddej ZdS)a> DropBlock, DropPath PyTorch implementations of DropBlock and DropPath (Stochastic Depth) regularization layers. Papers: DropBlock: A regularization method for convolutional networks (https://arxiv.org/abs/1810.12890) Deep Networks with Stochastic Depth (https://arxiv.org/abs/1603.09382) Code: DropBlock impl inspired by two Tensorflow impl that I liked: - https://github.com/tensorflow/tpu/blob/master/models/official/resnet/resnet_model.py#L74 - https://github.com/clovaai/assembled-cnn/blob/master/nets/blocks.py Hacked together by / Copyright 2020 Ross Wightman N皙??F) drop_prob block_size gamma_scale with_noiseinplace batchwisecCs|j\}}} } | | } t|t| | } ||| | d| |d| |d} tt| |jt| |j\}}|| dk|| | ddk@|| dk|| | ddk@@}t|dd| | fj|jd}|rtj d|| | f|j|jd}n t |}d| ||dkj|jd}t j | | d| dd }|r|rhtj d|| | f|j|jdnt|}|r|||d|n|||d|}nJ||jtjdd|j}|r|||n |||}|S)a  DropBlock. See https://arxiv.org/pdf/1810.12890.pdf DropBlock with an experimental gaussian noise option. This layer has been tested on a few training runs with success, but needs further validation and possibly optimization for lower runtime impact. dtype)rdeviceZ kernel_sizeZstridepaddinggHz>)shapemintorchZmeshgridZarangetorZreshaperZrandZ rand_likeF max_pool2dZrandnZ randn_likemul_add_numelfloat32sumadd)xrrrrr r BCHW total_sizeclipped_block_sizegammaZw_iZh_iZ valid_blockZ uniform_noise block_mask normal_noisenormalize_scaler)Y/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/timm/layers/drop.py drop_block_2ds>, ,( r+)rrrrrr cCs|j\}}}} | |} t|t| |} ||| | d| |d||d} t|| } tj| |j| d| dd} |rt| }|r| d|  || n|d| || }nTd| } | | jtj ddj|jd}|r| | |n || |}|S)z DropBlock. See https://arxiv.org/pdf/1810.12890.pdf DropBlock with an experimental gaussian noise option. Simplied from above without concern for valid block mask at edges. r r rrr gư>)rrrZ empty_like bernoulli_rrrrZnormal_rrrrrr)rrrrrr rr r!r"r#r$r%r&r'r(r)r)r*drop_block_fast_2dFs** r-c s<eZdZdZd eeeeeeedfdd Zd d ZZ S) DropBlock2dz9 DropBlock. See https://arxiv.org/pdf/1810.12890.pdf rrrFT)rrrrr r fastcs<tt|||_||_||_||_||_||_||_ dSN) superr.__init__rrrrr r r/)selfrrrrr r r/ __class__r)r*r2ls zDropBlock2d.__init__cCsV|jr |js|S|jr2t||j|j|j|j|jSt||j|j|j|j|j|j SdSr0) trainingrr/r-rrrr r+r r3rr)r)r*forward~s zDropBlock2d.forward)rrrFFFT) __name__ __module__ __qualname____doc__floatintboolr2r8 __classcell__r)r)r4r*r.hs$r.T)rr6 scale_by_keepcCs`|dks |s|Sd|}|jdfd|jd}|||}|dkrX|rX||||S)a(Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). This is the same as the DropConnect impl I created for EfficientNet, etc networks, however, the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper... See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted for changing the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use 'survival rate' as the argument. rAr r)r )rndimZ new_emptyr,Zdiv_)rrr6rBZ keep_probrZ random_tensorr)r)r* drop_paths   rDcs:eZdZdZd eedfdd ZddZd d ZZ S) DropPathz^Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). rAT)rrBcstt|||_||_dSr0)r1rEr2rrB)r3rrBr4r)r*r2szDropPath.__init__cCst||j|j|jSr0)rDrr6rBr7r)r)r*r8szDropPath.forwardcCsdt|jddS)Nz drop_prob=z0.3f)roundr)r3r)r)r* extra_reprszDropPath.extra_repr)rAT) r9r:r;r<r=r?r2r8rHr@r)r)r4r*rEsrE)rrrFFF)rrrFF)rAFT)r<rZtorch.nnnnZtorch.nn.functionalZ functionalrr=r>r?r+ZTensorr-Moduler.rDrEr)r)r)r*s$  1 "!