a d&@sddlmZddlZddlmZGdddejZeeejdddZdeeeeejd d d ZGdddejZ GdddejZ GdddejZ GdddejZ GdddejZ GdddejZGdddejZdS))TupleN)nnc@s$eZdZdZejejdddZdS)SiLUzModule SiLU (Sigmoid Linear Units) This implementation is to support pytorch < 1.8, and will be deprecated after 1.8. Paper: https://arxiv.org/abs/1702.03118 xreturncCs|t|SN)torchZsigmoidselfrr b/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/kornia/contrib/vit_mobile.pyforwardsz SiLU.forwardN)__name__ __module__ __qualname____doc__r Tensorrr r r r rsr)inpouprc Cs(ttj||dddddt|tS)NrFbiasr SequentialConv2d BatchNorm2dr)rrr r r conv_1x1_bnsrr)rr kernal_sizestriderc Cs(ttj||||dddt|tS)NrFrr)rrrr r r r conv_nxn_bnsr!cs<eZdZeejddfdd ZejejdddZ Z S)PreNormN)dimfnrcs tt||_||_dSr)super__init__rZ LayerNormnormr$)r r#r$ __class__r r r&s  zPreNorm.__init__rcKs|j||fi|Sr)r$r')r rkwargsr r r r!szPreNorm.forward) rrrintrModuler&r rr __classcell__r r r(r r"sr"cs>eZdZd eeeddfdd ZejejdddZZ S) FeedForwardN)r# hidden_dimdropoutrc s@ttt||tt|t||t||_dSr)r%r&rrLinearrDropoutnet)r r#r0r1r(r r r&&s (zFeedForward.__init__rcCs ||Sr)r4r r r r r,szFeedForward.forward)r/ rrrr+floatr&r rrr-r r r(r r.%sr.cs@eZdZd eeeeddfdd Zejejdd d ZZ S) Attention@r/N)r#headsdim_headr1rcst||}|dko ||k }||_|d|_tjdd|_tj||ddd|_|rvt t||t |nt |_ dS)Nrgr#rFr) r%r&r:scalerZSoftmaxattendr2to_qkvrr3ZIdentityto_out)r r#r:r;r1Z inner_dimZ project_outr(r r r&1s  zAttention.__init__rc s|jddd}|dj\tfdd|\}}}t||ddj}|}t||}|j\} } | | | } |S)Nrr<r=rcs|jjSr) contiguousviewr:)tBZHDNPr r r Bz#Attention.forward..) r@chunkshapemapr matmulZ transposer>r?rCrA) r rZqkvqkvdotsattnoutHDr rEr r>s   zAttention.forward)r8r9r/r5r r r(r r70s r7c sHeZdZdZd eeeeeeddfdd Zejejddd Z Z S) TransformeraTransformer block described in ViT. Paper: https://arxiv.org/abs/2010.11929 Based on: https://github.com/lucidrains/vit-pytorch Args: dim: input dimension. depth: depth for transformer block. heads: number of heads in multi-head attention layer. dim_head: head size. mlp_dim: dimension of the FeedForward layer. dropout: dropout ratio, defaults to 0. r/N)r#depthr:r;mlp_dimr1rc s\ttg|_t|D]8}|jtt|t||||t|t |||gqdSr) r%r&r ModuleListlayersrangeappendr"r7r.)r r#rYr:r;rZr1_r(r r r&[s   zTransformer.__init__rcCs,|jD] \}}|||}|||}q|Sr)r\)r rrTffr r r rhs zTransformer.forward)r/) rrrrr+r6r&r rrr-r r r(r rXLs rXcsDeZdZdZd eeeeddfdd Zejejdd d ZZ S) MV2BlockapMV2 block described in MobileNetV2. Paper: https://arxiv.org/pdf/1801.04381 Based on: https://github.com/tonylins/pytorch-mobilenet-v2 Args: inp: input channel. oup: output channel. stride: stride for convolution, defaults to 1, set to 2 if down-sample. expansion: expansion ratio for hidden dimension, defaults to 4. rN)rrr expansionrcst||_t||}|jdko,||k|_|dkrttj||d|d|ddt|t tj||dddddt||_ nhttj||dddddt|t tj||d|d|ddt|t tj||dddddt||_ dS)NrrF)groupsrrr) r%r&r r+use_res_connectrrrrrconv)r rrr rcr0r(r r r&|s,   zMV2Block.__init__rcCs"|jr|||S||SdSr)rerfr r r r rszMV2Block.forward)rrb) rrrrr+r&r rrr-r r r(r raos  rac sReZdZdZd eeeeeeefeeddfdd Zej ej ddd Z Z S) MobileViTBlockaxMobileViT block mentioned in MobileViT. Args: dim: input dimension of Transformer. depth: depth of Transformer. channel: input channel. kernel_size: kernel size. patch_size: patch size for folding and unfloding. mlp_dim: dimension of the FeedForward layer in Transformer. dropout: dropout ratio, defaults to 0. r/N)r#rYchannel kernel_size patch_sizerZr1rcsft|\|_|_t||||_t|||_t||dd|||_ t|||_ td||||_ dS)Nrbr8) r%r&phpwr!conv1rconv2rX transformerconv3conv4)r r#rYrhrirjrZr1r(r r r&s    zMobileViTBlock.__init__rcCs|}||}||}|j\}}}}|||j|j||j||j|}||}|||||}||}t ||fd}| |}|S)Nr) clonernrorMrCrlrmrprqr catrr)r rybdhwr r r rs  (   zMobileViTBlock.forward)r/) rrrrr+rr6r&r rrr-r r r(r rgs rgcsLeZdZdZd eeeeefeddfdd Ze j e j d d d Z Z S) MobileViTa=Module MobileViT. Default arguments is for MobileViT XXS. Paper: https://arxiv.org/abs/2110.02178 Based on: https://github.com/chinhsuanwu/mobilevit-pytorch Args: mode: 'xxs', 'xs' or 's', defaults to 'xxs'. in_channels: the number of channels for the input image. patch_size: image_size must be divisible by patch_size. dropout: dropout ratio in Transformer. Example: >>> img = torch.rand(1, 3, 256, 256) >>> mvit = MobileViT(mode='xxs') >>> mvit(img).shape torch.Size([1, 320, 8, 8]) xxsrrkrkr/N)mode in_channelsrjr1rc s&t|dkr(d}gd}gd}n:|dkrFd}gd}gd}n|d krbd}gd }gd }d }gd } t||ddd|_tg|_|jt|d|dd||jt|d|dd||jt|d|d d||jt|d|d d||jt|d |dd||jt|d|dd||jt|d|dd|tg|_ |j t |d| d|d||t |dd|d|j t |d| d|d||t |dd|d|j t |d| d|d||t |dd|dt |d|d|_ dS)Nr{rk)r9P`) rr0rr9r9rri@xsrb)rx) r rrr9r9rrrris)r) rrr9r9rrrrir)rkrbrr)r rr8)r1 rKr<)r%r&r!rnrr[mv2r^ramvitrgr+rro) r r}r~rjr1rcdimsZchannelsrirYr(r r r&sF     ,,,zMobileViT.__init__rcCs||}|jd|}|jd|}|jd|}|jd|}|jd|}|jd|}|jd|}|jd|}|jd|}|jd|}||}|S)Nrrrkrrbrr)rnrrror r r r rs  zMobileViT.forward)r{rr|r/) rrrrstrr+rr6r&r rrr-r r r(r rzs,rz)rr)typingrr rr,rr+rr!r"r.r7rXrargrzr r r r s     #46