a d@sdZddlmZmZmZddlZddlmZdgZGdddejZ Gdd d ej Z Gd d d ejZ Gd d d ej Z GdddejZGdddejZGdddejZdS)a_Module that implement Vision Transformer (ViT). Paper: https://paperswithcode.com/paper/an-image-is-worth-16x16-words-transformers-1 Based on: https://towardsdatascience.com/implementing-visualttransformer-in-pytorch-184f9f16f632 Added some tricks from https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/vision_transformer.py )ListOptionalTupleN)nnVisionTransformercs0eZdZddfdd ZddddZZS) ResidualAddN)returncst||_dSN)super__init__fn)selfr  __class__[/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/kornia/contrib/vit.pyr s zResidualAdd.__init__cKs"|}|j|fi|}||7}|Sr )r )r xkwargsresrrrforwardszResidualAdd.forward)__name__ __module__ __qualname__r r __classcell__rrrrrsrcs,eZdZdeeeeddfdd ZZS) FeedForwardN) in_featureshidden_features out_features dropout_raterc s8tt||tt|t||t|dSr )r r rLinearZGELUDropout)r rrrrrrrr s  zFeedForward.__init__)rrrrintfloatr rrrrrrsrcs>eZdZeeeeddfdd ZejejdddZZ S)MultiHeadAttentionN)emb_size num_headsatt_drop proj_droprcsht||_||_||}|d|_tj||ddd|_t||_ t|||_ t||_ dS)NgF)Zbias) r r r&r'scalerr qkvr!r( projectionprojection_drop)r r&r'r(r)Z head_sizerrrr (s   zMultiHeadAttention.__init__rrc Cs|j\}}}||||dd|ddddd}|d|d|d}}}td|||j} | jdd} || } td | |} | dddd ||d} | | } | | } | S) Nr*rzbhqd, bhkd -> bhqkZdimzbhal, bhlv -> bhav ) shaper,ZreshapepermutetorchZeinsumr+Zsoftmaxr( contiguousviewr-r.) r rBNCr,qkvZattoutrrrr5s &    zMultiHeadAttention.forward rrrr#r$r r7Tensorrrrrrrr%'s r%cs*eZdZeeeeddfdd ZZS)TransformerEncoderBlockN) embed_dimr'r dropout_attnrc s\tttt|t||||t|ttt|t||||dt|dS)N)r) r r rr SequentialZ LayerNormr%r!r)r rDr'rrErrrr Ks  z TransformerEncoderBlock.__init__r"rrrrrCJsrCcsBeZdZd eeeeeddfdd Zejejdd d ZZ S) TransformerEncoder rN)rDdepthr'rrErcs8ttjfddt|D|_g|_dS)Nc3s|]}tVqdSr )rC).0_rErrDr'rr iz.TransformerEncoder.__init__..)r r rrFrangeblocksresults)r rDrJr'rrErrMrr _s  zTransformerEncoder.__init__r/cCs2g|_|}|jD]}||}|j|q|Sr )rRrQchildrenappend)r rr@mrrrrms zTransformerEncoder.forward)rHrIrIrrrArrrrrG^srGcsneZdZdZdeeeeeejddfdd Ze eeefe eefd d d Z e j e j d ddZ ZS)PatchEmbeddingzJCompute the 2d image patch embedding ready to pass to transformer encoder.r*rHN) in_channels out_channels patch_size image_sizebackbonercst||_||_||_|p0tj||||d|_|durX||||f\}}||_n ||d}t t dd||_ t t |d||_ dS)N)Z kernel_sizeZstrider1r2)r r rYrZr[rZConv2dr]_compute_feats_dims Parameterr7Zrandn cls_token positions)r rYrZr[r\r]Z feat_sizerrrr ys  zPatchEmbedding.__init__)r\rcCs:|tjdg|R}|jd|jd|jdfS)Nr2r0)r]r7zerosdetachr5)r r\r@rrrr^sz"PatchEmbedding._compute_feats_dimsr/cCs`||}|j\}}}}|||dddd}|j|dd}tj||gdd}||j7}|S)Nr0rr1r2r4) r]r5r9r6r`repeatr7catra)r rr:r;rLZ cls_tokensrrrrs  zPatchEmbedding.forward)r*rHrWrXN)rrr__doc__r#rrModuler rr^r7rBrrrrrrrVvs "rVc s`eZdZdZdeeeeeeeeeejdd fd d Z e d d Z e j e j dddZZS)raVision transformer (ViT) module. The module is expected to be used as operator for different vision tasks. The method is inspired from existing implementations of the paper :cite:`dosovitskiy2020vit`. .. warning:: This is an experimental API subject to changes in favor of flexibility. Args: image_size: the size of the input image. patch_size: the size of the patch to compute the embedding. in_channels: the number of channels for the input. embed_dim: the embedding dimension inside the transformer encoder. depth: the depth of the transformer. num_heads: the number of attention heads. dropout_rate: dropout rate. dropout_attn: attention dropout rate. backbone: an nn.Module to compute the image patches embeddings. Example: >>> img = torch.rand(1, 3, 224, 224) >>> vit = VisionTransformer(image_size=224, patch_size=16) >>> vit(img).shape torch.Size([1, 197, 768]) rXrWr*rHrIrN) r\r[rYrDrJr'rrEr]rc sRt||_||_||_||_t||||| |_|jj} t | |||||_ dSr ) r r r\r[rYZ embed_sizerVpatch_embeddingrZrGencoder) r r\r[rYrDrJr'rrEr]Z hidden_dimrrrr s zVisionTransformer.__init__cCs|jjSr )rkrR)r rrrencoder_resultssz!VisionTransformer.encoder_resultsr/c Cst|tjstdt||jg|jddRvrp|jd|jkrptd|jd|jd|jd|j| |}| |}|S)Nz)Input x type is not a torch.Tensor. Got: rcrbzInput image shape must be Bxrz. Got: ) isinstancer7rB TypeErrortyper\r5rY ValueErrorrjrk)r rr@rrrrs *  zVisionTransformer.forward) rXrWr*rHrIrIrrN)rrrrhr#r$rrrir propertyrlr7rBrrrrrrrs2 )rhtypingrrrr7r__all__rirrFrr%rCrGrVrrrrrs   #+