a  d-]@sdZddlZddlZddlmZddlmZmZddlZddl m Z ddl m Z ddl mZddlmZmZddlmZmZmZmZmZmZd d lmZd d lmZmZd gZee Z!Gd dde j"Z#Gddde j"Z$Gddde j"Z%Gddde j"Z&Gdd d e j"Z'dGddZ(dHddZ)ee)dddde)dd d!e)d"dd#e)d$dd#e)d%dd#e)d&dd#e)d'dd#e)d(dd#e)e)d)dd#e)e)d*dd#e)d+ Z*edIe'd,d-d.Z+edJe'd,d/d0Z,edKe'd,d1d2Z-edLe'd,d3d4Z.edMe'd,d5d6Z/edNe'd,d7d8Z0edOe'd,d9d:Z1edPe'd,d;d<Z2edQe'd,d=d>Z3edRe'd,d?d@Z4edSe'd,dAdBZ5edTe'd,dCdDZ6edUe'd,dEdFZ7dS)Vz Relative Position Vision Transformer (ViT) in PyTorch NOTE: these models are experimental / WIP, expect changes Hacked together by / Copyright 2022, Ross Wightman N)partial)OptionalTuple)Final) checkpointIMAGENET_INCEPTION_MEANIMAGENET_INCEPTION_STD) PatchEmbedMlpDropPath RelPosMlp RelPosBiasuse_fused_attn)build_model_with_cfg)generate_default_cfgsregister_modelVisionTransformerRelPoscsReZdZUeeed<ddddddejffdd Zd e e j dd d Z Z S) RelPosAttention fused_attnFNc st||dksJd||_|||_|jd|_t|_tj||d|d|_ |rf||jnt |_ |r~||jnt |_ |r||dnd|_ t||_t|||_t||_dS)Nrz$dim should be divisible by num_headsg)Zbias num_heads)super__init__rhead_dimscalerrnnLinearqkvIdentityq_normk_normrel_posDropout attn_dropproj proj_drop) selfdimrqkv_biasqk_norm rel_pos_clsr(r* norm_layer __class__n/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/timm/models/vision_transformer_relpos.pyrs    zRelPosAttention.__init__shared_rel_posc Cs6|j\}}}||||d|j|jddddd}|d\}}} ||}||}|j r|j durv|j } n|dur|} nd} t j jj||| | |jjd}nb||j}||dd} |j dur|j | |d } n|dur| |} | jdd } || } | | }|dd|||}||}||}|S) Nrrr)Z attn_maskZ dropout_pr5r,)shaper"ZreshaperrZpermuteZunbindr$r%rr&get_biastorchr Z functionalZscaled_dot_product_attentionr(prZ transposeZsoftmaxr)r*) r+xr6BNCr"qkvZ attn_biasattnr3r3r4forward8s: *          zRelPosAttention.forward)N)__name__ __module__ __qualname__rbool__annotations__r LayerNormrrr>TensorrH __classcell__r3r3r1r4rs  rcs&eZdZdfdd ZddZZS) LayerScaleh㈵>Fcs*t||_t|t||_dSN)rrinplacer Parameterr>Zonesgamma)r+r, init_valuesrTr1r3r4r^s zLayerScale.__init__cCs|jr||jS||jSrS)rTZmul_rVr+r@r3r3r4rHcszLayerScale.forward)rRF)rIrJrKrrHrPr3r3r1r4rQ]srQc sLeZdZddddddddejejf fdd Zd eej ddd Z Z S) RelPosBlock@FNrc st| ||_t|||||| |d|_|rrOrHrPr3r3r1r4rYgs(rYc sTeZdZddddddddejejf fdd ZddZd ee j d d d Z Z S) ResPostRelPosBlockrZFNrc st||_t|||||| |d|_| ||_| dkrBt| nt|_ t |t ||| |d|_ | ||_ | dkrt| nt|_|dS)Nr[rr\)rrrWrrGr^r r r#r`r rbrcrare init_weightsrfr1r3r4rs,    zResPostRelPosBlock.__init__cCs6|jdur2tj|jj|jtj|jj|jdSrS)rWr initZ constant_r^Zweightrar+r3r3r4rms zResPostRelPosBlock.init_weightsr5c Cs<||||j||d}|||||}|Sri)r`r^rGrerarcrjr3r3r4rHszResPostRelPosBlock.forward)N) rIrJrKr rkrNrrmrr>rOrHrPr3r3r1r4rls)rlcseZdZdZddddddddd d d d d d d dd dddddeddeffdd Zd(ddZej j ddZ ej j d)ddZ ej j d*ddZ ej j ddZd+eddd Zd!d"Zd,ed#d$d%Zd&d'ZZS)-rah Vision Transformer w/ Relative Position Bias Differing from classic vit, this impl * uses relative position index (swin v1 / beit) or relative log coord + mlp (swin v2) pos embed * defaults to no class token (can be enabled) * defaults to global avg pool for head (can be changed) * layer-scale (residual branch gain) enabled ravg rZTFư>rcNrskipcst|dvsJ| s&|dks&Jp6ttjddp@tj||_||_|_|_ | rbdnd|_ d|_ ||||d|_ |j j }t||j d }|d r|r||d <d |vrd |d <ttfi| nttfi| d|_|r d|_d | rttd|j nd|_ddtd||Dt f ddt|D|_|svnt|_|rnt|_t||_|dkrt |j |nt|_!|dkr|"|dS)a Args: img_size (int, tuple): input image size patch_size (int, tuple): patch size in_chans (int): number of input channels num_classes (int): number of classes for classification head global_pool (str): type of global pooling for final sequence (default: 'avg') embed_dim (int): embedding dimension depth (int): depth of transformer num_heads (int): number of attention heads mlp_ratio (int): ratio of mlp hidden dim to embedding dim qkv_bias (bool): enable bias for qkv if True qk_norm (bool): Enable normalization of query and key in attention init_values: (float): layer-scale init values class_token (bool): use class token (default: False) fc_norm (bool): use pre classifier norm instead of pre-pool rel_pos_ty pe (str): type of relative position shared_rel_pos (bool): share relative pos across all blocks drop_rate (float): dropout rate proj_drop_rate (float): projection dropout rate attn_drop_rate (float): attention dropout rate drop_path_rate (float): stochastic depth rate weight_init (str): weight init scheme embed_layer (nn.Module): patch embedding layer norm_layer: (nn.Module): normalization layer act_layer: (nn.Module): MLP activation layer rstokenrzrv)ZepsrrF)img_size patch_sizein_chans embed_dim)Z window_sizeZ prefix_tokensrcZ hidden_dimZswinmodeNrcSsg|] }|qSr3)item).0r@r3r3r4 5z4VisionTransformerRelPos.__init__..cs0g|](}  |d qS)) r,rrgr-r.r/rWr*r(rhr0r]r3)ri r]attn_drop_rateblock_fnZdprr~rWrgr0rproj_drop_rater.r-r/r3r4r6srw)#rrrr rNrk num_classes global_poolZ num_featuresr~num_prefix_tokensgrad_checkpointing patch_embedZ grid_sizedict startswithr rr6rUr>zeros cls_tokenZlinspaceZ ModuleListrangeblocksr#normfc_normr' head_dropr!headrm)r+r{r|r}rrr~depthrrgr-r.rW class_tokenrZ rel_pos_type rel_pos_dimr6Z drop_raterrZdrop_path_rateZ weight_initZ embed_layerr0r]rZ feat_sizeZ rel_pos_argsr1rr4rsP8      "&  " z VisionTransformerRelPos.__init__rycCs,|dvs J|jdur(tjj|jdddS)N)ZjaxZmocoryrv)std)rr rnZnormal_)r+rr3r3r4rmPs  z$VisionTransformerRelPos.init_weightscCsdhS)Nrr3ror3r3r4no_weight_decayWsz'VisionTransformerRelPos.no_weight_decaycCstdddgdS)Nz^cls_token|patch_embed)z^blocks\.(\d+)N)z^norm)i)stemr)r)r+Zcoarser3r3r4 group_matcher[sz%VisionTransformerRelPos.group_matchercCs ||_dSrS)r)r+enabler3r3r4set_grad_checkpointingbsz.VisionTransformerRelPos.set_grad_checkpointingcCs|jSrS)rror3r3r4get_classifierfsz&VisionTransformerRelPos.get_classifier)rcCsD||_|dur |dvsJ||_|dkr6t|j|nt|_dS)Nrxr)rrr r!r~r#r)r+rrr3r3r4reset_classifierjs  z(VisionTransformerRelPos.reset_classifiercCs||}|jdur8tj|j|jddd|fdd}|jdurL|jnd}|jD]0}|j rztj szt |||d}qV|||d}qV| |}|S)Nrr:rr;r5)rrr>catexpandr<r6r=rrjitZ is_scriptingrr)r+r@r6Zblkr3r3r4forward_featuresqs  $  z(VisionTransformerRelPos.forward_features) pre_logitscCsd|jr>|jdkr.|dd|jdfjddn|dddf}||}||}|rZ|S||S)Nrsrr;r)rrmeanrrr)r+r@rr3r3r4 forward_heads 8  z$VisionTransformerRelPos.forward_headcCs||}||}|SrS)rrrXr3r3r4rHs  zVisionTransformerRelPos.forward)ry)F)T)N)F)rIrJrK__doc__r rYrrmr>rignorerrrrrbrrrLrrHrPr3r3r1r4rsR x     FcKs,|ddrtdtt||fi|}|S)NZ features_onlyzZtorch.nnr Z torch.jitrZtorch.utils.checkpointrZ timm.datarr Z timm.layersr r r r rrZ_builderr _registryrr__all__ getLoggerrIZ_loggerModulerrQrYrlrrrZ default_cfgsrrrrrrrrrrrrrr3r3r3r4s       B 07?   (