a  dZ@sldZddlmZddlmZddlZddlmZddlmm Z ddlm Z ddl m Z mZddlmZmZmZmZmZmZmZddlmZmZd d lmZd d lmZd d lmZd d lm Z m!Z!dgZ"Gdddej#Z$Gdddej#Z%Gdddej#Z&Gdddej#Z'Gdddej#Z(e ee)e)fdddZ*ee ee)e)fe)e)dddZ+Gdd d ej#Z,Gd!d"d"ej#Z-Gd#d$d$ej#Z.Gd%ddej#Z/d&d'Z0d>d)d*Z1d?d,d-Z2e e2d.d/e2d.d/e2d.d/e2e2e2d0Z3e!d@e/d1d2d3Z4e!dAe/d1d4d5Z5e!dBe/d1d6d7Z6e!dCe/d1d8d9Z7e!dDe/d1d:d;Z8e!dEe/d1dr; norm_layerr(r*r+r!3s z Stem.__init__r,c Cs|j\}}}}t|d|jd||jd|jdf}t|ddd|jd||jd|jdf}||}||}|S)Nrr)shapeFpadr;r?r@r'r-BCHWr*r*r+r/Is .2  z Stem.forward) r0r1r2__doc__r r!rr/r5r*r*r(r+r6.sr6cs.eZdZeffdd ZedddZZS) Downsamplecs:t||_||_|||_tj||dddd|_dS)Nrrr:)r r!r=r>r@r"r#r?)r'r=r>rAr(r*r+r!Ss  zDownsample.__init__r,c Csb|j\}}}}||}t|dd|ddf}t|dddd|ddf}||}|S)Nrr)rBr@rCrDr?rEr*r*r+r/fs    zDownsample.forward)r0r1r2r r!rr/r5r*r*r(r+rKRsrKcs,eZdZdfdd ZedddZZS) ChannelAttentionFcsJt||_||}|d|_tj||d|d|_t|||_dS)Nrbias)r r! num_headsscaler"Linearqkvr$)r'rrQqkv_biashead_dimr(r*r+r!qs   zChannelAttention.__init__r,c Cs|j\}}}||||d|j||jddddd}|d\}}}||j}|dd|} | jdd} | |dddd}|dd|||}| |}|S) Nrrrrr8r) rBrTreshaperQpermuteunbindrR transposesoftmaxr$) r'r-rFNrGrTqrvZ attentionr*r*r+r/zs .   zChannelAttention.forward)rMF)r0r1r2r!rr/r5r*r*r(r+rLos rLcs>eZdZdddejejddffdd Zeddd ZZ S) ChannelBlock@FTc stt|d| d|_||_|||_t|||d|_|dkrJt|nt |_ t|d| d|_ |jr|||_ t|t|||d|_|dkrt|nt |_nd|_ d|_d|_dSNrr)rQrUrd)Z in_featuresZhidden_features act_layer)r r!rcpe1ffnnorm1rLattnrr"r& drop_path1cpe2norm2r r3mlp drop_path2) r'rrQ mlp_ratiorU drop_pathrfrArhcpe_actr(r*r+r!s$    zChannelBlock.__init__r,cCs|j\}}}}||ddd}||}||}|||}||dd||||}|j dur|ddd}|| | | |}|dd||||}|S)Nrr) rBrgflattenr]rirjrkrlviewrnrorm)r'r-rFrGrHrIcurr*r*r+r/s   zChannelBlock.forward) r0r1r2r"r% LayerNormr!rr/r5r*r*r(r+rbs"rb)r- window_sizecCsj|j\}}}}||||d|d||d|d|}|ddddddd|d|d|}|S)z Args: x: (B, H, W, C) window_size (int): window size Returns: windows: (num_windows*B, window_size, window_size, C) rrrrr8rWrBrtr[ contiguous)r-rwrFrHrIrGwindowsr*r*r+window_partitions,,r|)r{rwrHrIcCs^|jd}|d||d||d|d|d|}|ddddddd|||}|S)z Args: windows: (num_windows*B, window_size, window_size, C) window_size (int): Window size H (int): Height of image W (int): Width of image Returns: x: (B, H, W, C) rWrrrrr8rxry)r{rwrHrIrGr-r*r*r+window_reverses ,$r}csBeZdZUdZejjeed<d fdd Z e dddZ Z S) WindowAttentiona Window based multi-head self attention (W-MSA) module with relative position bias. It supports both of shifted and non-shifted window. Args: dim (int): Number of input channels. window_size (tuple[int]): The height and width of the window. num_heads (int): Number of attention heads. qkv_bias (bool, optional): If True, add a learnable bias to query, key, value. Default: True fused_attnTcslt||_||_||_||}|d|_t|_tj ||d|d|_ t |||_ tj dd|_ dS)NrNrrOrWrY)r r!rrwrQrRrrr"rSrTr$ZSoftmaxr^)r'rrwrQrUrVr(r*r+r!s  zWindowAttention.__init__r,c Cs|j\}}}||||d|j||jddddd}|d\}}}|jr`t|||}n,||j }|| dd} | | } | |}| dd|||}| |}|S)Nrrrrr8rXrW) rBrTrZrQr[r\rrCZscaled_dot_product_attentionrRr]r^r$) r'r-ZB_r_rGrTr`rrarjr*r*r+r/s .   zWindowAttention.forward)T) r0r1r2rJtorchjitFinalr4__annotations__r!rr/r5r*r*r(r+r~s r~csDeZdZdZddddejejddffdd Zed d d Z Z S) SpatialBlocka< Windows Block. Args: dim (int): Number of input channels. num_heads (int): Number of attention heads. window_size (int): Window size. mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. qkv_bias (bool, optional): If True, add a learnable bias to query, key, value. Default: True drop_path (float, optional): Stochastic depth rate. Default: 0.0 act_layer (nn.Module, optional): Activation layer. Default: nn.GELU norm_layer (nn.Module, optional): Normalization layer. Default: nn.LayerNorm r9rcTrdFc st||_| |_||_t||_||_t|d| d|_ |||_ t ||j||d|_ |dkrjt |nt|_t|d| d|_|jr|||_t||} t|| |d|_|dkrt |nt|_nd|_d|_d|_dSre)r r!rrhrQr rwrprrgrir~rjrr"r&rkrlrmr3r rnro) r'rrQrwrprUrqrfrArhrrZmlp_hidden_dimr(r*r+r!s8     zSpatialBlock.__init__r,c Cs|j\}}}}||ddd}||}|||||}d}}|jd||jd|jd} |jd||jd|jd} t|dd|| || f}|j\} } } } t ||j}|d|jd|jd|}| |}|d|jd|jd|}t ||j| | }|ddd|d|ddf }|||||}|| |}||dd||||}|jdur|ddd}|||||}|dd||||}|S)NrrrrW)rBrgrsr]rirtrwrCrDr|rjr}rzrkrlrnrorm)r'r-rFrGrHrIZshortcutZpad_lZpad_tZpad_rZpad_b_ZHpZWpZ x_windowsZ attn_windowsr*r*r+r/Es. ""  $ zSpatialBlock.forward) r0r1r2rJr"r%rvr!rr/r5r*r*r(r+r s-rc sXeZdZddddddddeejddf fd d Zejj dd d Z e d ddZ Z S) DaVitStagerTspatialchannelrr9r8)rrFcstd|_|r&t||| d|_n t|_g}t|D]}g}t|D]`\}}|dkr| t |||| | || | ||d qL|dkrL| t |||| | || | |dqL| tj |qdepthr attn_typesrQrwrprUdrop_path_ratesrA norm_layer_clrhrrZ stage_blocksZ block_idxZdual_attention_blockZattn_idxZ attn_typer(r*r+r!nsD     zDaVitStage.__init__cCs ||_dSr.)r)r'enabler*r*r+set_grad_checkpointingsz!DaVitStage.set_grad_checkpointingr,cCs6||}|jr(tjs(t|j|}n ||}|Sr.)rrrr is_scriptingrrr'r-r*r*r+r/s   zDaVitStage.forward)T)r0r1r2r r"rvr!rrignorerrr/r5r*r*r(r+rms @ rcsveZdZdZd#fdd ZddZejjd$ddZ ejjddZ d%ddZ ddZ d&e ddd Zd!d"ZZS)'ra DaViT A PyTorch implementation of `DaViT: Dual Attention Vision Transformers` - https://arxiv.org/abs/2204.03645 Supports arbitrary input sizes and pyramid feature extraction Args: in_chans (int): Number of input image channels. Default: 3 num_classes (int): Number of classes for classification head. Default: 1000 depths (tuple(int)): Number of blocks in each stage. Default: (1, 1, 3, 1) embed_dims (tuple(int)): Patch embedding dimension. Default: (96, 192, 384, 768) num_heads (tuple(int)): Number of attention heads in different layers. Default: (3, 6, 12, 24) window_size (int): Window size. Default: 7 mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. Default: 4 qkv_bias (bool): If True, add a learnable bias to query, key, value. Default: True drop_path_rate (float): Stochastic depth rate. Default: 0.1 norm_layer (nn.Module): Normalization layer. Default: nn.LayerNorm. rrrrrr7r r9r8T layernorm2d layernormh㈵>rFrdavgcstt|}|t|kr.t|ks4nJtt|| d}tt| | d} ||_|d|_||_d|_g|_ t ||d|d|_ |d}ddt d|t||D}g}t|D]n}||}t|||||dk| ||||||||| | | d}|}|||j t|d d |d g7_ qtj||_|rf||j|_t|j|||jd |_n"t|_t|j|||j|d |_||jdS)N)ZepsrWFrrcSsg|] }|qSr*)tolist).0r-r*r*r+ z"DaVit.__init__..) rrrrQrwrprUrrArrhrrrzstages.)Znum_chsZ reductionmodule) pool_type drop_rate)rrrA)r r!lenrr num_classesZ num_featuresrrZ feature_infor6stemrZlinspacesumsplitrrrdictr"rstagesnorm_prerheadr&rapply _init_weights)r'Zin_chansdepths embed_dimsrQrwrprUrArZnorm_epsrrhrrrZdrop_path_rater global_poolZhead_norm_firstZ num_stagesr=ZdprrZ stage_idxr>stager(r*r+r!sj " "  "    zDaVit.__init__cCsDt|tjr@t|jddt|tjr@|jdur@tj|jddS)Ng{Gz?)stdr) isinstancer"rSr ZweightrPinitZ constant_)r'mr*r*r+r s zDaVit._init_weightscCs"||_|jD]}|j|dq dS)N)r)rrr)r'rrr*r*r+r&s zDaVit.set_grad_checkpointingcCs|jjSr.)rfc)r'r*r*r+get_classifier,szDaVit.get_classifierNcCs|jj||ddS)N)r)rreset)r'rrr*r*r+reset_classifier0szDaVit.reset_classifiercCs@||}|jr(tjs(t|j|}n ||}||}|Sr.)rrrrrrrrrr*r*r+forward_features3s    zDaVit.forward_features) pre_logitscCsD|j|}|j|}|j|}|j|}|r8|S|j|Sr.)rrr@rsZdropr)r'r-rr*r*r+ forward_head<s     zDaVit.forward_headcCs||}||}|Sr.)rrrr*r*r+r/Cs  z DaVit.forward)rrrrr9r8TrrrrTFrdrdrrF)T)N)F)r0r1r2rJr!rrrrrrrrr4rr/r5r*r*r(r+rs:S    cCsd|vr |Sd|vr|d}ddl}i}|D]t\}}|dd|}|dd|}|d d }|d d }|d d}|dd}|dd}|dd}|||<q0|S)z Remap MSFT checkpoints -> timm zhead.fc.weight state_dictrNzpatch_embeds.([0-9]+)zstages.\1.downsamplezmain_blocks.([0-9]+)zstages.\1.blockszdownsample.projzdownsample.convzstages.0.downsamplerzhead.zhead.fc.znorms.z head.norm.zcpe.0rgzcpe.1rl)reitemssubreplace)rmodelrZout_dictrrar*r*r+checkpoint_filter_fnIs"       rFcKsPtddt|ddD}|d|}tt||fttd|dd|}|S) Ncss|]\}}|VqdSr.r*)rirr*r*r+ arz _create_davit..rr out_indicesT)Zflatten_sequentialr)Zpretrained_filter_fnZ feature_cfg)tuplergetpoprrrr)variant pretrainedkwargsZdefault_out_indicesrrr*r*r+ _create_davit`s  rc Ks|dddddttddd |S) Nr)rr)r9r9gffffff?Zbicubicz stem.convzhead.fc) urlrZ input_sizeZ pool_sizeZcrop_pct interpolationmeanrZ first_conv classifierr)rrr*r*r+_cfgosrztimm/)Z hf_hub_id)zdavit_tiny.msft_in1kzdavit_small.msft_in1kzdavit_base.msft_in1k davit_large davit_huge davit_giant)returncKs(tfdddd|}tdd|i|S)NrrrrrrQ davit_tinyr)rrrrrZ model_kwargsr*r*r+rs rcKs(tfdddd|}tdd|i|S)Nrr rrrr davit_smallr)rrrr*r*r+rs rcKs(tfdddd|}tdd|i|S)Nr))r8rM r davit_baser)rrrr*r*r+rs rcKs(tfdddd|}tdd|i|S)Nr)rrr)rrr0rrr)rrrr*r*r+rs rcKs(tfdddd|}tdd|i|S)Nr)rrri)rMrr@rrr)rrrr*r*r+rs rcKs(tfdddd|}tdd|i|S)N)rrrr)rrri )rrrr7rrr)rrrr*r*r+rs r)F)r)F)F)F)F)F)F):rJ functoolsrtypingrrZtorch.nnr"Ztorch.nn.functionalZ functionalrCrZ timm.datarrZ timm.layersrr r r r r rrrZ_builderrZ _features_fxrZ _manipulater _registryrr__all__Modulerr6rKrLrbr3r|r}r~rrrrrrZ default_cfgsrrrrrrr*r*r*r+sp    $    $7,bN