a  dԈ7@sdZddlmZddlmZddlZddlmZddlmm Z ddlm Z ddl m Z ddlmZmZddlmZmZmZmZmZmZmZmZd d lmZd d lmZd d lmZm Z d gZ!Gdddej"Z#Gdddej"Z$Gdddej"Z%Gdddej"Z&Gdddej"Z'Gdddej"Z(GdddeZ)GdddeZ*GdddejZ+Gd d!d!ej"Z,Gd"d#d#ej"Z-Gd$d%d%ej"Z.Gd&d'd'ej"Z/Gd(d)d)ej"Z0Gd*d d ej"Z1d+d,Z2ded.d/Z3dfd1d2Z4ee4d3d4d5e4d3d4d5e4d3d4d5e4d3d6d5e4d3d6d5e4d3d7e4d3d7e4d3d7e4d3d7e4d3d7e4d3d8d9e4d3d8d:d;d<e4d3d8d9e4d3d8d:d;d<e4d3d8d=d>e4d3d8d9e4d3d8d:d;d<e4d3d8d9e4d3d8d:d;d<e4d3d8d=d>e4d3d8d9e4d3d8d:d;d<e4d3d8d9e4d3d8d:d;d<e4d3d8d=d>e4d3d8d9e4d3d8d:d;d<e4d3d8d9e4d3d8d:d;d<e4d3d8d=d>e4d3d8d9e4d3d8d:d;d<e4d3d8d9e4d3d8d:d;d<e4d3d8d=d>e4d3d8d9e4d3d8d:d;d<e4d3d8d9e4d3d8d:d;d<e4d3d8d=d>e4d3d8d9e4d3d8d:d;d<e4d3d8d9e4d3d8d:d;d<e4d3d8d=d>e4d3d8d9e4d3d8d:d;d<e4d3d8d9e4d3d8d:d;d<e4d3d8d=d>d?2Z5e dge1d@dAdBZ6e dhe1d@dCdDZ7e die1d@dEdFZ8e dje1d@dGdHZ9e dke1d@dIdJZ:e dle1d@dKdLZ;e dme1d@dMdNZe dpe1d@dSdTZ?e dqe1d@dUdVZ@e dre1d@dWdXZAe dse1d@dYdZZBe dte1d@d[d\ZCe due1d@d]d^ZDe dve1d@d_d`ZEe dwe1d@dadbZFe dxe1d@dcddZGdS)ya Poolformer from MetaFormer is Actually What You Need for Vision https://arxiv.org/abs/2111.11418 IdentityFormer, RandFormer, PoolFormerV2, ConvFormer, and CAFormer from MetaFormer Baselines for Vision https://arxiv.org/abs/2210.13452 All implemented models support feature extraction and variable input resolution. Original implementation by Weihao Yu et al., adapted for timm by Fredo Guan and Ross Wightman. Adapted from https://github.com/sail-sg/metaformer, original copyright below ) OrderedDict)partialN)Tensor)FinalIMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD) trunc_normal_DropPathSelectAdaptivePool2d GroupNorm1 LayerNorm LayerNorm2dMlpuse_fused_attn)build_model_with_cfg)checkpoint_seq)generate_default_cfgsregister_model MetaFormercs*eZdZdZdfdd ZddZZS)Stemzc Stem implemented by a layer of convolution. Conv2d params constant across all models. Ncs:ttj||dddd|_|r,||nt|_dS)N kernel_sizestridepadding)super__init__nnConv2dconvIdentitynorm)self in_channels out_channels norm_layer __class___/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/timm/models/metaformer.pyr 7s z Stem.__init__cCs||}||}|SN)r#r%r&xr,r,r-forwardGs  z Stem.forward)N__name__ __module__ __qualname____doc__r r1 __classcell__r,r,r*r-r1s rcs*eZdZdZd fdd ZddZZS) Downsamplingz= Downsampling implemented by a layer of convolution. rrNcs:t|r||nt|_tj|||||d|_dS)Nr)rr r!r$r%r"r#)r&r'r(rrrr)r*r,r-r Rs zDownsampling.__init__cCs||}||}|Sr.)r%r#r/r,r,r-r1es  zDownsampling.forward)rrNr2r,r,r*r-r8Ms  r8cs*eZdZdZdfdd ZddZZS) Scalez2 Scale vector by element multiplications. ?Tcs>t|r|ddfn|f|_tj|t||d|_dS)NrZ requires_grad)rr shaper! Parametertorchonesscale)r&dim init_valueZ trainableuse_nchwr*r,r-r ps zScale.__init__cCs||j|jSr.)r@viewr<r/r,r,r-r1usz Scale.forward)r:TTr2r,r,r*r-r9ksr9cs*eZdZdZdfdd ZddZZS) SquaredReLUz< Squared ReLU: https://arxiv.org/abs/2109.08668 Fcsttj|d|_dS)Ninplace)rr r!ReLUrelu)r&rGr*r,r-r ~s zSquaredReLU.__init__cCst||Sr.)r>ZsquarerIr/r,r,r-r1szSquaredReLU.forward)Fr2r,r,r*r-rEysrEcs*eZdZdZd fdd Zd d ZZS) StarReLUz( StarReLU: s * relu(x) ** 2 + b r:TNFcsVt||_tj|d|_tj|td|d|_ tj|td|d|_ dS)NrFrr;) rr rGr!rHrIr=r>r?r@bias)r&Z scale_valueZ bias_valueZscale_learnableZbias_learnablemoderGr*r,r-r s zStarReLU.__init__cCs|j||d|jS)Nr)r@rIrLr/r,r,r-r1szStarReLU.forward)r:rKTTNFr2r,r,r*r-rJsrJcs8eZdZUdZeeed<d fdd Zd d ZZ S) Attentionzl Vanilla self-attention from Transformer: https://arxiv.org/abs/1706.03762. Modified from timm. fused_attn NFrKc  st||_|d|_t|_|r*|n|||_|jdkrDd|_|j|j|_tj ||jd|d|_ t ||_ tj |j||d|_ t ||_dS)NgrrrL)rr head_dimr@rrO num_headsZ attention_dimr!LinearqkvDropout attn_dropproj proj_drop) r&rArSrTZqkv_biasrXrZZ proj_biaskwargsr*r,r-r s    zAttention.__init__c Cs|j\}}}||||d|j|jddddd}|d\}}}|jrdtj ||||j j d}n4|| dd|j } | jdd } | | } | |}| dd|||}||}||}|S) NrQrrrr)Z dropout_p)rA)r<rVreshaperTrSZpermuteZunbindrOFZscaled_dot_product_attentionrXp transposer@ZsoftmaxrYrZ) r&r0BNCrVqkvZattnr,r,r-r1s  *    zAttention.forward)rPNFrKrKF) r3r4r5r6rbool__annotations__r r1r7r,r,r*r-rNs  rNcseZdZfddZZS)GroupNorm1NoBiasc s,tj|fi||dd|_d|_dSNepsư>rr getrlrLr&Z num_channelsr[r*r,r-r szGroupNorm1NoBias.__init__r3r4r5r r7r,r,r*r-rjsrjcseZdZfddZZS)LayerNorm2dNoBiasc s,tj|fi||dd|_d|_dSrkrnrpr*r,r-r szLayerNorm2dNoBias.__init__rqr,r,r*r-rrsrrcseZdZfddZZS)LayerNormNoBiasc s,tj|fi||dd|_d|_dSrkrnrpr*r,r-r szLayerNormNoBias.__init__rqr,r,r*r-rssrscs8eZdZdZdeejdddffdd Zdd ZZ S) SepConvz\ Inverted separable convolution from MobileNetV2: https://arxiv.org/abs/1801.04381. rFrrQc  sjtt||} tj|| d|d|_||_tj| | ||| |d|_||_tj| |d|d|_ dS)Nr)rrL)rrgroupsrL) rr intr!r"pwconv1act1dwconvact2pwconv2) r&rAZexpansion_ratioZ act1_layerZ act2_layerrLrrr[Z mid_channelsr*r,r-r s  zSepConv.__init__cCs6||}||}||}||}||}|Sr.)rwrxryrzr{r/r,r,r-r1 s      zSepConv.forward) r3r4r5r6rJr!r$r r1r7r,r,r*r-rtsrtcs*eZdZdZdfdd ZddZZS)PoolingzT Implementation of pooling for PoolFormer: https://arxiv.org/abs/2111.11418 rQc s&ttj|d|ddd|_dS)NrrF)rrZcount_include_pad)rr r!Z AvgPool2dpool)r& pool_sizer[r*r,r-r s  zPooling.__init__cCs||}||Sr.)r})r&r0yr,r,r-r1s zPooling.forward)rQr2r,r,r*r-r|sr|cs6eZdZdZddeeddffdd Zdd ZZS) MlpHeadz MLP classification head rrKTc s\tt||}tj|||d|_||_|||_tj|||d|_t ||_ dS)NrR) rr rvr!rUfc1actr%fc2rW head_drop) r&rA num_classesZ mlp_ratio act_layerr) drop_raterLZhidden_featuresr*r,r-r %s   zMlpHead.__init__cCs6||}||}||}||}||}|Sr.)rrr%rrr/r,r,r-r17s      zMlpHead.forward) r3r4r5r6rEr r r1r7r,r,r*r-r!src s<eZdZdZeededddddf fdd Zdd ZZ S) MetaFormerBlockz1 Implementation of one MetaFormer block. FrKTNc  s ttt|| |d} tt|| |d} |||_|f||d| |_|dkrZt|nt|_ | durr| nt|_ | dur| nt|_ |||_ t |td|||||d|_|dkrt|nt|_| dur| nt|_| dur| nt|_dS)N)rArBrC)rArZrKr)rrLdropZuse_conv)rr rr9norm1 token_mixerr r!r$ drop_path1 layer_scale1 res_scale1norm2rrvmlp drop_path2 layer_scale2 res_scale2)r&rArmlp_actmlp_biasr)rZ drop_pathrClayer_scale_init_valueres_scale_init_valuer[Zls_layerZrs_layerr*r,r-r Es(    zMetaFormerBlock.__init__c CsP|||||||}|||||| |}|Sr.) rrrrrrrrrrr/r,r,r-r1jszMetaFormerBlock.forward) r3r4r5r6r|rJrr r1r7r,r,r*r-r@s%rc sZeZdZdejedeeddgdddf fdd Zej j d dd Z e d d d Z ZS)MetaFormerStagerFrKNc  sztd _t t _|kr.tnt|ddd|d _ tj  f ddt |D _ dS)NFrQrr)rrrr)cs6g|].}tf | jd qS)) rArrrr)rZrrrrC)rrC).0i dp_ratesr[rrrr)out_chsrZrr&rr,r- s  z,MetaFormerStage.__init__..) rr grad_checkpointing issubclassrNrCr!r$r8 downsample Sequentialrangeblocks)r&Zin_chsrdepthrrrdownsample_normr)rZrrrr[r*rr-r |s  " zMetaFormerStage.__init__TcCs ||_dSr.)r)r&enabler,r,r-set_grad_checkpointingsz&MetaFormerStage.set_grad_checkpointingr0cCs~||}|j\}}}}|js4|||ddd}|jrRtjsRt |j |}n | |}|jsz|dd||||}|S)Nr]rr) rr<rCr^rarr>jit is_scriptingrr)r&r0rbrdHWr,r,r-r1s  zMetaFormerStage.forward)T)r3r4r5r!r$rJrr r>rignorerrr1r7r,r,r*r-rzs- rcseZdZdZdddddeeddddd d eeed ffd d ZddZ e j j dddZ e j j ddZd ddZd!eedddZedddZedddZZS)"raM MetaFormer A PyTorch impl of : `MetaFormer Baselines for Vision` - https://arxiv.org/abs/2210.13452 Args: in_chans (int): Number of input image channels. num_classes (int): Number of classes for classification head. global_pool: Pooling for classifier head. depths (list or tuple): Number of blocks at each stage. dims (list or tuple): Feature dimension at each stage. token_mixers (list, tuple or token_fcn): Token mixer for each stage. mlp_act: Activation layer for MLP. mlp_bias (boolean): Enable or disable mlp bias term. drop_path_rate (float): Stochastic depth rate. drop_rate (float): Dropout rate. layer_scale_init_values (list, tuple, float or None): Init value for Layer Scale. None means not use the layer scale. Form: https://arxiv.org/abs/2103.17239. res_scale_init_values (list, tuple, float or None): Init value for res Scale on residual connections. None means not use the res scale. From: https://arxiv.org/abs/2110.09456. downsample_norm (nn.Module): Norm layer used in stem and downsampling layers. norm_layers (list, tuple or norm_fcn): Norm layers for each stage. output_norm: Norm layer before classifier head. use_mlp_head: Use MLP classification head. rQravgrrr@i@FrKN)NNr:r:Tc sTt||_|d|_| |_||_t||_t|t t fsD|g}t|t t fsX|g}t|t t fsr|g|j}t|t t fs|g|j}t| t t fs| g|j} t| t t fs| g|j} d|_ g|_ t ||d|d|_g}|d}ddtd| t||D}t|jD]|}|t|||f||||||| ||| || ||||d |g7}||}|j t||dd |d g7_ qtj||_|dkr|jrt|j||jd }nt|j|}nt}ttd t|d fd||jfd|rtdntfd|jr,t| ntfd|fg|_ |!|j"dS)Nr]Fr)r)cSsg|] }|qSr,)tolist)rr0r,r,r-rz'MetaFormer.__init__..) rrrrrZrrrrr)rzstages.)Znum_chsZ reductionmoduler global_poolZ pool_typer%flattenrrfc)#rr r num_featuresr use_mlp_headlenZ num_stages isinstancelisttuplerZ feature_inforstemr>Zlinspacesumsplitrrdictr!rstagesrrUr$rr FlattenrWheadapply _init_weights)r&Zin_chansrrdepthsdims token_mixersrrZdrop_path_rateZproj_drop_raterlayer_scale_init_valuesres_scale_init_valuesr norm_layersZ output_normrr[rZprev_dimrrfinalr*r,r-r s|       "  (     zMetaFormer.__init__cCs>t|tjtjfr:t|jdd|jdur:tj|jddS)Ng{Gz?)stdr) rr!r"rUr ZweightrLinitZ constant_)r&mr,r,r-r7s zMetaFormer._init_weightscCs"||_|jD]}|j|dq dS)N)r)rrr)r&rZstager,r,r-r=s z!MetaFormer.set_grad_checkpointingcCs|jjSr.)rr)r&r,r,r-get_classifierCszMetaFormer.get_classifierrcCsv|dur0t|d|j_|r$tdnt|j_|dkrb|jrRt|j ||j d}qjt |j |}nt}||j_ dS)Nrrrr) r rrr!rr$rrrrrrUr)r&rrrr,r,r-reset_classifierGszMetaFormer.reset_classifier)r0 pre_logitscCsD|j|}|j|}|j|}|j|}|r8|S|j|Sr.)rrr%rrr)r&r0rr,r,r- forward_headTs     zMetaFormer.forward_headrcCs6||}|jr(tjs(t|j|}n ||}|Sr.)rrr>rrrrr/r,r,r-forward_features\s   zMetaFormer.forward_featurescCs||}||}|Sr.)rrr/r,r,r-r1ds  zMetaFormer.forward)T)rN)F)r3r4r5r6r|rJrrrr rr>rrrrrrrhrrr1r7r,r,r*r-rs8^   cCsdd|vr |Sddl}i}d|v}|}|D],\}}|r|dd|}|dd}|dd }|d d }|d d}|d d }|dd}|dd}|dd|}|dd}|dd}|dd|}|dd}|dd}|dd}|dd}|dd |}|d!d"|}|j||krV|||krV|||j}|||<q0|S)#Nzstem.conv.weightrznetwork.0.0.mlp.fc1.weightzlayer_scale_([0-9]+)zlayer_scale\1.scalez network.1zdownsample_layers.1z network.3zdownsample_layers.2z network.5zdownsample_layers.3z network.2z network.4z network.6networkrzdownsample_layers.([0-9]+)zstages.\1.downsamplezdownsample.projzdownsample.convzpatch_embed.projzpatch_embed.convz([0-9]+).([0-9]+)z \1.blocks.\2zstages.0.downsampleZ patch_embedrZ post_normr%Zpre_normz^headhead.fcz^normz head.norm)re state_dictitemssubreplacer<Znumelr^)rmodelrZout_dictZis_poolformerv1Zmodel_state_dictrfrgr,r,r-checkpoint_filter_fnks<             & rFcKsPtddt|ddD}|d|}tt||fttd|dd|}|S) Ncss|]\}}|VqdSr.r,)rr_r,r,r- rz%_create_metaformer..rr out_indicesT)Zflatten_sequentialr)Zpretrained_filter_fnZ feature_cfg)r enumerateropoprrrr)variant pretrainedr[Zdefault_out_indicesrrr,r,r-_create_metaformers   rc Ks|dddddttddd |S) Nr)rQr)rrr:Zbicubicrz stem.conv) urlr input_sizer~crop_pct interpolationmeanr classifierZ first_convr)rr[r,r,r-_cfgsrztimm/g?) hf_hub_idrgffffff?)rz head.fc.fc2)rr)rQr) r)rrrr~iQU)rrr)2zpoolformer_s12.sail_in1kzpoolformer_s24.sail_in1kzpoolformer_s36.sail_in1kzpoolformer_m36.sail_in1kzpoolformer_m48.sail_in1kzpoolformerv2_s12.sail_in1kzpoolformerv2_s24.sail_in1kzpoolformerv2_s36.sail_in1kzpoolformerv2_m36.sail_in1kzpoolformerv2_m48.sail_in1kzconvformer_s18.sail_in1kzconvformer_s18.sail_in1k_384z!convformer_s18.sail_in22k_ft_in1kz%convformer_s18.sail_in22k_ft_in1k_384zconvformer_s18.sail_in22kzconvformer_s36.sail_in1kzconvformer_s36.sail_in1k_384z!convformer_s36.sail_in22k_ft_in1kz%convformer_s36.sail_in22k_ft_in1k_384zconvformer_s36.sail_in22kzconvformer_m36.sail_in1kzconvformer_m36.sail_in1k_384z!convformer_m36.sail_in22k_ft_in1kz%convformer_m36.sail_in22k_ft_in1k_384zconvformer_m36.sail_in22kzconvformer_b36.sail_in1kzconvformer_b36.sail_in1k_384z!convformer_b36.sail_in22k_ft_in1kz%convformer_b36.sail_in22k_ft_in1k_384zconvformer_b36.sail_in22kzcaformer_s18.sail_in1kzcaformer_s18.sail_in1k_384zcaformer_s18.sail_in22k_ft_in1kz#caformer_s18.sail_in22k_ft_in1k_384zcaformer_s18.sail_in22kzcaformer_s36.sail_in1kzcaformer_s36.sail_in1k_384zcaformer_s36.sail_in22k_ft_in1kz#caformer_s36.sail_in22k_ft_in1k_384zcaformer_s36.sail_in22kzcaformer_m36.sail_in1kzcaformer_m36.sail_in1k_384zcaformer_m36.sail_in22k_ft_in1kz#caformer_m36.sail_in22k_ft_in1k_384zcaformer_m36.sail_in22kzcaformer_b36.sail_in1kzcaformer_b36.sail_in1k_384zcaformer_b36.sail_in22k_ft_in1kz#caformer_b36.sail_in22k_ft_in1k_384zcaformer_b36.sail_in22k)returnc Ks>tfgdgddtjdtdddd |}td d|i|S) NrrTh㈵>F rrrrrrrrrpoolformer_s12r)rrr!ZGELUr rrr[Z model_kwargsr,r,r-rDs  rc Ks>tfgdgddtjdtdddd |}td d|i|S) NrrrrrTrFrpoolformer_s24r)rrrr,r,r-rTs  rc Ks>tfgdgddtjdtdddd |}td d|i|S) NrrrrTrmFrpoolformer_s36r)rrrr,r,r-rds  rc Ks>tfgdgddtjdtdddd |}td d|i|S) Nr`rTrmFrpoolformer_m36r)r rrr,r,r-r ts  r c Ks>tfgdgddtjdtdddd |}td d|i|S) Nr r rTrmFrpoolformer_m48r)rrrr,r,r-rs  rcKs2tfgdgdtdd|}tdd|i|S)NrrFrrrrpoolformerv2_s12r)rrrjrrr,r,r-rsrcKs2tfgdgdtdd|}tdd|i|S)NrrFrpoolformerv2_s24r)rrrr,r,r-rsrcKs2tfgdgdtdd|}tdd|i|S)NrrFrpoolformerv2_s36r)rrrr,r,r-rsrcKs2tfgdgdtdd|}tdd|i|S)NrrFrpoolformerv2_m36r)rrrr,r,r-rsrcKs2tfgdgdtdd|}tdd|i|S)Nr rFrpoolformerv2_m48r)rrrr,r,r-rsrcKs2tfgdgdttd|}tdd|i|S)NrQrQ rQrrrrrconvformer_s18r)rrrtrrrrr,r,r-rsrcKs2tfgdgdttd|}tdd|i|S)NrQrrrQrrconvformer_s36r)rrrr,r,r-rsrcKs2tfgdgdttd|}tdd|i|S)Nrrrri@rconvformer_m36r)rrrr,r,r-rsrcKs2tfgdgdttd|}tdd|i|S)Nrrrr rconvformer_b36r)r!rrr,r,r-r!sr!cKsJtfgdgdttttgtgdtgdd|}tdd|i|S)Nrrrr caformer_s18r)r"rrtrNrrrsrrr,r,r-r"s r"cKsJtfgdgdttttgtgdtgdd|}tdd|i|S)Nrrrr caformer_s36r)r$r#rr,r,r-r$s r$cKsJtfgdgdttttgtgdtgdd|}tdd|i|S)Nrrrr caformer_m36r)r%r#rr,r,r-r% s r%cKsJtfgdgdttttgtgdtgdd|}tdd|i|S)Nrrrr caformer_b36r)r&r#rr,r,r-r&s r&)F)r)F)F)F)F)F)F)F)F)F)F)F)F)F)F)F)F)F)F)Hr6 collectionsr functoolsrr>Ztorch.nnr!Ztorch.nn.functionalZ functionalr_rZ torch.jitrZ timm.datarrZ timm.layersr r r r r rrrZ_builderrZ _manipulater _registryrr__all__Modulerr8r9rErJrNrjrrrsrtr|rrrrrrrZ default_cfgsrrrr rrrrrrrrrr!r"r$r%r&r,r,r,r-s     (   <#:E-%