a  d}(@stdZddlmZddlmZmZddlmZddlm Z m Z m Z ddl Z ddl mZddlmZmZddlmZmZmZmZmZmZmZmZmZd d lmZd d lmZd d l m!Z!d d l"m#Z#m$Z$ddgZ%eGdddZ&Gdddej'Z(de)dddZ*Gdddej'Z+eGdddej'Z,de-e-e.e e e e e/dddZ0e1dd d!d"d#d$d%d&d'd(d)d*d+d,d-d.Z2Gd/ddej'Z3dd2d3Z4dd5d6Z5dd=d>Z6dd?d@Z7e1e7dAdBe7dCdBe7dDdBe7dEdBe7dFdBe7dGdBe7dHdBe6dAdBe6dCdBe6dDdBe6dEdBe6dFdBe6dGdBe6dHdBe6dIdBe6dAdJdKdLe1dLdMdNdOdPe6dAdJdKdLdQe1dOdRe6dCdSdKdLdQe1dOdRe6dDdSdKdLdQe1dOdRe6dEdSdKdLdQe1dOdRe5dTdBe5dUdBe5dVdWdXe5dYdZdXe5d[d\dXe5d]d^dXe4d_dBe4d`dBe4dadBe4d_de$de3dddZ?e$de3dddZ@e$de3dddZAe$de3dddZBe$de3dddZCe$de3dddZDe$de3dddZEe$de3dddZFe$de3dddZGe$de3dddZHe$de3ddd„ZIe$de3dddĄZJe$de3dddƄZKe$de3dddȄZLe$de3dddʄZMe$de3ddd̄ZNe$de3ddd΄ZOe$d e3dddЄZPe$d e3ddd҄ZQe$d e3dddԄZRe$d e3dddքZSe$d e3ddd؄ZTe$de3dddڄZUe$de3ddd܄ZVe$de3dddބZWe$de3dddZXe$de3dddZYe$de3dddZZe$de3dddZ[e$de3dddZ\e$de3dddZ]e$de3dddZ^dS(a Normalization Free Nets. NFNet, NF-RegNet, NF-ResNet (pre-activation) Models Paper: `Characterizing signal propagation to close the performance gap in unnormalized ResNets` - https://arxiv.org/abs/2101.08692 Paper: `High-Performance Large-Scale Image Recognition Without Normalization` - https://arxiv.org/abs/2102.06171 Official Deepmind JAX code: https://github.com/deepmind/deepmind-research/tree/master/nfnets Status: * These models are a work in progress, experiments ongoing. * Pretrained weights for two models so far, more to come. * Model details updated to closer match official JAX code now that it's released * NF-ResNet, NF-RegNet-B, and NFNet-F models supported Hacked together by / copyright Ross Wightman, 2021. ) OrderedDict) dataclassreplace)partial)CallableTupleOptionalNIMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD) ClassifierHeadDropPath AvgPool2dSameScaledStdConv2dScaledStdConv2dSame get_act_layer get_act_fnget_attnmake_divisible)build_model_with_cfg)register_notrace_module)checkpoint_seq)generate_default_cfgsregister_model NormFreeNetNfCfgc@s&eZdZUeeeeefed<eeeeefed<dZeed<dZe ed<dZ e eed<dZ e eed <dZ e e ed <dZeed <d Zeed <dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZe ed <dS)!rdepthschannelsg?alpha3x3 stem_typeNstem_chs group_size attn_layer attn_kwargs@ attn_gain? width_factor? bottle_ratior num_featuresch_divFreg extra_conv gamma_in_act same_paddinggh㈵> std_conv_epsskipinit zero_init_fcsilu act_layer)__name__ __module__ __qualname__rint__annotations__rfloatr!strr"rr#r$r%dictr'r)r+r,r.r/boolr0r1r2r3r4r5r7rArAZ/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/timm/models/nfnet.pyr&s*                cs,eZdZd edfdd ZddZZS) GammaActrelur(Fgammacs$tt||_||_||_dSN)super__init__ract_fnrFinplace)selfact_typerFrK __class__rArBrI@s  zGammaAct.__init__cCs|j||jd|jS)NrK)rJrKmul_rFrLxrArArBforwardFszGammaAct.forward)rDr(F)r8r9r:r=rIrT __classcell__rArArNrBrC?srCr(rEcsdfdd }|S)NFcst|dS)N)rFrK)rCrPrMrFrArB_createKszact_with_gamma.._create)FrA)rMrFrWrArVrBact_with_gammaJsrXcsBeZdZdddefeeeeeeedfdd ZddZZ S) DownsampleAvgrN)in_chsout_chsstridedilationfirst_dilation conv_layerc s|tt||dkr|nd}|dks.|dkr\|dkrB|dkrBtntj}|d|ddd|_n t|_|||ddd|_dS)zF AvgPool Downsampling as in 'D' ResNet variants. Support for dilation.rTF)Z ceil_modeZcount_include_pad)r\N) rHrYrIrnnZ AvgPool2dpoolIdentityconv) rLrZr[r\r]r^r_Z avg_strideZ avg_pool_fnrNrArBrIQs  zDownsampleAvg.__init__cCs|||SrG)rdrbrRrArArBrTdszDownsampleAvg.forward) r8r9r:rr;rrrIrTrUrArArNrBrYPsrYcseZdZdZddddddddddddddded feeeeeeeeeeeeeeeeee eee e ed fd d Z d dZ Z S) NormFreeBlockz-Normalization-Free pre-activation block. Nrr(?TFr&)rZr[r\r]r^rbetar+r#r.r/r0r4r$r'r7r_drop_path_ratecst|p|}|p|}t| r(||n||| }| srHt,j9j?|j;ddd|j=durHt,j9:|j=qHdS)a Args: cfg: Model architecture configuration. num_classes: Number of classifier classes. in_chans: Number of input channels. global_pool: Global pooling type. output_stride: Output stride of network, one of (8, 16, 32). drop_rate: Dropout rate. drop_path_rate: Stochastic depth drop-path rate. **kwargs: Extra kwargs overlayed onto cfg. Fz3Please add non-linearity constants for activation (z).rE)eps)rFrNr)r_r7cSsg|] }|qSrA)tolist).0rSrArArB Tz(NormFreeNet.__init__..rr(r`)rr`r*)rZr[rrhr\r]r^r#r+r.r/r0r4r$r'r7r_rizstages.r final_convrP)Z pool_typerfcrgg{Gz?Zfan_inZlinear)modeZ nonlinearity)@rHrIrrgrad_checkpointingrr7 _nonlin_gammar2rrr1rXrr3rr$rr%rr"rr)r.rr!rZ feature_inforyZlinspacesumrsplitrrangererr#r/r+r0r4r'r?rarstagesr,rrc final_actr headZ named_modules isinstanceZLinearr5initZzeros_ZweightZnormal_ZbiasZConv2dZkaiming_normal_)rLrrrrrrrikwargsr_r7r$r"rZ stem_featZdrop_path_ratesZprev_chsZ net_strider]Z expected_varrZ stage_idxZ stage_depthr\r^blocksZ block_idxZ first_blockr[nmrNrArBrI$s  "&        zNormFreeNet.__init__FcCs td|r dnddfdgd}|S)Nz^stemz^stages\.(\d+)z^stages\.(\d+)\.(\d+))z ^final_conv)i)rr)r?)rLZcoarseZmatcherrArArB group_matcherszNormFreeNet.group_matcherTcCs ||_dSrG)r)rLenablerArArBset_grad_checkpointingsz"NormFreeNet.set_grad_checkpointingcCs|jjSrG)rr)rLrArArBget_classifierszNormFreeNet.get_classifiercCs|j||dSrG)rreset)rLrrrArArBreset_classifierszNormFreeNet.reset_classifiercCsJ||}|jr(tjs(t|j|}n ||}||}||}|SrG) rrryjitZ is_scriptingrrrrrRrArArBforward_featuress    zNormFreeNet.forward_features pre_logitscCs|r|j||dS||S)Nr)r)rLrSrrArArB forward_headszNormFreeNet.forward_headcCs||}||}|SrG)rrrRrArArBrTs  zNormFreeNet.forward)rrjrrrgrg)F)T)r)F)r8r9r:r|rr;r>r=rIryrignorerrrrrr@rrTrUrArArNrBrs4|     iirDc Cs&|pi}t||ddd||||d }|S)Nr@rf) rrr!r"r+r#r7r$r%)r)rrr#r7r$r%rrArArB _nfres_cfgs r0hc Cs:d|dd}tdd}t||dddd |d d |d }|S) Nirrr*rd_ratior r-g?g@Tse) rrr!r#r)r+r,r/r$r%)r?r)rrr,r%rrArArB _nfreg_cfgs  rrrrr*r&rrc CsHt|d|}|dur|ntdd}t||dd||d||||d } | S)Nrr*rrrT) rrr!r"r#r+r0r,r7r$r%)r;r?r) rrr#r+ feat_multr7r$r%r,rrArArB _nfnet_cfgs  rcCs:t||ddddddd|t|dd|dtddd }|S) Nrrr*Trr&rr)rrr!r"r#r+r0r1r2r4r,r7r$r%)rr;r?)rrr7r4rrArArB _dm_nfnet_cfgs"r)rr`rj)r)r`r r)rjr )rr-r) )rr$r)r*)r-rrg?rrfr-)rZ rd_divisorr6)rrr#r+r%r7Zeca)rrr#r+r$r%r7r`)rrjrr)r`rrr)r`rr-r-)8pi)rr)r`rrr)rri)r`r r)r ih)rjrrr)PiPi)r`r`r`r`)rjrrrj)rjrrjg?r)rr$r%)# dm_nfnet_f0 dm_nfnet_f1 dm_nfnet_f2 dm_nfnet_f3 dm_nfnet_f4 dm_nfnet_f5 dm_nfnet_f6nfnet_f0nfnet_f1nfnet_f2nfnet_f3nfnet_f4nfnet_f5nfnet_f6nfnet_f7nfnet_l0 eca_nfnet_l0 eca_nfnet_l1 eca_nfnet_l2 eca_nfnet_l3 nf_regnet_b0 nf_regnet_b1 nf_regnet_b2 nf_regnet_b3 nf_regnet_b4 nf_regnet_b5 nf_resnet26 nf_resnet50 nf_resnet101 nf_seresnet26 nf_seresnet50nf_seresnet101nf_ecaresnet26nf_ecaresnet50nf_ecaresnet101FcKs,t|}tdd}tt||f||d|S)NT)Zflatten_sequential) model_cfg feature_cfg) model_cfgsr?rr)variant pretrainedrrrrArArB_create_normfreenetds rc Ks|dddddttddd |S) Nrrjrrr?Zbicubicz stem.conv1zhead.fc) urlr input_size pool_sizecrop_pct interpolationmeanZstd first_conv classifierr )rrrArArB_dcfgqsr#ztimm/zmhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-dnf-weights/dm_nfnet_f0-604f9c3a.pth)rr)rjr$)rjrrrZsquash) hf_hub_idrrrtest_input_sizerZ crop_modezmhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-dnf-weights/dm_nfnet_f1-fc540f82.pthrr)rj@r'gQ?zmhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-dnf-weights/dm_nfnet_f2-89875923.pth)r-r-)rj`r(gq= ףp?zmhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-dnf-weights/dm_nfnet_f3-d74ab3aa.pth)rr)rjr)gGz?zmhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-dnf-weights/dm_nfnet_f4-0ac5b10b.pth)rr)rjr*)rjrrg;On?zmhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-dnf-weights/dm_nfnet_f5-ecb20ab1.pth) r+)rj r,gI +?zmhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-dnf-weights/dm_nfnet_f6-e0f12116.pth)rr)rjr-)rj@r.gd;O?)rrrr&)rr)rjr/)rj`r0zjhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/nfnet_l0_ra2-45c6688d.pth)rjrr)r%rrrr& test_crop_pctzmhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/ecanfnet_l0_ra2-e3e9ac50.pthzmhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/ecanfnet_l1_ra2-7dce93cd.pthzmhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/ecanfnet_l2_ra3-da781a61.pth)rr)rrrr&r1r)rrrr&r!zrhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/nf_regnet_b1_256_ra2-ad85cfef.pth)r%rrrr&r!)rjr2)rjr3)rr)rjr4)rr!zmhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/nf_resnet50_ra2-9f236009.pth)r%rrrr&rr!)#zdm_nfnet_f0.dm_in1kzdm_nfnet_f1.dm_in1kzdm_nfnet_f2.dm_in1kzdm_nfnet_f3.dm_in1kzdm_nfnet_f4.dm_in1kzdm_nfnet_f5.dm_in1kzdm_nfnet_f6.dm_in1krrrrrrrrznfnet_l0.ra2_in1kzeca_nfnet_l0.ra2_in1kzeca_nfnet_l1.ra2_in1kzeca_nfnet_l2.ra3_in1krrznf_regnet_b1.ra2_in1krrrrrznf_resnet50.ra2_in1kr r r r rrr)returncKstdd|i|S)z NFNet-F0 (DeepMind weight compatible) `High-Performance Large-Scale Image Recognition Without Normalization` - https://arxiv.org/abs/2102.06171 rr)rrrrrArArBrsrcKstdd|i|S)z NFNet-F1 (DeepMind weight compatible) `High-Performance Large-Scale Image Recognition Without Normalization` - https://arxiv.org/abs/2102.06171 rr)rr6r7rArArBrsrcKstdd|i|S)z NFNet-F2 (DeepMind weight compatible) `High-Performance Large-Scale Image Recognition Without Normalization` - https://arxiv.org/abs/2102.06171 rr)rr6r7rArArBrsrcKstdd|i|S)z NFNet-F3 (DeepMind weight compatible) `High-Performance Large-Scale Image Recognition Without Normalization` - https://arxiv.org/abs/2102.06171 rr)rr6r7rArArBrsrcKstdd|i|S)z NFNet-F4 (DeepMind weight compatible) `High-Performance Large-Scale Image Recognition Without Normalization` - https://arxiv.org/abs/2102.06171 rr)rr6r7rArArBrsrcKstdd|i|S)z NFNet-F5 (DeepMind weight compatible) `High-Performance Large-Scale Image Recognition Without Normalization` - https://arxiv.org/abs/2102.06171 rr)rr6r7rArArBr srcKstdd|i|S)z NFNet-F6 (DeepMind weight compatible) `High-Performance Large-Scale Image Recognition Without Normalization` - https://arxiv.org/abs/2102.06171 rr)rr6r7rArArBrsrcKstdd|i|S)z NFNet-F0 `High-Performance Large-Scale Image Recognition Without Normalization` - https://arxiv.org/abs/2102.06171 rr)rr6r7rArArBrsrcKstdd|i|S)z NFNet-F1 `High-Performance Large-Scale Image Recognition Without Normalization` - https://arxiv.org/abs/2102.06171 rr)rr6r7rArArBr'srcKstdd|i|S)z NFNet-F2 `High-Performance Large-Scale Image Recognition Without Normalization` - https://arxiv.org/abs/2102.06171 rr)rr6r7rArArBr0srcKstdd|i|S)z NFNet-F3 `High-Performance Large-Scale Image Recognition Without Normalization` - https://arxiv.org/abs/2102.06171 rr)rr6r7rArArBr9srcKstdd|i|S)z NFNet-F4 `High-Performance Large-Scale Image Recognition Without Normalization` - https://arxiv.org/abs/2102.06171 rr)rr6r7rArArBrBsrcKstdd|i|S)z NFNet-F5 `High-Performance Large-Scale Image Recognition Without Normalization` - https://arxiv.org/abs/2102.06171 rr)rr6r7rArArBrKsrcKstdd|i|S)z NFNet-F6 `High-Performance Large-Scale Image Recognition Without Normalization` - https://arxiv.org/abs/2102.06171 rr)rr6r7rArArBrTsrcKstdd|i|S)z NFNet-F7 `High-Performance Large-Scale Image Recognition Without Normalization` - https://arxiv.org/abs/2102.06171 rr)rr6r7rArArBr]srcKstdd|i|S)z NFNet-L0b w/ SiLU My experimental 'light' model w/ F0 repeats, 1.5x final_conv mult, 64 group_size, .25 bottleneck & SE ratio rr)rr6r7rArArBrfsrcKstdd|i|S)z ECA-NFNet-L0 w/ SiLU My experimental 'light' model w/ F0 repeats, 1.5x final_conv mult, 64 group_size, .25 bottleneck & ECA attn rr)rr6r7rArArBrnsrcKstdd|i|S)z ECA-NFNet-L1 w/ SiLU My experimental 'light' model w/ F1 repeats, 2.0x final_conv mult, 64 group_size, .25 bottleneck & ECA attn rr)rr6r7rArArBrvsrcKstdd|i|S)z ECA-NFNet-L2 w/ SiLU My experimental 'light' model w/ F2 repeats, 2.0x final_conv mult, 64 group_size, .25 bottleneck & ECA attn rr)rr6r7rArArBr~srcKstdd|i|S)z ECA-NFNet-L3 w/ SiLU My experimental 'light' model w/ F3 repeats, 2.0x final_conv mult, 64 group_size, .25 bottleneck & ECA attn rr)rr6r7rArArBrsrcKstdd|i|S)z Normalization-Free RegNet-B0 `Characterizing signal propagation to close the performance gap in unnormalized ResNets` - https://arxiv.org/abs/2101.08692 rr)rr6r7rArArBrsrcKstdd|i|S)z Normalization-Free RegNet-B1 `Characterizing signal propagation to close the performance gap in unnormalized ResNets` - https://arxiv.org/abs/2101.08692 rr)rr6r7rArArBrsrcKstdd|i|S)z Normalization-Free RegNet-B2 `Characterizing signal propagation to close the performance gap in unnormalized ResNets` - https://arxiv.org/abs/2101.08692 rr)rr6r7rArArBrsrcKstdd|i|S)z Normalization-Free RegNet-B3 `Characterizing signal propagation to close the performance gap in unnormalized ResNets` - https://arxiv.org/abs/2101.08692 rr)rr6r7rArArBrsrcKstdd|i|S)z Normalization-Free RegNet-B4 `Characterizing signal propagation to close the performance gap in unnormalized ResNets` - https://arxiv.org/abs/2101.08692 rr)rr6r7rArArBrsrcKstdd|i|S)z Normalization-Free RegNet-B5 `Characterizing signal propagation to close the performance gap in unnormalized ResNets` - https://arxiv.org/abs/2101.08692 rr)rr6r7rArArBrsrcKstdd|i|S)z Normalization-Free ResNet-26 `Characterizing signal propagation to close the performance gap in unnormalized ResNets` - https://arxiv.org/abs/2101.08692 rr)rr6r7rArArBrsrcKstdd|i|S)z Normalization-Free ResNet-50 `Characterizing signal propagation to close the performance gap in unnormalized ResNets` - https://arxiv.org/abs/2101.08692 r r)r r6r7rArArBr sr cKstdd|i|S)z Normalization-Free ResNet-101 `Characterizing signal propagation to close the performance gap in unnormalized ResNets` - https://arxiv.org/abs/2101.08692 r r)r r6r7rArArBr sr cKstdd|i|S)z$ Normalization-Free SE-ResNet26 r r)r r6r7rArArBr sr cKstdd|i|S)z$ Normalization-Free SE-ResNet50 r r)r r6r7rArArBr sr cKstdd|i|S)z% Normalization-Free SE-ResNet101 r r)r r6r7rArArBr sr cKstdd|i|S)z% Normalization-Free ECA-ResNet26 rr)rr6r7rArArBrsrcKstdd|i|S)z% Normalization-Free ECA-ResNet50 rr)rr6r7rArArBrsrcKstdd|i|S)z& Normalization-Free ECA-ResNet101 rr)rr6r7rArArBrsr)r()r}NNT)rNrDNN)r)rrr*r&rrN)rrT)F)r})F)F)F)F)F)F)F)F)F)F)F)F)F)F)F)F)F)F)F)F)F)F)F)F)F)F)F)F)F)F)F)F)F)F)F)_r| collectionsrZ dataclassesrr functoolsrtypingrrrryZtorch.nnraZ timm.datar r Z timm.layersr r rrrrrrrZ_builderrZ _features_fxrZ _manipulater _registryrr__all__rModulerCr=rXrYrer;r>r@rr?rrrrrrrrr#Z default_cfgsrrrrrrrrrrrrrrrrrrrrrrrrrrrr r r r r rrrrArArArBsp   ,    e 0>         =                      c