a  dH@sFdZddlmZddlmZmZddlZddlmZddlm Z m Z ddl m Z ddl mZmZmZd d lmZd d lmZd d lmZmZmZd gZGdd d eZdaddZdbddZeedddedddedddedddddedddd edd!dd edd"dd edd#dddd$edd%dedd&dddedd'dedd(dedd)dddedd*dedd+dddedd,dedd-dd.edd/dddedd0dd.edd1dd.edd2dddedd3dd.edd4dddedd5dd.d6Zedced7d8d9Zedded7d:d;Zedeed7dd?Z!edged7d@dAZ"edhed7dBdCZ#edied7dDdEZ$edjed7dFdGZ%edked7dHdIZ&edled7dJdKZ'edmed7dLdMZ(edned7dNdOZ)edoed7dPdQZ*edped7dRdSZ+edqed7dTdUZ,edred7dVdWZ-ee.dXdYdZd[d\d]d^d_d`dS)sa[ DeiT - Data-efficient Image Transformers DeiT model defs and weights from https://github.com/facebookresearch/deit, original copyright below paper: `DeiT: Data-efficient Image Transformers` - https://arxiv.org/abs/2012.12877 paper: `DeiT III: Revenge of the ViT` - https://arxiv.org/abs/2204.07118 Modifications copyright 2021, Ross Wightman )partial)SequenceUnionN)nnIMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD)resample_abs_pos_embed)VisionTransformer trunc_normal_checkpoint_filter_fn)build_model_with_cfg)checkpoint_seq)generate_default_cfgsregister_modelregister_model_deprecationsVisionTransformerDistilledcseZdZdZfddZdfdd Zejjddd Z ejjd d Z dd dZ ejjdddZ d ej eeefdddZej dddZd!eej dddZZS)"rz Vision Transformer w/ Distillation Token and Head Distillation token & head support for `DeiT: Data-efficient Image Transformers` - https://arxiv.org/abs/2012.12877 cs|dd}tj|i|ddi|jdvs4Jd|_ttdd|j |_ ttd|j j |j|j |_ |jdkrt|j |jnt|_d|_||dS) N weight_initskip)tokenr rF)popsuper__init__ global_poolZnum_prefix_tokensr Parametertorchzeros embed_dim dist_token patch_embedZ num_patches pos_embed num_classesLinearIdentity head_distdistilled_training init_weights)selfargskwargsr __class__Y/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/timm/models/deit.pyr$s $z#VisionTransformerDistilled.__init__rcs t|jddtj|ddS)Ng{Gz?)std)mode)r r!rr))r*r2r-r/r0r)2sz'VisionTransformerDistilled.init_weightsFcCstdddgdS)Nz+^cls_token|pos_embed|patch_embed|dist_token)z^blocks\.(\d+)N)z^norm)i)stemblocks)dict)r*Zcoarser/r/r0 group_matcher6s z(VisionTransformerDistilled.group_matchercCs |j|jfSNheadr')r*r/r/r0get_classifier?sz)VisionTransformerDistilled.get_classifierNcCsL||_|dkrt|j|nt|_|dkr>t|j|jnt|_dS)Nr)r$rr%r r&r9r')r*r$rr/r/r0reset_classifierCs z+VisionTransformerDistilled.reset_classifierTcCs ||_dSr7)r()r*enabler/r/r0set_distilled_trainingHsz1VisionTransformerDistilled.set_distilled_trainingr )xncCsgt|j}}tt|tr*t|||n|}||}tj|j |j ddd|j |j ddd|fdd}| ||j}||}||}t|jD]"\}}||}||vr||q|SNrr )Zdim)lenr4set isinstanceintranger"rcat cls_tokenexpandshaper!pos_dropr#Z patch_dropZnorm_pre enumerateappend)r*r>r?outputsZ num_blocksZ take_indicesiZblkr/r/r0_intermediate_layersLs$     z/VisionTransformerDistilled._intermediate_layersreturncCs||}tj|j|jddd|j|jddd|fdd}|||j}|j rptj spt |j |}n | |}||}|Sr@)r"rrGrHrIrJr!rKr#Zgrad_checkpointingjit is_scriptingrr4Znorm)r*r>r/r/r0forward_featureses   z+VisionTransformerDistilled.forward_features) pre_logitsrRcCst|dddf|dddf}}|r2||dS||}||}|jrd|jrdtjsd||fS||dSdS)Nrr r)r9r'r(ZtrainingrrSrT)r*r>rVZx_distr/r/r0 forward_headts"   z'VisionTransformerDistilled.forward_head)r)F)N)T)r )F)__name__ __module__ __qualname____doc__rr)rrSignorer6r:r;r=ZTensorrrErrPrUboolrW __classcell__r/r/r-r0rs"      FcKsD|ddrtd|rtnt}t|||fdttddi|}|S)NZ features_onlyzhttps://dl.fbaipublicfiles.com/deit/deit_3_huge_224_21k_v1.pth)zdeit_tiny_patch16_224.fb_in1kzdeit_small_patch16_224.fb_in1kzdeit_base_patch16_224.fb_in1kzdeit_base_patch16_384.fb_in1kz'deit_tiny_distilled_patch16_224.fb_in1kz(deit_small_distilled_patch16_224.fb_in1kz'deit_base_distilled_patch16_224.fb_in1kz'deit_base_distilled_patch16_384.fb_in1kzdeit3_small_patch16_224.fb_in1kzdeit3_small_patch16_384.fb_in1kz deit3_medium_patch16_224.fb_in1kzdeit3_base_patch16_224.fb_in1kzdeit3_base_patch16_384.fb_in1kzdeit3_large_patch16_224.fb_in1kzdeit3_large_patch16_384.fb_in1kzdeit3_huge_patch14_224.fb_in1k(deit3_small_patch16_224.fb_in22k_ft_in1k(deit3_small_patch16_384.fb_in22k_ft_in1k)deit3_medium_patch16_224.fb_in22k_ft_in1k'deit3_base_patch16_224.fb_in22k_ft_in1k'deit3_base_patch16_384.fb_in22k_ft_in1k(deit3_large_patch16_224.fb_in22k_ft_in1k(deit3_large_patch16_384.fb_in22k_ft_in1k'deit3_huge_patch14_224.fb_in22k_ft_in1krQcKs2tddddd}tdd|it|fi|}|S) z DeiT-tiny model @ 224x224 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit.  rf patch_sizer depth num_headsdeit_tiny_patch16_224rb)rr5rerbr,Z model_argsrdr/r/r0rsrcKs2tddddd}tdd|it|fi|}|S) z DeiT-small model @ 224x224 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. ryrpr{r|deit_small_patch16_224rb)rrrr/r/r0rsrcKs2tddddd}tdd|it|fi|}|S)z DeiT base model @ 224x224 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. ryr{r|deit_base_patch16_224rb)rrrr/r/r0rsrcKs2tddddd}tdd|it|fi|}|S)z DeiT base model @ 384x384 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. ryrr{r|deit_base_patch16_384rb)rrrr/r/r0rsrcKs4tddddd}td |ddt|fi|}|S) z DeiT-tiny distilled model @ 224x224 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. ryrzr{rfr|deit_tiny_distilled_patch16_224Trbrc)rrrr/r/r0r$srcKs4tddddd}td |ddt|fi|}|S) z DeiT-small distilled model @ 224x224 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. ryrpr{rr| deit_small_distilled_patch16_224Tr)rrrr/r/r0r/srcKs4tddddd}td|ddt|fi|}|S) z DeiT-base distilled model @ 224x224 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. ryrr{r|deit_base_distilled_patch16_224Tr)rrrr/r/r0r:srcKs4tddddd}td|ddt|fi|}|S) z DeiT-base distilled model @ 384x384 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. ryrr{r|deit_base_distilled_patch16_384Tr)rrrr/r/r0rEsrcKs6tddddddd}td d |it|fi|}|S) z DeiT-3 small model @ 224x224 from paper (https://arxiv.org/abs/2204.07118). ImageNet-1k weights from https://github.com/facebookresearch/deit. ryrpr{rTư>r}r r~rZno_embed_classZ init_valuesdeit3_small_patch16_224rb)rrrr/r/r0rPsrcKs6tddddddd}td d |it|fi|}|S) z DeiT-3 small model @ 384x384 from paper (https://arxiv.org/abs/2204.07118). ImageNet-1k weights from https://github.com/facebookresearch/deit. ryrpr{rTrrdeit3_small_patch16_384rb)rrrr/r/r0rZsrcKs6tddddddd}td d |it|fi|}|S) z DeiT-3 medium model @ 224x224 (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. ryir{Trrdeit3_medium_patch16_224rb)rrrr/r/r0rdsrcKs6tddddddd}td d|it|fi|}|S) z DeiT-3 base model @ 224x224 from paper (https://arxiv.org/abs/2204.07118). ImageNet-1k weights from https://github.com/facebookresearch/deit. ryrr{Trrdeit3_base_patch16_224rb)rrrr/r/r0rnsrcKs6tddddddd}td d|it|fi|}|S) DeiT-3 base model @ 384x384 from paper (https://arxiv.org/abs/2204.07118). ImageNet-1k weights from https://github.com/facebookresearch/deit. ryrr{Trrdeit3_base_patch16_384rb)rrrr/r/r0rxsrcKs6tddddddd}td d|it|fi|}|S) z DeiT-3 large model @ 224x224 from paper (https://arxiv.org/abs/2204.07118). ImageNet-1k weights from https://github.com/facebookresearch/deit. ryTrrdeit3_large_patch16_224rb)rrrr/r/r0rsrcKs6tddddddd}td d|it|fi|}|S) z DeiT-3 large model @ 384x384 from paper (https://arxiv.org/abs/2204.07118). ImageNet-1k weights from https://github.com/facebookresearch/deit. ryrrTrrdeit3_large_patch16_384rb)rrrr/r/r0rsrcKs6tddddddd}td d |it|fi|}|S) ri ryTrrdeit3_huge_patch14_224rb)rrrr/r/r0rsrrqrrrsrtrurvrwrx)Z deit3_small_patch16_224_in21ft1kZ deit3_small_patch16_384_in21ft1kZ!deit3_medium_patch16_224_in21ft1kZdeit3_base_patch16_224_in21ft1kZdeit3_base_patch16_384_in21ft1kZ deit3_large_patch16_224_in21ft1kZ deit3_large_patch16_384_in21ft1kZdeit3_huge_patch14_224_in21ft1k)FF)r)F)F)F)F)F)F)F)F)F)F)F)F)F)F)F)F)/r[ functoolsrtypingrrrrZ timm.datarrZ timm.layersr Ztimm.models.vision_transformerr r r Z_builderrZ _manipulater _registryrrr__all__rrernZ default_cfgsrrrrrrrrrrrrrrrrrXr/r/r/r0sV     e  a