a  d7@s|dZddlZddlmZmZmZmZmZddlZddl Zddlm Z ddl m Z m Z mZddlmZddlmZmZmZmZdd lmZdd lmZmZdd lmZmZmZmZm Z d d l!m"Z"e#e$Z%dZ&dZ'gdZ(dZ)dZ*gdZ+dBe,e,ee,e,dddZ-Gddde j.Z/Gddde j.Z0Gddde j.Z1Gddde j.Z2Gdd d e j.Z3Gd!d"d"e j.Z4Gd#d$d$e j.Z5Gd%d&d&e j.Z6Gd'd(d(e j.Z7Gd)d*d*e j.Z8Gd+d,d,e j.Z9Gd-d.d.e j.Z:Gd/d0d0eZ;d1Zed6e<Gd7d8d8e;Z?Gd9d:d:e j.Z@Gd;d<dd>e j.ZBed?e<Gd@dAdAe;ZCdS)Cz PyTorch MobileViT model.N)DictOptionalSetTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)BaseModelOutputWithNoAttention(BaseModelOutputWithPoolingAndNoAttention$ImageClassifierOutputWithNoAttentionSemanticSegmenterOutput)PreTrainedModel) find_pruneable_heads_and_indicesprune_linear_layer)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings)MobileViTConfigrapple/mobilevit-small)rirztabby, tabby cat)rzapple/mobilevit-x-smallzapple/mobilevit-xx-smallzapple/deeplabv3-mobilevit-smallz!apple/deeplabv3-mobilevit-x-smallz"apple/deeplabv3-mobilevit-xx-smallr)valuedivisor min_valuereturncCsF|dur |}t|t||d||}|d|kr>||7}t|S)a Ensure that all layers have a channel count that is divisible by `divisor`. This function is taken from the original TensorFlow repo. It can be seen here: https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet/mobilenet.py Ng?)maxint)rrr new_valuer%y/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/transformers/models/mobilevit/modeling_mobilevit.pymake_divisibleIs  r'c sTeZdZd eeeeeeeeeeeefdd fdd Ze j e j dd d Z Z S) MobileViTConvLayerrFTN) config in_channels out_channels kernel_sizestridegroupsbiasdilationuse_normalizationuse_activationr c stt|dd|} ||dkr@td|d|d||dkrbtd|d|dtj||||| |||dd |_| rtj|d d d d d |_nd|_| rt | t rt | |_ qt |j t rt |j |_ q|j |_ nd|_ dS)Nrr!rzInput channels (z) are not divisible by z groups.zOutput channels (zeros) r*r+r,r-paddingr0r.r/Z padding_modegh㈵>g?T)Z num_featuresepsZmomentumZaffineZtrack_running_stats)super__init__r# ValueErrorrConv2d convolutionZ BatchNorm2d normalization isinstancestrr activation hidden_act) selfr)r*r+r,r-r.r/r0r1r2r4 __class__r%r&r7YsB         zMobileViTConvLayer.__init__featuresr cCs6||}|jdur||}|jdur2||}|SN)r:r;r>)r@rDr%r%r&forwards      zMobileViTConvLayer.forward)rrFrTT) __name__ __module__ __qualname__rr#boolrr=r7torchTensorrF __classcell__r%r%rAr&r(Xs( 6r(csFeZdZdZd eeeeeddfdd Zejejddd Z Z S) MobileViTInvertedResidualzQ Inverted residual block (MobileNetv2): https://arxiv.org/abs/1801.04381 rN)r)r*r+r-r0r c stttt||jd}|dvr:td|d|dkoH||k|_t|||dd|_ t|||d|||d|_ t|||dd d |_ dS) Nr)rr!zInvalid stride .rr*r+r,r )r*r+r,r-r.r0Fr*r+r,r2) r6r7r'r#roundZ expand_ratior8 use_residualr( expand_1x1conv_3x3 reduce_1x1)r@r)r*r+r-r0Zexpanded_channelsrAr%r&r7s0  z"MobileViTInvertedResidual.__init__rCcCs4|}||}||}||}|jr0||S|SrE)rTrUrVrS)r@rDresidualr%r%r&rFs    z!MobileViTInvertedResidual.forward)r) rGrHrI__doc__rr#r7rKrLrFrMr%r%rAr&rNs !rNcsBeZdZd eeeeeddfdd ZejejdddZZ S) MobileViTMobileNetLayerrN)r)r*r+r- num_stagesr csRtt|_t|D]0}t||||dkr4|ndd}|j||}qdS)Nrr)r*r+r-)r6r7r ModuleListlayerrangerNappend)r@r)r*r+r-rZir\rAr%r&r7s    z MobileViTMobileNetLayer.__init__rCcCs|jD] }||}q|SrEr\)r@rD layer_moduler%r%r&rFs  zMobileViTMobileNetLayer.forward)rr rGrHrIrr#r7rKrLrFrMr%r%rAr&rYs  rYcsNeZdZeeddfdd ZejejdddZejejdd d Z Z S) MobileViTSelfAttentionNr) hidden_sizer cst||jdkr2td|fd|jd|j|_t||j|_|j|j|_tj||j|j d|_ tj||j|j d|_ tj||j|j d|_ t |j|_dS)NrzThe hidden size z4 is not a multiple of the number of attention heads rO)r/)r6r7num_attention_headsr8r#attention_head_size all_head_sizerLinearZqkv_biasquerykeyrDropoutZattention_probs_dropout_probdropoutr@r)rerAr%r&r7s  zMobileViTSelfAttention.__init__)xr cCs6|dd|j|jf}|j|}|ddddS)Nrr!rr )sizerfrgviewpermute)r@roZ new_x_shaper%r%r&transpose_for_scoress z+MobileViTSelfAttention.transpose_for_scores hidden_statesr c Cs||}|||}|||}||}t||dd}|t|j }t j j |dd}| |}t||}|dddd}|dd|jf} |j| }|S)Nrpdimrr!rr )rjrtrkrrKmatmul transposemathsqrtrgr functionalZsoftmaxrmrs contiguousrqrhrr) r@rvZmixed_query_layerZ key_layerZ value_layerZ query_layerZattention_scoresZattention_probsZ context_layerZnew_context_layer_shaper%r%r&rFs     zMobileViTSelfAttention.forward) rGrHrIrr#r7rKrLrtrFrMr%r%rAr&rcsrccs:eZdZeeddfdd ZejejdddZZ S)MobileViTSelfOutputNrdcs*tt|||_t|j|_dSrEr6r7rridenserlZhidden_dropout_probrmrnrAr%r&r7s zMobileViTSelfOutput.__init__rucCs||}||}|SrErrmr@rvr%r%r&rFs  zMobileViTSelfOutput.forwardrbr%r%rAr&rsrcsNeZdZeeddfdd ZeeddddZej ej dd d Z Z S) MobileViTAttentionNrdcs.tt|||_t|||_t|_dSrE)r6r7rc attentionroutputset pruned_headsrnrAr%r&r7s   zMobileViTAttention.__init__)headsr cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrrx)lenrrrfrgrrrjrkrrrrhunion)r@rindexr%r%r& prune_heads%s zMobileViTAttention.prune_headsrucCs||}||}|SrE)rr)r@rvZ self_outputsattention_outputr%r%r&rF7s  zMobileViTAttention.forward) rGrHrIrr#r7rrrKrLrFrMr%r%rAr&rsrcs<eZdZeeeddfdd ZejejdddZZ S)MobileViTIntermediateNr)reintermediate_sizer cs>tt|||_t|jtr2t|j|_ n|j|_ dSrE) r6r7rrirr<r?r=r intermediate_act_fnr@r)rerrAr%r&r7>s   zMobileViTIntermediate.__init__rucCs||}||}|SrE)rrrr%r%r&rFFs  zMobileViTIntermediate.forwardrbr%r%rAr&r=srcs@eZdZeeeddfdd ZejejejdddZZ S)MobileViTOutputNrcs*tt|||_t|j|_dSrErrrAr%r&r7Ms zMobileViTOutput.__init__)rv input_tensorr cCs ||}||}||}|SrEr)r@rvrr%r%r&rFRs  zMobileViTOutput.forwardrbr%r%rAr&rLsrcs<eZdZeeeddfdd ZejejdddZZ S)MobileViTTransformerLayerNrcsZtt|||_t||||_t||||_tj ||j d|_ tj ||j d|_ dS)Nr5) r6r7rrr intermediaterrr LayerNormlayer_norm_epslayernorm_beforelayernorm_afterrrAr%r&r7Zs   z"MobileViTTransformerLayer.__init__rucCs<|||}||}||}||}|||}|SrE)rrrrr)r@rvrZ layer_outputr%r%r&rFbs    z!MobileViTTransformerLayer.forwardrbr%r%rAr&rYsrcs<eZdZeeeddfdd ZejejdddZZ S)MobileViTTransformerN)r)rerZr csJtt|_t|D](}t||t||jd}|j |qdS)N)rer) r6r7rr[r\r]rr#Z mlp_ratior^)r@r)rerZ_transformer_layerrAr%r&r7ms    zMobileViTTransformer.__init__rucCs|jD] }||}q|SrEr`)r@rvrar%r%r&rFys  zMobileViTTransformer.forwardrbr%r%rAr&rls rc s|eZdZdZdeeeeeeeddfdd Zeje eje fddd Z eje ejd d d Z ejejdd dZ ZS)MobileViTLayerz; MobileViT block: https://arxiv.org/abs/2110.02178 rN)r)r*r+r-rerZr0r cst|j|_|j|_|dkrXt||||dkr6|nd|dkrH|dndd|_|}nd|_t||||jd|_ t|||dddd|_ t |||d|_ t j||jd|_t|||dd|_t|d|||jd|_dS) Nr!r)r*r+r-r0rPF)r*r+r,r1r2)rerZr)r6r7Z patch_size patch_width patch_heightrNdownsampling_layerr(Zconv_kernel_sizeconv_kxkconv_1x1r transformerrrr layernormconv_projectionfusion)r@r)r*r+r-rerZr0rAr%r&r7sN  zMobileViTLayer.__init__rCcCs|j|j}}t||}|j\}}}}tt|||} tt|||} d} | |ksh| |krtjj|| | fddd}d} | |} | |} | | }| ||| || |}| dd}| ||||}| dd}| |||d}||f||| || | d }||fS) NFbilinearrqmodeZ align_cornersTrr!r rp) orig_size batch_sizechannels interpolate num_patchesnum_patches_widthnum_patches_height) rrr#shaper|ceilrr~rreshaper{)r@rDrr patch_arearrZ orig_heightZ orig_widthZ new_heightZ new_widthrnum_patch_widthnum_patch_heightrpatches info_dictr%r%r& unfoldings<     zMobileViTLayer.unfolding)rrr c Cs|j|j}}t||}|d}|d}|d}|d} |d} ||||d} | dd} | ||| | ||} | dd } | ||| || |} |d rtjj | |d d d d} | S)Nrrrrrrprr r!rrrFr) rrr#rrrr{rrr~r) r@rrrrrrrrrrrDr%r%r&foldings*    zMobileViTLayer.foldingcCs||jr||}|}||}||}||\}}||}||}|||}||}|t j ||fdd}|SNrrx) rrrrrrrrrrKcat)r@rDrWrrr%r%r&rFs       zMobileViTLayer.forward)r)rGrHrIrXrr#r7rKrLrrrrrFrMr%r%rAr&rs :+rcsDeZdZeddfdd Zd ejeeee e fddd Z Z S) MobileViTEncoderNr)r c sZt||_t|_d|_d}}|jdkr.create_custom_forward..custom_forwardr%)rrr%rr&create_custom_forwardvs z7MobileViTEncoder.forward..create_custom_forwardcss|]}|dur|VqdSrEr%).0vr%r%r& z+MobileViTEncoder.forward..)last_hidden_staterv) enumerater\rZtrainingrKutils checkpointtupler )r@rvrrZall_hidden_statesr_rarr%r%r&rFks   zMobileViTEncoder.forward)FT) rGrHrIrr7rKrLrJrrr rFrMr%r%rAr&r sM rc@sJeZdZdZeZdZdZdZe e j e j e j fddddZd d d ZdS) MobileViTPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. mobilevit pixel_valuesTN)rr cCsjt|tjtjfr@|jjjd|jjd|j durf|j j n&t|tj rf|j j |jj ddS)zInitialize the weightsg)meanZstdNg?) r<rrir9ZweightdataZnormal_r)Zinitializer_ranger/Zzero_rZfill_)r@rr%r%r& _init_weightss   z&MobileViTPreTrainedModel._init_weightsFcCst|tr||_dSrE)r<rr)r@rrr%r%r&_set_gradient_checkpointings z4MobileViTPreTrainedModel._set_gradient_checkpointing)F)rGrHrIrXr config_classZbase_model_prefixZmain_input_nameZsupports_gradient_checkpointingrrrir9rrrr%r%r%r&rs raK This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`MobileViTConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. aF Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`MobileViTImageProcessor.__call__`] for details. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zWThe bare MobileViT model outputting raw hidden-states without any specific head on top.c steZdZd eedfdd ZddZeee e e e de ddeejeeeeeee fd d d ZZS)MobileViTModelT)r) expand_outputcsnt|||_||_t||j|jdddd|_t||_ |jrbt||jd|jddd|_ | dS) Nrr r!)r*r+r,r-rrrP) r6r7r)rr(Z num_channelsr conv_stemrencoder conv_1x1_exp post_init)r@r)rrAr%r&r7s&  zMobileViTModel.__init__cCsF|D]8\}}|jj|}t|tr|jjD]}|j|q.qdS)zPrunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrr\r<rrrr)r@Zheads_to_pruneZ layer_indexrZmobilevit_layerrr%r%r& _prune_headss    zMobileViTModel._prune_headsZvision)r output_typerZmodalityexpected_outputN)rrrr c Cs|dur |n|jj}|dur |n|jj}|dur8td||}|j|||d}|jr|||d}tj |ddgdd}n |d}d}|s|dur||fn|f}||ddSt |||j d S) Nz You have to specify pixel_valuesrrrrwrpF)ryZkeepdimr)r pooler_outputrv) r)ruse_return_dictr8rrrrrKrrrv) r@rrrZembedding_outputZencoder_outputsr pooled_outputrr%r%r&rFs0 zMobileViTModel.forward)T)NNN)rGrHrIrrJr7rrMOBILEVIT_INPUTS_DOCSTRINGr_CHECKPOINT_FOR_DOCr_CONFIG_FOR_DOC_EXPECTED_OUTPUT_SHAPErrKrLrrrFrMr%r%rAr&rs&   rz MobileViT model with an image classification head on top (a linear layer on top of the pooled features), e.g. for ImageNet. c speZdZeddfdd Zeeeee e e dd e e je ee e je eeee fdddZZS) MobileViTForImageClassificationNrcsdt||j|_t||_tj|jdd|_|jdkrNt |j d|jnt |_ | dS)NT)Zinplacerrp)r6r7 num_labelsrrrrlclassifier_dropout_probrmrirZIdentity classifierrr@r)rAr%r&r7#s  $z(MobileViTForImageClassification.__init__)rrrr)rrlabelsrr c Csr|dur |n|jj}|j|||d}|r.|jn|d}|||}d}|dur2|jjdur|jdkrtd|j_n4|jdkr|jt j ks|jt j krd|j_nd|j_|jjdkrt } |jdkr| | | }n | ||}nN|jjdkrt} | |d|j|d}n|jjdkr2t} | ||}|sb|f|dd} |dur^|f| S| St|||jd S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss). If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NrrZ regressionZsingle_label_classificationZmulti_label_classificationrpr!)losslogitsrv)r)rrrrrmZ problem_typerZdtyperKlongr#r Zsqueezer rrrrrv) r@rrrroutputsrrrloss_fctrr%r%r&rF2s>    "     z'MobileViTForImageClassification.forward)NNNN)rGrHrIrr7rrr_IMAGE_CLASS_CHECKPOINTrr_IMAGE_CLASS_EXPECTED_OUTPUTrrKrLrJrrrFrMr%r%rAr&rs& rcs<eZdZeeeddfdd ZejejdddZZ S)MobileViTASPPPoolingN)r)r*r+r c s4ttjdd|_t|||ddddd|_dS)Nr)Z output_sizeTrelu)r*r+r,r-r1r2)r6r7rZAdaptiveAvgPool2d global_poolr(r)r@r)r*r+rAr%r&r7qs zMobileViTASPPPooling.__init__rCcCs:|jdd}||}||}tjj||ddd}|S)NrwrFr)rrrrr~r)r@rDZ spatial_sizer%r%r&rFs   zMobileViTASPPPooling.forwardrbr%r%rAr&rpsrcs<eZdZdZeddfdd ZejejdddZZ S) MobileViTASPPzs ASPP module defined in DeepLab papers: https://arxiv.org/abs/1606.00915, https://arxiv.org/abs/1706.05587 Nrcstjdjtjdkr0tdt|_ t ddd}|j ||j fddjDt }|j |t d ddd|_tjjd |_dS) Nrwr z"Expected 3 values for atrous_ratesrrrQc s g|]}td|ddqS)r r)r*r+r,r0r2)r()rZrater)r*r+r%r& s z*MobileViTASPP.__init__..r)p)r6r7raspp_out_channelsrZ atrous_ratesr8rr[convsr(r^extendrprojectrlZaspp_dropout_probrm)r@r)Z in_projectionZ pool_layerrArr&r7s2       zMobileViTASPP.__init__rCcCsDg}|jD]}|||q tj|dd}||}||}|Sr)rr^rKrr rm)r@rDZpyramidconvZpooled_featuresr%r%r&rFs   zMobileViTASPP.forward rGrHrIrXrr7rKrLrFrMr%r%rAr&rs+rcs<eZdZdZeddfdd ZejejdddZZ S) MobileViTDeepLabV3zB DeepLabv3 architecture: https://arxiv.org/abs/1706.05587 Nrc sBtt||_t|j|_t||j |j ddddd|_ dS)NrFT)r*r+r,r1r2r/) r6r7raspprZ Dropout2drrmr(rrrrrAr%r&r7s  zMobileViTDeepLabV3.__init__rucCs&||d}||}||}|S)Nrp)rrmr)r@rvrDr%r%r&rFs  zMobileViTDeepLabV3.forwardr r%r%rAr&r sr zX MobileViT model with a semantic segmentation head on top, e.g. for Pascal VOC. c sleZdZeddfdd Zeeeee dd e e j e e j e e e e eeefdddZZS) MobileViTForSemanticSegmentationNrcs8t||j|_t|dd|_t||_|dS)NF)r)r6r7rrrr segmentation_headrrrAr%r&r7s   z)MobileViTForSemanticSegmentation.__init__)rr)rrrrr c Cs |dur |n|jj}|dur |n|jj}|j|d|d}|rB|jn|d}||}d}|dur|jjdkrvtdn6tj j ||j ddddd } t |jj d } | | |}|s|r|f|dd} n|f|d d} |dur|f| S| St|||r|jnddd S) at labels (`torch.LongTensor` of shape `(batch_size, height, width)`, *optional*): Ground truth semantic segmentation maps for computing the loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1`, a classification loss is computed (Cross-Entropy). Returns: Examples: ```python >>> from transformers import AutoImageProcessor, MobileViTForSemanticSegmentation >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("apple/deeplabv3-mobilevit-small") >>> model = MobileViTForSemanticSegmentation.from_pretrained("apple/deeplabv3-mobilevit-small") >>> inputs = image_processor(images=image, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) >>> # logits are of shape (batch_size, num_labels, height, width) >>> logits = outputs.logits ```NTrrz/The number of labels should be greater than onerwrFr)Z ignore_indexr!)rrrvZ attentions)r)rrrrvrrr8rr~rrr Zsemantic_loss_ignore_indexr) r@rrrrrZencoder_hidden_statesrrZupsampled_logitsrrr%r%r&rFs<&    z(MobileViTForSemanticSegmentation.forward)NNNN)rGrHrIrr7rrrrrrrKrLrJrrrFrMr%r%rAr&rs   r)rN)DrXr|typingrrrrrrKZtorch.utils.checkpointrZtorch.nnrr r Z activationsr Zmodeling_outputsr rrrZmodeling_utilsrZ pytorch_utilsrrrrrrrrZconfiguration_mobilevitrZ get_loggerrGloggerrrrrrZ'MOBILEVIT_PRETRAINED_MODEL_ARCHIVE_LISTr#r'Moduler(rNrYrcrrrrrrrrrZMOBILEVIT_START_DOCSTRINGrrrrrr rr%r%r%r&sl      @03  "l  WN;