a  d@sdZddlZddlZddlmZmZmZmZm Z m Z ddl Z ddl Z ddl m Z ddlmZmZmZddlmZddlmZmZmZmZdd lmZdd lmZmZdd lmZm Z m!Z!m"Z"m#Z#d d l$m%Z%e"&e'Z(dZ)dZ*gdZ+dZ,dZ-dgZ.Gddde j/Z0Gddde j/Z1Gddde j/Z2Gddde j/Z3Gddde j/Z4Gddde j/Z5Gdd d e j/Z6Gd!d"d"e j/Z7Gd#d$d$e j/Z8Gd%d&d&eZ9d'Z:d(Z;e d)e:Gd*d+d+e9Ze d1e:Gd2d3d3e9Z?dS)4z PyTorch ViT model.N)DictListOptionalSetTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)BaseModelOutputBaseModelOutputWithPoolingImageClassifierOutputMaskedImageModelingOutput)PreTrainedModel) find_pruneable_heads_and_indicesprune_linear_layer)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) ViTConfigrz!google/vit-base-patch16-224-in21k)rizgoogle/vit-base-patch16-224z Egyptian catcsdeZdZdZd eeddfdd Zeje e ejddd Z deje ej eejd d d Z ZS) ViTEmbeddingszb Construct the CLS token, position and patch embeddings. Optionally, also the mask token. FN)configuse_mask_tokenreturncstttdd|j|_|rr*r(expandrHZtype_asr&r$rGrIr,r/) r0rJrKrI batch_size num_channelsr6r7r5Z seq_lengthZ mask_tokensmaskZ cls_tokensr3r3r4forwardss   zViTEmbeddings.forward)F)NF)__name__ __module__ __qualname____doc__rboolr"r$TensorrArIr BoolTensorrQ __classcell__r3r3r1r4rBs $rcs8eZdZdZfddZdejeejdddZZ S) r)z This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. cst|j|j}}|j|j}}t|tjj r8|n||f}t|tjj rR|n||f}|d|d|d|d}||_||_||_||_ t j ||||d|_ dS)Nrr) kernel_sizeZstride)r!r" image_sizer?rOr% isinstance collectionsabcIterabler+rConv2d projection)r0rr[r?rOr%r+r1r3r4r"s  zViTPatchEmbeddings.__init__F)rJrIr c Cs|j\}}}}||jkr0td|jd|d|s~||jdksP||jdkr~td|d|d|jdd|jdd ||d dd }|S) NzoMake sure that the channel dimension of the pixel values match with the one set in the configuration. Expected z but got .rrzInput image size (*z) doesn't match model (z).r9)r>rO ValueErrorr[raflatten transpose)r0rJrIrNrOr6r7r5r3r3r4rQs( zViTPatchEmbeddings.forward)F) rRrSrTrUr"r$rWrVrQrYr3r3r1r4r)s r)csleZdZeddfdd ZejejdddZd eeje e e ejejfe ejfd d d Z Z S) ViTSelfAttentionNrr cst|j|jdkr@t|ds@td|jfd|jd|j|_t|j|j|_|j|j|_t j |j|j|j d|_ t j |j|j|j d|_ t j |j|j|j d|_t |j|_dS)NrZembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads rb)bias)r!r"r%num_attention_headshasattrrdrAattention_head_size all_head_sizerLinearZqkv_biasquerykeyvaluer-Zattention_probs_dropout_probr/r0rr1r3r4r"s  zViTSelfAttention.__init__)xr cCs6|dd|j|jf}||}|ddddS)Nr8rr9rr )sizerjrlrFrD)r0rsZ new_x_shaper3r3r4transpose_for_scoress z%ViTSelfAttention.transpose_for_scoresF) head_maskoutput_attentionsr c Cs||}|||}|||}||}t||dd}|t|j }t j j |dd} | | } |dur| |} t| |} | dddd} | dd|jf} | | } |r| | fn| f} | S)Nr8r;r<rr9rr )rorurprqr$matmulrfrBrCrlrrEZsoftmaxr/rD contiguousrtrmrF) r0 hidden_statesrvrwZmixed_query_layerZ key_layerZ value_layerZ query_layerZattention_scoresZattention_probsZ context_layerZnew_context_layer_shapeoutputsr3r3r4rQs      zViTSelfAttention.forward)NF)rRrSrTrr"r$rWrurrVrrrQrYr3r3r1r4rgs rgcs@eZdZdZeddfdd ZejejejdddZZ S) ViTSelfOutputz The residual connection is defined in ViTLayer instead of here (as is the case with other models), due to the layernorm applied before each block. Nrhcs.tt|j|j|_t|j|_dSN) r!r"rrnr%denser-r.r/rrr1r3r4r"s zViTSelfOutput.__init__rz input_tensorr cCs||}||}|Sr}r~r/r0rzrr3r3r4rQs  zViTSelfOutput.forward) rRrSrTrUrr"r$rWrQrYr3r3r1r4r|sr|cspeZdZeddfdd ZeeddddZd ej e ej e e e ej ej fe ej fd d d ZZS) ViTAttentionNrhcs*tt||_t||_t|_dSr})r!r"rg attentionr|outputset pruned_headsrrr1r3r4r"s   zViTAttention.__init__)headsr cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrr<)lenrrrjrlrrrorprqrr~rmunion)r0rindexr3r3r4 prune_headss zViTAttention.prune_headsFrzrvrwr cCs4||||}||d|}|f|dd}|S)Nrr)rr)r0rzrvrwZ self_outputsattention_outputr{r3r3r4rQ szViTAttention.forward)NF)rRrSrTrr"rrArr$rWrrVrrrQrYr3r3r1r4rsrcs8eZdZeddfdd ZejejdddZZS)ViTIntermediateNrhcsBtt|j|j|_t|jt r6t |j|_ n|j|_ dSr}) r!r"rrnr%intermediate_sizer~r\Z hidden_actstrr intermediate_act_fnrrr1r3r4r"/s   zViTIntermediate.__init__)rzr cCs||}||}|Sr})r~r)r0rzr3r3r4rQ7s  zViTIntermediate.forward rRrSrTrr"r$rWrQrYr3r3r1r4r.srcs<eZdZeddfdd ZejejejdddZZS) ViTOutputNrhcs.tt|j|j|_t|j|_ dSr}) r!r"rrnrr%r~r-r.r/rrr1r3r4r"?s zViTOutput.__init__rcCs ||}||}||}|Sr}rrr3r3r4rQDs  zViTOutput.forwardrr3r3r1r4r>srcs`eZdZdZeddfdd Zd ejeeje e e ejejfe ejfddd Z Z S) ViTLayerz?This corresponds to the Block class in the timm implementation.Nrhcsbt|j|_d|_t||_t||_t||_ t j |j |j d|_t j |j |j d|_dS)NrZeps)r!r"Zchunk_size_feed_forwardZ seq_len_dimrrr intermediaterrr LayerNormr%layer_norm_epslayernorm_beforelayernorm_afterrrr1r3r4r"Ps    zViTLayer.__init__FrcCs`|j||||d}|d}|dd}||}||}||}|||}|f|}|S)Nrwrr)rrrrr)r0rzrvrwZself_attention_outputsrr{Z layer_outputr3r3r4rQZs     zViTLayer.forward)NF)rRrSrTrUrr"r$rWrrVrrrQrYr3r3r1r4rMs rc sNeZdZeddfdd Zd ejeejeeee e e fddd Z Z S) ViTEncoderNrhcs:t|_tfddtjD|_d|_dS)Ncsg|] }tqSr3)r).0_rr3r4 {z'ViTEncoder.__init__..F) r!r"rrZ ModuleListrangenum_hidden_layerslayergradient_checkpointingrrr1rr4r"xs  zViTEncoder.__init__FT)rzrvrwoutput_hidden_states return_dictr c s|rdnd}rdnd}t|jD]\}} |r8||f}|durH||nd} |jr||jr|fdd} tjj| | || } n | || } | d}r"|| df}q"|r||f}|stdd|||fDSt|||dS) Nr3csfdd}|S)Ncsg|RSr}r3)inputs)modulerwr3r4custom_forwardszIViTEncoder.forward..create_custom_forward..custom_forwardr3)rrr)rr4create_custom_forwardsz1ViTEncoder.forward..create_custom_forwardrrcss|]}|dur|VqdSr}r3)rvr3r3r4 rz%ViTEncoder.forward..)last_hidden_staterz attentions) enumeraterrZtrainingr$utils checkpointtupler) r0rzrvrwrrZall_hidden_statesZall_self_attentionsiZ layer_moduleZlayer_head_maskrZ layer_outputsr3rr4rQ~s4       zViTEncoder.forward)NFFT)rRrSrTrr"r$rWrrVrrrrQrYr3r3r1r4rws  rc@sXeZdZdZeZdZdZdZgZ e e j e j e jfddddZd eedd d d ZdS)ViTPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. vitrJTN)rr cCst|tjtjfrZtjj|jjt j d|j j d|jj |j_|jdur|jjnt|tjr|jj|jjdnjt|trtjj|jjt j d|j j d|jj |j_tjj|jjt j d|j j d|jj |j_dS)zInitialize the weightsg)meanZstdNrL)r\rrnr`initZ trunc_normal_weightdatator$float32rZinitializer_rangedtyperiZzero_rZfill_rr,r&)r0rr3r3r4 _init_weightss4    z ViTPreTrainedModel._init_weightsF)rrqr cCst|tr||_dSr})r\rr)r0rrqr3r3r4_set_gradient_checkpointings z.ViTPreTrainedModel._set_gradient_checkpointing)F)rRrSrTrUr config_classZbase_model_prefixZmain_input_nameZsupports_gradient_checkpointingZ_no_split_modulesrrrnr`rrrrVrr3r3r3r4rs raE This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`ViTConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`ViTImageProcessor.__call__`] for details. head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. interpolate_pos_encoding (`bool`, *optional*): Whether to interpolate the pre-trained position encodings. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z]The bare ViT Model transformer outputting raw hidden-states without any specific head on top.c seZdZdeeedfdd ZedddZee e e fd d d d Z e e eeeed eddeejeejeejeeeeeeeeeeefdddZZS)ViTModelTF)radd_pooling_layerrcs\t|||_t||d|_t||_tj|j |j d|_ |rJt |nd|_ |dS)N)rr)r!r"rrr5rencoderrrr%r layernorm ViTPoolerpooler post_init)r0rrrr1r3r4r"s  zViTModel.__init__)r cCs|jjSr})r5r*)r0r3r3r4get_input_embeddingsszViTModel.get_input_embeddingsN)heads_to_pruner cCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)r0rrrr3r3r4 _prune_headsszViTModel._prune_headsZvision)r output_typerZmodalityexpected_outputrJrKrvrwrrIrr cCs |dur |n|jj}|dur |n|jj}|dur4|n|jj}|durLtd|||jj}|jjj j j }|j |kr~| |}|j|||d} |j | ||||d} | d} || } |jdur|| nd} |s| dur| | fn| f} | | ddSt| | | j| jdS)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Nz You have to specify pixel_values)rKrI)rvrwrrrr)rZ pooler_outputrzr)rrwruse_return_dictrdZ get_head_maskrr5r*rarrrrrrrrzr)r0rJrKrvrwrrIrZexpected_dtypeZembedding_outputZencoder_outputssequence_output pooled_outputZ head_outputsr3r3r4rQs@   zViTModel.forward)TF)NNNNNNN)rRrSrTrrVr"r)rrrArrrVIT_INPUTS_DOCSTRINGr_CHECKPOINT_FOR_DOCr_CONFIG_FOR_DOC_EXPECTED_OUTPUT_SHAPErr$rWrXrrrQrYr3r3r1r4rs8   rcs*eZdZedfdd ZddZZS)rrcs*tt|j|j|_t|_dSr})r!r"rrnr%r~ZTanh activationrrr1r3r4r"cs zViTPooler.__init__cCs(|dddf}||}||}|S)Nr)r~r)r0rzZfirst_token_tensorrr3r3r4rQhs  zViTPooler.forward)rRrSrTrr"rQrYr3r3r1r4rbsraVViT Model with a decoder on top for masked image modeling, as proposed in [SimMIM](https://arxiv.org/abs/2111.09886). Note that we provide a script to pre-train this model on custom data in our [examples directory](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining). c seZdZeddfdd Zeeeee dd e e j e e j e e j e ee ee ee eeeefdddZZS) ViTForMaskedImageModelingNrhcsXt|t|ddd|_ttj|j|jd|j ddt |j|_ | dS)NFT)rrr9r)Z in_channelsZ out_channelsrZ) r!r"rrrZ Sequentialr`r%encoder_striderOZ PixelShuffledecoderrrrr1r3r4r"~s   z"ViTForMaskedImageModeling.__init__)rrrc Cs|dur |n|jj}|durJ|jj|jjkrJtd|jjd|jjd|j|||||||d}|d} | ddddf} | j\} } } t| d} }| dd d | | | |} | | }d}|durD|jj |jj}| d ||}| |jjd |jjd d}tjj||d d }|||d |jj}|st|f|dd}|durp|f|S|St|||j|jdS)a= bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Returns: Examples: ```python >>> from transformers import AutoImageProcessor, ViTForMaskedImageModeling >>> import torch >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("google/vit-base-patch16-224-in21k") >>> model = ViTForMaskedImageModeling.from_pretrained("google/vit-base-patch16-224-in21k") >>> num_patches = (model.config.image_size // model.config.patch_size) ** 2 >>> pixel_values = image_processor(images=image, return_tensors="pt").pixel_values >>> # create random boolean mask of shape (batch_size, num_patches) >>> bool_masked_pos = torch.randint(low=0, high=2, size=(1, num_patches)).bool() >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos) >>> loss, reconstructed_pixel_values = outputs.loss, outputs.reconstruction >>> list(reconstructed_pixel_values.shape) [1, 3, 224, 224] ```NzWhen `bool_masked_pos` is provided, `patch_size` must be equal to `encoder_stride` to ensure that the reconstructed image has the same dimensions as the input.Got `patch_size` = z and `encoder_stride` = rb)rKrvrwrrIrrrg?r9r8none)Z reductiongh㈵>)lossZreconstructionrzr)rrr?rrdrr>rBfloorrDr@rr[Zrepeat_interleaverHryrrEZl1_losssumrOrrzr)r0rJrKrvrwrrIrr{rrNZsequence_lengthrOr6r7Zreconstructed_pixel_valuesZmasked_im_lossrtrPZreconstruction_lossrr3r3r4rQsZ)     z!ViTForMaskedImageModeling.forward)NNNNNNN)rRrSrTrr"rrrrrrr$rWrXrVrrrQrYr3r3r1r4rqs(   ra ViT Model transformer with an image classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for ImageNet. Note that it's possible to fine-tune ViT on higher resolution images than the ones it has been trained on, by setting `interpolate_pos_encoding` to `True` in the forward of the model. This will interpolate the pre-trained position embeddings to the higher resolution. c seZdZeddfdd Zeeeee e e dd e e je e je e je ee ee ee eeee fdddZZS) ViTForImageClassificationNrhcsRt||j|_t|dd|_|jdkr 1` a classification loss is computed (Cross-Entropy). N)rvrwrrIrrrZ regressionZsingle_label_classificationZmulti_label_classificationr8)rlogitsrzr)rrrrrZdeviceZ problem_typerrr$longrAr Zsqueezer rFr rrzr)r0rJrvrrwrrIrr{rrrZloss_fctrr3r3r4rQ sP      "     z!ViTForImageClassification.forward)NNNNNNN)rRrSrTrr"rrr_IMAGE_CLASS_CHECKPOINTrr_IMAGE_CLASS_EXPECTED_OUTPUTrr$rWrVrrrQrYr3r3r1r4rs2  r)@rUcollections.abcr]rBtypingrrrrrrr$Ztorch.utils.checkpointrZtorch.nnr r r Z activationsr Zmodeling_outputsrrrrZmodeling_utilsrZ pytorch_utilsrrrrrrrrZconfiguration_vitrZ get_loggerrRloggerrrrrrZ!VIT_PRETRAINED_MODEL_ARCHIVE_LISTModulerr)rgr|rrrrrrZVIT_START_DOCSTRINGrrrrrr3r3r3r4sb      P'<'*9+ _  r