a  d@sdZddlZddlZddlmZddlmZmZm Z m Z ddl Z ddl Z ddl m Z ddlmZddlmZdd lmZmZmZmZmZmZdd lmZdd lmZmZmZmZdd l m!Z!m"Z"m#Z#m$Z$d dl%m&Z&e#'e(Z)ese)*de j+ddZ,dZ-dgZ.eGdddeZ/Gddde j0Z1Gddde j0Z2Gddde j0Z3Gddde j0Z4Gddde j0Z5Gdd d e j0Z6Gd!d"d"e j0Z7Gd#d$d$e j0Z8Gd%d&d&e j0Z9Gd'd(d(e j0Z:Gd)d*d*eZ;d+Ze!d.e<Gd/d0d0e;Z?Gd1d2d2e j0Z@e!d3e<Gd4d5d5e;ZAGd6d7d7e j0ZBGd8d9d9e j0ZCe!d:e<Gd;d<dd?d?e;ZEe!d@e>GdAdBdBe;ZFe!dCe<GdDdEdEe;ZGdS)Fz PyTorch ViLT model.N) dataclass)ListOptionalTupleUnion)nn)CrossEntropyLoss)ACT2FN)BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutput ModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel) find_pruneable_heads_and_indices#is_torch_greater_or_equal_than_1_10meshgridprune_linear_layer)add_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) ViltConfigzYou are using torch==zG, but torch>=1.10.0 is required to use ViltModel. Please upgrade torch.rzdandelin/vilt-b32-mlmc@sfeZdZUdZdZeejed<dZ ejed<dZ ee e ejed<dZ ee e ejed<dS)(ViltForImagesAndTextClassificationOutputa Class for outputs of [`ViltForImagesAndTextClassification`]. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Classification (or regression if config.num_labels==1) loss. logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Classification (or regression if config.num_labels==1) scores (before SoftMax). hidden_states (`List[tuple(torch.FloatTensor)]`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): List of tuples of `torch.FloatTensor` (one for each image-text pair, each tuple containing the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`List[tuple(torch.FloatTensor)]`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): List of tuples of `torch.FloatTensor` (one for each image-text pair, each tuple containing the attention weights of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nlosslogits hidden_states attentions)__name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__rrrrr r(r(o/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/transformers/models/vilt/modeling_vilt.pyr@s rcs4eZdZdZfddZd ddZd dd ZZS) ViltEmbeddingsz Construct the text and patch embeddings. Text embeddings are equivalent to BERT embeddings. Patch embeddings are equivalent to ViT embeddings. cstt||_ttdd|j|_ t ||_ |j j }ttd|d|j|_ t|j|j|_t|j|_||_dSNr)super__init__TextEmbeddingstext_embeddingsr Parameterr%zeros hidden_size cls_tokenViltPatchEmbeddingspatch_embeddings num_patchesposition_embeddings EmbeddingZmodality_type_vocab_sizetoken_type_embeddingsDropouthidden_dropout_probdropoutconfig)selfr=r6 __class__r(r)r-cs   zViltEmbeddings.__init__cs|jjjj\}}}}||}|dddddddf}tjj||jd|jdfd}|dddfj dddddf} |dddfj dddddf} |j\} } |j j |j j } |j ddddddfddd| | | tjfddt| | Ddd}|ddd}|ddd}tjtt|jd t|jd d d d dj|jd }|ddddddddf}||jd|jdd d d }|dd}|d}dksdustts| | }|n| | }t||jddd|jdddddf}fdd|D}fdd|D}dd|D}dd|D}fdd|D}g}t t|||D]\}\}}}|dkrt!t"|}|#|||n>tj!t"||dd}|#tj|||||gddqtj|dd}||dddf|dddff| d | }||dddf|dddff| d }||dddf|dddff| d d}||dddf|dddff| d | }|j$| d d }tj||fdd}tj|j dddddfdddddf| d d |fdd}||}|%|}tjt"|jdd||gdd}|||fffS)Nr sizerrZdimc sBg|]:\}}tjtjj||fdddd|d|fqS)ZbilinearT)rDmodeZ align_cornersr)r functionalpad interpolate).0hw)height spatial_poswidthr(r) s z/ViltEmbeddings.visual_embed..Zij)ZindexingdeviceF)as_tuplecs$g|]}dddf|kqSNrr(rJu) valid_idxr(r)rPcs$g|]}dddf|kqSrVr(rW) non_valid_idxr(r)rPrZcSsg|]}|dqSrrCrJvr(r(r)rPrZcSsg|]}|dqSr\rCr]r(r(r)rPrZcsg|] }|qSr(r(r]max_image_lengthr(r)rPrZT) replacement)&r5 projectionweightshapefloatrrGrIlongsumr= image_size patch_sizer7 transposeviewr%catzipflattenstackrarangetorTexpand isinstanceintmaxminZnonzerounique enumerateZ multinomialonesappendr3r<)r> pixel_values pixel_maskr`_phpwxZx_maskZx_hZx_w batch_size num_channelsZ patch_dimZ pos_embed patch_indexZeffective_resolutionZ unique_rowsZ valid_row_idxZnon_valid_row_idxZ valid_numsZnon_valid_numsZpad_numsselectir^nvpZ valid_choiceZ pad_choiceZ cls_tokensr()rMr`r[rNrYrOr) visual_embedrsx  $$$0  &      (.,..8 &zViltEmbeddings.visual_embedrc  Cs|j|||d} |dur4|j|||jjd\}} } n |d} |durJd}| |tj|tj| j d} ||tj | |tj| j d}tj | |gdd} tj || gdd} | | fS)N) input_idstoken_type_ids inputs_embedsr_rdtyperTrE) r/rr=r`rnr9r%Z zeros_likerfrTZ full_likerl)r>rattention_maskrr{r|r image_embedsimage_token_type_idxZ text_embedsZ image_masksr embeddingsmasksr(r(r)forwards&   zViltEmbeddings.forward)rA)r)r!r"r#r$r-rr __classcell__r(r(r?r)r*Zs   ar*cs*eZdZdZfddZdddZZS)r.zGConstruct the embeddings from word, position and token_type embeddings.csttj|j|j|jd|_t|j|j|_ t|j |j|_ tj |j|j d|_ t|j|_t|dd|_|dt|jd|jdtj|jtjdd d dS) N) padding_idxZepsposition_embedding_typeabsolute position_ids)rrRr)rF) persistent)r,r-rr8 vocab_sizer2Z pad_token_idword_embeddingsZmax_position_embeddingsr7Ztype_vocab_sizer9 LayerNormlayer_norm_epsr:r;r<getattrrZregister_bufferr%rprrr1rrDrfr>r=r?r(r)r-s zTextEmbeddings.__init__Nc Cs|dur|}n|dd}|d}|durH|jddd|f}|durt|dr|jddd|f}||d|}|}ntj|tj|jjd}|dur| |}| |} || } |j dkr| |} | | 7} | | } || } | S)NrRrrrrr)rDrhasattrrrrr%r1rfrTrr9rr7rr<) r>rrrr input_shape seq_lengthZbuffered_token_type_idsZ buffered_token_type_ids_expandedr9rr7r(r(r)rs,        zTextEmbeddings.forward)NNNNr!r"r#r$r-rrr(r(r?r)r.s r.cs(eZdZdZfddZddZZS)r4z# Image to Patch Embedding. cst|j|j}}|j|j}}t|tjj r8|n||f}t|tjj rR|n||f}|d|d|d|d}||_||_||_||_ t j ||||d|_ dS)Nrr)Z kernel_sizeZstride)r,r-rhrirr2rs collectionsabcIterabler6rConv2drb)r>r=rhrirr2r6r?r(r)r-0s  zViltPatchEmbeddings.__init__cCs.|j\}}}}||jkr td||}|S)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.)rdr ValueErrorrb)r>r{rrrMrOrr(r(r)r?s  zViltPatchEmbeddings.forwardrr(r(r?r)r4+s r4cs.eZdZfddZddZd ddZZS) ViltSelfAttentioncst|j|jdkr@t|ds@td|jfd|jd|j|_t|j|j|_|j|j|_t j |j|j|j d|_ t j |j|j|j d|_ t j |j|j|j d|_t |j|_dS)NrZembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .bias)r,r-r2num_attention_headsrrrtattention_head_size all_head_sizerLinearZqkv_biasquerykeyvaluer:Zattention_probs_dropout_probr<rr?r(r)r-Js  zViltSelfAttention.__init__cCs6|dd|j|jf}|j|}|ddddS)NrRrrBrr )rDrrrkpermute)r>rZ new_x_shaper(r(r)transpose_for_scores\s z&ViltSelfAttention.transpose_for_scoresNFcCs||}|||}|||}||}t||dd} | t|j } |durh| |} t j dd| } | | } |dur| |} t| |} | dddd} | dd|jf} | j| } |r| | fn| f} | S)NrRrQrErrBrr )rrrrr%matmulrjmathsqrtrrZSoftmaxr<r contiguousrDrrk)r>rr head_maskoutput_attentionsZmixed_query_layerZ key_layerZ value_layerZ query_layerZattention_scoresZattention_probsZ context_layerZnew_context_layer_shapeoutputsr(r(r)ras$     zViltSelfAttention.forward)NNF)r!r"r#r-rrrr(r(r?r)rIs rcs@eZdZdZeddfdd ZejejejdddZZ S) ViltSelfOutputz The residual connection is defined in ViltLayer instead of here (as is the case with other models), due to the layernorm applied before each block. Nr=returncs.tt|j|j|_t|j|_dSN) r,r-rrr2denser:r;r<rr?r(r)r-s zViltSelfOutput.__init__r input_tensorrcCs||}||}|Srrr<r>rrr(r(r)rs  zViltSelfOutput.forward) r!r"r#r$rr-r%Tensorrrr(r(r?r)rsrcs.eZdZfddZddZd ddZZS) ViltAttentioncs*tt||_t||_t|_dSr)r,r-r attentionroutputset pruned_headsrr?r(r)r-s   zViltAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)NrrrE)lenrrrrrrrrrrrrunion)r>headsindexr(r(r) prune_headss zViltAttention.prune_headsNFcCs6|||||}||d|}|f|dd}|S)Nrr)rr)r>rrrrZ self_outputsattention_outputrr(r(r)rszViltAttention.forward)NNF)r!r"r#r-rrrr(r(r?r)rs rcs8eZdZeddfdd ZejejdddZZS)ViltIntermediateNrcsBtt|j|j|_t|jt r6t |j|_ n|j|_ dSr) r,r-rrr2intermediate_sizerrs hidden_actstrr intermediate_act_fnrr?r(r)r-s   zViltIntermediate.__init__)rrcCs||}||}|Sr)rrr>rr(r(r)rs  zViltIntermediate.forward r!r"r#rr-r%rrrr(r(r?r)rsrcs<eZdZeddfdd ZejejejdddZZS) ViltOutputNrcs.tt|j|j|_t|j|_ dSr) r,r-rrrr2rr:r;r<rr?r(r)r-s zViltOutput.__init__rcCs ||}||}||}|Srrrr(r(r)rs  zViltOutput.forwardrr(r(r?r)rsrcs*eZdZdZfddZdddZZS) ViltLayerz?This corresponds to the Block class in the timm implementation.csbt|j|_d|_t||_t||_t||_ t j |j |j d|_t j |j |j d|_dS)Nrr)r,r-Zchunk_size_feed_forwardZ seq_len_dimrrr intermediaterrrrr2rlayernorm_beforelayernorm_afterrr?r(r)r-s    zViltLayer.__init__NFc Csj|j|||||d}|d}|dd}|||j}||}||}|||}|f|}|S)Nrrr)rrrqrTrrr) r>rrrrZself_attention_outputsrrZ layer_outputr(r(r)rs     zViltLayer.forward)NNFrr(r(r?r)rs rcs&eZdZfddZdddZZS) ViltEncodercs:t|_tfddtjD|_d|_dS)Ncsg|] }tqSr()r)rJr}r=r(r)rPrZz(ViltEncoder.__init__..F) r,r-r=rZ ModuleListrangenum_hidden_layerslayergradient_checkpointingrr?rr)r-s  zViltEncoder.__init__NFTcs|rdnd}rdnd}t|jD]\} } |r8||f}|durH|| nd} |jr~|jr~fdd} tjj| | ||| } n| ||| } | d}r"|| df}q"|r||f}|stdd|||fDSt|||dS) Nr(csfdd}|S)Ncsg|RSrr()inputs)modulerr(r)custom_forwardszJViltEncoder.forward..create_custom_forward..custom_forwardr()rrr)rr)create_custom_forwardsz2ViltEncoder.forward..create_custom_forwardrrcss|]}|dur|VqdSrr(r]r(r(r) 4rZz&ViltEncoder.forward..)last_hidden_staterr ) rxrrZtrainingr%utils checkpointtupler )r>rrrroutput_hidden_states return_dictZall_hidden_statesZall_self_attentionsrZ layer_moduleZlayer_head_maskrZ layer_outputsr(rr)rs6      zViltEncoder.forward)NNFFTr!r"r#r-rrr(r(r?r)rs rc@s4eZdZdZeZdZdZdgZddZ d dd Z d S) ViltPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. viltTrcCst|tjtjfr@|jjjd|jjd|j dur|j j nft|tj r|jjjd|jjd|j dur|jj|j  n&t|tj r|j j |jjddS)zInitialize the weightsg)meanZstdNg?)rsrrrrcdataZnormal_r=Zinitializer_rangerZzero_r8rrZfill_)r>rr(r(r) _init_weightsGs     z!ViltPreTrainedModel._init_weightsFcCst|tr||_dSr)rsrr)r>rrr(r(r)_set_gradient_checkpointingWs z/ViltPreTrainedModel._set_gradient_checkpointingN)F) r!r"r#r$r config_classZbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesrrr(r(r(r)r<sraH This model is a PyTorch `torch.nn.Module `_ subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`ViltConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`BertTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`ViltImageProcessor.__call__`] for details. pixel_mask (`torch.LongTensor` of shape `(batch_size, height, width)`, *optional*): Mask to avoid performing attention on padding pixel values. Mask values selected in `[0, 1]`: - 1 for pixels that are real (i.e. **not masked**), - 0 for pixels that are padding (i.e. **masked**). `What are attention masks? <../glossary.html#attention-mask>`__ head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. a Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`BertTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) pixel_values (`torch.FloatTensor` of shape `(batch_size, num_images, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`ViltImageProcessor.__call__`] for details. pixel_mask (`torch.LongTensor` of shape `(batch_size, num_images, height, width)`, *optional*): Mask to avoid performing attention on padding pixel values. Mask values selected in `[0, 1]`: - 1 for pixels that are real (i.e. **not masked**), - 0 for pixels that are padding (i.e. **masked**). `What are attention masks? <../glossary.html#attention-mask>`__ head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. image_embeds (`torch.FloatTensor` of shape `(batch_size, num_images, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z^The bare ViLT Model transformer outputting raw hidden-states without any specific head on top.cseZdZdfdd ZddZddZdd Zeee e e d de e je e je e je e je e je e je e je e je ee ee ee eee ee jfd d dZZS) ViltModelTcsXt|||_t||_t||_tj|j |j d|_ |rFt |nd|_ |dSNr)r,r-r=r*rrencoderrrr2r layernorm ViltPoolerpooler post_init)r>r=add_pooling_layerr?r(r)r-s   zViltModel.__init__cCs |jjjSrrr/rr>r(r(r)get_input_embeddingsszViltModel.get_input_embeddingscCs||jj_dSrr)r>rr(r(r)set_input_embeddingsszViltModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)r>Zheads_to_prunerrr(r(r) _prune_headsszViltModel._prune_heads output_typerN) rrrr{r|rrrrrrrrc  Cs| dur | n|jj} | dur | n|jj} | dur4| n|jj} |durV|durVtdn4|durh|} n"|dur|dd} ntd| \}}|dur|jn|j}|durtj||f|d}|dur|durtdn|dur|durtd|dur|j dn|j d}||kr"td |durHtj||jj |jj f|d}| ||jj }|j |||||||| d \}}||| }|j|||| | | d }|d}||}|jdur||nd}| s||f|d dSt|||j|jd S)a Returns: Examples: ```python >>> from transformers import ViltProcessor, ViltModel >>> from PIL import Image >>> import requests >>> # prepare image and text >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "hello world" >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm") >>> model = ViltModel.from_pretrained("dandelin/vilt-b32-mlm") >>> inputs = processor(image, text, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timerRz5You have to specify either input_ids or inputs_embedsrSzFYou cannot specify both pixel_values and image_embeds at the same timez7You have to specify either pixel_values or image_embedsrzAThe text inputs and image inputs need to have the same batch size)r)rrrrrr)r pooler_outputrr )r=rruse_return_dictrrDrTr%ryrdrhZ get_head_maskrrZget_extended_attention_maskrrrr rr )r>rrrr{r|rrrrrrrrZtext_batch_sizerrTZimage_batch_sizeZembedding_outputZextended_attention_maskZencoder_outputssequence_output pooled_outputr(r(r)rsn'       zViltModel.forward)T) NNNNNNNNNNNN)r!r"r#r-rrrrVILT_INPUTS_DOCSTRINGrr _CONFIG_FOR_DOCrr% LongTensorr&rtboolrrrrr(r(r?r)rsB  rcs$eZdZfddZddZZS)rcs*tt|j|j|_t|_dSr)r,r-rrr2rZTanh activationrr?r(r)r-ks zViltPooler.__init__cCs(|dddf}||}||}|SrV)rr)r>rZfirst_token_tensorr r(r(r)rps  zViltPooler.forwardrr(r(r?r)rjs rzU ViLT Model with a language modeling head on top as done during pretraining. cseZdZdgZfddZddZddZee de e e d de eje eje eje eje eje eje eje eje eje ee ee eee eejfd d d ZZS)ViltForMaskedLMzmlm_score.decoder.biascs,t|t||_t||_|dSr)r,r-rr ViltMLMHead mlm_scorerrr?r(r)r-s   zViltForMaskedLM.__init__cCs|jjSrrdecoderrr(r(r)get_output_embeddingssz%ViltForMaskedLM.get_output_embeddingscCs ||j_dSrr)r>Znew_embeddingsr(r(r)set_output_embeddingssz%ViltForMaskedLM.set_output_embeddingszbatch_size, sequence_lengthrN rrrr{r|rrrlabelsrrrrc  Cs| dur | n|jj} |j||||||||| | | d } | dd\}}|durV|jdn|jd}|ddd|f|dd|df}}||}d}| durt}| |j} ||d|jj | d}| s|f| dd}|dur|f|S|St ||| j | j dS)a labels (*torch.LongTensor* of shape *(batch_size, sequence_length)*, *optional*): Labels for computing the masked language modeling loss. Indices should be in *[-100, 0, ..., config.vocab_size]* (see *input_ids* docstring) Tokens with indices set to *-100* are ignored (masked), the loss is only computed for the tokens with labels in *[0, ..., config.vocab_size]* Returns: Examples: ```python >>> from transformers import ViltProcessor, ViltForMaskedLM >>> import requests >>> from PIL import Image >>> import re >>> import torch >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "a bunch of [MASK] laying on a [MASK]." >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm") >>> model = ViltForMaskedLM.from_pretrained("dandelin/vilt-b32-mlm") >>> # prepare inputs >>> encoding = processor(image, text, return_tensors="pt") >>> # forward pass >>> outputs = model(**encoding) >>> tl = len(re.findall("\[MASK\]", text)) >>> inferred_token = [text] >>> # gradually fill in the MASK tokens, one by one >>> with torch.no_grad(): ... for i in range(tl): ... encoded = processor.tokenizer(inferred_token) ... input_ids = torch.tensor(encoded.input_ids) ... encoded = encoded["input_ids"][0][1:-1] ... outputs = model(input_ids=input_ids, pixel_values=encoding.pixel_values) ... mlm_logits = outputs.logits[0] # shape (seq_len, vocab_size) ... # only take into account text features (minus CLS and SEP token) ... mlm_logits = mlm_logits[1 : input_ids.shape[1] - 1, :] ... mlm_values, mlm_ids = mlm_logits.softmax(dim=-1).max(dim=-1) ... # only take into account text ... mlm_values[torch.tensor(encoded) != 103] = 0 ... select = mlm_values.argmax().item() ... encoded[select] = mlm_ids[select].item() ... inferred_token = [processor.decode(encoded)] >>> selected_token = "" >>> encoded = processor.tokenizer(inferred_token) >>> output = processor.decode(encoded.input_ids[0], skip_special_tokens=True) >>> print(output) a bunch of cats laying on a couch. ```N rrr{r|rrrrrrrBrrRrrrr ) r=r rrdrrrqrTrkrr rr )r>rrrr{r|rrrrrrrrr r Z text_seq_lenZ text_featuresr}Z mlm_logitsZmasked_lm_lossloss_fctrr(r(r)rs@I*  zViltForMaskedLM.forward) NNNNNNNNNNNN)r!r"r#Z_keys_to_ignore_on_load_missingr-rrrr formatrr rrr%rr&rrrrrr(r(r?r)rysB   rcs$eZdZfddZddZZS)ViltPredictionHeadTransformcsVtt|j|j|_t|jtr6t |j|_ n|j|_ tj |j|j d|_ dSr) r,r-rrr2rrsrrr transform_act_fnrrrr?r(r)r-s   z$ViltPredictionHeadTransform.__init__cCs"||}||}||}|Sr)rr rrr(r(r)rs   z#ViltPredictionHeadTransform.forwardrr(r(r?r)rs rcs&eZdZdfdd ZddZZS)rNcsbt||_t||_tj|j|jdd|_ t t |j|_ |durT||j _|j |j _ dS)NFr)r,r-r=r transformrrr2rrr0r%r1rrc)r>r=rcr?r(r)r-s  zViltMLMHead.__init__cCs||}||}|Sr)r!r)r>rr(r(r)r"s  zViltMLMHead.forward)Nrr(r(r?r)rs rz Vilt Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the [CLS] token) for visual question answering, e.g. for VQAv2. cseZdZfddZeeeeedde e j e e j e e j e e j e e j e e j e e j e e j e e j e e e e e e eeee j fd ddZZS) ViltForQuestionAnsweringc slt||j|_t||_tt|j|jdt |jdt t|jd|j|_ | dS)NrB) r,r- num_labelsrrr Sequentialrr2rGELU classifierrrr?r(r)r-0s  z!ViltForQuestionAnswering.__init__rNrc  Cs| dur | n|jj} |j||||||||| | | d } | r>| jn| d}||}d}| dur| |j} tj || | j d}| s|f| dd}|dur|f|S|St ||| j | j dS)a labels (`torch.FloatTensor` of shape `(batch_size, num_labels)`, *optional*): Labels for computing the visual question answering loss. This tensor must be either a one-hot encoding of all answers that are applicable for a given example in the batch, or a soft encoding indicating which answers are applicable, where 1.0 is the highest score. Returns: Examples: ```python >>> from transformers import ViltProcessor, ViltForQuestionAnswering >>> import requests >>> from PIL import Image >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "How many cats are there?" >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-vqa") >>> model = ViltForQuestionAnswering.from_pretrained("dandelin/vilt-b32-finetuned-vqa") >>> # prepare inputs >>> encoding = processor(image, text, return_tensors="pt") >>> # forward pass >>> outputs = model(**encoding) >>> logits = outputs.logits >>> idx = logits.argmax(-1).item() >>> print("Predicted answer:", model.config.id2label[idx]) Predicted answer: 2 ```NrrrBr)r=r rr r&rqrTrrGZ binary_cross_entropy_with_logitsrdrrr r>rrrr{r|rrrrrrrrr rrrr(r(r)rAs:1  z ViltForQuestionAnswering.forward) NNNNNNNNNNNNr!r"r#r-rr rrrrr%rr&rrrrrr(r(r?r)r"(s<  r"z Vilt Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the [CLS] token) for image-to-text or text-to-image retrieval, e.g. MSCOCO and F30K. cseZdZfddZeeeeedde e j e e j e e j e e j e e j e e j e e j e e j e e j e e e e e e eeee j fd ddZZS) ViltForImageAndTextRetrievalcs2t|t||_t|jd|_|dSr+) r,r-rrrrr2 rank_outputrrr?r(r)r-s  z%ViltForImageAndTextRetrieval.__init__rNrc  Cs| dur | n|jj} |j||||||||| | | d } | r>| jn| d}||}d}| durp| |j} td| s|f| dd}|dur|f|S|St||| j | j dS)a' labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels are currently not supported. Returns: Examples: ```python >>> from transformers import ViltProcessor, ViltForImageAndTextRetrieval >>> import requests >>> from PIL import Image >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> texts = ["An image of two cats chilling on a couch", "A football player scoring a goal"] >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-coco") >>> model = ViltForImageAndTextRetrieval.from_pretrained("dandelin/vilt-b32-finetuned-coco") >>> # forward pass >>> scores = dict() >>> for text in texts: ... # prepare inputs ... encoding = processor(image, text, return_tensors="pt") ... outputs = model(**encoding) ... scores[text] = outputs.logits[0, :].item() ```NrrzTraining is not yet supported.rBr) r=r rr r*rqrTNotImplementedErrorrrr r'r(r(r)rs:-  z$ViltForImageAndTextRetrieval.forward) NNNNNNNNNNNNr(r(r(r?r)r)s<  r)zq Vilt Model transformer with a classifier head on top for natural language visual reasoning, e.g. NLVR2. cseZdZfddZeeeeedde e j e e j e e j e e j e e j e e j e e j e e j e e j e e e e e e eeee j fd ddZZS) "ViltForImagesAndTextClassificationc svt||j|_t||_|j}tt|j ||j |t |j |t t|j ||j|_ | dSr)r,r-r#rr num_imagesrr$rr2rr%r&r)r>r=r-r?r(r)r-s  z+ViltForImagesAndTextClassification.__init__rNrc CsN| dur | n|jj} | dur | n|jj} | dur4| n|jj} |durX|jdkrX|d}|durt|jdkrt|d}|dur|jdnd} | dur|dur|jdnd} | |jjkrtdg}| rgnd}| rgnd}t | D]}|j ||||dur|dd|ddddddfnd|durB|dd|ddddfnd|||durn|dd|ddddfnd|d| | | d }| r|j n|d}| || r| |j | r| |jqtj|dd}||}d}| durt}| |j} ||d|j| d}| s>|||f}|dur:|f|S|St||||d S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Binary classification labels. Returns: Examples: ```python >>> from transformers import ViltProcessor, ViltForImagesAndTextClassification >>> import requests >>> from PIL import Image >>> image1 = Image.open(requests.get("https://lil.nlp.cornell.edu/nlvr/exs/ex0_0.jpg", stream=True).raw) >>> image2 = Image.open(requests.get("https://lil.nlp.cornell.edu/nlvr/exs/ex0_1.jpg", stream=True).raw) >>> text = "The left image contains twice the number of dogs as the right image." >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-nlvr2") >>> model = ViltForImagesAndTextClassification.from_pretrained("dandelin/vilt-b32-finetuned-nlvr2") >>> # prepare inputs >>> encoding = processor([image1, image2], text, return_tensors="pt") >>> # forward pass >>> outputs = model(input_ids=encoding.input_ids, pixel_values=encoding.pixel_values.unsqueeze(0)) >>> logits = outputs.logits >>> idx = logits.argmax(-1).item() >>> print("Predicted answer:", model.config.id2label[idx]) Predicted answer: True ```Nrr z\Make sure to match the number of images in the model with the number of images in the input.) rrr{r|rrrrrrrrRrEr)r=rrr ndimZ unsqueezerdr-rrrr rzrr r%rlr&rrqrTrkr#r)r>rrrr{r|rrrrrrrr-Zpooler_outputsrr rrr r rrrrr(r(r)rsn/      .((      z*ViltForImagesAndTextClassification.forward) NNNNNNNNNNNN)r!r"r#r-rr rrrrr%rr&rrrrrr(r(r?r)r,s<  r,z ViLT Model with a token classification head on top (a linear layer on top of the final hidden-states of the text tokens) e.g. for Named-Entity-Recognition (NER) tasks. cseZdZdgZfddZeeeee dd e e j e e j e e j e e j e e j e e j e e j e e j e e j e ee ee eeeee j fd ddZZS) ViltForTokenClassificationrcsNt||j|_t|dd|_t|j|_t |j |j|_ | dS)NF)r) r,r-r#rrrr:r;r<rr2r&rrr?r(r)r-s  z#ViltForTokenClassification.__init__rNrc  Cs| dur | n|jj} |j||||||||| | | d } | d}|durN|jdn|jd}||}||ddd|f}d}| durt}| |j} || d|j | d}| s|f| dd}|dur|f|S|St ||| j | j dS)z labels (`torch.LongTensor` of shape `(batch_size, text_sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Returns: NrrrrRrBr)r=r rrdr<r&rrqrTrkr#rrr )r>rrrr{r|rrrrrrrrr Ztext_input_sizerrrrr(r(r)rs@  z"ViltForTokenClassification.forward) NNNNNNNNNNNN)r!r"r#Z"_keys_to_ignore_on_load_unexpectedr-rr rrrrr%rr&rrrrrr(r(r?r)r0s>  r0)Hr$collections.abcrrZ dataclassesrtypingrrrrr%Ztorch.utils.checkpointrZtorch.nnrZ activationsr Zmodeling_outputsr r r rrrZmodeling_utilsrZ pytorch_utilsrrrrrrrrrZconfiguration_viltrZ get_loggerr!loggerwarning __version__rZ_CHECKPOINT_FOR_DOCZ"VILT_PRETRAINED_MODEL_ARCHIVE_LISTrModuler*r.r4rrrrrrrrZVILT_START_DOCSTRINGr Z4VILT_IMAGES_AND_TEXT_CLASSIFICATION_INPUTS_DOCSTRINGrrrrrr"r)r,r0r(r(r(r)s         7=#&;  78j_