a  d/@sdZddlZddlmZddlmZmZmZmZm Z ddl Z ddl Z ddl Z ddl mZddlmZddlmZmZdd lmZdd lmZmZmZmZmZmZd d lmZmZm Z erdd l!m"Z"e#e$Z%dZ&gdZ'dQe j(e j)ee*dddZ+e j(e j(dddZ,e j(e j(dddZ-eGdddeZ.e j(e j(dddZ/e j(e j(ddd Z0e j(e j(e j(d!d"d#Z1d$d%Z2eGd&d'd'eZ3eGd(d)d)eZ4Gd*d+d+ej5Z6Gd,d-d-ej5Z7Gd.d/d/ej5Z8Gd0d1d1ej5Z9Gd2d3d3ej5Z:Gd4d5d5eZ;d6Zd9Z?d:Z@d;ZAGdd?d@ZEGdAdBdBej5ZFGdCdDdDe;ZGGdEdFdFej5ZHGdGdHdHe;ZIee<GdIdJdJe;ZJGdKdLdLej5ZKGdMdNdNej5ZLGdOdPdPe;ZMdS)Sz PyTorch OWL-ViT model.N) dataclass)AnyDictOptionalTupleUnion)nn)ACT2FN)BaseModelOutputBaseModelOutputWithPooling)PreTrainedModel) ModelOutputadd_start_docstrings%add_start_docstrings_to_model_forwardis_vision_availableloggingreplace_return_docstrings) OwlViTConfigOwlViTTextConfigOwlViTVisionConfig)center_to_corners_formatgoogle/owlvit-base-patch32)rzgoogle/owlvit-base-patch16zgoogle/owlvit-large-patch14)maskdtypetgt_lencCsj|\}}|dur|n|}|ddddddf|d|||}d|}||tjt|jS)z_ Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`. Nr?)sizeexpandtoZ masked_filltorchboolfinfomin)rrrbszsrc_lenZ expanded_maskZ inverted_maskr's/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/transformers/models/owlvit/modeling_owlvit.py _expand_mask:s  *r))logitsreturncCstj|tjt||jdS)Ndevice)r functionalZ cross_entropyr!arangelenr-)r*r'r'r(contrastive_lossIsr1) similarityr+cCs t|}t|}||dS)Ng@)r1t)r2Z caption_lossZ image_lossr'r'r( owlvit_lossNs r4c@seZdZUdZdZeejed<dZ ejed<dZ ejed<dZ ejed<dZ ejed<dZ eed<dZeed <eed d d ZdS) OwlViTOutputa% Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. text_embeds (`torch.FloatTensor` of shape `(batch_size * num_max_text_queries, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`OwlViTTextModel`]. image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`OwlViTVisionModel`]. text_model_output (Tuple[`BaseModelOutputWithPooling`]): The output of the [`OwlViTTextModel`]. vision_model_output (`BaseModelOutputWithPooling`): The output of the [`OwlViTVisionModel`]. Nlosslogits_per_imagelogits_per_text text_embeds image_embedstext_model_outputvision_model_outputr+cstfddDS)Nc3s,|]$}|dvr|n t|VqdS)r;r<Ngetattrto_tuple.0kselfr'r( tsz(OwlViTOutput.to_tuple..tuplekeysrEr'rEr(rAss zOwlViTOutput.to_tuple)__name__ __module__ __qualname____doc__r6rr! FloatTensor__annotations__r7r8r9r:r;r r<rrrAr'r'r'r(r5Ts   r5)r3r+cCsH|r&|jtjtjfvr|S|S|jtjtjfvr<|S|SdSN) Zis_floating_pointrr!float32float64floatint32int64int)r3r'r'r(_upcast{srX)boxesr+cCsHt|}|dddf|dddf|dddf|dddfS)a Computes the area of a set of bounding boxes, which are specified by its (x1, y1, x2, y2) coordinates. Args: boxes (`torch.FloatTensor` of shape `(number_of_boxes, 4)`): Boxes for which the area will be computed. They are expected to be in (x1, y1, x2, y2) format with `0 <= x1 < x2` and `0 <= y1 < y2`. Returns: `torch.FloatTensor`: a tensor containing the area for each box. Nrr r)rX)rYr'r'r(box_areas r[)boxes1boxes2r+c Cst|}t|}t|dddddf|ddddf}t|dddddf|ddddf}||jdd}|dddddf|dddddf}|dddf||}||} | |fS)NrZrr$r)r[r!maxr$clamp) r\r]Zarea1Zarea2Zleft_topZ right_bottom width_heightinterunioniour'r'r(box_ious..,recCs*|ddddf|ddddfks:td||ddddf|ddddfksttd|t||\}}t|dddddf|ddddf}t|dddddf|ddddf}||jdd}|dddddf|dddddf}||||S)z Generalized IoU from https://giou.stanford.edu/. The boxes should be in [x0, y0, x1, y1] (corner) format. Returns: `torch.FloatTensor`: a [N, M] pairwise matrix, where N = len(boxes1) and M = len(boxes2) NrZzr?rBrEr'r(rGsz7OwlViTObjectDetectionOutput.to_tuple..rHrEr'rEr(rAs z$OwlViTObjectDetectionOutput.to_tuple)rKrLrMrNr6rr!rOrPrkrr*rlr9r:rmr;r r<rrrAr'r'r'r(rjs   rjc@seZdZUdZdZejed<dZejed<dZ ejed<dZ ejed<dZ ejed<dZ ejed<dZ eed <dZeed <eed d d ZdS)&OwlViTImageGuidedObjectDetectionOutputa Output type of [`OwlViTForObjectDetection.image_guided_detection`]. Args: logits (`torch.FloatTensor` of shape `(batch_size, num_patches, num_queries)`): Classification logits (including no-object) for all queries. target_pred_boxes (`torch.FloatTensor` of shape `(batch_size, num_patches, 4)`): Normalized boxes coordinates for all queries, represented as (center_x, center_y, width, height). These values are normalized in [0, 1], relative to the size of each individual target image in the batch (disregarding possible padding). You can use [`~OwlViTImageProcessor.post_process_object_detection`] to retrieve the unnormalized bounding boxes. query_pred_boxes (`torch.FloatTensor` of shape `(batch_size, num_patches, 4)`): Normalized boxes coordinates for all queries, represented as (center_x, center_y, width, height). These values are normalized in [0, 1], relative to the size of each individual query image in the batch (disregarding possible padding). You can use [`~OwlViTImageProcessor.post_process_object_detection`] to retrieve the unnormalized bounding boxes. image_embeds (`torch.FloatTensor` of shape `(batch_size, patch_size, patch_size, output_dim`): Pooled output of [`OwlViTVisionModel`]. OWL-ViT represents images as a set of image patches and computes image embeddings for each patch. query_image_embeds (`torch.FloatTensor` of shape `(batch_size, patch_size, patch_size, output_dim`): Pooled output of [`OwlViTVisionModel`]. OWL-ViT represents images as a set of image patches and computes image embeddings for each patch. class_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`): Class embeddings of all image patches. OWL-ViT represents images as a set of image patches where the total number of patches is (image_size / patch_size)**2. text_model_output (Tuple[`BaseModelOutputWithPooling`]): The output of the [`OwlViTTextModel`]. vision_model_output (`BaseModelOutputWithPooling`): The output of the [`OwlViTVisionModel`]. Nr*r:query_image_embedstarget_pred_boxesquery_pred_boxesrmr;r<r=cstfddDS)Nc3s,|]$}|dvr|n t|VqdSr>r?rBrEr'r(rGszBOwlViTImageGuidedObjectDetectionOutput.to_tuple..rHrEr'rEr(rAs z/OwlViTImageGuidedObjectDetectionOutput.to_tuple)rKrLrMrNr*r!rOrPr:rorprqrmr;r r<rrrAr'r'r'r(rns   rncs6eZdZedfdd ZejejdddZZ S)OwlViTVisionEmbeddingsconfigcst||_|j|_tt|j|_ tj |j |j|j |j dd|_ |j|j d|_|jd|_t|j|j|_|dt|jddS)NF)Z in_channelsZ out_channelsZ kernel_sizeZstridebiasrZr position_idsr)super__init__rt hidden_size embed_dimr Parameterr!Zrandnclass_embeddingZConv2dZ num_channelsZ patch_sizepatch_embeddingZ image_size num_patchesZ num_positions Embeddingposition_embeddingregister_bufferr/rrFrt __class__r'r(rz"s  zOwlViTVisionEmbeddings.__init__) pixel_valuesr+cCs\|jd}||}|ddd}|j|dd}tj||gdd}|||j }|S)NrrZrrxdim) shaperflatten transposer~rr!catrrv)rFr batch_sizeZ patch_embedsrm embeddingsr'r'r(forward5s  zOwlViTVisionEmbeddings.forward) rKrLrMrrzr!rOTensorr __classcell__r'r'rr(rr!srrcsLeZdZedfdd Zdeejeejeejej dddZ Z S) OwlViTTextEmbeddingsrscsLtt|j|j|_t|j|j|_| dt |j ddS)Nrvrw) ryrzrrZ vocab_sizer{token_embeddingZmax_position_embeddingsrrr!r/rrrr'r(rzBs zOwlViTTextEmbeddings.__init__N) input_idsrv inputs_embedsr+cCsb|dur|jdn|jd}|dur:|jddd|f}|durL||}||}||}|S)Nrx)rrvrr)rFrrvrZ seq_lengthZposition_embeddingsrr'r'r(rJs  zOwlViTTextEmbeddings.forward)NNN) rKrLrMrrzrr! LongTensorrOrrrr'r'rr(rAs rc szeZdZdZfddZejeedddZd eje eje eje e e eje eje e ejfd d d Z Z S) OwlViTAttentionz=Multi-headed attention from 'Attention Is All You Need' papercst||_|j|_|j|_|j|j|_|j|j|jkrZtd|jd|jd|jd|_ |j |_ t |j|j|_t |j|j|_t |j|j|_t |j|j|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).)ryrzrtr{r|Znum_attention_heads num_headshead_dimrgscaleZattention_dropoutdropoutrLineark_projv_projq_projout_projrrr'r(rzas"   zOwlViTAttention.__init__)tensorseq_lenr%cCs ||||j|jddS)NrrZ)viewrrr contiguous)rFrrr%r'r'r(_shapetszOwlViTAttention._shapeNF hidden_statesattention_maskcausal_attention_maskoutput_attentionsr+cCs|\}}}|||j}|||d|} |||d|} ||jd|jf} ||||j| }| j| } | j| } | d} t || dd} | ||j|| fkrt d||j|| fd| |durD||d|| fkrt d|d|| fd|| ||j|| |} | ||j|| } |dur||d|| fkrt d|d|| fd|| ||j|| |} | ||j|| } t jj| dd} |r| ||j|| }|||j|| } nd}t jj| |j|jd }|| j}t || }|||j||jfkr^t d ||j||jfd||||j||j}| dd}||||}||}||fS) z#Input shape: Batch x Time x ChannelrxrrZz$Attention weights should be of size z , but is Nz!Attention mask should be of size r)ptrainingz `attn_output` should be of size )rrrrrrrrrr!Zbmmrrgrr.Zsoftmaxrrr rreshaper)rFrrrrr%rr|Z query_statesZ key_statesZ value_statesZ proj_shaper& attn_weightsZattn_weights_reshapedZ attn_probsZ attn_outputr'r'r(rwsf          zOwlViTAttention.forward)NNF)rKrLrMrNrzr!rrWrrr"rrrr'r'rr(r^s rcs0eZdZfddZejejdddZZS) OwlViTMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSrQ) ryrzrtr Z hidden_act activation_fnrrr{Zintermediate_sizefc1fc2rrr'r(rzs   zOwlViTMLP.__init__)rr+cCs"||}||}||}|SrQ)rrr)rFrr'r'r(rs   zOwlViTMLP.forward)rKrLrMrzr!rrrr'r'rr(rs rcsJeZdZedfdd Zdejejejeee ej dddZ Z S) OwlViTEncoderLayerrscsRt|j|_t||_tj|j|jd|_ t ||_ tj|j|jd|_ dSN)Zeps) ryrzr{r|r self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2rrr'r(rzs    zOwlViTEncoderLayer.__init__FrcCsd|}||}|j||||d\}}||}|}||}||}||}|f}|r`||f7}|S)aI Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. `(config.encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rrrr)rrrr)rFrrrrZresidualroutputsr'r'r(rs"     zOwlViTEncoderLayer.forward)F) rKrLrMrrzr!rrr"rrOrrr'r'rr(rs rc@s:eZdZdZeZdZdZdgZdgZ ddZ d d d Z d S) OwlViTPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. owlvitTrvrcCsX|jj}t|trF|jjjjd|dd|jjjjd|ddnt|t r|jj}t j j|j d|j d|dt j j|jj|jj|dt j j|jj|jj|dnTt|trD|jj}|j dd|jjd|}|j d|}t j j|jj|dt j j|jj|dt j j|jj|dt j j|jj|dnt|tr|jj}|jjdd|jjd|}d|jjd|}t j j|jj|dt j j|jj|dnPt|trt j j|jj|jd|jjdt j j|jj|jd|jjdt|t jr.|j j!|jj"dt|t j#rT|j durT|j j!dS) zInitialize the weightsg{Gz?)meanstdr)rrZrN)$rtZinitializer_factor isinstancerrweightdataZnormal_rrrrinitr~r|rZinitializer_rangernum_hidden_layersrrrrrr{rr OwlViTModeltext_projectiontext_embed_dimvisual_projectionvision_embed_dimrruZzero_Zfill_r)rFmoduleZfactorZ in_proj_stdZ out_proj_stdZfc_stdr'r'r( _init_weightssL       z#OwlViTPreTrainedModel._init_weightsFcCst|tr||_dSrQ)r OwlViTEncodergradient_checkpointing)rFrvaluer'r'r(_set_gradient_checkpointingBs z1OwlViTPreTrainedModel._set_gradient_checkpointingN)F) rKrLrMrNr config_classZbase_model_prefixZsupports_gradient_checkpointingZ_keys_to_ignore_on_load_missingZ_no_split_modulesrrr'r'r'r(r s*raP Parameters: This model is a PyTorch [torch.nn.Module](https: //pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. config ([`OwlViTConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`torch.LongTensor` of shape `(batch_size * num_max_text_queries, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, num_max_text_queries, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. a Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. a Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. input_ids (`torch.LongTensor` of shape `(batch_size * num_max_text_queries, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids). attention_mask (`torch.Tensor` of shape `(batch_size, num_max_text_queries, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_hidden_states (`bool`, *optional*): Whether or not to return the last hidden state. See `text_model_last_hidden_state` and `vision_model_last_hidden_state` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. a_ Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. query_pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values of query image(s) to be detected. Pass in one query image per target image. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. c s`eZdZdZedfdd Zd eejeejee ee ee e e e fdddZ ZS) rz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`OwlViTEncoderLayer`]. Args: config: OwlViTConfig rscs4ttfddtjD|_d|_dS)Ncsg|] }tqSr')r)rC_rsr'r( z*OwlViTEncoder.__init__..F)ryrzrZ ModuleListrangerlayersrrrrsr(rzs  zOwlViTEncoder.__init__N)rrroutput_hidden_states return_dictr+c sdur n|jj|dur |n|jj}|dur4|n|jj}|rDdnd}rPdnd}|} |jD]n} |rp|| f}|jr|jrfdd} tjj | | | ||} n| | ||d} | d} r^|| df}q^|r|| f}|st dd | ||fDSt | ||d S) a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`). attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Causal mask for the text model. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr'csfdd}|S)Ncsg|RSrQr')inputs)rrr'r(custom_forwardszLOwlViTEncoder.forward..create_custom_forward..custom_forwardr')rrr)rr(create_custom_forwardsz4OwlViTEncoder.forward..create_custom_forwardrrrcss|]}|dur|VqdSrQr')rCvr'r'r(rG rz(OwlViTEncoder.forward..)last_hidden_stater attentions) rtrruse_return_dictrrrr!utils checkpointrIr ) rFrrrrrrZencoder_statesZall_attentionsrZ encoder_layerrZ layer_outputsr'rr(rsD       zOwlViTEncoder.forward)NNNNN)rKrLrMrNrrzrr!rr"rrr rrr'r'rr(rs r)input_ids_shaperr-past_key_values_lengthcCs|\}}tj||ftjt|j|d|d}tj|d|d}|||d|ddkd| |}|dkrtj tj ||||d|gdd}|ddddddf |d|||S)zB Make causal mask used for bi-directional self-attention. r,rxrr)rr-rN) r!fullrr#r$r/rZ masked_fill_rr rzerosr)rrr-rr%rrZ mask_condr'r'r(_make_causal_masks&"  rc steZdZedfdd Zeeeeedd e j e e j e e j e e e e e e e eefdddZZS) OwlViTTextTransformerrscs@t||_|j}t||_t||_tj ||j d|_ dSr) ryrzrtr{rrrencoderrrrfinal_layer_norm)rFrtr|rr'r(rz%s    zOwlViTTextTransformer.__init__ output_typerN)rrrvrrrr+c Cs|dur |n|jj}|dur |n|jj}|dur4|n|jj}|}|d|d}|j||d}t||j|j d} |durt ||j}|j ||| |||d} | d} | | } | t j| jd| j d|t jjdd| j f} |s| | f| ddSt| | | j| jd S)  Returns: Nrx)rrvr,)rrrrrrrrrrZ pooler_outputrr)rtrrrrrrrrr-r)rrr!r/rr rWZargmaxr rr) rFrrrvrrrZ input_shaperrencoder_outputsr pooled_outputr'r'r(r-sB   zOwlViTTextTransformer.forward)NNNNN)rKrLrMrrzrOWLVIT_TEXT_INPUTS_DOCSTRINGrr r!rrr"rrrrr'r'rr(r$s"  rc seZdZeZedfdd ZejdddZddZ e e e e ed dejeejeeeeeeeee fd d d ZZS)OwlViTTextModelrscs"t|t||_|dSrQ)ryrzr text_model post_initrrr'r(rzos  zOwlViTTextModel.__init__r=cCs |jjjSrQrrrrEr'r'r(get_input_embeddingsusz$OwlViTTextModel.get_input_embeddingscCs||jj_dSrQr)rFrr'r'r(set_input_embeddingsxsz$OwlViTTextModel.set_input_embeddingsrNrrrrrr+cCs|j|||||dS)ay Returns: Examples: ```python >>> from transformers import AutoProcessor, OwlViTTextModel >>> model = OwlViTTextModel.from_pretrained("google/owlvit-base-patch32") >>> processor = AutoProcessor.from_pretrained("google/owlvit-base-patch32") >>> inputs = processor( ... text=[["a photo of a cat", "a photo of a dog"], ["photo of a astranaut"]], return_tensors="pt" ... ) >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled (EOS token) states ```rrrrr)r)rFrrrrrr'r'r(r{szOwlViTTextModel.forward)NNNN)rKrLrMrrrzrModulerrrrrr r!rrr"rrrrr'r'rr(rls$  rc sdeZdZedfdd Zeeeeedd e j e e e e e e e eefdddZZS) OwlViTVisionTransformerrscsPt||_t||_tj|j|jd|_ t ||_ tj|j|jd|_ dSr) ryrzrtrrrrrr{r pre_layernormrrpost_layernormrrr'r(rzs    z OwlViTVisionTransformer.__init__rNrrrrr+c Cs|dur |n|jj}|dur |n|jj}|dur4|n|jj}|jjjj}||}||}| |}|j ||||d}|d}|dddddf} | | } |s|| f|ddSt || |j |jdS)rN)rrrrrrr)rtrrrrrrrr rrrr rr) rFrrrrZexpected_input_dtyperrrrr'r'r(rs2      zOwlViTVisionTransformer.forward)NNN)rKrLrMrrzrOWLVIT_VISION_INPUTS_DOCSTRINGrr r!rOrr"rrrrr'r'rr(rs   rc seZdZeZdZedfdd ZejdddZ e e e e edd eejeeeeeeeee fd d d ZZS)OwlViTVisionModelrrscs"t|t||_|dSrQ)ryrzr vision_modelrrrr'r(rzs  zOwlViTVisionModel.__init__r=cCs |jjjSrQ)rrrrEr'r'r(rsz&OwlViTVisionModel.get_input_embeddingsrNrcCs|j||||dS)a Returns: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, OwlViTVisionModel >>> model = OwlViTVisionModel.from_pretrained("google/owlvit-base-patch32") >>> processor = AutoProcessor.from_pretrained("google/owlvit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled CLS states ```rrrr)r)rFrrrrr'r'r(rs zOwlViTVisionModel.forward)NNNN)rKrLrMrrZmain_input_namerzrrrrrrr rr!rOr"rrrrr'r'rr(rs"  rcseZdZeZedfdd Zeedee j ee j ee ee ee e j dddZ eedee j ee ee ee e j dd d Zeeeeed dee jee j ee j ee ee ee ee ee eeefd d dZZS)rrscst|t|jts.tdt|jdt|jtsPtdt|jd|j}|j}|j |_ |j |_ |j |_ t ||_t||_tj|j |j dd|_tj|j |j dd|_ttg|j|_|dS)NzMconfig.text_config is expected to be of type OwlViTTextConfig but is of type .zQconfig.vision_config is expected to be of type OwlViTVisionConfig but is of type F)ru)ryrzr text_configrrgtype vision_configrZprojection_dimr{rrrrrrrrrrr}r!ZonesZlogit_scale_init_value logit_scaler)rFrtrr rr'r(rzs0     zOwlViTModel.__init__Nrc Cs:|dur |n|jj}|j|||d}|d}||}|S)a Returns: text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`OwlViTTextModel`]. Examples: ```python >>> from transformers import AutoProcessor, OwlViTModel >>> model = OwlViTModel.from_pretrained("google/owlvit-base-patch32") >>> processor = AutoProcessor.from_pretrained("google/owlvit-base-patch32") >>> inputs = processor( ... text=[["a photo of a cat", "a photo of a dog"], ["photo of a astranaut"]], return_tensors="pt" ... ) >>> text_features = model.get_text_features(**inputs) ```N)rrrr)rtrrr) rFrrrrrZ text_outputrZ text_featuresr'r'r(get_text_features0s  zOwlViTModel.get_text_featuresrcCsd|dur |n|jj}|dur |n|jj}|dur4|n|jj}|j||||d}|d}||}|S)aB Returns: image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`OwlViTVisionModel`]. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, OwlViTModel >>> model = OwlViTModel.from_pretrained("google/owlvit-base-patch32") >>> processor = AutoProcessor.from_pretrained("google/owlvit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> image_features = model.get_image_features(**inputs) ```Nrr)rtrrrrr)rFrrrrvision_outputsrimage_featuresr'r'r(get_image_featuresSs zOwlViTModel.get_image_featuresr) rrr return_lossrrreturn_base_image_embedsrr+c  Csd|dur |n|jj}|dur |n|jj}|dur4|n|jj}|j||||d} |j|||||d} | d} || } | d} || } | tj j | dddd} | tj j | dddd} |j | j}t| | |}|}d}|rt|}|rtd t| d }|j|} n| } |sN||| | | | f}|durJ|f|S|St|||| | | | d S) a[ Returns: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, OwlViTModel >>> model = OwlViTModel.from_pretrained("google/owlvit-base-patch32") >>> processor = AutoProcessor.from_pretrained("google/owlvit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(text=[["a photo of a cat", "a photo of a dog"]], images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score >>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities ```NrrrrZrxT)ordrkeepdimz`return_base_image_embeds` is deprecated and will be removed in v4.27 of Transformers, one can obtain the base (unprojected) image embeddings from outputs.vision_model_output.r)r6r7r8r9r:r;r<)rtrrrrrrrr!linalgnormr expr r-matmulr3r4warningswarn FutureWarningrr5)rFrrrrrrrrr  text_outputsr9r:Ztext_embeds_normr r8r7r6routputr'r'r(rsb   zOwlViTModel.forward)NNNNN)NNNN)NNNNNNNN)rKrLrMrrrzrrrr!rr"rOr rrOWLVIT_INPUTS_DOCSTRINGrr5rrrrrr'r'rr(r sb ",  rcs6eZdZedfdd ZejejdddZZ S)OwlViTBoxPredictionHeadrscsJt|jj}t|||_t|||_t|_ t|d|_ dS)N) ryrzr r{rrdense0dense1ZGELUgeludense2)rFrtwidthrr'r(rzs   z OwlViTBoxPredictionHead.__init__)r r+cCs6||}||}||}||}||}|SrQ)rr!r r")rFr rr'r'r(rs      zOwlViTBoxPredictionHead.forward) rKrLrMrrzr!rrOrrr'r'rr(rs rcsJeZdZedfdd Zejeejeeje ejdddZ Z S)OwlViTClassPredictionHeadrscsZt|jj}|jj|_t|j||_t|jd|_ t|jd|_ t |_ dS)Nr) ryrzrr{r  query_dimrrr logit_shiftr ZELUelu)rFrtZout_dimrr'r(rzs  z"OwlViTClassPredictionHead.__init__)r: query_embeds query_maskr+c Cs||}|durJ|j}|jdd\}}t|||jf|}||fS|tjj|dddd}|tjj|dddd}t d||}| |} | |} | | d} || | }|dur|j dkrtj|dd }|tj}t|d kd |}|tj}||fS) NrZrxT)rrgư>z...pd,...qd->...pqrrrrg.)rr-rr!rr%r rreinsumr&r r'ndimZ unsqueezerSwhererR) rFr:r(r)image_class_embedsr-rr pred_logitsr&r r'r'r(rs(        z!OwlViTClassPredictionHead.forward) rKrLrMrrzr!rOrrrrrr'r'rr(r$s  r$c seZdZeZedfdd ZejdddZejejddd Z ejejejd d d Z d!eje eje ej e ejdddZd"ej ejej e ee ee ejdddZd#eje ee ee ejdddZejejejdddZeeeeedd$eje eje ee ee eedddZeeeeedd%ej eje ej e ee ee eeddd ZZS)&OwlViTForObjectDetectionrscsPt|t||_t||_t||_tj |j j |j j d|_ t|_dSr)ryrzrrr$ class_headrbox_headrrr r{r layer_normZSigmoidsigmoidrrr'r(rz*s     z!OwlViTForObjectDetection.__init__) feature_mapc Cs|jdkstd|j}|jd}tjttd|dtd|dddtj }|t ||gtj }| |jd|jd|jd}t ||}|S)NrzJExpected input shape is [batch_size, num_patches, num_patches, hidden_dim]rrxZaxisrrZ)r+rgr-rnpstackZmeshgridr/ZastyperRarrayrr!Z from_numpyr )rFr4r-rbox_coordinatesr'r'r(!normalize_grid_corner_coordinates4s  $z:OwlViTForObjectDetection.normalize_grid_corner_coordinates)r4r+cCs||}t|dd}t|dt| d}t|d|jd}t|dt| d}tj||gdd}|S)Nrrg-C6?rrxr)r:r!Zcliploglog1pZ full_likerr)rFr4r9Zbox_coord_biasZbox_sizeZ box_size_biasZbox_biasr'r'r(compute_box_biasIs z)OwlViTForObjectDetection.compute_box_bias) image_featsr4r+cCs&||}|||7}||}|S)a Args: image_feats: Features extracted from the image, returned by the `image_text_embedder` method. feature_map: A spatial re-arrangement of image_features, also returned by the `image_text_embedder` method. Returns: pred_boxes: List of predicted boxes (cxcywh normalized to 0, 1) nested within a dictionary. )r1r=r3)rFr>r4rlr'r'r( box_predictorYs  z&OwlViTForObjectDetection.box_predictorN)r>r(r)r+cCs||||\}}||fS)a8 Args: image_feats: Features extracted from the `image_text_embedder`. query_embeds: Text query embeddings. query_mask: Must be provided with query_embeddings. A mask indicating which query embeddings are valid. )r0)rFr>r(r)r.r-r'r'r(class_predictorpsz(OwlViTForObjectDetection.class_predictor)rrrrrr+c Cs|j|||||dd}|jd}|jj|}tt|jtd} t |ddddddf| } |ddddddf| }| |}|jdt t |jdt t |jd|jdf} | | }|d} | ||fS)NT)rrrrrrrrrrrrx)rr<rrrIr6r8rr! broadcast_tor2rWsqrtr) rFrrrrrrrr:new_sizeclass_token_outr9r'r'r(image_text_embedders,  "  z,OwlViTForObjectDetection.image_text_embedder)rrrr+c Cs|jj|dd}|d}|jj|}tt|jtd}t|ddddddf|}|ddddddf|}| |}|jdt t |jdt t |jd|jdf}| |}||fS)NT)rrrrArrx) rrrrIr6r8rr!rCr2rWrDr) rFrrrr rr:rErFr'r'r(image_embedders"  z'OwlViTForObjectDetection.image_embedder)query_image_featuresquery_feature_mapr+cCs:||\}}|||}t|}g}g}|j} t|jdD]} tjgdg| d} || } t| | \} }t | ddkrt | | } t | d}| d|k }| r>|| |d}tj|| dd}td||}|t|}||| |||q>|r(t|}t|}nd \}}|||fS) Nr)rrrrr,rg?rr5zd,id->i)NN)r@r?rr-rrr!rrerfrir_ZnonzeroZnumelZsqueezerr*Zargminappendr7)rFrIrJrrmrlZpred_boxes_as_cornersZbest_class_embedsbest_box_indicesZpred_boxes_deviceiZeach_query_boxZeach_query_pred_boxesZiousZ iou_thresholdZ selected_indsZselected_embeddingsZ mean_embedsZmean_simZ best_box_indr(Z box_indicesr'r'r(embed_image_querys4     z*OwlViTForObjectDetection.embed_image_queryr)rquery_pixel_valuesrrrr+c Cs"|dur |n|jj}|dur |n|jj}|dur4|n|jj}|j|dd}|j|||d\}}|j\} } } } t|| | | | f} |j\} } } } t|| | | | f} || |\}}}|j | |d\}}| | |}|s ||||||| f}t dd|D}|St ||||||d|dS) a3 Returns: Examples: ```python >>> import requests >>> from PIL import Image >>> import torch >>> from transformers import AutoProcessor, OwlViTForObjectDetection >>> processor = AutoProcessor.from_pretrained("google/owlvit-base-patch16") >>> model = OwlViTForObjectDetection.from_pretrained("google/owlvit-base-patch16") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> query_url = "http://images.cocodataset.org/val2017/000000001675.jpg" >>> query_image = Image.open(requests.get(query_url, stream=True).raw) >>> inputs = processor(images=image, query_images=query_image, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model.image_guided_detection(**inputs) >>> # Target image sizes (height, width) to rescale box predictions [batch_size, 2] >>> target_sizes = torch.Tensor([image.size[::-1]]) >>> # Convert outputs (bounding boxes and class logits) to COCO API >>> results = processor.post_process_image_guided_detection( ... outputs=outputs, threshold=0.6, nms_threshold=0.3, target_sizes=target_sizes ... ) >>> i = 0 # Retrieve predictions for the first image >>> boxes, scores = results[i]["boxes"], results[i]["scores"] >>> for box, score in zip(boxes, scores): ... box = [round(i, 2) for i in box.tolist()] ... print(f"Detected similar object with confidence {round(score.item(), 3)} at location {box}") Detected similar object with confidence 0.856 at location [10.94, 50.4, 315.8, 471.39] Detected similar object with confidence 1.0 at location [334.84, 25.33, 636.16, 374.71] ```N)rr)rrr)r>r(css|]}|dur|VqdSrQr'rCxr'r'r(rGGrzBOwlViTForObjectDetection.image_guided_detection..)r:rorprqr*rmr;r<)rtrrrrHrr!rrNr@r?rArIrn)rFrrOrrrrJr4r rr hidden_dimr>Zquery_image_featsr(rLrqr.rmrprr'r'r(image_guided_detectionsL+   z/OwlViTForObjectDetection.image_guided_detection)rrrrrrr+c Cs2|dur |n|jj}|dur |n|jj}|dur4|n|jj}|j|||||d\}}} | j} | j} |j\} } } }t || | | |f}|jd| }| | ||jd}| | ||jd}|ddk}| |||\}}| ||}|s|||||| | f}t dd|D}|St|||||| | dS) a Returns: Examples: ```python >>> import requests >>> from PIL import Image >>> import torch >>> from transformers import AutoProcessor, OwlViTForObjectDetection >>> processor = AutoProcessor.from_pretrained("google/owlvit-base-patch32") >>> model = OwlViTForObjectDetection.from_pretrained("google/owlvit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> texts = [["a photo of a cat", "a photo of a dog"]] >>> inputs = processor(text=texts, images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> # Target image sizes (height, width) to rescale box predictions [batch_size, 2] >>> target_sizes = torch.Tensor([image.size[::-1]]) >>> # Convert outputs (bounding boxes and class logits) to final bounding boxes and scores >>> results = processor.post_process_object_detection( ... outputs=outputs, threshold=0.1, target_sizes=target_sizes ... ) >>> i = 0 # Retrieve predictions for the first image for the corresponding text queries >>> text = texts[i] >>> boxes, scores, labels = results[i]["boxes"], results[i]["scores"], results[i]["labels"] >>> for box, score, label in zip(boxes, scores, labels): ... box = [round(i, 2) for i in box.tolist()] ... print(f"Detected {text[label]} with confidence {round(score.item(), 3)} at location {box}") Detected a photo of a cat with confidence 0.707 at location [324.97, 20.44, 640.58, 373.29] Detected a photo of a cat with confidence 0.717 at location [1.46, 55.26, 315.55, 472.17] ```N)rrrrrrrx).rcss|]}|dur|VqdSrQr'rPr'r'r(rGrz3OwlViTForObjectDetection.forward..)r:r9rlr*rmr;r<)rtrrrrGr;r<rr!rr@r?rArIrj)rFrrrrrrr(r4rrr rrrRr>Zmax_text_queriesr)r.rmrlrr'r'r(rUsR/    z OwlViTForObjectDetection.forward)NN)NN)NN)NNNN)NNNN)rKrLrMrrrzr!rOr:r=r?rrrr@r"rGrHrNr5OWLVIT_IMAGE_GUIDED_OBJECT_DETECTION_INPUTS_DOCSTRINGrrnrS(OWLVIT_OBJECT_DETECTION_INPUTS_DOCSTRINGrjrrr'r'rr(r/'s    - ! ) ] r/)N)r)NrNrZ dataclassesrtypingrrrrrnumpyr6r!Ztorch.utils.checkpointrZ activationsr Zmodeling_outputsr r Zmodeling_utilsr rrrrrrrZconfiguration_owlvitrrrZtransformers.image_transformsrZ get_loggerrKloggerZ_CHECKPOINT_FOR_DOCZ$OWLVIT_PRETRAINED_MODEL_ARCHIVE_LISTrrrWr)r1r4r5rXr[rerirjrnrrrrrrrrZOWLVIT_START_DOCSTRINGrrrrUrTrSizer-rrrrrrrr$r/r'r'r'r(sr       & 10 l2; a H4:2W1