a  d@sdZddlZddlZddlmZddlmZddlm Z m Z m Z ddl Z ddl Z ddl mZddlmZmZmZdd lmZdd lmZmZdd lmZdd lmZmZdd lmZmZm Z m!Z!m"Z"ddl#m$Z$e!%e&Z'dZ(dZ)dgZ*eGdddeZ+eGdddeZ,eGdddeZ-dLddZ.dMddZ/dNdd Z0Gd!d"d"ej1Z2Gd#d$d$ej1Z3Gd%d&d&ej1Z4Gd'd(d(ej1Z5Gd)d*d*ej1Z6Gd+d,d,ej1Z7Gd-d.d.ej1Z8Gd/d0d0ej1Z9Gd1d2d2ej1Z:Gd3d4d4ej1Z;Gd5d6d6ej1Zd:Z?ed;e>Gdd?d?ej1ZAed@e>GdAdBdBe=ZBGdCdDdDej1ZCGdEdFdFej1ZDGdGdHdHej1ZEedIe>GdJdKdKe=ZFdS)Oz PyTorch TVLT model.N)deepcopy) dataclass)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)BaseModelOutputSequenceClassifierOutput)PreTrainedModel) find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) TvltConfigrzZinengTang/tvlt-basec@seZdZUdZdZejed<dZejed<dZ ejed<dZ ej ed<dZ ej ed<dZ ej ed<dZej ed <dZeeejed <dZeeejed <dS) TvltModelOutputa Class for TvltModel's outputs, with potential hidden states and attentions. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. last_pixel_hidden_state (`torch.FloatTensor` of shape `(batch_size, pixel_sequence_length, hidden_size)`): Pixel sequence of hidden-states at the output of the last layer of the model. last_audio_hidden_state (`torch.FloatTensor` of shape `(batch_size, audio_sequence_length, hidden_size)`): Audio sequence of hidden-states at the output of the last layer of the model. pixel_label_masks (`torch.FloatTensor` of shape `(batch_size, pixel_patch_length)`): Tensor indicating which pixel patches are masked (1) and which are not (0). audio_label_masks (`torch.FloatTensor` of shape `(batch_size, audio_patch_length)`): Tensor indicating which audio patches are masked (1) and which are not (0). pixel_ids_restore (`torch.LongTensor` of shape `(batch_size, pixel_patch_length)`): Tensor containing the ids permutation of pixel masking. audio_ids_restore (`torch.LongTensor` of shape `(batch_size, audio_patch_length)`): Tensor containing the ids permutation of audio masking. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings and one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nlast_hidden_statelast_pixel_hidden_statelast_audio_hidden_statepixel_label_masksaudio_label_maskspixel_ids_restoreaudio_ids_restore hidden_states attentions)__name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__rrrZ LongTensorrrr r!rrr"r*r*o/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/transformers/models/tvlt/modeling_tvlt.pyr6s rc@sLeZdZUdZdZejed<dZe e ejed<dZ e e ejed<dS)TvltDecoderOutputaM Class for TvltDecoder's outputs, with potential hidden states and attentions. Args: logits (`torch.FloatTensor` of shape `(batch_size, patch_size ** 2 * num_channels)`): Pixel reconstruction logits. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings and one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nlogitsr!r") r#r$r%r&r-r'r(r)r!rrr"r*r*r*r+r,_s r,c@szeZdZUdZdZeejed<dZ ejed<dZ ejed<dZ ejed<dZ ee ejed<dZee ejed<dS) TvltForPreTrainingOutputa  Class for TvltForPreTraining's outputs, with potential hidden states and attentions. Args: loss (`torch.FloatTensor` of shape `(1,)`): Pixel reconstruction loss. matching_logits (`torch.FloatTensor` of shape `(batch_size, 1)`): Matching objective logits. pixel_logits (`torch.FloatTensor` of shape `(batch_size, pixel_patch_length, image_patch_size ** 3 * pixel_num_channels)`): Pixel reconstruction logits. audio_logits (`torch.FloatTensor` of shape `(batch_size, audio_patch_length, image_patch_size[0] * image_patch_size[1])`): Audio reconstruction logits. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings and one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nlossmatching_logits pixel_logits audio_logitsr!r")r#r$r%r&r/rr'r(r)r0r1r2r!rr"r*r*r*r+r.vs r.?cCs>|jdd\}}tj||f|jd}t|d|}||fS)!Generate noise for audio masking.Ndevicer)shaper'randr7int) pixel_values pixel_mask mask_ratio batch_sizeseq_lennoiselen_keepr*r*r+generate_pixel_mask_noisesrB patch-levelc Cs|jdd\}}|dkrN||}tj|||jdddd|||}n|dkrhtj|||jd}t|d|} || fS)r4Nr5z frame-levelr6rrC)r8r'r9r7 unsqueezerepeatviewr:) audio_values audio_maskr= mask_typefreq_lenr>r?num_time_patchesr@rAr*r*r+generate_audio_mask_noisesrNc Cs|j\}}}tj|dd}tj|dd}|ddd|f} tj|d| ddd|d} tj||g|jd} d| ddd|f<tj| d|d} |dur| |9} tj|d| d}| || |fS)z Perform random masking by per-sample shuffling on frame-level. Per-sample shuffling is done by argsort random noise. sequence: [batch_size, seq_len, hidden_dim], sequence rdimNrErPindexr6r)r8r'ZargsortgatherrFrGZonesr7) sequencer@rAattention_masksr>r?Z hidden_dimZ ids_shuffle ids_restoreZids_keepZsequence_maskedZ label_masksr*r*r+random_maskings  rWcs*eZdZdZfddZdddZZS)TvltPixelEmbeddings,Construct the patch and position embeddings.csttt||_|jj|_ttdd|j |_ ttd|j |j |_ ttd|j|j |_ ||_dSNr)super__init__TvltPixelPatchEmbeddingspatch_embeddingsnum_patches_per_imager Parameterr'zeros hidden_size type_embed_v num_framestemporal_embed pos_embed_vconfigselfrg __class__r*r+r\s   zTvltPixelEmbeddings.__init__Nc Csh|j\}}}}}||}||jd|d7}|tj|jddd|f|jdd7}||j7}||fSNrrO) r8r^rfrGr'repeat_interleaverer_rc) rir;rUr>rd num_channelsheightwidth embeddingsr*r*r+forwards  ( zTvltPixelEmbeddings.forward)Nr#r$r%r&r\rr __classcell__r*r*rjr+rXs rXcs*eZdZdZfddZdddZZS)TvltAudioEmbeddingsrYcstt||_|jj|_ttdd|j |_ |j |j d|_ ttd|j|j |j |_ttd|j |j |_|j |j d|_ ||_dSrZ)r[r\TvltAudioPatchEmbeddingsr^ num_patchesrr`r'rarb type_embed_afrequency_lengthaudio_patch_sizenum_freq_patches pos_embed_a freq_embedrgrhrjr*r+r\s    zTvltAudioEmbeddings.__init__NcCsh||}|d|j}||jd|d7}|tj|jddd|f|jdd7}||j7}||fSrl) r^sizer{r}rGr'rmr|rx)rirIrUrqrMr*r*r+rrs  ( zTvltAudioEmbeddings.forward)Nrsr*r*rjr+rus rucs4eZdZdZfddZejejdddZZS)r]z This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. cst|j|j}}|j|j}}t|tjj r8|n||f}t|tjj rR|n||f}|d|d|d|d}||_||_ ||_ ||_ ||_t j||||d|_dSNrr)Z kernel_sizeZstride)r[r\ image_sizeimage_patch_sizenum_image_channelsrb isinstance collectionsabcIterable patch_sizernr_rConv2d projection)rirgrrrnrbr_rjr*r+r\s  z!TvltPixelPatchEmbeddings.__init__)r;returnc Cs|j\}}}}}||jkr"td||jdks>||jdkrltd|d|d|jdd|jdd ||||||}||ddd}||||j|j }|S) NeMake sure that the channel dimension of the pixel values match with the one set in the configuration.rrzInput image size (*) doesn't match model ().r5) r8rn ValueErrorrreshaperflatten transposer_rb)rir;r>rdrnrorprqr*r*r+rrs (z TvltPixelPatchEmbeddings.forward r#r$r%r&r\r'Tensorrrrtr*r*rjr+r]s r]cs4eZdZdZfddZejejdddZZS)rvz This class turns `audio_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. c st|j|j|j}}}|j|j}}||f}t|tj j rH|n||f}|d|d|d|d}|d|d|d|df} ||_ ||_ ||_ ||_| |_tj||||d|_dSr)r[r\spectrogram_lengthryrznum_audio_channelsrbrrrrspectrogram_sizerrnrw patch_shaperrr) rirgrryrrnrbrrwrrjr*r+r\8s     z!TvltAudioPatchEmbeddings.__init__)rIrc Cs|j\}}}}||jkr td||jdks<||jdkrjtd|d|d|jdd|jdd ||ddd}|S) NrrrzInput audio size (rrrr5)r8rnrrrrr)rirIr>rnrorprqr*r*r+rrMs z TvltAudioPatchEmbeddings.forwardrr*r*rjr+rv1s rvcs.eZdZfddZddZd ddZZS) TvltSelfAttentioncst|j|jdkr@t|ds@td|jfd|jd|j|_t|j|j|_|j|j|_t j |j|j|j d|_ t j |j|j|j d|_ t j |j|j|j d|_t |j|_dS)NrZembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .bias)r[r\rbnum_attention_headshasattrrr:attention_head_size all_head_sizerLinearZqkv_biasquerykeyvalueDropoutZattention_probs_dropout_probdropoutrhrjr*r+r\_s  zTvltSelfAttention.__init__cCs6|dd|j|jf}|j|}|ddddS)NrErr5rr )r~rrrHpermute)rixZ new_x_shaper*r*r+transpose_for_scoresqs z&TvltSelfAttention.transpose_for_scoresNFcCs||}|||}|||}||}t||dd} | t|j } |durh| |} t j dd| } | | } |dur| |} t| |} | dddd} | dd|jf} | j| } |r| | fn| f} | S)NrErOrr5rr )rrrrr'matmulrmathsqrtrrZSoftmaxrr contiguousr~rrH)rir!attention_mask head_maskoutput_attentionsZmixed_query_layerZ key_layerZ value_layerZ query_layerZattention_scoresZattention_probsZ context_layerZnew_context_layer_shapeoutputsr*r*r+rrvs$     zTvltSelfAttention.forward)NNF)r#r$r%r\rrrrtr*r*rjr+r^s rcs@eZdZdZeddfdd ZejejejdddZZ S) TvltSelfOutputz The residual connection is defined in TvltLayer instead of here (as is the case with other models), due to the layernorm applied before each block. Nrgrcs.tt|j|j|_t|j|_dSN) r[r\rrrbdenserhidden_dropout_probrrhrjr*r+r\s zTvltSelfOutput.__init__r! input_tensorrcCs||}||}|Srrrrir!rr*r*r+rrs  zTvltSelfOutput.forward) r#r$r%r&rr\r'rrrrtr*r*rjr+rsrcs.eZdZfddZddZd ddZZS) TvltAttentioncs*tt||_t||_t|_dSr)r[r\r attentionroutputset pruned_headsrhrjr*r+r\s   zTvltAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)NrrrO)lenrrrrrrrrrrrrunion)riheadsrRr*r*r+ prune_headss zTvltAttention.prune_headsNFcCs6|||||}||d|}|f|dd}|S)Nrr)rr)rir!rrrZ self_outputsattention_outputrr*r*r+rrszTvltAttention.forward)NNF)r#r$r%r\rrrrtr*r*rjr+rs rcs8eZdZeddfdd ZejejdddZZS)TvltIntermediateNrcsBtt|j|j|_t|jt r6t |j|_ n|j|_ dSr) r[r\rrrbintermediate_sizerrZ hidden_actstrr intermediate_act_fnrhrjr*r+r\s   zTvltIntermediate.__init__)r!rcCs||}||}|Sr)rrrir!r*r*r+rrs  zTvltIntermediate.forward r#r$r%rr\r'rrrrtr*r*rjr+rsrcs<eZdZeddfdd ZejejejdddZZS) TvltOutputNrcs.tt|j|j|_t|j|_ dSr) r[r\rrrrbrrrrrhrjr*r+r\s zTvltOutput.__init__rcCs ||}||}||}|Srrrr*r*r+rrs  zTvltOutput.forwardrr*r*rjr+rsrcs*eZdZdZfddZdddZZS) TvltLayerz?This corresponds to the Block class in the timm implementation.csbt|j|_d|_t||_t||_t||_ t j |j |j d|_t j |j |j d|_dSNrZeps)r[r\Zchunk_size_feed_forwardZ seq_len_dimrrr intermediaterrr LayerNormrblayer_norm_epslayernorm_beforelayernorm_afterrhrjr*r+r\s    zTvltLayer.__init__NFc Csj|j|||||d}|d}|dd}|||j}||}||}|||}|f|}|S)Nrrr)rrtor7rrr) rir!rrrZself_attention_outputsrrZ layer_outputr*r*r+rrs     zTvltLayer.forward)NNFrsr*r*rjr+rs rcs&eZdZfddZdddZZS) TvltEncodercs:t|_tfddtjD|_d|_dS)Ncsg|] }tqSr*r.0_rgr*r+ z(TvltEncoder.__init__..F) r[r\rgr ModuleListrangenum_hidden_layerslayergradient_checkpointingrhrjrr+r\s  zTvltEncoder.__init__NFTcs|rdnd}rdnd}t|jD]\} } |r8||f}|durH|| nd} |jr~|jr~fdd} tjj| | ||| } n| ||| } | d}r"|| df}q"|r||f}|stdd|||fDSt|||dS) Nr*csfdd}|S)Ncsg|RSrr*inputsmodulerr*r+custom_forward5szJTvltEncoder.forward..create_custom_forward..custom_forwardr*rrrrr+create_custom_forward4sz2TvltEncoder.forward..create_custom_forwardrrcss|]}|dur|VqdSrr*rvr*r*r+ Lrz&TvltEncoder.forward..)rr!r") enumeraterrtrainingr'utils checkpointtupler )rir!rrroutput_hidden_states return_dictall_hidden_statesall_self_attentionsi layer_moduleZlayer_head_maskr layer_outputsr*rr+rr s6      zTvltEncoder.forward)NNFFTr#r$r%r\rrrtr*r*rjr+rs rc@s2eZdZdZeZdZdZdZddZ d dd Z d S) TvltPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. tvltr;TcCsjt|tjtjfr@|jjjd|jjd|j durf|j j n&t|tj rf|j j |jj ddS)zInitialize the weights)meanZstdNg?) rrrrZweightdataZnormal_rgZinitializer_rangerZzero_rZfill_)rirr*r*r+ _init_weights_s   z!TvltPreTrainedModel._init_weightsFcCst|tr||_dSr)rrr)rirrr*r*r+_set_gradient_checkpointingks z/TvltPreTrainedModel._set_gradient_checkpointingN)F) r#r$r%r&r config_classZbase_model_prefixZmain_input_nameZsupports_gradient_checkpointingrrr*r*r*r+rTs raF This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`TvltConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_frames, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`TvltProcessor`]. See [`TvltProcessor.__call__`] for details. audio_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Audio values. Audio values can be obtained using [`TvltProcessor`]. See [`TvltProcessor.__call__`] for details. pixel_mask (`torch.FloatTensor` of shape `(batch_size, num_pixel_patches)`): Pixel masks. Pixel masks can be obtained using [`TvltProcessor`]. See [`TvltProcessor.__call__`] for details. audio_mask (`torch.FloatTensor` of shape `(batch_size, num_audio_patches)`): Audio masks. Audio masks can be obtained using [`TvltProcessor`]. See [`TvltProcessor.__call__`] for details. pixel_values_mixed (`torch.FloatTensor` of shape `(batch_size, num_frames, num_channels, height, width)`): Pixel values that mix positive and negative samples in Tvlt vision-audio matching. Pixel values mixed can be obtained using [`TvltProcessor`]. See [`TvltProcessor.__call__`] for details. pixel_mask_mixed (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel masks of pixel_values_mixed. Pixel masks mixed can be obtained using [`TvltProcessor`]. See [`TvltProcessor.__call__`] for details. mask_pixel (`bool`, *optional*): Whether to mask pixel for MAE tasks. Only set to True in TvltForPreTraining. mask_audio (`bool`, *optional*): Whether to mask audio for MAE tasks. Only set to True in TvltForPreTraining. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z^The bare TVLT Model transformer outputting raw hidden-states without any specific head on top.c sXeZdZfddZddZddZeeee e dd e e e fd d d Z ZS) TvltModelcsvt|||_t||_t||_t||_t t dd|j |_|jrVd|_nt j|j |jd|_|dSr)r[r\rgrXpixel_embeddingsruaudio_embeddingsrencoderrr`r'rarb cls_embeddingZuse_mean_pooling layernormrr post_initrhrjr*r+r\s    zTvltModel.__init__cCs|jj|jjfSr)rr^r)rir*r*r+get_input_embeddingsszTvltModel.get_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)riZheads_to_prunerrr*r*r+ _prune_headsszTvltModel._prune_heads output_typerNFrc  Cs|dur |n|jj}|dur |n|jj}| dur4| n|jj} |||\} }|||\} }d} d} |rt| ||jjd\}}t| |||d\} }} } d}d}|r|jj |jj d}t | ||jj |jj |d\}}t| |||d\} }}}|d}t|j|dd| | gd}| d}d}|durX|durXt|ddddf||gd}|}d}|durz|||}|j||||| d}|d}|jdur||}|dddd|f}|ddd|df}| s|||| || |f|ddSt|||| || ||j|jd S) a Returns: Examples: ```python >>> from transformers import TvltProcessor, TvltModel >>> import numpy as np >>> import torch >>> num_frames = 8 >>> images = list(np.random.randn(num_frames, 3, 224, 224)) >>> audio = list(np.random.randn(10000)) >>> processor = TvltProcessor.from_pretrained("ZinengTang/tvlt-base") >>> model = TvltModel.from_pretrained("ZinengTang/tvlt-base") >>> input_dict = processor(images, audio, sampling_rate=44100, return_tensors="pt") >>> outputs = model(**input_dict) >>> loss = outputs.loss ```N)r<r=)rUr)rJr=rKrLr)rrrr) rrrrrrr r!r")rgrruse_return_dictrrrBZpixel_mask_ratiorWryrzrNZaudio_mask_ratioZaudio_mask_typer~r'catrrGZget_extended_attention_maskrr rr!r")rir;rIr<rJ mask_pixel mask_audiorrrZpixel_embedding_outputZaudio_embedding_outputrrZpixel_mask_noiseZpixel_len_keeprr r{Zaudio_mask_noiseZaudio_len_keepr>Zembedding_outputZmasked_pixel_lenrZ input_shapeZextended_attention_maskZencoder_outputssequence_outputpixel_sequence_outputaudio_sequence_outputr*r*r+rrs%    "      zTvltModel.forward)NNFFNNN)r#r$r%r\r r rTVLT_INPUTS_DOCSTRINGrr_CONFIG_FOR_DOCrrrrrtr*r*rjr+rs   rcs&eZdZfddZdddZZS) TvltDecodercsvtt||j_|j_|j_|j _ t fddt |jD|_t j|j|jd|_d|_||_dS)Ncsg|] }tqSr*rrZdecoder_configr*r+rYrz(TvltDecoder.__init__..rF)r[r\rdecoder_hidden_sizerbZdecoder_num_hidden_layersrZdecoder_num_attention_headsrZdecoder_intermediate_sizerrrrdecoder_layersrrr rrgrhrjrr+r\Ps zTvltDecoder.__init__FTc s|rdnd}rdnd}t|jD]l\}}|r8||f}|jrh|jrhfdd} tjj| ||d} n ||d} | d}r"|| df}q"|r||f}||} |stdd| ||fDSt | ||d S) Nr*csfdd}|S)Ncsg|RSrr*rrr*r+rrszJTvltDecoder.forward..create_custom_forward..custom_forwardr*rrrr+rqsz2TvltDecoder.forward..create_custom_forwardrrrcss|]}|dur|VqdSrr*rr*r*r+rrz&TvltDecoder.forward..)r-r!r") rrrrr'rrr rr,) rir!rrrrrrrrrr-r*rr+rras,        zTvltDecoder.forward)FFTrr*r*rjr+rOs  rzTThe TVLT Model transformer with the decoder on top for self-supervised pre-training.c speZdZfddZddZddZddZd d Zd d Ze e e e e d deee fdddZZS)TvltForPreTrainingc st|||_|j|_|j|_|js6|js6tdt||_|jrPt||_ |jrt j |j |j dd|_t tdd|j |_t tdd|j |_t||_|j }|j}|jjj}t td|||_t td|j||_t tdd||_|jjj}|j|jd}t td||||_ t td|||_!t tdd||_"|jj#dd|jj$}t%|||_&|jjd|jjd|jj'}t%|||_(||_||_||_)|j#|_#|j|_|*dS)Nz;Must set at least one of matching task and MAE task to trueTrrrr5)+r[r\rg task_matchingtask_maerrrTvltMatchingHead matching_headrrrbrencoder_to_decoderr`r'rapixel_mask_tokenaudio_mask_tokenrdecoderrdrr_decoder_pixel_pos_embeddecoder_temporal_embeddecoder_pixel_type_embedrrwryrzdecoder_audio_pos_embeddecoder_freq_embeddecoder_audio_type_embedrr TvltMAEHeadpixel_mae_headraudio_mae_headr{r ) rirgrrdr_Znum_audio_patchesr{Zpixel_mae_output_dimZaudio_mae_output_dimrjr*r+r\sL         zTvltForPreTraining.__init__c Cs|j\}}}}}|jd|jd}|jd|jd}|j|||||jd||jdfd} td| } | j|||||jd|jd|fd} | S)zJ pixel_values: [batch_size, num_frames, 3, height, width] r rrr8zntchpwq->nthwpqc)r8rrr'einsum) rir;r>rdrnrorpnum_patches_heightnum_patches_widthpatchified_pixel_valuesr*r*r+patchify_pixels*  z!TvltForPreTraining.patchify_pixelc Cs|j\}}}}||jd}||jd}|j||||jd||jdfd}td|}|j||||jd|jd|fd}|S)z> audio_values: [batch_size, 1, height, width] rrr1znchpwq->nhwpqc)r8rzrr'r2) rirIr>rnrorpr3r4patchified_audio_valuesr*r*r+patchify_audios( z!TvltForPreTraining.patchify_audiocCs:||}||d}|jdd}|||}|SNr5rErO)r6rsum)rir;Zpixel_predictionsmaskr5r/r*r*r+pixel_mae_losss    z!TvltForPreTraining.pixel_mae_losscCs:||}||d}|jdd}|||}|Sr9)r8rr:)rirIZaudio_predictionsr;r7r/r*r*r+audio_mae_losss    z!TvltForPreTraining.audio_mae_lossc CsZ|j\}}}|||jd|d}tj||gdd}tj|d|ddd|d}|S)NrrOrErQ)r8rGr'rrSrF) riZ mask_tokenrTrVr>Z seq_lengthrPZ mask_tokensZpadded_sequencer*r*r+concatenate_masks z#TvltForPreTraining.concatenate_maskrNrc Cs| dur | n|jj} d} |jr|dur.td|dur>td|j|||||| | d} | d} || }t}||d|d}| |7} d}d}|jr\|j r\|j||||dd|| | d } | r| j n| d }| r| j n| d }| r| j n| d }| r| j n| d }| r| jn| d}| r*| jn| d}||}||}|d }||j||}||jd |d }|tj|jddd|f|jd d}||j}||}||j}||j||}|d |j}||j d |d }|tj|j!ddd|f|jd d}||j"}||}|#|j}|$||||%|||}| |7} | s|||f| dd}|dur| f|S|St&| |||| j'| j(dS)aF pixel_values_mixed (`torch.FloatTensor` of shape `(batch_size, num_frames, num_channels, height, width)`): Pixel values that mix positive and negative samples in Tvlt vision-audio matching. Audio values can be obtained using [`TvltProcessor`]. See [`TvltProcessor.__call__`] for details. pixel_mask_mixed (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel masks of pixel_values_mixed. Pixel values mixed can be obtained using [`TvltProcessor`]. See [`TvltProcessor.__call__`] for details. labels (`torch.LongTensor` of shape `(batch_size, num_labels)`, *optional*): Labels for computing the vision audio matching loss. Indices should be in `[0, 1]`. num_labels has to be 1. Return: Examples: ```python >>> from transformers import TvltProcessor, TvltForPreTraining >>> import numpy as np >>> import torch >>> num_frames = 8 >>> images = list(np.random.randn(num_frames, 3, 224, 224)) >>> images_mixed = list(np.random.randn(num_frames, 3, 224, 224)) >>> audio = list(np.random.randn(10000)) >>> processor = TvltProcessor.from_pretrained("ZinengTang/tvlt-base") >>> model = TvltForPreTraining.from_pretrained("ZinengTang/tvlt-base") >>> input_dict = processor( ... images, audio, images_mixed, sampling_rate=44100, mask_pixel=True, mask_audio=True, return_tensors="pt" ... ) >>> outputs = model(**input_dict) >>> loss = outputs.loss ```NrzMatching task requires labelsz)Matching task requires pixel_values_mixedr<rJrrrrrET)r<rJrrrrrrr5r r0rO)r/r0r1r2r!r"))rgrrrrr"rrHr rrrrrrr r#r~r>r$r'rGr'rmr(r_r)r&r.r-r%r{r+r*r,r/r<r=r.r!r") rir;rIr<rJlabelsZpixel_values_mixedZpixel_mask_mixedrrrZ total_lossrrr0loss_fctr/r1r2rrrrrr Zpixel_decoder_inputZaudio_decoder_inputrdZpixel_decoder_outputsrMZaudio_decoder_outputsrr*r*r+rrs1          zTvltForPreTraining.forward)NNNNNNNN)r#r$r%r\r6r8r<r=r>rrrr.rrrrrrtr*r*rjr+rs& 6   rcs$eZdZfddZddZZS) TvltPoolercs*tt|j|j|_t|_dSr)r[r\rrrbrZTanh activationrhrjr*r+r\s zTvltPooler.__init__cCs(|dddf}||}||}|S)Nr)rrF)rir!Zfirst_token_tensorZ pooled_outputr*r*r+rrs  zTvltPooler.forwardrr*r*rjr+rEs rEcs$eZdZfddZddZZS)r!cs(tt||_t|jd|_dSrZ)r[r\rEpoolerrrrbfcrhrjr*r+r\s  zTvltMatchingHead.__init__cCs|||}|Sr)rHrGrr*r*r+rrszTvltMatchingHead.forwardrr*r*rjr+r!s r!cs&eZdZdfdd ZddZZS)r-Ncs$t||_t|j||_dSr)r[r\rgrrrr&)rirgZ output_dimrjr*r+r\s zTvltMAEHead.__init__cCs||}|Sr)r&rr*r*r+rrs zTvltMAEHead.forward)Nrr*r*rjr+r-sr-z Tvlt Model transformer with a classifier head on top (an MLP on top of the final hidden state of the [CLS] token) for audiovisual classification tasks, e.g. CMU-MOSEI Sentiment Analysis and Audio to Video Retrieval. csHeZdZfddZeeeeedde e efdddZ Z S) TvltForAudioVisualClassificationc spt|t||_tt|j|jdtj|jd|j dt t|jd|j |_ ||_ |dS)Nr5r)r[r\rrrZ SequentialrrbrrZGELUZ num_labels classifierrgr rhrjr*r+r\s  z)TvltForAudioVisualClassification.__init__rNrc  Cs|dur |n|jj}|j|||||||d} | ddddf} || } d} |dur|jjdkrtt} | | |} n|jjdkrt} | | |} |s| f| dd}| dur| f|S|St| | | j| j dS)a labels (`torch.LongTensor` of shape `(batch_size, num_labels)`, *optional*): Labels for computing the audiovisual loss. Indices should be in `[0, ..., num_classes-1]` where num_classes refers to the number of classes in audiovisual tasks. Return: Examples: ```python >>> from transformers import TvltProcessor, TvltForAudioVisualClassification >>> import numpy as np >>> import torch >>> num_frames = 8 >>> images = list(np.random.randn(num_frames, 3, 224, 224)) >>> audio = list(np.random.randn(10000)) >>> processor = TvltProcessor.from_pretrained("ZinengTang/tvlt-base") >>> model = TvltForAudioVisualClassification.from_pretrained("ZinengTang/tvlt-base") >>> input_dict = processor(images, audio, sampling_rate=44100, return_tensors="pt") >>> outputs = model(**input_dict) >>> loss = outputs.loss ```Nr?rZ regressionZclassificationr0)r/r-r!r") rgrrrJZ loss_typer r rr!r")rir;rIr<rJrrrrCrrr-r/rDrr*r*r+rrs:$      z(TvltForAudioVisualClassification.forward)NNNNNN) r#r$r%r\rrrrrrrrrrtr*r*rjr+rIs   rI)Nr3)Nr3rCrD)N)Gr&collections.abcrrcopyrZ dataclassesrtypingrrrr'Ztorch.utils.checkpointrZtorch.nnrr r Z activationsr Zmodeling_outputsr rZmodeling_utilsrZ pytorch_utilsrrrrrrrrZconfiguration_tvltrZ get_loggerr#loggerrZ_CHECKPOINT_FOR_DOCZ"TVLT_PRETRAINED_MODEL_ARCHIVE_LISTrr,r.rBrNrWModulerXrur]rvrrrrrrrrZTVLT_START_DOCSTRINGrrrrrEr!r-rIr*r*r*r+s       (!  )-=#'; -$@