a  d@sdZddlZddlZddlmZmZmZddlZddlZddlm Z ddl m Z ddl m Z ddlmZmZmZdd lmZmZdd lmZmZmZmZdd lmZd d lmZeeZ d Z!dZ"dZ#gdZ$dZ%dZ&dgZ'd4ej(ej)ee*dddZ+Gddde j,Z-Gddde j,Z.Gddde j,Z/Gddde j,Z0Gddde j,Z1Gd d!d!e j,Z2Gd"d#d#e j,Z3Gd$d%d%e j,Z4Gd&d'd'e j,Z5Gd(d)d)eZ6d*Z7d+Z8Gd,d-d-e6Z9ed.e7Gd/d0d0e6Z:ed1e7Gd2d3d3e6Z;dS)5z PyTorch M-CTC-T model.N)OptionalTupleUnion)nn)ACT2FN)is_deepspeed_zero3_enabled)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forward)BaseModelOutputCausalLMOutput)PreTrainedModelapply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)logging) MCTCTConfigrzspeechbrain/m-ctc-t-large)rizY"Mr. Quilter is the apostle of the middle classes, and we're glad to welcome his gospel."gv@)maskdtypetgt_lencCsj|\}}|dur|n|}|ddddddf|d|||}d|}||tjt|jS)z_ Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`. Nr?)sizeexpandtoZ masked_filltorchboolZfinfomin)rrrbszZsrc_lenZ expanded_maskZ inverted_maskr!q/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/transformers/models/mctct/modeling_mctct.py _expand_mask>s  *r#cs(eZdZdZfddZddZZS)MCTCTConv1dSubsamplerz Convolutional subsampler: a stack of 1D convolution (along temporal dimension) followed by non-linear activation via gated linear units (https://arxiv.org/abs/1911.08460) cst|_|j_t|j_|j _ |j |j _ j dkrb|jdurXtd|j_nd_|jd_|j_|j_tfddtjD_dS)NrzbNeed to specify `conv_channels` configuration in `MCTCTConfig` to use multiple convolution layers.c3sZ|]R\}}tj|dkrjnj||jdkr<j|nj|j|ddVqdS)rrZvalid) kernel_sizestridepaddingN)rConv1d in_channels mid_channels num_layers out_channelsr').0ikselfr!r" nsz1MCTCTConv1dSubsampler.__init__..)super__init__configZ conv_glu_dimglu_dimrDropoutZ conv_dropoutdropoutnum_conv_layersr,Zinput_feat_per_channelZinput_channelsr*Z conv_channels ValueErrorr+ hidden_sizer- conv_kernelr& conv_strider' ModuleList enumerate conv_layersr2r6 __class__r1r"r5Rs&     zMCTCTConv1dSubsampler.__init__cCstdd|jD}tjj|dd||fdd}|dd}|jD](}||}tjj ||j d}| |}qF|dd}|S)NcSsg|] }|dqS)r%r!)r.rr!r!r" |z1MCTCTConv1dSubsampler.forward..rZconstantrr%dim) sumr&rr functionalpad transpose contiguousrAZglur7r9)r2input_featuresr( hidden_statesconvr!r!r"forwardys  zMCTCTConv1dSubsampler.forward__name__ __module__ __qualname____doc__r5rQ __classcell__r!r!rCr"r$Ls 'r$cs*eZdZdZfddZdddZZS) MCTCTEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.csttj|j|j|jd|_t|j|j|_ t|j |j|_ t |_ t|j|_|dt|jd|jdtj|jtj|jjddddS)N) padding_idx position_ids)rtoken_type_idsrdeviceF) persistent)r4r5r Embedding vocab_sizer< pad_token_idword_embeddingsmax_position_embeddingsZposition_embeddingsZtype_vocab_sizetoken_type_embeddingsMCTCTLayerNorm LayerNormr8hidden_dropout_probr9Zregister_bufferrarangerzerosrZrlongr^rBrCr!r"r5s zMCTCTEmbeddings.__init__Nrc Cs|dur|n|dd}|d}|durJ|jdd|||f}|durt|dr|jddd|f}||d|} | }ntj|tj|jjd}|dur| |}| |} || } | | } | | } | S)Nr[rr\rr]) rrZhasattrr\rrrjrkr^rcrergr9) r2rNr\rZ inputs_embedsZpast_key_values_lengthZ input_shapeZ seq_lengthZbuffered_token_type_idsZ buffered_token_type_ids_expandedreZ embeddingsr!r!r"rQs"      zMCTCTEmbeddings.forward)NNNNrrRr!r!rCr"rXs rXcs>eZdZfddZddZddZddZd d d ZZS)MCTCTSelfAttentioncst|j|jdkr>t|ds>td|jd|jd|j|_|j|_|j|j|_t j |j|jdd|_ t j |j|jdd|_ t j |j|jdd|_ t |j|_|j|_t d|jd |j|_|j|_dS) NrZembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()Fbiasr%r)r4r5r<num_attention_headsrlr;Zattention_head_dimattention_head_size all_head_sizerLinearquerykeyvaluer8Zattention_probs_dropout_probr9rdr`distance_embedding is_decoderrBrCr!r"r5s"  zMCTCTSelfAttention.__init__cCs6|dd|j|jf}|j|}|ddddS)Nr[rr%rr)rrrrsviewpermute)r2xZ new_x_shaper!r!r"transpose_for_scoress z'MCTCTSelfAttention.transpose_for_scorescCsFt|jdkr&|jttt|j}|jt|jttt|S)Nr)lenshaper|reversedrangeZreshape)r2r}rr!r!r"reshape_fortransz"MCTCTSelfAttention.reshape_fortrancCs|dddd}|j\}}}}tj|tj||||f|jdfdd}||||||d|g}|ddd||d|f}|||||d||g}|d}|dd|||fdd}|ddddS)Nrr%rrr^rG)r|rrcatrjr^rrL)r2ZscoresbatchZ hidden_stateZseq_lenheadsZ halfpointr!r!r""relative_position_embedding_rotates&  z5MCTCTSelfAttention.relative_position_embedding_rotateNFcCs||}|t|j}|||}|||}||}t|| dd} |j j } t d| | dd} | | } | | } |dur| |} tjj| dd} || } |dur| |} t| |} | ddddjdd } |r| | fn| f}|S) Nr[zlh, bche -> bcler%rrGrr)Z start_dim)rvmathsqrtrsr~rwrxrmatmulrLryweightZeinsumrrrJZsoftmaxr9r|flatten)r2rOattention_mask head_maskoutput_attentionsZmixed_query_layerZ key_layerZ value_layerZ query_layerZattention_scoresZpositional_embeddingZrelative_position_scoresZattention_probsZ context_layeroutputsr!r!r"rQs(     zMCTCTSelfAttention.forward)NNF) rSrTrUr5r~rrrQrWr!r!rCr"rns rncs$eZdZfddZddZZS)rfcs2tttd|_ttd|_dSNr) r4r5r Parameterronessingleton_weightrjsingleton_biasr1rCr!r"r5.s zMCTCTLayerNorm.__init__cCs||j|jSN)rrr2rOr!r!r"rQ3szMCTCTLayerNorm.forwardrSrTrUr5rQrWr!r!rCr"rf-s rfcs$eZdZfddZddZZS)MCTCTSelfOutputcsLt||_tj|j|jdd|_tj|j|jd|_t |j |_ dSNFrp)Zeps) r4r5r6rrur<denserglayer_norm_epsr8rhr9rBrCr!r"r58s  zMCTCTSelfOutput.__init__cCs&||}||}|||}|Srrr9rgr2rOZ input_tensorr!r!r"rQ?s  zMCTCTSelfOutput.forwardrr!r!rCr"r7s rcs.eZdZfddZddZd ddZZS) MCTCTAttentioncs*tt||_t||_t|_dSr)r4r5rnr2routputset pruned_headsrBrCr!r"r5Gs   zMCTCTAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)NrrrG)rrr2rrrsrrrvrwrxrrrtunion)r2rindexr!r!r" prune_headsMs zMCTCTAttention.prune_headsNFcCs6|||||}||d|}|f|dd}|S)Nrr)r2r)r2rOrrrZ self_outputsattention_outputrr!r!r"rQ_szMCTCTAttention.forward)NNF)rSrTrUr5rrQrWr!r!rCr"rFs  rcs$eZdZfddZddZZS)MCTCTIntermediatecsFttj|j|jdd|_t|jt r:t |j|_ n|j|_ dS)NFrp) r4r5rrur<intermediate_sizer isinstanceZ hidden_actstrrintermediate_act_fnrBrCr!r"r5ss   zMCTCTIntermediate.__init__cCs||}||}|Sr)rrrr!r!r"rQ{s  zMCTCTIntermediate.forwardrr!r!rCr"rrs rcs$eZdZfddZddZZS) MCTCTOutputcsFttj|j|jdd|_tj|j|jd|_t |j |_ dSr) r4r5rrurr<rrgrr8rhr9rBrCr!r"r5s zMCTCTOutput.__init__cCs&||}||}|||}|Srrrr!r!r"rQs  zMCTCTOutput.forwardrr!r!rCr"rs rcs4eZdZedfdd Zd ddZdd ZZS) MCTCTLayerr6csBtd|_|j|_t||_t||_|j|_t ||_ dSr) r4r5 seq_len_dimchunk_size_feed_forwardr intermediater attentionrzrrrBrCr!r"r5s   zMCTCTLayer.__init__NFc CsH|j||||d}|d}|dd}t|j|j|j|}|f|}|S)Nrrr)rrfeed_forward_chunkrr) r2rOrrrZself_attention_outputsrr layer_outputr!r!r"rQs  zMCTCTLayer.forwardcCs||}|||}|Sr)rr)r2rZintermediate_outputrr!r!r"rs  zMCTCTLayer.feed_forward_chunk)NNF)rSrTrUrr5rQrrWr!r!rCr"rs  rc@sPeZdZdZeZdZdZdgZdZ ddZ e j dd d Z d d ZdddZdS)MCTCTPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. mctctrNrZTcCs|jj}t|tjr>|jjjd|d|jdur|jj nt|tj rz|jjjd|d|j dur|jj|j  nLt|tj r|jj |jj dn$t|tr|jj d|jj t|tjtjfr|jjjd|d|jdur|jj dS)zInitialize the weightsg)meanstdNr)r6Zinitializer_rangerrrurdataZnormal_rqZzero_r`rYrgZfill_rfrrr))r2modulerr!r!r" _init_weightss&         z"MCTCTPreTrainedModel._init_weights) input_lengthscCshd}tt|jj|jj|jjD]B\}}}|d}|d|||dd}tj||ddd}q |S)zH Computes the output length of the convolutional layers rr%trunc)Z rounding_mode)ziprr6r:r=r>rdiv)r2rZdilation_Z kernel_szr'r(r!r!r" _get_feat_extract_output_lengthssz5MCTCTPreTrainedModel._get_feat_extract_output_lengthscCst|jdkr$|dddddf}||d}|d}tj||f|j|jd}d|tj ||jd|df<| dg d dg }|S)Nr%r[rr]rr) rrrrIrrrjrr^riZflipZcumsumrk)r2Zfeature_vector_lengthrZsubsampled_lengthsr r!r!r""_get_feature_vector_attention_masks z7MCTCTPreTrainedModel._get_feature_vector_attention_maskFcCst|tr||_dSr)r MCTCTEncodergradient_checkpointing)r2rrxr!r!r"_set_gradient_checkpointings z0MCTCTPreTrainedModel._set_gradient_checkpointingN)F)rSrTrUrVr config_classZbase_model_prefixZmain_input_nameZ_keys_to_ignore_on_load_missingZsupports_gradient_checkpointingrr LongTensorrrrr!r!r!r"rsraH This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`MCTCTConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_features (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`Wav2Vec2CTCTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~file_utils.ModelOutput`] instead of a plain tuple. c sLeZdZedfdd Zd ejejejeeeee e fdddZ Z S) rrcsPtj|_t|_t|_tfddt j D|_ d|_ dS)Ncsg|] }tqSr!)r)r.rrr!r"rE2rFz)MCTCTEncoder.__init__..F) r4r5rhrf layer_normr$rPrr?rZnum_hidden_layerslayersrrBrCrr"r5,s    zMCTCTEncoder.__init__FTrNrrroutput_hidden_states return_dictreturncsdur n|jj|dur |n|jj}|dur4|n|jj}||}||}|durj||jd|}tj j ||j |j d}|durt ||j}|rdnd} rdnd} |dur|dt|jkrtdt|jd|ddt} t|jD]\} } |r| |f} tdd}|j r<||jjkr.create_custom_forward..custom_forwardr!)rrr)rr"create_custom_forwardlsz3MCTCTEncoder.forward..create_custom_forward)rOrr)NNcss|]}|dur|VqdSrr!)r.vr!r!r"r3rFz'MCTCTEncoder.forward..Zlast_hidden_staterO attentions)r6rruse_return_dictrrPrrrrJr9rhrr#rrrrr;rr@randomuniformZ layerdroprrutils checkpointtupler )r2rNrrrrrrmrOZencoder_statesZall_attentionsZdeepspeed_zero3_is_enabledidxZ encoder_layerZdropout_probabilityZskip_the_layerrZ layer_outputsr!rr"rQ6sh            zMCTCTEncoder.forward)FFT) rSrTrUrr5rTensorrrrr rQrWr!r!rCr"r+s rzaThe bare M-CTC-T Model transformer outputting raw hidden-states without any specific head on top.c szeZdZfddZeedeee e de dd e j ee j ee j eeeeeeeee fddd ZZS) MCTCTModelcs(t|||_t||_|dSr)r4r5r6rencoder post_initrBrCr!r"r5s  zMCTCTModel.__init__zbatch_size, sequence_lengthaudio)r output_typerZmodalityexpected_outputNrc Cs|dur |n|jj}|dur |n|jj}|dur4|n|jj}|durLtd|j||||||d}|d}|s|f|ddSt||j|jdS)Nz#You have to specify input_features.rrrrrrrr) r6rrrr;rr rOr) r2rNrrrrrZencoder_outputsZsequence_outputr!r!r"rQs,zMCTCTModel.forward)NNNNN)rSrTrUr5r MCTCT_INPUTS_DOCSTRINGformatr _CHECKPOINT_FOR_DOCr _CONFIG_FOR_DOC_EXPECTED_OUTPUT_SHAPErrrrrrrQrWr!r!rCr"rs.   rzcMCTCT Model with a `language modeling` head on top for Connectionist Temporal Classification (CTC).c s|eZdZfddZeeeeee e e dde j ee j ee j eeeeeeee jeeefdddZZS) MCTCTForCTCcsTt|t||_|jdur2td|jd|j}t ||j|_ | dS)NzYou are trying to instantiate z with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `MCTCTForCTC.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration.) r4r5rrrar;rDr<rructc_headr)r2r6Zoutput_hidden_sizerCr!r"r5s    zMCTCTForCTC.__init__)rrrrZ expected_lossN)rNrrrrrlabelsrc Cs~|dur |n|jj}|j||||||d}|d} || } d} |dur:||jjkrltd|jj|durx|ntj|j ddtj d}| | d tj } |dk} | d}|| }tjj| dtjddd}tjjjd d 6tjj||| ||jj|jj|jjd } Wdn1s00Y|sj| f|td}| durf| f|S|St| | |j|jd S) a labels (`torch.LongTensor` of shape `(batch_size, target_length)`, *optional*): Labels for connectionist temporal classification. Note that `target_length` has to be smaller or equal to the sequence length of the output logits. Indices are selected in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size - 1]`. Nrrz$Label values must be <= vocab_size: r[)r)rHrrF)Zenabled)ZblankZ reductionZ zero_infinity)losslogitsrOr)r6rrrmaxrar;rrrrkrrIrZ masked_selectrrJZ log_softmaxfloat32rLbackendsZcudnnflagsZctc_lossrbZctc_loss_reductionZctc_zero_infinity_HIDDEN_STATES_START_POSITIONr rOr)r2rNrrrrrrrrOrrrZ labels_maskZtarget_lengthsZflattened_targetsZ log_probsrr!r!r"rQsP     &  zMCTCTForCTC.forward)NNNNNN)rSrTrUr5r rr rr r_CTC_EXPECTED_OUTPUT_CTC_EXPECTED_LOSSrrrrrrrrQrWr!r!rCr"rs2   r)N)s\      =8l ,'J  f8