a  d@sdZddlZddlZddlmZmZmZddlZddlZddlm Z ddl m Z m Z m Z ddlmZddlmZmZmZmZdd lmZdd lmZmZmZmZd d lmZeeZ d Z!dZ"d dgZ#d,ej$ej%ej&e'dddZ(d-ej)ej%ee'dddZ*Gddde j+Z,Gddde j-Z.Gddde j-Z/GdddeZ0dZ1dZ2ed e1Gd!d"d"e0Z3ed#e1Gd$d%d%e0Z4ed&e1Gd'd(d(e0Z5ed)e1Gd*d+d+e0Z6dS).z PyTorch BioGPT model.N)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentions SequenceClassifierOutputWithPastTokenClassifierOutput)PreTrainedModel)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardlogging) BioGptConfigzmicrosoft/biogptrzmicrosoft/BioGPT-Large)input_ids_shapedtypedevicepast_key_values_lengthcCs|\}}tj||ftjt|j|d|d}tj|d|d}|||d|ddkd| |}|dkrtj tj ||||d|gdd}|ddddddf |d|||S)zB Make causal mask used for bi-directional self-attention. rrrrrZdimN) torchfulltensorfinfominarangesizeZ masked_fill_viewtocatzerosexpand)rrrrbsztgt_lenmaskZ mask_condr-s/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/transformers/models/biogpt/modeling_biogpt.py_make_causal_mask:s&"  r/)r,rr+cCsj|\}}|dur|n|}|ddddddf|d|||}d|}||tjt|jS)z_ Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`. Nr?)r$r)r&Z masked_fillrboolr!r")r,rr+r*src_lenZ expanded_maskZ inverted_maskr-r-r. _expand_maskLs  *r3cs@eZdZdZeedfdd Zd ejedfdd ZZ S) BioGptLearnedPositionalEmbeddingzN This module learns positional embeddings up to a fixed maximum size. )num_embeddings embedding_dimcsd|_t||j|dS)N)offsetsuper__init__)selfr5r6 __class__r-r.r:`sz)BioGptLearnedPositionalEmbedding.__init__r)attention_maskrcsN|}tj|dd||d}|dd|df}t||jS)z3`input_ids_shape` is expected to be [bsz x seqlen].rrN)longrZcumsumtype_asr9forwardr8)r;r>r positionsr<r-r.rAfs z(BioGptLearnedPositionalEmbedding.forward)r) __name__ __module__ __qualname____doc__intr:r LongTensorrA __classcell__r-r-r<r.r4[sr4c seZdZdZdeeeeedfdd Zej eedd d Z dej e ej e e ej e ej e ej ee ej e ej e e ej fd d dZ ZS)BioGptAttentionz=Multi-headed attention from 'Attention Is All You Need' paperFT) embed_dim num_headsdropout is_decoderbiascst||_||_||_|||_|j||jkrNtd|jd|d|jd|_||_t j |||d|_ t j |||d|_ t j |||d|_ t j |||d|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g࿩rP)r9r:rLrMrNhead_dim ValueErrorscalingrOrLineark_projv_projq_projout_proj)r;rLrMrNrOrPr<r-r.r:ws"    zBioGptAttention.__init__)r seq_lenr*cCs ||||j|jddS)Nrr7)r%rMrR transpose contiguous)r;r rZr*r-r-r._shapeszBioGptAttention._shapeN) hidden_stateskey_value_statespast_key_valuer>layer_head_maskoutput_attentionsreturncCsx|du}|\}} } |||j} |r\|dur\|djd|jdkr\|d} |d} n|r|||d|} |||d|} n|dur|||d|} |||d|} tj|d| gdd} tj|d| gdd} n(|||d|} |||d|} |j r | | f}||j d|j f}|| | |j |} | j |} | j |} | d}t| | dd}|||j | |fkrtd||j | |fd||dur ||d| |fkrtd |d| |fd|| ||j | ||}| ||j | |}tjj|dd}|dur||j fkrhtd |j fd|| dddd| ||j | |}| ||j | |}|r| ||j | |}| ||j | |}nd}tjj||j|jd }t|| }|||j | |j fkr4td ||j | |j fd|| ||j | |j }|dd}| || |j}||}|||fS) z#Input shape: Batch x Time x ChannelNrr7rrrz$Attention weights should be of size z , but is z!Attention mask should be of size z/Head mask for a single layer should be of size ptrainingz `attn_output` should be of size )r$rXrTshaper]rVrWrr'rOrMrRr%ZreshapeZbmmr[rSr functionalZsoftmaxrNrfrLrY)r;r^r_r`r>rarbZis_cross_attentionr*r+_Z query_statesZ key_statesZ value_statesZ proj_shaper2Z attn_weightsZattn_weights_reshapedZ attn_probsZ attn_outputr-r-r.rAs        "   zBioGptAttention.forward)rKFT)NNNNF)rCrDrErFrGfloatr1r:rTensorr]rrrArIr-r-r<r.rJts4 rJc sxeZdZedfdd Zd ejeejeejeeejee ee eej eeej ej ffddd Z Z S) BioGptDecoderLayerconfigcst|j|_t|j|j|jdd|_|j|_ t |j |_ |j |_ t|j|_t|j|j|_t|j|j|_t|j|_dS)NT)rLrMrNrO)r9r: hidden_sizerLrJZnum_attention_headsZattention_probs_dropout_prob self_attnhidden_dropout_probrNr Z hidden_act activation_fnactivation_dropoutr LayerNormself_attn_layer_normrUZintermediate_sizefc1fc2final_layer_normr;rnr<r-r.r:s  zBioGptDecoderLayer.__init__NFT)r^r>rar`rb use_cachercc Cs|}||}|dur"|ddnd}|j|||||d\}} } tjj||j|jd}||}|}||}||}||}tjj||j |jd}| |}tjj||j|jd}||}|f} |r| | f7} |r| | f7} | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. past_key_value (`Tuple(torch.FloatTensor)`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). Nr7)r^r`r>rarbrd) rurprrhrNrfrxrvrrrsrw) r;r^r>rar`rbrzZresidualZself_attn_past_key_valueZself_attn_weightsZpresent_key_valueoutputsr-r-r.rA#s4        zBioGptDecoderLayer.forward)NNNFT) rCrDrErr:rrkrrr1 FloatTensorrArIr-r-r<r.rls rlc@s.eZdZdZeZdZdZddZd ddZ d S) BioGptPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. biogptTcCst|tjr:|jjjd|jjd|jdur|jj nft|tj rz|jjjd|jjd|j dur|jj|j  n&t|tj r|jj |jj ddS)zInitialize the weightsrK)meanZstdNr0) isinstancerrUZweightdataZnormal_rnZinitializer_rangerPZzero_ Embedding padding_idxrtZfill_)r;moduler-r-r. _init_weightsls      z#BioGptPreTrainedModel._init_weightsFcCst|tr||_dSN)r BioGptModelgradient_checkpointing)r;rvaluer-r-r._set_gradient_checkpointing|s z1BioGptPreTrainedModel._set_gradient_checkpointingN)F) rCrDrErFr config_classZbase_model_prefixZsupports_gradient_checkpointingrrr-r-r-r.r}bs r}aJ This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`~BioGptConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z`The bare BioGPT Model transformer outputting raw hidden-states without any specific head on top.cseZdZedfdd ZddZddZdd Zee d e e e ed deejeejeejeejeeeejeeeeeeeeeee fd ddZZS)rrmcst|_j|_j|_j|_j|_ j rDt jnd|_ tj|j|j |_tj|j|_tfddtjD|_t|j|_d|_|dS)Nr0csg|] }tqSr-)rl).0rirmr-r. z(BioGptModel.__init__..F)r9r:rn layerdroprqrNrorL pad_token_idrZscale_embeddingmathsqrt embed_scalerr vocab_size embed_tokensr4Zmax_position_embeddingsembed_positionsZ ModuleListrangeZnum_hidden_layerslayersrt layer_normr post_initryr<rmr.r:s  zBioGptModel.__init__cCs|jSrrr;r-r-r.get_input_embeddingssz BioGptModel.get_input_embeddingscCs ||_dSrrr;rr-r-r.set_input_embeddingssz BioGptModel.set_input_embeddingscCs`d}|ddkr$t||j|j|d}|dur\t||j|dd|j}|durT|n||}|S)Nrr)rr)r+)r/rrr3r&)r;r> input_shape inputs_embedsrZcombined_attention_maskZexpanded_attn_maskr-r-r._prepare_decoder_attention_masks z+BioGptModel._prepare_decoder_attention_maskbatch_size, sequence_length checkpoint output_typerN) input_idsr> head_maskrpast_key_valuesrzrboutput_hidden_states return_dictrcc  sJdur n|jj|dur |n|jj}dur4n|jj| durH| n|jj} |durj|durjtdnN|dur|} | } n8|dur|dd} |dddddf} ntd|dur|ddjdnd} |dur|| |j }|durt j |jddt j |j d}n<|jd| | dkrTtd|jdd | | dd ||| } ||| || }|| }tjj||j|jd }|jr|jrrtd d |rdnd}rdnd}d}rdnd}t|jD]\}}|r||f7}tdd}|jr(||jkr(q|dur:||nd}|jr|jrfdd}t jj|||||dur||ndd}n&||||dur||nd|d}|d}r||rdndf7}r||df7}q|r||f7}||}r|nd}| s8tdd|||||fDSt |||||dS)NzDYou cannot specify both input_ids and inputs_embeds at the same timerz5You have to specify either input_ids or inputs_embedsrr7rrz'The provided attention mask has length z, but its length should be z0 (sum of the lengths of current and past inputs)rdzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fr-csfdd}|S)Ncsg|RSrr-)inputs)rrbrzr-r.custom_forwardOszJBioGptModel.forward..create_custom_forward..custom_forwardr-)rrrbrz)rr.create_custom_forwardNsz2BioGptModel.forward..create_custom_forward)r>rar`rbrzcss|]}|dur|VqdSrr-)rvr-r-r. wsz&BioGptModel.forward..)Zlast_hidden_staterr^ attentionscross_attentions)!rnrbrrzuse_return_dictrSr$rgrrrZonesr1rrrrrhrNrfrloggerZ warning_once enumeraterrandomuniformrutilsrrtupler )r;rr>rrrrzrbrrinputrrrBr^Zall_hidden_statesZall_self_attnsZall_cross_attentionsZnext_decoder_cacheidxZ decoder_layerZdropout_probabilityr`rZ layer_outputsZ next_cacher-rr.rAs            zBioGptModel.forward) NNNNNNNNN)rCrDrErr:rrrrBIOGPT_INPUTS_DOCSTRINGformatr_CHECKPOINT_FOR_DOCr _CONFIG_FOR_DOCrrrHr|rrkr1rrArIr-r-r<r.rs>  rzHBioGPT Model with a `language modeling` head on top for CLM fine-tuning.cseZdZdgZfddZddZddZee de e e e d deejeejeejeejeeeejeejeeeeeeeeeee fd d d ZdddZeddZZS)BioGptForCausalLMzoutput_projection.weightcs8t|t||_tj|j|jdd|_| dSNFrQ) r9r:rr~rrUroroutput_projectionrryr<r-r.r:s  zBioGptForCausalLM.__init__cCs|jSrrrr-r-r.get_output_embeddingssz'BioGptForCausalLM.get_output_embeddingscCs ||_dSrr)r;Znew_embeddingsr-r-r.set_output_embeddingssz'BioGptForCausalLM.set_output_embeddingsrrN) rr>rrrlabelsrzrbrrrcc  Cs| dur | n|jj} |j|||||||| | d } | d} || } d}|dur| ddddddf}|ddddf}t}||d|jj|d}| s| f| dd}|dur|f|S|St|| | j | j | j | j dS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` N)r>rrrrzrbrrrrr)losslogitsrr^rr) rnrr~rr\rr%rr rr^rr)r;rr>rrrrrzrbrrr{Zsequence_outputZprediction_scoresZlm_lossZshifted_prediction_scoresloss_fctoutputr-r-r.rAs>  zBioGptForCausalLM.forwardcKsX|r|dddfd}|dur4|dur4d|i}nd|i}||||dd|S)Nrrrrz)r>rrz)Z unsqueezeupdateget)r;rr>rrkwargsZ model_inputsr-r-r.prepare_inputs_for_generations z/BioGptForCausalLM.prepare_inputs_for_generationcs.d}|D] }|tfdd|Df7}q|S)Nr-c3s|]}|dVqdS)rN)Z index_select)rZ past_statebeam_idxr-r.rrz3BioGptForCausalLM._reorder_cache..)r)rrZreordered_pastZ layer_pastr-rr._reorder_cachesz BioGptForCausalLM._reorder_cache) NNNNNNNNNN)NN)rCrDrEZ_keys_to_ignore_on_load_missingr:rrrrrrrr rrrrHr|rrkr1rrAr staticmethodrrIr-r-r<r.rsJ   : rz BioGPT Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. cseZdZfddZeeeeee dde e j e e j e e j e e j e eee je e j e e j e ee ee ee eeeefd ddZZS) BioGptForTokenClassificationcsjt||j|_t||_t|dr:|jdur:|j}n|j}t ||_ t |j |j|_ |dS)Nclassifier_dropout)r9r: num_labelsrr~hasattrrrqrZDropoutrNrUro classifierr)r;rnrr<r-r.r:s   z%BioGptForTokenClassification.__init__rN) rtoken_type_idsr>rrrrrzrbrrrcc  Cs | dur | n|jj} |j||||||| | | d } | d} || } || }d}|durt}|dur|ddk}|d|j}t ||dt |j  |}|||}n||d|j|d}| s|f| dd}|dur|f|S|St ||| j| jdS) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrr>rrrzrbrrrrrr7)rrr^r)rnrr~rNrrr%rrwherer Z ignore_indexr@rr^r)r;rrr>rrrrrzrbrrtransformer_outputsr^rrrZ active_lossZ active_logitsZ active_labelsrr-r-r.rA sF    z$BioGptForTokenClassification.forward) NNNNNNNNNNN)rCrDrEr:rrrrrrrrrHr|rrkr1rrArIr-r-r<r.rs@  ra The BioGpt Model transformer with a sequence classification head on top (linear layer). [`BioGptForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it is required to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). cseZdZedfdd Zeeeee e dd e e j e e je e je eee je e je e j e ee ee ee eeee fd ddZd d Zd d ZZS)BioGptForSequenceClassificationrmcs@t||j|_t||_tj|j|jdd|_| dSr) r9r:rrr~rrUroscorerryr<r-r.r:cs   z(BioGptForSequenceClassification.__init__rN) rr>rrrrrzrbrrrcc  Cs | dur | n|jj} |j|||||||| | d } | d} || } |dur^|jdd\}}n|jdd\}}|jjdurd}nD|durt||jjdd | j }nd}t |j jd| tj|| j d|f}d}|dur|jjdurJ|jdkrd |j_n:|jdkrB|jtjks8|jtjkrBd |j_nd |j_|jjd krt}|jdkr~|||}n |||}nN|jjd krt}||d|j|d}n|jjd krt}|||}| s|f| dd}|dur|f|S|St||| j| j| jd S) rNrrr7rrz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`rZ regressionZsingle_label_classificationZmulti_label_classification)rrrr^r)rnrr~rrgrrnesumr&rrwarningr=rCr#Z problem_typerrr?rGrZsqueezerr%rr rr^r)r;rr>rrrrrzrbrrrr^rZ batch_sizeZsequence_lengthZ pooled_logitsrrrr-r-r.rAlsn   $    (    z'BioGptForSequenceClassification.forwardcCs|jjSrr~rrr-r-r.rsz4BioGptForSequenceClassification.get_input_embeddingscCs ||j_dSrrrr-r-r.rsz4BioGptForSequenceClassification.set_input_embeddings) NNNNNNNNNN)rCrDrErr:rrrrr rrrrHr|rrkr1rrArrrIr-r-r<r.rSs@  Xr)r)N)7rFrrtypingrrrrZtorch.utils.checkpointrZtorch.nnrrrZ activationsr Zmodeling_outputsr r r rZmodeling_utilsrrrrrrZconfiguration_biogptrZ get_loggerrCrrrZ$BIOGPT_PRETRAINED_MODEL_ARCHIVE_LISTSizerrrGr/rkr3rr4ModulerJrlr}ZBIOGPT_START_DOCSTRINGrrrrrr-r-r-r.sd       T 6@oU