a  dT@sddlZddlmZddlmZmZmZmZmZddl Z ddl m Z m Z mZddlmZeeZdZdd d Zd d d dddddddddddddiZdddddddddddddddZgdgddZdd d!d"d#d$d%d&ZGd'd(d(eZdS))N)copyfile)AnyDictListOptionalTuple) AddedToken BatchEncodingPreTrainedTokenizer)loggingu▁zsentencepiece.bpe.modelztokenizer.json) vocab_filetokenizer_filer zOhttps://huggingface.co/uclanlp/plbart-base/resolve/main/sentencepiece.bpe.modelzahttps://huggingface.co/uclanlp/plbart-c-cpp-defect-detection/resolve/main/sentencepiece.bpe.modelzRhttps://huggingface.co/uclanlp/plbart-cs-java/resolve/main/sentencepiece.bpe.modelzUhttps://huggingface.co/uclanlp/plbart-en_XX-java/resolve/main/sentencepiece.bpe.modelzShttps://huggingface.co/uclanlp/plbart-go-en_XX/resolve/main/sentencepiece.bpe.modelz_https://huggingface.co/uclanlp/plbart-java-clone-detection/resolve/main/sentencepiece.bpe.modelzRhttps://huggingface.co/uclanlp/plbart-java-cs/resolve/main/sentencepiece.bpe.modelzUhttps://huggingface.co/uclanlp/plbart-java-en_XX/resolve/main/sentencepiece.bpe.modelz[https://huggingface.co/uclanlp/plbart-javascript-en_XX/resolve/main/sentencepiece.bpe.modelzThttps://huggingface.co/uclanlp/plbart-php-en_XX/resolve/main/sentencepiece.bpe.modelzWhttps://huggingface.co/uclanlp/plbart-python-en_XX/resolve/main/sentencepiece.bpe.modelz]https://huggingface.co/uclanlp/plbart-refine-java-medium/resolve/main/sentencepiece.bpe.modelz\https://huggingface.co/uclanlp/plbart-refine-java-small/resolve/main/sentencepiece.bpe.modelzUhttps://huggingface.co/uclanlp/plbart-ruby-en_XX/resolve/main/sentencepiece.bpe.model)zuclanlp/plbart-basez%uclanlp/plbart-c-cpp-defect-detectionzuclanlp/plbart-cs-javazuclanlp/plbart-en_XX-javazuclanlp/plbart-go-en_XXz#uclanlp/plbart-java-clone-detectionzuclanlp/plbart-java-cszuclanlp/plbart-java-en_XXzuclanlp/plbart-javascript-en_XXzuclanlp/plbart-php-en_XXzuclanlp/plbart-python-en_XXz!uclanlp/plbart-refine-java-mediumz uclanlp/plbart-refine-java-smallzuclanlp/plbart-ruby-en_XXi)__java__ __python__ __en_XX__)rrr__javascript____php____ruby____go__)baseZmultirrrrrrr)javapythonen_XXZ javascriptZphprubyZgoc seZdZUdZeZeZeZ ddgZ gZ e e ed<gZe e ed<dEeeeefd fdd ZddZddZeddZeedddZejed dddZdFe e ee e ee e dfdd ZdGe e ee e e e dd d!ZdHe e ee e e e dd"d#Zeeeeed$d%d&Zd'd(Z ee ed)d*d+Z!d,d-Z"d.d/Z#d0d1Z$dIeeee%ed2d3d4Z&dJe eeee eee'd7fd8d9 Z(d:d;Z)dd?Z+ed d@dAdBZ,eed@dCdDZ-Z.S)KPLBartTokenizera Construct an PLBART tokenizer. Adapted from [`RobertaTokenizer`] and [`XLNetTokenizer`]. Based on [SentencePiece](https://github.com/google/sentencepiece). The tokenization method is ` ` for source language documents, and ` ` for target language documents. Args: vocab_file (`str`): Path to the vocabulary file. src_lang (`str`, *optional*): A string representing the source language. tgt_lang (`str`, *optional*): A string representing the target language. bos_token (`str`, *optional*, defaults to `""`): The start of sequence token. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. sep_token (`str`, *optional*, defaults to `""`): The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens. cls_token (`str`, *optional*, defaults to `""`): The cls token, which is a special token used as the first token for all tasks. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. mask_token(`str`, *optional*, defaults to `""`): The token used for masking values. This is the token used when training this model with masking tasks. This is only used in the `"base"` tokenizer type. For `"multi"` tokenizer, masking is never done for the downstream tasks. language_codes (`str`, *optional*, defaults to `"base"`): What language codes to use. Should be one of `"base"` or `"multi"`. sp_model_kwargs (`dict`, *optional*): Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things, to set: - `enable_sampling`: Enable subword regularization. - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout. - `nbest_size = {0,1}`: No sampling is performed. - `nbest_size > 1`: samples from the nbest_size results. - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. Examples: ```python >>> from transformers import PLBartTokenizer >>> tokenizer = PLBartTokenizer.from_pretrained("uclanlp/plbart-python-en_XX", src_lang="python", tgt_lang="en_XX") >>> example_python_phrase = "def maximum(a,b,c):NEW_LINE_INDENTreturn max([a,b,c])" >>> expected_translation_english = "Returns the maximum value of a b c." >>> inputs = tokenizer(example_python_phrase, text_target=expected_translation_english, return_tensors="pt") ```Z input_idsZattention_mask prefix_tokens suffix_tokensrN)sp_model_kwargsc st|trt|dddn|}| dur(in| _tjf|||||||| | | | |jd || } | } tjfij_ j t||_ | _ t j }ddddd _d_tj _fd d t|D_d d jD_j d kr(tj tjjjd<jjdd jD_tj_|durjfdd|Dj d kr| _jdurjjnj_n"| dur| nd_jj_| _jdS)NTF)lstriprstrip) bos_token eos_token unk_token sep_token cls_token pad_token mask_tokenlanguage_codesrsrc_langtgt_langadditional_special_tokensr#rr)rr!rr cs"i|]\}}|j|jqS) sp_model_sizefairseq_offset).0icodeselfr3w/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/transformers/models/plbart/tokenization_plbart.py sz,PLBartTokenizer.__init__..cSsi|]\}}||qSr3r3r6kvr3r3r;r<rr"cSsi|]\}}||qSr3r3r=r3r3r;r<r@csg|]}|jvr|qSr3)_additional_special_tokens)r6tr9r3r; r@z,PLBartTokenizer.__init__..r) isinstancestrr r#super__init__!_convert_lang_code_special_formatspmSentencePieceProcessorsp_modelLoadr r-FAIRSEQ_LANGUAGE_CODESfairseq_tokens_to_idsr5lenr4 enumeratelang_code_to_iditemsZid_to_lang_codeupdatefairseq_ids_to_tokenslistkeysrAextend _src_langZcur_lang_code_idr/set_src_lang_special_tokens)r:r r&r'r)r*r(r+r,r-rr.r/r#r0kwargsZfairseq_language_codes __class__r9r;rGsd         zPLBartTokenizer.__init__cCs$|j}d|d<|j|d<|S)NrKsp_model_proto)__dict__copyrKserialized_model_proto)r:stater3r3r; __getstate__ s zPLBartTokenizer.__getstate__cCs<||_t|dsi|_tjfi|j|_|j|jdS)Nr#)r^hasattrr#rIrJrKZLoadFromSerializedProtor])r:dr3r3r; __setstate__s  zPLBartTokenizer.__setstate__cCsF|jdkr(t|jt|j|jdSt|jt|j|jSdS)Nrr1)r-rOrKrQr5r9r3r3r; vocab_sizes zPLBartTokenizer.vocab_size)returncCs|jSN)rXr9r3r3r;r.%szPLBartTokenizer.src_lang) new_src_langrgcCs ||}||_||jdSrh)rHrXrY)r:rir3r3r;r.)s F) token_ids_0 token_ids_1already_has_special_tokensrgcsx|rtj||ddSdgt|j}dgt|j}|durT|dgt||S|dgt|dgt||S)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rjrkrlr1Nr)rFget_special_tokens_maskrOrr)r:rjrkrlZ prefix_onesZ suffix_onesr[r3r;rm/sz'PLBartTokenizer.get_special_tokens_mask)rjrkrgcCs,|dur|j||jS|j|||jS)ac Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. An PLBART sequence has the following format, where `X` represents the sequence: - `input_ids` (for encoder) `X [eos, src_lang_code]` - `decoder_input_ids`: (for decoder) `X [eos, tgt_lang_code]` BOS is never used. Pairs of sequences are not the expected use case, but they will be handled without a separator. Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. N)rr)r:rjrkr3r3r; build_inputs_with_special_tokensMsz0PLBartTokenizer.build_inputs_with_special_tokenscCsP|jg}|jg}|dur.t|||dgSt||||||dgS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. PLBart does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of zeros. Nr)Z sep_token_idZ cls_token_idrO)r:rjrksepclsr3r3r;$create_token_type_ids_from_sequenceshs z4PLBartTokenizer.create_token_type_ids_from_sequences)return_tensorsr.r/cKs^|dus|durtd|||_|||_||fd|d|}||j}||d<|S)zIUsed by translation pipeline, to prepare inputs for the generate functionNzATranslation requires a `src_lang` and a `tgt_lang` for this modelT)Zadd_special_tokensrrZforced_bos_token_id) ValueErrorrHr.r/Zconvert_tokens_to_ids)r:Z raw_inputsrrr.r/ extra_kwargsinputsZ tgt_lang_idr3r3r;_build_translation_inputss   z)PLBartTokenizer._build_translation_inputscs(fddtjD}|j|S)Ncsi|]}||qSr3)Zconvert_ids_to_tokens)r6r7r9r3r;r<r@z-PLBartTokenizer.get_vocab..)rangerfrSZadded_tokens_encoder)r:Zvocabr3r9r; get_vocabs zPLBartTokenizer.get_vocab)textrgcCs|jj|tdS)N)Zout_type)rKencoderE)r:ryr3r3r; _tokenizeszPLBartTokenizer._tokenizecCs4||jvr|j|S|j|}|r.||jS|jS)z0Converts a token (str) in an id using the vocab.)rNrKZ PieceToIdr5Z unk_token_id)r:tokenZspm_idr3r3r;_convert_token_to_ids   z$PLBartTokenizer._convert_token_to_idcCs&||jvr|j|S|j||jS)z=Converts an index (integer) in a token (str) using the vocab.)rTrKZ IdToPiecer5)r:indexr3r3r;_convert_id_to_tokens  z$PLBartTokenizer._convert_id_to_tokencCsd|td}|S)zIConverts a sequence of tokens (strings for sub-words) in a single string. )joinreplaceSPIECE_UNDERLINEstrip)r:tokensZ out_stringr3r3r;convert_tokens_to_stringsz(PLBartTokenizer.convert_tokens_to_string)save_directoryfilename_prefixrgcCstj|s"td|ddStj||r6|dndtd}tj|jtj|krztj |jrzt |j|nLtj |jst |d$}|j }||Wdn1s0Y|fS)NzVocabulary path (z) should be a directory-rr wb)ospathisdirloggererrorrVOCAB_FILES_NAMESabspathr isfileropenrKr`write)r:rrZout_vocab_filefiZcontent_spiece_modelr3r3r;save_vocabularys (  (zPLBartTokenizer.save_vocabularyrr) src_textsr. tgt_textsr/rgc s.|||_|||_tj||fi|Srh)rHr.r/rFprepare_seq2seq_batch)r:rr.rr/rZr[r3r;rs  z%PLBartTokenizer.prepare_seq2seq_batchcCs ||jSrh)rYr.r9r3r3r;_switch_to_input_modesz%PLBartTokenizer._switch_to_input_modecCs ||jSrh)set_tgt_lang_special_tokensr/r9r3r3r;_switch_to_target_modesz&PLBartTokenizer._switch_to_target_modecCsP||}|dur|j|nd|_g|_|jdurB|j|jg|_n |jg|_dS)z_Reset the special tokens to the source lang setting. No prefix and suffix=[eos, src_lang_code].NrHrQZ cur_lang_coderZ eos_token_idr)r:r.r3r3r;rYs   z+PLBartTokenizer.set_src_lang_special_tokens)langrgcCsP||}|dur|j|nd|_g|_|jdurB|j|jg|_n |jg|_dS)zcReset the special tokens to the target language setting. No prefix and suffix=[eos, tgt_lang_code].Nrr:rr3r3r;rs   z+PLBartTokenizer.set_tgt_lang_special_tokenscCs|tvrt|n|}|S)z;Convert Language Codes to format tokenizer uses if required)FAIRSEQ_LANGUAGE_CODES_MAPrVrr3r3r;rHsz1PLBartTokenizer._convert_lang_code_special_format) rrrrr r!r"rNNNNN)NF)N)N)N)rNr)/__name__ __module__ __qualname____doc__rZvocab_files_names&PRETRAINED_POSITIONAL_EMBEDDINGS_SIZESZmax_model_input_sizesPRETRAINED_VOCAB_FILES_MAPZpretrained_vocab_files_mapZmodel_input_namesrrint__annotations__rrrrErrGrbrepropertyrfr.setterboolrmrnrqrvrxr{r}rrrrr rrrrYrrH __classcell__r3r3r[r;rjs =\          r)rshutilrtypingrrrrrZ sentencepiecerIZtokenization_utilsr r r utilsr Z get_loggerrrrrrrrMrrr3r3r3r;sh    *