a  dy9@sddlZddlmZddlmZmZmZmZmZddl Z ddl m Z m Z mZddlmZeeZdZdd iZdd d d iZd d d ZgdZGdddeZdS)N)copyfile)AnyDictListOptionalTuple) AddedToken BatchEncodingPreTrainedTokenizer)loggingu▁ vocab_filezsentencepiece.bpe.modelzVhttps://huggingface.co/facebook/mbart-large-en-ro/resolve/main/sentencepiece.bpe.modelzUhttps://huggingface.co/facebook/mbart-large-cc25/resolve/main/sentencepiece.bpe.model)zfacebook/mbart-large-en-rozfacebook/mbart-large-cc25i)Zar_ARcs_CZde_DEen_XXZes_XXet_EEfi_FIZfr_XXgu_INhi_INit_ITZja_XXkk_KZko_KRlt_LTlv_LVZmy_MMne_NPZnl_XXro_ROru_RUsi_LKtr_TRvi_VNzh_CNc seZdZUdZeZeZeZ ddgZ gZ e e ed<gZe e ed<dBeeeefd fd d ZddZddZeddZeedddZejed dddZdCe e ee e ee e dfdd ZdDe e ee e e e ddd ZdEe e ee e e e dd!d"Zeeeeed#d$d%Zd&d'Z ee ed(d)d*Z!d+d,Z"d-d.Z#d/d0Z$dFeeee%ed1d2d3Z&dGe eeee eee'd6fd7d8 Z(d9d:Z)d;d<Z*d dd=d>Z+ed d?d@dAZ,Z-S)HMBartTokenizeruT Construct an MBART tokenizer. Adapted from [`RobertaTokenizer`] and [`XLNetTokenizer`]. Based on [SentencePiece](https://github.com/google/sentencepiece). The tokenization method is ` ` for source language documents, and ` ` for target language documents. Examples: ```python >>> from transformers import MBartTokenizer >>> tokenizer = MBartTokenizer.from_pretrained("facebook/mbart-large-en-ro", src_lang="en_XX", tgt_lang="ro_RO") >>> example_english_phrase = " UN Chief Says There Is No Military Solution in Syria" >>> expected_translation_romanian = "Şeful ONU declară că nu există o soluţie militară în Siria" >>> inputs = tokenizer(example_english_phrase, text_target=expected_translation_romanian, return_tensors="pt") ```Z input_idsZattention_mask prefix_tokens suffix_tokensN)sp_model_kwargsc st|trt|dddn|}| dur(in| _tjf|||||||d| | | jd |tjfij_j t||_ ddddd _ d_ t j_fd d ttD_d d jD_t jt jj j d <j jdd j D_tj_| durNjfdd| D| dur\| nd_jj_| _jdS)NTF)lstriprstrip) bos_token eos_token unk_token sep_token cls_token pad_token mask_tokentokenizer_filesrc_langtgt_langadditional_special_tokensr)rr)r$r'r%r&cs"i|]\}}|j|jqS) sp_model_sizefairseq_offset).0icodeselfr9u/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/transformers/models/mbart/tokenization_mbart.py sz+MBartTokenizer.__init__..cSsi|]\}}||qSr9r9r<kvr9r9rArBr(cSsi|]\}}||qSr9r9rCr9r9rArBrFcsg|]}|jvr|qSr9)_additional_special_tokens)r<tr?r9rA rFz+MBartTokenizer.__init__..r) isinstancestrr r)super__init__spmSentencePieceProcessorsp_modelLoadr fairseq_tokens_to_idsr;lenr: enumerateFAIRSEQ_LANGUAGE_CODESlang_code_to_iditemsZid_to_lang_codeupdatefairseq_ids_to_tokenslistkeysrGextend _src_langZcur_lang_code_idr5set_src_lang_special_tokens)r@r r,r-r/r0r.r1r2r3r4r5r)r6kwargs __class__r?rArMRsP      zMBartTokenizer.__init__cCs$|j}d|d<|j|d<|S)NrPsp_model_proto)__dict__copyrPserialized_model_proto)r@stater9r9rA __getstate__s zMBartTokenizer.__getstate__cCs<||_t|dsi|_tjfi|j|_|j|jdS)Nr))rchasattrr)rNrOrPZLoadFromSerializedProtorb)r@dr9r9rA __setstate__s  zMBartTokenizer.__setstate__cCst|jt|j|jdS)Nr7)rSrPrVr;r?r9r9rA vocab_sizeszMBartTokenizer.vocab_size)returncCs|jSN)r]r?r9r9rAr4szMBartTokenizer.src_lang) new_src_langrlcCs||_||jdSrm)r]r^)r@rnr9r9rAr4sF) token_ids_0 token_ids_1already_has_special_tokensrlcsx|rtj||ddSdgt|j}dgt|j}|durT|dgt||S|dgt|dgt||S)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rorprqr7Nr)rLget_special_tokens_maskrSr"r#)r@rorprqZ prefix_onesZ suffix_onesr`r9rArrsz&MBartTokenizer.get_special_tokens_mask)rorprlcCs,|dur|j||jS|j|||jS)ab Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. An MBART sequence has the following format, where `X` represents the sequence: - `input_ids` (for encoder) `X [eos, src_lang_code]` - `decoder_input_ids`: (for decoder) `X [eos, tgt_lang_code]` BOS is never used. Pairs of sequences are not the expected use case, but they will be handled without a separator. Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. N)r"r#)r@rorpr9r9rA build_inputs_with_special_tokenssz/MBartTokenizer.build_inputs_with_special_tokenscCsP|jg}|jg}|dur.t|||dgSt||||||dgS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. mBART does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of zeros. Nr)Z sep_token_idZ cls_token_idrS)r@rorpsepclsr9r9rA$create_token_type_ids_from_sequencess z3MBartTokenizer.create_token_type_ids_from_sequences)return_tensorsr4r5cKsJ|dus|durtd||_||fd|d|}||}||d<|S)zIUsed by translation pipeline, to prepare inputs for the generate functionNzATranslation requires a `src_lang` and a `tgt_lang` for this modelT)Zadd_special_tokensrwZforced_bos_token_id) ValueErrorr4Zconvert_tokens_to_ids)r@Z raw_inputsrwr4r5 extra_kwargsinputsZ tgt_lang_idr9r9rA_build_translation_inputs s z(MBartTokenizer._build_translation_inputscs(fddtjD}|j|S)Ncsi|]}||qSr9)Zconvert_ids_to_tokens)r<r=r?r9rArBrFz,MBartTokenizer.get_vocab..)rangerkrXZadded_tokens_encoder)r@Zvocabr9r?rA get_vocabs zMBartTokenizer.get_vocab)textrlcCs|jj|tdS)N)Zout_type)rPencoderK)r@r~r9r9rA _tokenizeszMBartTokenizer._tokenizecCs4||jvr|j|S|j|}|r.||jS|jS)z0Converts a token (str) in an id using the vocab.)rRrPZ PieceToIdr;Z unk_token_id)r@tokenZspm_idr9r9rA_convert_token_to_id!s   z#MBartTokenizer._convert_token_to_idcCs&||jvr|j|S|j||jS)z=Converts an index (integer) in a token (str) using the vocab.)rYrPZ IdToPiecer;)r@indexr9r9rA_convert_id_to_token*s  z#MBartTokenizer._convert_id_to_tokencCsd|td}|S)zIConverts a sequence of tokens (strings for sub-words) in a single string. )joinreplaceSPIECE_UNDERLINEstrip)r@tokensZ out_stringr9r9rAconvert_tokens_to_string0sz'MBartTokenizer.convert_tokens_to_string)save_directoryfilename_prefixrlcCstj|s"td|ddStj||r6|dndtd}tj|jtj|krztj |jrzt |j|nLtj |jst |d$}|j }||Wdn1s0Y|fS)NzVocabulary path (z) should be a directory-rr wb)ospathisdirloggererrorrVOCAB_FILES_NAMESabspathr isfileropenrPrewrite)r@rrZout_vocab_filefiZcontent_spiece_modelr9r9rAsave_vocabulary5s (  (zMBartTokenizer.save_vocabularyrr) src_textsr4 tgt_textsr5rlc s"||_||_tj||fi|Srm)r4r5rLprepare_seq2seq_batch)r@rr4rr5r_r`r9rArFsz$MBartTokenizer.prepare_seq2seq_batchcCs ||jSrm)r^r4r?r9r9rA_switch_to_input_modeRsz$MBartTokenizer._switch_to_input_modecCs ||jSrm)set_tgt_lang_special_tokensr5r?r9r9rA_switch_to_target_modeUsz%MBartTokenizer._switch_to_target_modecCs$|j||_g|_|j|jg|_dS)z_Reset the special tokens to the source lang setting. No prefix and suffix=[eos, src_lang_code].NrVZ cur_lang_coder"Z eos_token_idr#)r@r4r9r9rAr^Xs z*MBartTokenizer.set_src_lang_special_tokens)langrlcCs$|j||_g|_|j|jg|_dS)zcReset the special tokens to the target language setting. No prefix and suffix=[eos, tgt_lang_code].Nr)r@rr9r9rAr^s z*MBartTokenizer.set_tgt_lang_special_tokens) r$r%r%r$r&r'r(NNNNN)NF)N)N)N)rNr).__name__ __module__ __qualname____doc__rZvocab_files_names&PRETRAINED_POSITIONAL_EMBEDDINGS_SIZESZmax_model_input_sizesPRETRAINED_VOCAB_FILES_MAPZpretrained_vocab_files_mapZmodel_input_namesr"rint__annotations__r#rrrKrrMrgrjpropertyrkr4setterboolrrrsrvr{r}rrrrrrr rrrr^r __classcell__r9r9r`rAr!5s  L        r!)rshutilrtypingrrrrrZ sentencepiecerNZtokenization_utilsr r r utilsr Z get_loggerrrrrrrrUr!r9r9r9rAs$