a  dD@sddlZddlZddlZddlZddlmZddlmZddlm Z m Z m Z m Z m Z mZddlZddlmZddlmZeeZdd d d d d ZddiddiddiddidZddiZiZGdddeZee ee fejdddZeddddZeee e fdddZ dS)N)Path)copyfile)AnyDictListOptionalTupleUnion)PreTrainedTokenizer)loggingz source.spmz target.spmz vocab.jsonztarget_vocab.jsonztokenizer_config.json) source_spm target_spmvocabtarget_vocab_filetokenizer_config_filezHelsinki-NLP/opus-mt-en-dezIhttps://huggingface.co/Helsinki-NLP/opus-mt-en-de/resolve/main/source.spmzIhttps://huggingface.co/Helsinki-NLP/opus-mt-en-de/resolve/main/target.spmzIhttps://huggingface.co/Helsinki-NLP/opus-mt-en-de/resolve/main/vocab.jsonzThttps://huggingface.co/Helsinki-NLP/opus-mt-en-de/resolve/main/tokenizer_config.json)r rrrc seZdZdZeZeZeZ e Z ddgZ e dZdCeeeefdd fd d ZddZeedddZddZedddZeeedddZeedddZfddZfd d!Zeeed"d#d$ZdDeed%d&d'Z d(d)Z!d*d+Z"e#ed%d,d-Z$dEeeee%ed.d/d0Z&ed%d1d2Z'd3d4Z(d5d6Z)ed%d7d8Z*edd9d:d;Z+dd?Z-dFeeee.eed@dAdBZ/Z0S)GMarianTokenizeraB Construct a Marian tokenizer. Based on [SentencePiece](https://github.com/google/sentencepiece). This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: source_spm (`str`): [SentencePiece](https://github.com/google/sentencepiece) file (generally has a .spm extension) that contains the vocabulary for the source language. target_spm (`str`): [SentencePiece](https://github.com/google/sentencepiece) file (generally has a .spm extension) that contains the vocabulary for the target language. source_lang (`str`, *optional*): A string representing the source language. target_lang (`str`, *optional*): A string representing the target language. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. model_max_length (`int`, *optional*, defaults to 512): The maximum sentence length the model accepts. additional_special_tokens (`List[str]`, *optional*, defaults to `["", ""]`): Additional special tokens used by the tokenizer. sp_model_kwargs (`dict`, *optional*): Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things, to set: - `enable_sampling`: Enable subword regularization. - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout. - `nbest_size = {0,1}`: No sampling is performed. - `nbest_size > 1`: samples from the nbest_size results. - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. Examples: ```python >>> from transformers import MarianForCausalLM, MarianTokenizer >>> model = MarianForCausalLM.from_pretrained("Helsinki-NLP/opus-mt-en-de") >>> tokenizer = MarianTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-de") >>> src_texts = ["I am a small frog.", "Tom asked his teacher for advice."] >>> tgt_texts = ["Ich bin ein kleiner Frosch.", "Tom bat seinen Lehrer um Rat."] # optional >>> inputs = tokenizer(src_texts, text_target=tgt_texts, return_tensors="pt", padding=True) >>> outputs = model(**inputs) # should work ```Z input_idsZattention_maskz>>.+<rF)sp_model_kwargsreturnc  s*| dur in| |_tjf||||| | |j|| d | t|sTJd|| |_t||_|j|jvrxt d|j |jvsJ| rt||_ dd|j D|_ g|_n(dd|j D|_ dd|jD|_||_||_||g|_t||j|_t||j|_|j|_|j|_|dS) N) source_lang target_lang unk_token eos_token pad_tokenmodel_max_lengthrrseparate_vocabszcannot find spm source z token must be in vocabcSsi|]\}}||qSr .0kvr r w/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/transformers/models/marian/tokenization_marian.py z,MarianTokenizer.__init__..cSsi|]\}}||qSr r r!r r r%r&r'cSs$g|]}|dr|dr|qS)z>>z<<) startswithendswithr"r#r r r% r'z,MarianTokenizer.__init__..)rsuper__init__rexistsr load_jsonencoderrKeyErrorrtarget_encoderitemsdecoderZsupported_language_codesrr spm_filesload_spm spm_source spm_target current_spmcurrent_encoder_setup_normalizer)selfr rrrrrrrrrrrkwargs __class__r r%r-sD      zMarianTokenizer.__init__c CsNzddlm}||jj|_Wn*ttfyHtddd|_Yn0dS)Nr)MosesPunctNormalizerz$Recommended: pip install sacremoses.cSs|SNr )xr r r%r'z3MarianTokenizer._setup_normalizer..) Z sacremosesr@r normalizepunc_normalizer ImportErrorFileNotFoundErrorwarningswarn)r<r@r r r%r;s   z!MarianTokenizer._setup_normalizer)rBrcCs|r||SdS)zHCover moses empty string edge case. They return empty list for '' input!)rE)r<rBr r r%rDszMarianTokenizer.normalizecCs|j||j|jSrA)r:getr)r<tokenr r r%_convert_token_to_idsz$MarianTokenizer._convert_token_to_id)textcCs2|j|}|r|dgng}||jd|fS)z6Remove language codes like >>fr<< before sentencepiecerrJ)language_code_rematchgroupsub)r<rNrPcoder r r%remove_language_codes z$MarianTokenizer.remove_language_code)rNrcCs&||\}}|jj|td}||S)N)Zout_type)rTr9encodestr)r<rNrSpiecesr r r% _tokenizeszMarianTokenizer._tokenize)indexrcCs|j||jS)z?Converts an index (integer) in a token (str) using the decoder.)r4rKr)r<rYr r r%_convert_id_to_tokensz$MarianTokenizer._convert_id_to_tokenc stj|fi|S)ao Convert a list of lists of token ids into a list of strings by calling decode. Args: sequences (`Union[List[int], List[List[int]], np.ndarray, torch.Tensor, tf.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. clean_up_tokenization_spaces (`bool`, *optional*): Whether or not to clean up the tokenization spaces. If `None`, will default to `self.clean_up_tokenization_spaces` (available in the `tokenizer_config`). use_source_tokenizer (`bool`, *optional*, defaults to `False`): Whether or not to use the source tokenizer to decode sequences (only applicable in sequence-to-sequence problems). kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific decode method. Returns: `List[str]`: The list of decoded sentences. )r, batch_decode)r< sequencesr=r>r r%r[szMarianTokenizer.batch_decodec stj|fi|S)a Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up tokenization spaces. Similar to doing `self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))`. Args: token_ids (`Union[int, List[int], np.ndarray, torch.Tensor, tf.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. clean_up_tokenization_spaces (`bool`, *optional*): Whether or not to clean up the tokenization spaces. If `None`, will default to `self.clean_up_tokenization_spaces` (available in the `tokenizer_config`). use_source_tokenizer (`bool`, *optional*, defaults to `False`): Whether or not to use the source tokenizer to decode sequences (only applicable in sequence-to-sequence problems). kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific decode method. Returns: `str`: The decoded sentence. )r,decode)r<Z token_idsr=r>r r%r]szMarianTokenizer.decode)tokensrcCsj|jr |jn|j}g}d}|D]4}||jvrH||||d7}g}q||q|||7}|S)zQUses source spm if _decode_use_source_tokenizer is True, and target spm otherwiserJ )Z_decode_use_source_tokenizerr7r8Zall_special_tokensZ decode_piecesappendstrip)r<r^Zsp_modelZcurrent_sub_tokensZ out_stringrLr r r%convert_tokens_to_string s  z(MarianTokenizer.convert_tokens_to_string)rcCs$|dur||jgS|||jgS)z=Build model inputs from a sequence by appending eos_token_id.N)Z eos_token_id)r< token_ids_0 token_ids_1r r r% build_inputs_with_special_tokenss z0MarianTokenizer.build_inputs_with_special_tokenscCs|j|_|j|_dSrA)r7r9r0r:r<r r r%_switch_to_input_mode"sz%MarianTokenizer._switch_to_input_modecCs|j|_|jr|j|_dSrA)r8r9rr2r:rfr r r%_switch_to_target_mode&sz&MarianTokenizer._switch_to_target_modecCs t|jSrA)lenr0rfr r r% vocab_size+szMarianTokenizer.vocab_size)save_directoryfilename_prefixrc Cstj|s"td|ddSg}|jrtj||r@|dndtd}tj||rb|dndtd}t|j |t|j || || |n8tj||r|dndtd}t|j || |t tdtdg|j |j|jgD]\}}} tj||r|dnd|} tj|tj| kr`tj|r`t|| | | qtj|st| d "} | } | | Wdn1s0Y| | qt|S) NzVocabulary path (z) should be a directory-rJrrr rwb)ospathisdirloggererrorrjoinVOCAB_FILES_NAMES save_jsonr0r2r`zipr5r7r8abspathisfileropenZserialized_model_protowritetuple) r<rkrlZ saved_filesZout_src_vocab_fileZout_tgt_vocab_fileZout_vocab_fileZspm_save_filenameZ spm_orig_pathZ spm_modelZ spm_save_pathfiZcontent_spiece_modelr r r%save_vocabulary/sN        (    * zMarianTokenizer.save_vocabularycCs|SrA) get_src_vocabrfr r r% get_vocab\szMarianTokenizer.get_vocabcCst|jfi|jSrA)dictr0Zadded_tokens_encoderrfr r r%r_szMarianTokenizer.get_src_vocabcCst|jfi|jSrA)rr2Zadded_tokens_decoderrfr r r% get_tgt_vocabbszMarianTokenizer.get_tgt_vocabcCs"|j}|dddD|S)NcSsi|] }|dqSrAr r*r r r%r&hr'z0MarianTokenizer.__getstate__..)r7r8r9rEr)__dict__copyupdate)r<stater r r% __getstate__es   zMarianTokenizer.__getstate__)drcsF|_tdsi_fddjD\__j_dS)Nrc3s|]}t|jVqdSrA)r6r)r"frfr r% sr'z/MarianTokenizer.__setstate__..)rhasattrrr5r7r8r9r;)r<rr rfr% __setstate__ls  zMarianTokenizer.__setstate__cOsdS)zJust EOSr )r<argsr=r r r%num_special_tokens_to_addwsz)MarianTokenizer.num_special_tokens_to_addcs(t|j|jfdd|DS)Ncsg|]}|vrdndqS)rrr )r"rBall_special_idsr r%r+~r'z7MarianTokenizer._special_token_mask..)setrremoveZ unk_token_id)r<seqr rr%_special_token_mask{s  z#MarianTokenizer._special_token_mask)rcrdalready_has_special_tokensrcCs>|r||S|dur&||dgS|||dgSdS)zCGet list where entries are [1] if a token is [eos] or [pad] else 0.Nr)r)r<rcrdrr r r%get_special_tokens_masks  z'MarianTokenizer.get_special_tokens_mask) NNNrrrrNF)N)N)NF)1__name__ __module__ __qualname____doc__ruZvocab_files_namesPRETRAINED_VOCAB_FILES_MAPZpretrained_vocab_files_mapPRETRAINED_INIT_CONFIGURATIONZpretrained_init_configuration&PRETRAINED_POSITIONAL_EMBEDDINGS_SIZESZmax_model_input_sizesZmodel_input_namesrecompilerOrrrVrr-r;rDrMrTrrXintrZr[r]rbrergrhpropertyrjrr~rrrrrrrboolr __classcell__r r r>r%r=s^:  =   -  r)rprrcCstjfi|}|||SrA) sentencepieceSentencePieceProcessorLoad)rprZspmr r r%r6s r6)rprcCs>t|d }tj||ddWdn1s00YdS)Nw)indent)rzjsondump)datarprr r r%rvs rvcCs8t|d}t|WdS1s*0YdS)Nr)rzrload)rprr r r%r/s r/)!rrorrHpathlibrshutilrtypingrrrrrr rZtokenization_utilsr utilsr Z get_loggerrrrrurrrrrVrr6rvr/r r r r%sD       Q