a  d&@sdZddlZddlmZddlmZmZmZmZm Z ddl Z ddl m Z mZddlmZeeZdd iZd d id d id Zd diZGdddeZdS)zTokenization classes for LLaMA.N)copyfile)AnyDictListOptionalTuple) AddedTokenPreTrainedTokenizer)logging vocab_fileztokenizer.modelz#hf-internal-testing/llama-tokenizerzWhttps://huggingface.co/hf-internal-testing/llama-tokenizer/resolve/main/tokenizer.modelz]https://huggingface.co/hf-internal-testing/llama-tokenizer/resolve/main/tokenizer_config.json)r Ztokenizer_fileicseZdZdZeZeZeZ ddgZ d(e e e efd fd d Zd dZddZeddZddZddZddZddZddZd)e e ee dddZd*d d!Zd+eee eeeeed"fd#d$ Zd,eee eeeed%d&d'Z Z!S)-LlamaTokenizerz Construct a Llama tokenizer. Based on byte-level Byte-Pair-Encoding. Args: vocab_file (`str`): Path to the vocabulary file. Z input_idsZattention_maskNTF)sp_model_kwargsc s|dur in||_t|tr*t|dddn|}t|trFt|dddn|}t|trbt|dddn|}t|tr~t|dddn|}tjf|||||||j| d| ||_||_||_t j fi|j|_ |j |dS)NF)lstriprstrip) bos_token eos_token unk_token pad_token add_bos_token add_eos_tokenrclean_up_tokenization_spaces) r isinstancestrr super__init__r rrspmSentencePieceProcessorsp_modelLoad) selfr rrrrrrrrkwargs __class__u/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/transformers/models/llama/tokenization_llama.pyr?s,   zLlamaTokenizer.__init__cCs|j}d|d<|S)Nr!)__dict__copy)r#stater'r'r( __getstate__bs zLlamaTokenizer.__getstate__cCs,||_tjfi|j|_|j|jdSN)r)rr rr!r"r )r#dr'r'r( __setstate__gszLlamaTokenizer.__setstate__cCs |jS)zReturns vocab size)r!Zget_piece_sizer#r'r'r( vocab_sizelszLlamaTokenizer.vocab_sizecs(fddtjD}|j|S)zReturns vocab as a dictcsi|]}||qSr')Zconvert_ids_to_tokens).0ir0r'r( sz,LlamaTokenizer.get_vocab..)ranger1updateZadded_tokens_encoder)r#Zvocabr'r0r( get_vocabqs zLlamaTokenizer.get_vocabcCs|jj|tdS)zReturns a tokenized string.)Zout_type)r!encoder)r#textr'r'r( _tokenizewszLlamaTokenizer._tokenizecCs |j|S)z0Converts a token (str) in an id using the vocab.)r!Z piece_to_id)r#tokenr'r'r(_convert_token_to_id{sz#LlamaTokenizer._convert_token_to_idcCs|j|}|S)z=Converts an index (integer) in a token (str) using the vocab.)r!Z IdToPiece)r#indexr<r'r'r(_convert_id_to_tokens z#LlamaTokenizer._convert_id_to_tokencCs|g}d}d}t|D]R\}}||jvrX|s:|dkr:|d7}||j||7}d}g}q||d}q||j|7}|S)z:Converts a sequence of tokens (string) in a single string.Fr T) enumerateZall_special_tokensr!decodeappend)r#tokensZcurrent_sub_tokensZ out_stringZprev_is_specialr3r<r'r'r(convert_tokens_to_strings   z'LlamaTokenizer.convert_tokens_to_string)filename_prefixreturncCstj|s"td|ddStj||r6|dndtd}tj|jtj|krztj |jrzt |j|nLtj |jst |d$}|j }||Wdn1s0Y|fS)a Save the vocabulary and special tokens file to a directory. Args: save_directory (`str`): The directory in which to save the vocabulary. Returns: `Tuple(str)`: Paths to the files saved. zVocabulary path (z) should be a directoryN-r@r wb)ospathisdirloggererrorjoinVOCAB_FILES_NAMESabspathr isfileropenr!Zserialized_model_protowrite)r#Zsave_directoryrGZout_vocab_filefiZcontent_spiece_modelr'r'r(save_vocabularys (  (zLlamaTokenizer.save_vocabularycCsL|jr|jgng}|jr |jgng}|||}|durH||||}|Sr-)r bos_token_idr eos_token_idr# token_ids_0 token_ids_1rXrYoutputr'r'r( build_inputs_with_special_tokenss  z/LlamaTokenizer.build_inputs_with_special_tokens)r[r\already_has_special_tokensrHcs|rtj||ddS|jr"dgng}|jr2dgng}|durT|dgt||S|dgt|||dgt||S)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)r[r\r_Nr)rget_special_tokens_maskrrlen)r#r[r\r_rXrYr%r'r(ras(  z&LlamaTokenizer.get_special_tokens_mask)r[r\rHcCs`|jr|jgng}|jr |jgng}dgt|||}|dur\|dgt|||7}|S)a Creates a mask from the two sequences passed to be used in a sequence-pair classification task. An ALBERT sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` if token_ids_1 is None, only returns the first portion of the mask (0s). Args: token_ids_0 (`List[int]`): List of ids. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). rNr`)rrXrrYrbrZr'r'r($create_token_type_ids_from_sequencess z3LlamaTokenizer.create_token_type_ids_from_sequences)rrrNNTFF)N)N)NF)N)"__name__ __module__ __qualname____doc__rQZvocab_files_namesPRETRAINED_VOCAB_FILES_MAPZpretrained_vocab_files_map&PRETRAINED_POSITIONAL_EMBEDDINGS_SIZESZmax_model_input_sizesZmodel_input_namesrrrrrr,r/propertyr1r8r;r=r?rFrrWr^rintboolrarc __classcell__r'r'r%r(r 1sJ#  &r )rgrKshutilrtypingrrrrrZ sentencepiecerZtokenization_utilsr r utilsr Z get_loggerrdrNrQrhrir r'r'r'r(s