a  d@s@dZddlZddlZddlZddlmZddlmZmZm Z m Z m Z m Z ddl mZddlmZddlmZddlmZddlmZmZmZmZd d lmZd d lmZd d l m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*d d l+m,Z,m-Z-m.Z.e./e0Z1dZ2dZ3dZ4dZ5e!d7Z!eeeedZ6de2iZ7e-e!Gddde&Z8dS)z Tokenization classes for fast tokenizers (provided by HuggingFace's tokenizers library). For slow (python) tokenizers see tokenization_utils.py N) defaultdict)AnyDictListOptionalTupleUnion)Encoding) Tokenizer)Decoder) BpeTrainerUnigramTrainerWordLevelTrainerWordPieceTrainer)convert_slow_tokenizer)PreTrainedTokenizer) INIT_TOKENIZER_DOCSTRING AddedToken BatchEncodingPreTokenizedInputPreTokenizedInputPairPreTrainedTokenizerBaseSpecialTokensMixin TextInput TextInputPairTruncationStrategy)PaddingStrategyadd_end_docstringsloggingztokenizer.jsonzspecial_tokens_map.jsonztokenizer_config.jsonzadded_tokens.jsonu tokenizer_object ([`tokenizers.Tokenizer`]): A [`tokenizers.Tokenizer`] object from 🤗 tokenizers to instantiate from. See [Using tokenizers from 🤗 tokenizers](../fast_tokenizers) for more information. tokenizer_file ([`str`]): A path to a local JSON file representing a previously serialized [`tokenizers.Tokenizer`] object from 🤗 tokenizers. )BPEUnigram WordLevel WordPiecetokenizer_filecsheZdZUdZeZdZeed<dZ e ed<fddZ e e dd d Z e edd d Zeeefdd dZe eeefdddZeeefdddZedddZe edddZe edddZdFeee ee e e e e e eeeefeefd ddZe eeefe eeefdddZ!eed d!d"Z"eeed#d$d%Z#dGee ee$fed&d'd(Z%dHe ed)d*d+Z&dIe eeefe e eeefd,d-d.Z'dJeeee eed/d0d1Z(e)e*eeeed2d3d4Z+de)j,e*j-dd5ddddddddddfe ee.ee/ee0ee1fe e)e*eeee eeeeee ee e e e e e e2d6d7d8Z3dde)j,e*j-dd5ddddddddddfe e.e0fee e.e0fe e)e*eeee eeee ee ee e e e e e e2d9d:d;Z4eeeddd?d@Z6dLe ee7j8feeee eeeedAdBdCZ9dMdDdEZ:Z;S)NPreTrainedTokenizerFastaQ Base class for all fast tokenizers (wrapping HuggingFace tokenizers library). Inherits from [`~tokenization_utils_base.PreTrainedTokenizerBase`]. Handles all the shared methods for tokenization and special tokens, as well as methods for downloading/caching/loading pretrained tokenizers, as well as adding tokens to the vocabulary. This class also contains the added tokens in a unified way on top of all tokenizers so we don't have to handle the specific vocabulary augmentation methods of the various underlying dictionary structures (BPE, sentencepiece...). Nslow_tokenizer_classTcan_save_slow_tokenizercs|dd}|dd}|dd}|dd}|rN|durN|jdurNtd|durbt|}nV|durz|szt|}n>|durt|}n,|jdur|j|i|}t|}ntd||_|dur| |j d|_ t j fi|dS)Ntokenizer_objectZ__slow_tokenizerr$ from_slowFzCannot instantiate this tokenizer from a slow version. If it's based on sentencepiece, make sure you have sentencepiece installed.a-Couldn't instantiate the backend tokenizer from one of: (1) a `tokenizers` library serialization file, (2) a slow tokenizer instance to convert or (3) an equivalent slow tokenizer class to instantiate and convert. You need to have sentencepiece installed to convert a slow tokenizer to a fast one.)popr& ValueErrorcopydeepcopy TokenizerFast from_filer _tokenizerupdate init_kwargs_decode_use_source_tokenizersuper__init__)selfargskwargsr(Zslow_tokenizerZfast_tokenizer_filer)Zfast_tokenizer __class__m/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/transformers/tokenization_utils_fast.pyr5_s2           z PreTrainedTokenizerFast.__init__)returncCsdS)NTr;r6r;r;r<is_fastszPreTrainedTokenizerFast.is_fastcCs|jjddS)zP `int`: Size of the base vocabulary (without the added tokens). FZwith_added_tokensr0Zget_vocab_sizer>r;r;r< vocab_sizesz"PreTrainedTokenizerFast.vocab_sizecCs|jjddS)NTr@)r0 get_vocabr>r;r;r<rCsz!PreTrainedTokenizerFast.get_vocabcCs|SN)rCr>r;r;r<vocabszPreTrainedTokenizerFast.vocabcs6|jjdd|jjdd}fdd|D}|S)z Returns the added tokens in the vocabulary as a dictionary of token to index. Returns: `Dict[str, int]`: The added tokens. Fr@Tcsi|]\}}|vr||qSr;r;).0tokindexZ base_vocabr;r< z;PreTrainedTokenizerFast.get_added_vocab..)r0rCitems)r6Z full_vocab added_vocabr;rIr<get_added_vocabsz'PreTrainedTokenizerFast.get_added_vocabcCs|jjddS)zD Size of the full vocabulary with the added tokens. Tr@rAr>r;r;r<__len__szPreTrainedTokenizerFast.__len__cCs|jS)zc `tokenizers.implementations.BaseTokenizer`: The Rust tokenizer used as a backend. )r0r>r;r;r<backend_tokenizersz)PreTrainedTokenizerFast.backend_tokenizercCs|jjS)zU `tokenizers.decoders.Decoder`: The Rust decoder for this tokenizer. )r0decoderr>r;r;r<rQszPreTrainedTokenizerFast.decoderF) encodingreturn_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthverboser=c Cs|durd|jv}|dur$d|jv}|r@|jdur@|g|j} n|g} tt} | D]|} | d| j|rz| d| j|r| d| j|r| d| j|r| d| j |rR| dt | jqR| | fS)a Convert the encoding representation (from low-level HuggingFace tokenizer output) to a python Dict and a list of encodings, take care of building a batch from overflowing tokens. Overflowing tokens are converted to additional examples (like batches) so the output values of the dict are lists (overflows) of lists (tokens). Output shape: (overflows, sequence length) NZtoken_type_idsattention_mask input_idsspecial_tokens_maskZoffset_mappinglength) Zmodel_input_namesZ overflowingrlistappendidsZtype_idsrZr\offsetslen) r6rRrSrTrUrVrWrXrY encodingsZ encoding_dicter;r;r<_convert_encodings*  z)PreTrainedTokenizerFast._convert_encoding)tokensr=cs2|dur dSt|tr |Sfdd|DS)aT Converts a token string (or a sequence of tokens) in a single integer id (or a sequence of ids), using the vocabulary. Args: tokens (`str` or `List[str]`): One or several token(s) to convert to token id(s). Returns: `int` or `List[int]`: The token id or list of token ids. Ncsg|]}|qSr;)#_convert_token_to_id_with_added_vocrFtokenr>r;r< rKzAPreTrainedTokenizerFast.convert_tokens_to_ids..) isinstancestrrgr6rfr;r>r<convert_tokens_to_idss   z-PreTrainedTokenizerFast.convert_tokens_to_ids)rir=cCs|j|}|dur|jS|SrD)r0 token_to_idZ unk_token_id)r6rirHr;r;r<rgs z;PreTrainedTokenizerFast._convert_token_to_id_with_added_voc)rHr=cCs|jt|SrD)r0 id_to_tokenint)r6rHr;r;r<_convert_id_to_tokensz,PreTrainedTokenizerFast._convert_id_to_token) new_tokensr=cCs|r|j|S|j|SrD)r0add_special_tokensZ add_tokens)r6rsspecial_tokensr;r;r< _add_tokenss z#PreTrainedTokenizerFast._add_tokens)pairr=cCs |j|S)aG Returns the number of added tokens when encoding a sequence with special tokens. This encodes a dummy input and checks the number of added tokens, and is therefore not efficient. Do not put this inside your training loop. Args: pair (`bool`, *optional*, defaults to `False`): Whether the number of added tokens should be computed in the case of a sequence pair or a single sequence. Returns: `int`: Number of special tokens added to sequences. )r0num_special_tokens_to_add)r6rwr;r;r<rx sz1PreTrainedTokenizerFast.num_special_tokens_to_add)r`skip_special_tokensr=cCsRt|tr|j|Sg}|D].}t|}|r:||jvr:q||j|q|S)a Converts a single index or a sequence of indices in a token or a sequence of tokens, using the vocabulary and added tokens. Args: ids (`int` or `List[int]`): The token id (or token ids) to convert to tokens. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. Returns: `str` or `List[str]`: The decoded token(s). )rkrqr0rpZall_special_idsr_)r6r`ryrfrHr;r;r<convert_ids_to_tokens"s  z-PreTrainedTokenizerFast.convert_ids_to_tokens)textrwrtr=cKs|jf|||d|S)N)r{ text_pairrt)Z encode_plusrf)r6r{rwrtr8r;r;r<tokenize<sz PreTrainedTokenizerFast.tokenizepadding_strategytruncation_strategy max_lengthstridepad_to_multiple_ofc s|jj|jj}|tjkr.durz|jnL|||j|jd}durNd}nfdd|D}||krz|jjfi||t j kr|dur|j nF|t j kr|nd} | |j |j|j|j|d}||kr|jjfi|dS)a Define the truncation and the padding strategies for fast tokenizers (provided by HuggingFace tokenizers library) and restore the tokenizer settings afterwards. The provided tokenizer has no padding / truncation strategy before the managed section. If your tokenizer set a padding / truncation strategy before, then it will be reset to no padding / truncation when exiting the managed section. Args: padding_strategy ([`~utils.PaddingStrategy`]): The kind of padding that will be applied to the input truncation_strategy ([`~tokenization_utils_base.TruncationStrategy`]): The kind of truncation that will be applied to the input max_length (`int`): The maximum size of a sequence. stride (`int`): The stride to use when handling overflow. pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta). N)rrZstrategy directioncsi|]}||dqSrDget)rFkZ _truncationr;r<rJqrKzFPreTrainedTokenizerFast.set_truncation_and_padding..)r]rZpad_id pad_tokenZ pad_type_idr)r0Z truncationpaddingrDO_NOT_TRUNCATEZ no_truncationvalueZtruncation_sideZenable_truncationr DO_NOT_PADZ no_paddingZ MAX_LENGTHZ padding_sideZ pad_token_idrZpad_token_type_idZenable_padding) r6rrrrrZ_paddingtargetcurrentr]r;rr<set_truncation_and_padding?s8     z2PreTrainedTokenizerFast.set_truncation_and_paddingr)batch_text_or_text_pairsrtrrrris_split_into_wordsrreturn_tensorsrSrTrUrVrWrXrYr=cst|ttfs"tdt|dj|||||djj|||d}fdd|D}i}|ddD]fdd|D}||<q|d d|D}rg}t |D]"\}\}}||gt |d 7}q||d <|d D]} ||qt ||| d S) Nz:batch_text_or_text_pairs has to be a list or a tuple (got )r~)rtZis_pretokenizedc s&g|]}j|dqS))rRrSrTrUrVrWrXrY)re)rFrR)rTrXrWrUrVrSr6rYr;r<rjs z>PreTrainedTokenizerFast._batch_encode_plus..rcs"g|]\}}|D]}|qqSr;r;)rFitem_rd)keyr;r<rjrKcSsg|]\}}|D]}|qqSr;r;)rFrrrdr;r;r<rjrKr[overflow_to_sample_mapping)Z tensor_type) rktupler^ TypeErrortyperr0Z encode_batchkeys enumeraterb&_eventual_warn_about_too_long_sequencer)r6rrtrrrrrrrrSrTrUrVrWrXrYrcZtokens_and_encodingsZsanitized_tokensstackZsanitized_encodingsritoksrr[r;) rrTrXrWrUrVrSr6rYr<_batch_encode_pluss@    z*PreTrainedTokenizerFast._batch_encode_plus)r{r|rtrrrrrrrrSrTrUrVrWrXrYr=cKs|r||fgn|g}|j|f||||||| | | | | ||||d|}| durl| sltdd|D|j}||d|||S)N)rrtrrrrrrrSrTrUrVrWrXrYcSs8i|]0\}}|t|dkr0t|dtr0|dn|qS)r)rbrkr^)rFrrr;r;r<rJ sz8PreTrainedTokenizerFast._encode_plus..r[)rrrLrcr)r6r{r|rtrrrrrrrrSrTrUrVrWrXrYr8Z batched_inputZbatched_outputr;r;r< _encode_pluss> z$PreTrainedTokenizerFast._encode_pluscCs|jj|SrD)rPrQdecodermr;r;r<convert_tokens_to_stringsz0PreTrainedTokenizerFast.convert_tokens_to_string) token_idsryclean_up_tokenization_spacesr=cKsZ|dd|_t|tr|g}|jj||d}|dur:|n|j}|rR||}|S|SdS)NZuse_source_tokenizerF)ry)r*r3rkrqr0rrZclean_up_tokenization)r6rryrr8r{Z clean_textr;r;r<_decodes  zPreTrainedTokenizerFast._decode)save_directory file_names legacy_formatfilename_prefixr=c Cs*t|}|jdur"|dur"td|dus2|duo@|jduo@|j}|dupP|du}|rtj||rj|dndt}|}|rt |ddd 0} t j |d ddd d } | | Wdn1s0Y|j ||d } || |f}|r&tj||r|dndt} |j| || f}|S)z Save a tokenizer using the slow-tokenizer/legacy format: vocabulary + added tokens as well as in a unique JSON file containing {config + vocab + added-tokens}. NTzYour tokenizer does not have a legacy version defined and therefore cannot register this version. You might consider leaving the legacy_format at `None` or setting it to `False`.F-wzutf-8)rR)indent sort_keys ensure_ascii )r)rlr&r+r'ospathjoinADDED_TOKENS_FILErNopenjsondumpswriteZsave_vocabularyTOKENIZER_FILErPsave) r6rrrrZ save_slowZ save_fastZadded_tokens_filerMfZout_strZ vocab_filesr$r;r;r<_save_pretrained/s: (  z(PreTrainedTokenizerFast._save_pretrainedc st|j}|d}|d} d} |dddkrRi|dd<g|dd<n|ddd kr|dd dur|dd } |dd| d } dur| vr| } d |dd <| d gg|dd<n6|ddd vri|dd<ntd|ddddurBd|dvrB|ddvrB|dd|dd<tt|g} |D]v} | dd}| dd}|ddd kr|sqZdur| dvr| d| d<| t fi| qZ|dur| ||dddkr$d|vr$|dddur$|dd|d<|dddkrbd|vrb|dddurb|dd|d<|ddd kr| dur| |d<|ddur|dddkrt j |d<t|dd}|f|| d|}j|||d| durt}d| vrt| dD]`}| d|d}durBfdd|D}|| d|d<fdd|D| d|d <qd!D]N}|| vrx| |\}}dur|vr|}|}||g| |<qx| |d<tt||j}tj}|d"|D]}t|d#|durt||}durB|vrB|}t|d#|}t|t r~t ||j|j|j|jd$||<n|||<q|j}|dur| |t|d kr||d"<|jfd%i|S)&uG Trains a tokenizer on a new corpus with the same defaults (in terms of special tokens or tokenization pipeline) as the current one. Args: text_iterator (generator of `List[str]`): The training corpus. Should be a generator of batches of texts, for instance a list of lists of texts if you have everything in memory. vocab_size (`int`): The size of the vocabulary you want for your tokenizer. length (`int`, *optional*): The total number of sequences in the iterator. This is used to provide meaningful progress tracking new_special_tokens (list of `str` or `AddedToken`, *optional*): A list of new special tokens to add to the tokenizer you are training. special_tokens_map (`Dict[str, str]`, *optional*): If you want to rename some of the special tokens this tokenizer uses, pass along a mapping old special token name to new special token name in this argument. kwargs: Additional keyword arguments passed along to the trainer from the 🤗 Tokenizers library. Returns: [`PreTrainedTokenizerFast`]: A new tokenizer of the same type as the original one, trained on `text_iterator`. added_tokenspost_processorNmodelrr rEZmergesr!unk_idrg)r"r#z;This method does not support this type of tokenizer (found z-) only BPE, Unigram, WordLevel and WordPiece. unk_tokenspecialidcontentZcontinuing_subword_prefixZend_of_word_suffixZ pre_tokenizer ByteLevelZinitial_alphabet)rBru)r]trainerrurfcsg|]}||qSr;rrh)special_tokens_mapr;r<rjrKzCPreTrainedTokenizerFast.train_new_from_iterator..csg|]}|qSr;)rorh) tokenizerr;r<rjrKr`)clssepadditional_special_tokensr) single_wordlstriprstrip normalizedr() rloadsr0Zto_strr*r+r.Zfrom_strrr_rextendpre_tokenizers_fastralphabetMODEL_TO_TRAINER_MAPPINGZtrain_from_iteratorror2r,rZSPECIAL_TOKENS_ATTRIBUTESremovegetattrrkrrrrrrbr:)r6Z text_iteratorrBr]Znew_special_tokensrr8Ztokenizer_jsonrrrrruZ added_tokenrrZ trainer_classrZtrained_tokenizer_jsonrrfZ special_tokenriZtoken_idZspecial_tokens_listZspecial_token_fullrr;)rrr<train_new_from_iterator_s"               "            z/PreTrainedTokenizerFast.train_new_from_iterator)NNFFFFT)F)F)F)NF)FN)NN)NNN)<__name__ __module__ __qualname____doc__VOCAB_FILES_NAMESZvocab_files_namesr&r__annotations__r'boolr5propertyr?rqrBrrlrCrErNrOr.rP DecoderFastrQ EncodingFastrrrrrerrnrgrrrrvrxrzr}rrrrrrrrrrrrrrrPathLikerr __classcell__r;r;r9r<r%Ms&     +   /(  L X  9   4r%)9rr,rr collectionsrtypingrrrrrrZtokenizers.pre_tokenizersZpre_tokenizersrZ tokenizersr rr r.Ztokenizers.decodersr rZtokenizers.trainersr r rrrZtokenization_utilsrZtokenization_utils_baserrrrrrrrrrutilsrrrZ get_loggerrloggerrZSPECIAL_TOKENS_MAP_FILEZTOKENIZER_CONFIG_FILErrrr%r;r;r;r<s8        0