a  dO@sdZddlZddlZddlZddlmZddlmZmZm Z ddl Z ddl m Z mZmZmZmZddlmZeeZdd d Zd d id d id Zd diZd iiZeddZddZddZddZGdddeZ GdddeZ!dS)zTokenization classes for CLIP.N) lru_cache)ListOptionalTuple) AddedTokenPreTrainedTokenizer _is_control_is_punctuation_is_whitespace)loggingz vocab.jsonz merges.txt) vocab_file merges_filezopenai/clip-vit-base-patch32zKhttps://huggingface.co/openai/clip-vit-base-patch32/resolve/main/vocab.jsonzKhttps://huggingface.co/openai/clip-vit-base-patch32/resolve/main/merges.txtMcCstttdtddtttdtddtttdtdd}|dd}d }td D],}||vrf|||d ||d7}qfd d |D}tt||S) a8 Returns list of utf-8 byte and a mapping to unicode strings. We specifically avoids mapping to whitespace/control characters the bpe code barfs on. The reversible bpe codes work on unicode strings. This means you need a large # of unicode characters in your vocab if you want to avoid UNKs. When you're at something like a 10B token dataset you end up needing around 5K for decent coverage. This is a significant percentage of your normal, say, 32K bpe vocab. To avoid that, we want lookup tables between utf-8 bytes and unicode strings. !~¡¬®ÿNrcSsg|] }t|qS)chr).0nrrs/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/transformers/models/clip/tokenization_clip.py Lz$bytes_to_unicode..)listrangeordappenddictzip)bscsrbrrrbytes_to_unicode7s L    r(cCs6t}|d}|ddD]}|||f|}q|S)z Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings). rrN)setadd)wordpairsZ prev_charcharrrr get_pairsPs r.cCstdd|}|}|S)Nz\s+ )resubstrip)textrrrwhitespace_clean^sr4cCs|}|sgS|}|S)z@Runs basic whitespace cleaning and splitting on a piece of text.)r2split)r3tokensrrrwhitespace_tokenizees r7c@sNeZdZdZdddZdddZdd Zdd d Zd d ZddZ ddZ dS)BasicTokenizera Constructs a BasicTokenizer that will run basic tokenization (punctuation splitting, lower casing, etc.). Args: do_lower_case (`bool`, *optional*, defaults to `True`): Whether or not to lowercase the input when tokenizing. never_split (`Iterable`, *optional*): Collection of tokens which will never be split during tokenization. Only has an effect when `do_basic_tokenize=True` tokenize_chinese_chars (`bool`, *optional*, defaults to `True`): Whether or not to tokenize Chinese characters. This should likely be deactivated for Japanese (see this [issue](https://github.com/huggingface/transformers/issues/328)). strip_accents (`bool`, *optional*): Whether or not to strip all accents. If this option is not specified, then it will be determined by the value for `lowercase` (as in the original BERT). TNcCs,|dur g}||_t||_||_||_dSN) do_lower_caser) never_splittokenize_chinese_chars strip_accents)selfr:r;r<r=rrr__init__s  zBasicTokenizer.__init__cCs|r|jt|n|j}||}|jr4||}t|}g}|D]R}||vr|jrt|}|j dur| |}n|j r| |}| | ||qDtd |}|S)a Basic Tokenization of a piece of text. Split on "white spaces" only, for sub-word tokenization, see WordPieceTokenizer. Args: never_split (`List[str]`, *optional*) Kept for backward compatibility purposes. Now implemented directly at the base class level (see [`PreTrainedTokenizer.tokenize`]) List of token not to split. Fr/)r;unionr) _clean_textr<_tokenize_chinese_charsr7r:lowerr=_run_strip_accentsextend_run_split_on_puncjoin)r>r3r;Z orig_tokensZ split_tokenstokenZ output_tokensrrrtokenizes"      zBasicTokenizer.tokenizecCsBtd|}g}|D]"}t|}|dkr,q||qd|S)z$Strips accents from a piece of text.ZNFDZMn) unicodedata normalizecategoryr"rG)r>r3outputr-catrrrrDs   z!BasicTokenizer._run_strip_accentscCs|dur||vr|gSt|}d}d}g}|t|kr||}t|rX||gd}n |rf|gd}|d||d7}q*dd|DS) z&Splits punctuation on a piece of text.NrTFrcSsg|]}d|qS)rJ)rG)rxrrrrrz5BasicTokenizer._run_split_on_punc..)rlenr r")r>r3r;charsiZstart_new_wordrNr-rrrrFs"    z!BasicTokenizer._run_split_on_punccCsTg}|D]@}t|}||r>|d|||dq||qd|S)z)Adds whitespace around any CJK character.r/rJ)r!_is_chinese_charr"rGr>r3rNr-cprrrrBs     z&BasicTokenizer._tokenize_chinese_charscCs|dkr|dks|dkr |dks|dkr0|dks|dkr@|dks|d krP|d ks|d kr`|d ks|d krp|dks|dkr|dkrdSdS)z6Checks whether CP is the codepoint of a CJK character.iNii4iMiiߦii?i@ii iiiiiTFr)r>rWrrrrUsD  zBasicTokenizer._is_chinese_charcCsXg}|D]D}t|}|dks|dkst|r.qt|rB|dq||qd|S)zBPerforms invalid character removal and whitespace cleanup on text.rir/rJ)r!r r r"rGrVrrrrAs  zBasicTokenizer._clean_text)TNTN)N)N) __name__ __module__ __qualname____doc__r?rIrDrFrBrUrArrrrr8os  %  r8cseZdZdZeZeZeZ ddgZ d$fdd Z e d d Z d d Zd%eeeeeeedddZd&eeeeeeeedfdd Zd'eeeeeeedddZddZddZddZddZdd Zd(eeeeed!d"d#ZZS)) CLIPTokenizera< Construct a CLIP tokenizer. Based on byte-level Byte-Pair-Encoding. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): Path to the vocabulary file. merges_file (`str`): Path to the merges file. errors (`str`, *optional*, defaults to `"replace"`): Paradigm to follow when decoding bytes to UTF-8. See [bytes.decode](https://docs.python.org/3/library/stdtypes.html#bytes.decode) for more information. unk_token (`str`, *optional*, defaults to `<|endoftext|>`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str`, *optional*, defaults to `<|startoftext|>`): The beginning of sequence token. eos_token (`str`, *optional*, defaults to `<|endoftext|>`): The end of sequence token. Z input_idsZattention_maskreplace <|endoftext|><|startoftext|>c  st|trt|dddn|}t|tr4t|dddn|}t|trPt|dddn|}tjf|||||d|zddl} | j|_Wn.tyt dt dd|_ d|_Yn0t |dd } t | |_Wdn1s0Yd d |jD|_||_t|_d d |jD|_t |dd *} | d dd} Wdn1sf0Ydd| D} tt| tt| |_ddd|_t dtj!|_"dS)NF)lstriprstrip)errors unk_token bos_token eos_token pad_tokenrzKftfy or spacy is not installed using custom BasicTokenizer instead of ftfy.T)r:utf-8encodingcSsi|]\}}||qSrrrkvrrr Drz*CLIPTokenizer.__init__..cSsi|]\}}||qSrrrjrrrrmGr ricSsg|]}t|qSr)tupler5)rmergerrrrJrz*CLIPTokenizer.__init__..r_r^)r_r^z[<\|startoftext\|>|<\|endoftext\|>|'s|'t|'re|'ve|'m|'ll|'d|[\p{L}]+|[\p{N}]|[^\s\p{L}\p{N}]+)# isinstancestrrsuperr?ftfyfix_text ImportErrorloggerinfor8nlpopenjsonloadencoderitemsdecoderrbr( byte_encoder byte_decoderreadr2r5r#r$r rR bpe_rankscacher0compile IGNORECASEpat) r>r rrbrcrdrerfkwargsrtZ vocab_handleZ merges_handleZ bpe_merges __class__rrr?!sD       *: zCLIPTokenizer.__init__cCs t|jSr9)rRr}r>rrr vocab_sizeSszCLIPTokenizer.vocab_sizecCst|jfi|jSr9)r#r}Zadded_tokens_encoderrrrr get_vocabWszCLIPTokenizer.get_vocabN) token_ids_0 token_ids_1returncCs<|jg}|jg}|dur$|||S||||||S)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A CLIP sequence has the following format: - single sequence: `<|startoftext|> X <|endoftext|>` Pairs of sequences are not the expected use case, but they will be handled without a separator. Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. N) bos_token_id eos_token_idr>rrrdrerrr build_inputs_with_special_tokensZs  z.CLIPTokenizer.build_inputs_with_special_tokensF)rralready_has_special_tokensrcsl|rtj||ddS|dur8dgdgt|dgSdgdgt|dgdgdgt|dgS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrrNrr)rsget_special_tokens_maskrR)r>rrrrrrrusz%CLIPTokenizer.get_special_tokens_maskcCsP|jg}|jg}|dur.t|||dgSt||||||dgS)a Create a mask from the two sequences passed. CLIP does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of zeros. Nr)rrrRrrrr$create_token_type_ids_from_sequencess z2CLIPTokenizer.create_token_type_ids_from_sequencesc s||jvrj|St|dd|ddf}t|}|sF|dSt|fddd}|jvrhqd|\}}g}d}|t|kr:z|||} Wn*ty|||dYq:Yn0|||| | }|||kr"|t|dkr"||d|kr"| |||d7}qx| |||d7}qxt|}|}t|dkrZqdqFt|}qFd |}|j|<|S) NrPcsj|tdS)Ninf)rgetfloat)pairrrrrz#CLIPTokenizer.bpe..keyrrr/) rror.minrrRindex ValueErrorrEr"rG) r>rHr+r,ZbigramfirstsecondZnew_wordrTjrrrbpesB     2     zCLIPTokenizer.bpecsg}jdur"dj|}nt|}tj|D]B}dfdd| dD}| dd | dDqB|S)zTokenize a string.Nr/rJc3s|]}j|VqdSr9)r)rr'rrr sz*CLIPTokenizer._tokenize..rgcss|] }|VqdSr9r)rZ bpe_tokenrrrrr) rurGryrIr4rCr0findallrencoderErr5)r>r3 bpe_tokensrHrrr _tokenizes "zCLIPTokenizer._tokenizecCs|j||j|jS)z0Converts a token (str) in an id using the vocab.)r}rrc)r>rHrrr_convert_token_to_idsz"CLIPTokenizer._convert_token_to_idcCs |j|S)z=Converts an index (integer) in a token (str) using the vocab.)rr)r>rrrr_convert_id_to_tokensz"CLIPTokenizer._convert_id_to_tokencs@d|}tfdd|D}|jdjddd}|S)z:Converts a sequence of tokens (string) in a single string.rJcsg|]}j|qSr)r)rcrrrrrz:CLIPTokenizer.convert_tokens_to_string..rg)rbrr/)rG bytearraydecoderbr]r2)r>r6r3Z byte_arrayrrrconvert_tokens_to_strings z&CLIPTokenizer.convert_tokens_to_string)save_directoryfilename_prefixrc CsPtj|s td|dStj||r4|dndtd}tj||rV|dndtd}t|ddd.}| t j |j d d d d d Wdn1s0Yd}t|dddt}| dt |jdddD]B\}} || kr td|| }| d|d |d7}qWdn1s>0Y||fS)Nz*Vocabulary path ({}) should be a directory-rJr rwrgrhrTF)indent sort_keys ensure_asciirnrz#version: 0.2 cSs|dS)Nrr)kvrrrrrz/CLIPTokenizer.save_vocabulary..rzqSaving vocabulary to {}: BPE merge indices are not consecutive. Please check that the tokenizer is not corrupted!r/r)ospathisdirrwerrorformatrGVOCAB_FILES_NAMESrzwriter{dumpsr}sortedrr~warning) r>rrr Z merge_filefrwriterrZ token_indexrrrsave_vocabularys2 <  *zCLIPTokenizer.save_vocabulary)r]r^r_r^r^)N)NF)N)N)rXrYrZr[rZvocab_files_namesPRETRAINED_VOCAB_FILES_MAPZpretrained_vocab_files_map&PRETRAINED_POSITIONAL_EMBEDDINGS_SIZESZmax_model_input_sizesZmodel_input_namesr?propertyrrrintrrboolrrrrrrrrrrr __classcell__rrrrr\sF2   *r\)"r[r{rrK functoolsrtypingrrrregexr0Ztokenization_utilsrrr r r utilsr Z get_loggerrXrwrrrZPRETRAINED_INIT_CONFIGURATIONr(r.r4r7objectr8r\rrrrs: