a  d:@sdZddlZddlZddlZddlZddlmZmZddlm Z m Z m Z m Z ddl mZeeZddd Zd d id d id Zd d iZddZGdddeZddZddZGddde ZdS)z$Tokenization classes for OpenAI GPT.N)OptionalTuple)PreTrainedTokenizer _is_control_is_punctuation_is_whitespace)loggingz vocab.jsonz merges.txt) vocab_file merges_filez openai-gptz9https://huggingface.co/openai-gpt/resolve/main/vocab.jsonz9https://huggingface.co/openai-gpt/resolve/main/merges.txticCs|}|sgS|}|S)z@Runs basic whitespace cleaning and splitting on a piece of text.)stripsplit)texttokensrw/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/transformers/models/openai/tokenization_openai.pywhitespace_tokenize.s rc@sNeZdZdZdddZdddZdd Zdd d Zd d ZddZ ddZ dS)BasicTokenizera Constructs a BasicTokenizer that will run basic tokenization (punctuation splitting, lower casing, etc.). Args: do_lower_case (`bool`, *optional*, defaults to `True`): Whether or not to lowercase the input when tokenizing. never_split (`Iterable`, *optional*): Collection of tokens which will never be split during tokenization. Only has an effect when `do_basic_tokenize=True` tokenize_chinese_chars (`bool`, *optional*, defaults to `True`): Whether or not to tokenize Chinese characters. This should likely be deactivated for Japanese (see this [issue](https://github.com/huggingface/transformers/issues/328)). strip_accents (`bool`, *optional*): Whether or not to strip all accents. If this option is not specified, then it will be determined by the value for `lowercase` (as in the original BERT). TNcCs,|dur g}||_t||_||_||_dSN) do_lower_caseset never_splittokenize_chinese_chars strip_accents)selfrrrrrrr__init__Ls  zBasicTokenizer.__init__cCs|r|jt|n|j}||}|jr4||}t|}g}|D]R}||vr|jrt|}|j dur| |}n|j r| |}| | ||qDtd |}|S)a Basic Tokenization of a piece of text. Split on "white spaces" only, for sub-word tokenization, see WordPieceTokenizer. Args: never_split (`List[str]`, *optional*) Kept for backward compatibility purposes. Now implemented directly at the base class level (see [`PreTrainedTokenizer.tokenize`]) List of token not to split. F )runionr _clean_textr_tokenize_chinese_charsrrlowerr_run_strip_accentsextend_run_split_on_puncjoin)rrrZ orig_tokens split_tokenstokenZ output_tokensrrrtokenizeTs"      zBasicTokenizer.tokenizecCsBtd|}g}|D]"}t|}|dkr,q||qd|S)z$Strips accents from a piece of text.ZNFDZMn) unicodedata normalizecategoryappendr$)rroutputcharcatrrrr!ys   z!BasicTokenizer._run_strip_accentscCs|dur||vr|gSt|}d}d}g}|t|kr||}t|rX||gd}n |rf|gd}|d||d7}q*dd|DS) z&Splits punctuation on a piece of text.NrTFcSsg|]}d|qS)r()r$).0xrrr z5BasicTokenizer._run_split_on_punc..)listlenrr,)rrrcharsiZstart_new_wordr-r.rrrr#s"    z!BasicTokenizer._run_split_on_punccCsTg}|D]@}t|}||r>|d|||dq||qd|S)z)Adds whitespace around any CJK character.rr()ord_is_chinese_charr,r$rrr-r.cprrrrs     z&BasicTokenizer._tokenize_chinese_charscCs|dkr|dks|dkr |dks|dkr0|dks|dkr@|dks|d krP|d ks|d kr`|d ks|d krp|dks|dkr|dkrdSdS)z6Checks whether CP is the codepoint of a CJK character.iNii4iMiiߦii?i@ii iiiiiTFr)rr=rrrr;sD  zBasicTokenizer._is_chinese_charcCsXg}|D]D}t|}|dks|dkst|r.qt|rB|dq||qd|S)zBPerforms invalid character removal and whitespace cleanup on text.rirr()r:rrr,r$r<rrrrs  zBasicTokenizer._clean_text)TNTN)N)N) __name__ __module__ __qualname____doc__rr'r!r#rr;rrrrrr8s  %  rcCs6t}|d}|ddD]}|||f|}q|S)z Return set of symbol pairs in a word. word is represented as tuple of symbols (symbols being variable-length strings) rr1N)radd)wordpairsZ prev_charr.rrr get_pairss rEcCsn|dd}|dd}|dd}|dd}|dd}td d |}td d |}td d|}|S)zm fixes some issues the spacy tokenizer had on books corpus also does some whitespace standardization u—-u–u―u…z...´'zD(-+|~+|!+|"+|;+|\?+|\++|,+|\)+|\(+|\\+|\/+|\*+|\[+|\]+|}+|{+|\|+|_+)z \1 z\s*\n\s*z z[^\S\n]+r)replaceresubr )rrrrtext_standardizes     rLcseZdZdZeZeZeZ ddgZ dfdd Z e ddZ e d d Zd d Zd dZddZddZddZddZdeeeeedddZZS)OpenAIGPTTokenizera( Construct a GPT Tokenizer. Based on Byte-Pair-Encoding with the following peculiarities: - lowercases all inputs, - uses `SpaCy` tokenizer and `ftfy` for pre-BPE tokenization if they are installed, fallback to BERT's `BasicTokenizer` if not. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): Path to the vocabulary file. merges_file (`str`): Path to the merges file. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. Z input_idsZattention_maskc  s4tjfd|i|z.ddl}ddlm}|}|j|_|j|_Wn.tyrt dt dd|_d|_Yn0t |dd}t ||_Wdn1s0Yd d |jD|_t |dd&} | d d d } Wdn1s0Ydd| D} tt| tt| |_i|_dS)N unk_tokenr)EnglishzQftfy or spacy is not installed using BERT BasicTokenizer instead of SpaCy & ftfy.T)rutf-8encodingcSsi|]\}}||qSrr)r2kvrrr r5z/OpenAIGPTTokenizer.__init__.. r1r0cSsg|]}t|qSr)tupler )r2mergerrrr4r5z/OpenAIGPTTokenizer.__init__..)superrftfyZ spacy.lang.enrP tokenizernlpfix_text ImportErrorloggerwarningropenjsonloadencoderitemsdecoderreadr dictzipranger7 bpe_rankscache) rr r rOkwargsr[rPZ_nlpZ vocab_handleZ merges_handleZmerges __class__rrrs&      *4zOpenAIGPTTokenizer.__init__cCsdS)NTrrrrrrsz OpenAIGPTTokenizer.do_lower_casecCs t|jSr)r7rerqrrr vocab_sizeszOpenAIGPTTokenizer.vocab_sizecCst|jfi|jSr)rireZadded_tokens_encoderrqrrr get_vocab#szOpenAIGPTTokenizer.get_vocabc st|dd|ddf}|jvr2j|St|}|sF|dSt|fddd}|jvrhqd|\}}g}d}|t|kr:z|||} Wn*ty|||dYq:Yn0|||| | }|||kr"|t|dkr"||d|kr"| |||d7}qx| |||d7}qxt|}|}t|dkrZqdqFt|}qFd |}|d kr|d }|j|<|S) Nr0csj|tdS)Ninf)rlgetfloat)pairrqrr0r5z(OpenAIGPTTokenizer.bpe..keyrr1rz z ) rXrmrEminrlr7index ValueErrorr"r,r$) rr&rCrDZbigramfirstsecondZnew_wordr9jrrqrbpe&sF     2      zOpenAIGPTTokenizer.bpecCsg}|jdur@|j|}|D]}|t||dqn>|t||}|D]$}|t||j dqX|S)zTokenize a string.Nr) r^r]r'r"r6rr rLrr )rrr%r&rrr _tokenizeRs  "zOpenAIGPTTokenizer._tokenizecCs|j||j|jS)z0Converts a token (str) in an id using the vocab.)rervrO)rr&rrr_convert_token_to_idasz'OpenAIGPTTokenizer._convert_token_to_idcCs|j||jS)z0Converts an id in a token (BPE) using the vocab.)rgrvrO)rr~rrr_convert_id_to_tokenesz'OpenAIGPTTokenizer._convert_id_to_tokencCsd|dd}|S)z:Converts a sequence of tokens (string) in a single string.r(rtr)r$rIr )rrZ out_stringrrrconvert_tokens_to_stringisz+OpenAIGPTTokenizer.convert_tokens_to_stringN)save_directoryfilename_prefixreturnc CsTtj|s"td|ddStj||r6|dndtd}tj||rX|dndtd}t|ddd .}|t j |j d d d d dWdn1s0Yd}t|ddd v}|dt |j dddD]D\}} || krtd|d| }|d|d|d7}qWdn1sB0Y||fS)NzVocabulary path (z) should be a directoryrFr(r r wrQrRr|TF)indent sort_keys ensure_asciirWrz#version: 0.2 cSs|dS)Nr1r)kvrrrryr5z4OpenAIGPTTokenizer.save_vocabulary..rzzSaving vocabulary to zZ: BPE merge indices are not consecutive. Please check that the tokenizer is not corrupted!rr1)ospathisdirr`errorr$VOCAB_FILES_NAMESrbwritercdumpsresortedrlrfra) rrrr Z merge_filefr~writerZ bpe_tokensZ token_indexrrrsave_vocabularyns. <   *z"OpenAIGPTTokenizer.save_vocabulary)rN)N)r>r?r@rArZvocab_files_namesPRETRAINED_VOCAB_FILES_MAPZpretrained_vocab_files_map&PRETRAINED_POSITIONAL_EMBEDDINGS_SIZESZmax_model_input_sizesZmodel_input_namesrpropertyrrrrsrrrrrstrrrr __classcell__rrrorrMs"  ,rM)rArcrrJr)typingrrZtokenization_utilsrrrrutilsr Z get_loggerr>r`rrrrobjectrrErLrMrrrrs,