a  d+!8@sdZddlZddlZddlmZmZddlZddlm Z ddl m Z e e Zddd Zd d id d id Zd d iZddddddddddddddddddd d!d"d#d$d%d&d'd(d)d*d+d,d-d.d/d0d1d2d3d4d5d6d7d8d9d:d;dd?d@dAdBdCdDdE7ZdFdGZGdHdIdIe ZdS)Jz)Tokenization classes for Salesforce CTRL.N)OptionalTuple)PreTrainedTokenizer)loggingz vocab.jsonz merges.txt) vocab_file merges_fileZctrlzHhttps://raw.githubusercontent.com/salesforce/ctrl/master/ctrl-vocab.jsonzHhttps://raw.githubusercontent.com/salesforce/ctrl/master/ctrl-merges.txtiiiiii#jivi~i6iivii.iiiךiͨii%iii3iR-iniS.iKiiwii[i*iii/i?iin1iipiiiiϒi i)i-i(iiKiiiǢiihi)7Z PregnancyZ ChristianityZExplainZFitnessZSavingZAskZAssZJokeZ QuestionsZThoughtsZRetailZFeminismZWritingZAtheismZNetflixZ ComputingZOpinionZAloneFunnyZGamingZHumanZIndiaZJokerZDietZLegalZNormanZTipZWeightZMoviesZRunningZScienceZHorrorZ ConfessionZFinanceZPoliticsZScaryZSupportZ TechnologiesZTeenageEventZLearnedZNotionZ WikipediaZBooksZExtractZ ConfessionsZ ConspiracyZLinksZ NarcissusZ RelationshipZ RelationshipsZReviewsZNewsZ TranslationZ multilingualcCs>t}|d}|ddD]}|||f|}qt|}|S)z Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings). rN)setadd)wordpairsZ prev_charcharrs/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/transformers/models/ctrl/tokenization_ctrl.py get_pairsgsrcseZdZdZeZeZeZ e Z dfdd Z e ddZddZd d Zd d Zd dZddZddZdeeeeedddZZS) CTRLTokenizera` Construct a CTRL tokenizer. Based on Byte-Pair-Encoding. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): Path to the vocabulary file. merges_file (`str`): Path to the merges file. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. c stjfd|i|t|dd}t||_Wdn1sD0Ydd|jD|_t|dd&}| ddd}Wdn1s0Yd d |D}t t |t t ||_i|_dS) N unk_tokenutf-8encodingcSsi|]\}}||qSrr).0kvrrr z*CTRLTokenizer.__init__.. r cSsg|]}t|qSr)tuplesplit)rmergerrr rz*CTRLTokenizer.__init__..)super__init__openjsonloadencoderitemsdecoderreadr#dictziprangelen bpe_rankscache)selfrrrkwargsZ vocab_handleZ merges_handleZmerges __class__rrr's*4zCTRLTokenizer.__init__cCs t|jSN)r2r+r5rrr vocab_sizeszCTRLTokenizer.vocab_sizecCst|jfi|jSr9)r/r+Zadded_tokens_encoderr:rrr get_vocabszCTRLTokenizer.get_vocabc s|jvrj|St|}tt|dd|ddg}t|}|sN|St|fddd}|jvrpql|\}}g}d}|t|krBz|||} Wn*ty| ||dYqBYn0| ||| | }|||kr*|t|dkr*||d|kr*| |||d7}q| |||d7}qt|}|}t|dkrbqlqNt|}qNd |}|dd }|j|<|S) Nr!zcsj|tdS)Ninf)r3getfloat)pairr:rrrz#CTRLTokenizer.bpe..keyrr @@ ) r4r"listrminr3r2index ValueErrorextendappendjoin) r5tokenrrZbigramfirstsecondZnew_wordijrr:rbpesF  "   2      zCTRLTokenizer.bpecCs8g}td|}|D]}|t||dq|S)zTokenize a string.z\S+\n? )refindallrKrGrSr#)r5textZ split_tokenswordsrNrrr _tokenizes  zCTRLTokenizer._tokenizecCs|j||j|jS)z0Converts a token (str) in an id using the vocab.)r+r>r)r5rNrrr_convert_token_to_idsz"CTRLTokenizer._convert_token_to_idcCs|j||jS)z=Converts an index (integer) in a token (str) using the vocab.)r-r>r)r5rIrrr_convert_id_to_tokensz"CTRLTokenizer._convert_id_to_tokencCsd|dd}|S)z:Converts a sequence of tokens (string) in a single string.rTrE)rMreplacestrip)r5tokensZ out_stringrrrconvert_tokens_to_stringsz&CTRLTokenizer.convert_tokens_to_stringN)save_directoryfilename_prefixreturnc CsTtj|s"td|ddStj||r6|dndtd}tj||rX|dndtd}t|ddd .}|t j |j d d d d dWdn1s0Yd}t|ddd v}|dt |j dddD]D\}} || krtd|d| }|d|d|d7}qWdn1sB0Y||fS)NzVocabulary path (z) should be a directory-r\rrwrrrDTF)indent sort_keys ensure_asciir rz#version: 0.2 cSs|dS)Nr r)kvrrrrArz/CTRLTokenizer.save_vocabulary..rBzSaving vocabulary to zZ: BPE merge indices are not consecutive. Please check that the tokenizer is not corrupted!rTr )ospathisdirloggererrorrMVOCAB_FILES_NAMESr(writer)dumpsr+sortedr3r,warning) r5rarbrZ merge_filefrIwriterZ bpe_tokensZ token_indexrrrsave_vocabularys. <   *zCTRLTokenizer.save_vocabulary)r)N)__name__ __module__ __qualname____doc__roZvocab_files_namesPRETRAINED_VOCAB_FILES_MAPZpretrained_vocab_files_map&PRETRAINED_POSITIONAL_EMBEDDINGS_SIZESZmax_model_input_sizes CONTROL_CODESZ control_codesr'propertyr;r<rSrYrZr[r`strrrrv __classcell__rrr7rrws  , r)rzr)rjtypingrrregexrUZtokenization_utilsrutilsrZ get_loggerrwrmror{r|r}rrrrrrs   ;