a d@sdZddlZddlZddlZddlmZddlmZmZddl Z ddl Z ddl Z ddlZdej d<eddZed d Zd d Zd dZddZGdddeZeZe jdddZdeeeefee jdddZGdddZdS)zp CLIP tokenizer Copied from https://github.com/openai/CLIP. Originally MIT License, Copyright (c) 2021 OpenAI. N) lru_cache)UnionListfalseZTOKENIZERS_PARALLELISMcCstjtjtjtdS)Nzbpe_simple_vocab_16e6.txt.gz)ospathjoindirnameabspath__file__r r \/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/open_clip/tokenizer.py default_bpesrcCstttdtddtttdtddtttdtdd}|dd}d }td D],}||vrf|||d ||d7}qfd d |D}tt||S) a: Returns list of utf-8 byte and a corresponding list of unicode strings. The reversible bpe codes work on unicode strings. This means you need a large # of unicode characters in your vocab if you want to avoid UNKs. When you're at something like a 10B token dataset you end up needing around 5K for decent coverage. This is a significant percentage of your normal, say, 32K bpe vocab. To avoid that, we want lookup tables between utf-8 bytes and unicode strings. And avoids mapping to whitespace/control characters the bpe code barfs on. !~¡¬®ÿNrcSsg|] }t|qSr )chr).0nr r r ,z$bytes_to_unicode..)listrangeordappenddictzip)bscsrbr r r bytes_to_unicodes N    r%cCs6t}|d}|ddD]}|||f|}q|S)zReturn set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings). rrN)setadd)wordpairsZ prev_charcharr r r get_pairs0s r+cCs"t|}tt|}|SN)ftfyZfix_texthtmlunescapestriptextr r r basic_clean<s r3cCstdd|}|}|S)Nz\s+ )resubr0r1r r r whitespace_cleanBsr7c@s:eZdZedfedddZddZddZd d ZdS) SimpleTokenizerN)bpe_pathcsHt_ddjD_t|dd}|dd}dd|D}t t }|d d|D}|D]}| d |qv|sd d g}n d d g|}| |tt|tt|_d djD_tt|tt|_dd|D_d |}t|dtj_tj_fdd|D_dS)NcSsi|]\}}||qSr r rkvr r r Krz,SimpleTokenizer.__init__..utf-8 ricSsg|]}t|qSr )tuplesplit)rmerger r r rNrz,SimpleTokenizer.__init__..cSsg|] }|dqS)r )rr<r r r rPr cSsi|]\}}||qSr r r:r r r r=YrcSsi|] }||qSr r rtr r r r=[r|z:|'s|'t|'re|'ve|'m|'ll|'d|[\p{L}]+|[\p{N}]|[^\s\p{L}\p{N}]+csg|]}j|qSr encoderrGselfr r r`r)r% byte_encoderitems byte_decodergzipopenreaddecoderArvaluesrrextendr r!rlenrKdecoder bpe_rankscacher5compile IGNORECASEpatZ vocab_sizeZall_special_ids)rMr9Zspecial_tokensZmergesZvocabrBZspecialr rLr __init__Is*      zSimpleTokenizer.__init__c sv|jvrj|St|dd|ddf}t|}|sF|dSt|fddd}|jvrhq^|\}}g}d}|t|kr4z&|||} |||| | }Wn$|||dYq4Yn0|||kr|t|dkr||d|kr||||d7}qx||||d7}qxt|}|}t|dkrTq^qFt|}qFd |}|j|<|S) NrCcsj|tdS)Ninf)rYgetfloat)pairrLr r lrz%SimpleTokenizer.bpe..)keyrrr4) rZr@r+minrYrWindexrVrr) rMtokenr(r)ZbigramfirstsecondZnew_wordijr rLr bpebsB     2     zSimpleTokenizer.bpecsng}tt|}tj|D]F}dfdd|dD}|fdd | dDq"|S)NrDc3s|]}j|VqdSr,)rN)rr$rLr r rz)SimpleTokenizer.encode..r>c3s|]}j|VqdSr,rJ)rZ bpe_tokenrLr r rorr4) r7r3lowerr5findallr]rencoderVrnrA)rMr2Z bpe_tokensrir rLr rrs &zSimpleTokenizer.encodecsDdfdd|D}tfdd|Djddddd }|S) NrDcsg|]}j|qSr )rX)rrirLr r rrz*SimpleTokenizer.decode..csg|]}j|qSr )rP)rcrLr r rrr>replace)errorsrCr4)r bytearrayrTrt)rMtokensr2r rLr rTs(zSimpleTokenizer.decode) __name__ __module__ __qualname__rstrr^rnrrrTr r r r r8Hs)r8Z output_idscCs|}t|Sr,)cpunumpy _tokenizerrTr|r r r rTs rTMtextscontext_lengthreturncst|tr|g}tjdtjdfdd|D}tjt||tjd}t|D]B\}}t||kr~|d|}|d<t |||dt|f<qV|S)a Returns the tokenized representation of given input string(s) Parameters ---------- texts : Union[str, List[str]] An input string or a list of input strings to tokenize context_length : int The context length to use; all CLIP models use 77 as the context length Returns ------- A two-dimensional tensor containing the resulting tokens, shape = [number of input strings, context_length] rErFcs"g|]}gt|gqSr )rrrrr2Z eot_tokenZ sot_tokenr r rrztokenize..)ZdtypeNr_) isinstancer{rrKtorchzerosrWlong enumerateZtensor)rrZ all_tokensresultrlrwr rr tokenizes     rc@sHeZdZdZedddZddZd eeeefe e j dd d Z d S) HFTokenizerzHuggingFace tokenizer wrapper)tokenizer_namecCsddlm}|||_dS)Nr) AutoTokenizer)Z transformersrZfrom_pretrained tokenizer)rMrrr r r r^s zHFTokenizer.__init__cCs|j|dSr,)rsave_pretrained)rMdestr r r rszHFTokenizer.save_pretrainedrrcCs8t|tr|g}dd|D}|j|d|dddj}|S)NcSsg|]}tt|qSr )r7r3rr r r rrz(HFTokenizer.__call__..pt max_lengthT)Zreturn_tensorsrpaddingZ truncation)rr{r input_ids)rMrrrr r r __call__s zHFTokenizer.__call__N)r) rxryrz__doc__r{r^rrrintrTensorrr r r r rsr)r)rrQr.r functoolsrtypingrrr-regexr5renvironrr%r+r3r7objectr8rrrTr{rZ LongTensorrrr r r r s,     Q"