a Ad:@sddlZddlmZddlZddlmZmZmZddlm Z GdddZ eddd gZ Gd d d ej j ZGd d d eZGdddeZdS)N) namedtuple) prompt_parserdevices sd_hijack)optsc@seZdZdZddZdS) PromptChunkan This object contains token ids, weight (multipliers:1.4) and textual inversion embedding info for a chunk of prompt. If a prompt is short, it is represented by one PromptChunk, otherwise, multiple are necessary. Each PromptChunk contains an exact amount of tokens - 77, which includes one for start and end token, so just 75 tokens from prompt. cCsg|_g|_g|_dSN)tokens multipliersfixes)selfr >/var/www/html/stable-diffusion-webui/modules/sd_hijack_clip.py__init__szPromptChunk.__init__N)__name__ __module__ __qualname____doc__rr r r rr srPromptChunkFixoffset embeddingcsheZdZdZfddZddZddZdd Zd d Zd d Z ddZ ddZ ddZ ddZ ZS)%FrozenCLIPEmbedderWithCustomWordsBasezA pytorch module that is a wrapper for FrozenCLIPEmbedder module. it enhances FrozenCLIPEmbedder, making it possible to have unlimited prompt length and assign weights to tokens in prompt. csBt||_||_d|_t|dd|_t|dd|_d|_dS)NK is_trainableF input_keytxt) superrwrappedhijack chunk_lengthgetattrrrlegacy_ucg_valr rr __class__r rr#s z.FrozenCLIPEmbedderWithCustomWordsBase.__init__cCs8t}|jg|jg|jd|_dg|jd|_|S)z+creates an empty PromptChunk and returns it?)rid_startid_endrr r )r chunkr r r empty_chunk1sz1FrozenCLIPEmbedderWithCustomWordsBase.empty_chunkcCstt|d|j|jS)zreturns the maximum number of tokens a prompt of a known length can have before it requires one more PromptChunk to be representedr%)mathceilmaxr)r token_countr r rget_target_prompt_token_count9szCFrozenCLIPEmbedderWithCustomWordsBase.get_target_prompt_token_countcCstdS)z3Converts a batch of texts into a batch of token idsNNotImplementedError)r textsr r rtokenize>sz.FrozenCLIPEmbedderWithCustomWordsBase.tokenizecCstdS)a converts a batch of token ids (in python lists) into a single tensor with numeric respresentation of those tokens; All python lists with tokens are assumed to have same length, usually 77. if input is a list with B elements and each element has T tokens, expected output shape is (B, T, C), where C depends on model - can be 768 and 1024. Among other things, this call will read self.hijack.fixes, apply it to its inputs, and clear it (setting it to None). Nr1)r r r r rencode_with_transformersCs z>FrozenCLIPEmbedderWithCustomWordsBase.encode_with_transformerscCstdS)aConverts text into a tensor with this text's tokens' embeddings. Note that those are embeddings before they are passed through transformers. nvpt is used as a maximum length in tokens. If text produces less teokens than nvpt, only this many is returned.Nr1)r init_textnvptr r rencode_embedding_init_textNsz@FrozenCLIPEmbedderWithCustomWordsBase.encode_embedding_init_textcs4tjrt|}n |dgg}dd|D}gtdddfdd }t||D]\}\}}|d kr|dkr|qbd}|t|krb||} | jkrtj ntj dkrJtj j krJdkrJtj tj krJd } j | d } j | d } j d | _ j d | _ || _ | _ tj j krb|j j||\} }| d urj | j ||d 7}qt| j}tj |j kr|jttj | j dg|7_ j |g|7_ ||7}qqbj s"s,|d d fS)z this transforms a single prompt into a list of PromptChunk objects - as many as needed to represent the prompt. Returns the list and the total number of tokens in the prompt. r&cSsg|] \}}|qSr r ).0text_r r r `zGFrozenCLIPEmbedderWithCustomWordsBase.tokenize_line..rFcs|rtj7n j7jtj}|dkr`jjg|7_jdg|7_jgjjg_dgjdg_dtdS)zputs current chunk into the list of results and produces the next one - empty; if is_last is true, tokens tokens at the end won't add to token_countrr&r>N)lenr rr)r r(appendr)is_lastto_addr*chunks last_commar r/r r next_chunkgs  zGFrozenCLIPEmbedderWithCustomWordsBase.tokenize_line..next_chunkBREAKr%NT)rA)F)renable_emphasisrparse_prompt_attentionr4rzipr? comma_tokenr comma_padding_backtrackrr r embedding_dbfind_embedding_at_positionr@intvectorsr r)r lineparsed tokenizedrFr r:weightpositiontokenZbreak_locationZ reloc_tokensZ reloc_multsrZembedding_length_in_tokensemb_lenr rCr tokenize_lineTsZ     >      z3FrozenCLIPEmbedderWithCustomWordsBase.tokenize_linecCsZd}i}g}|D]@}||vr&||}n ||\}}t||}|||<||q||fS)z Accepts a list of texts and calls tokenize_line() on each, with cache. Returns the list of results and maximum length, in tokens, of all texts. r)rXr.r@)r r3r/cache batch_chunksrQrDZcurrent_token_countr r r process_textss   z3FrozenCLIPEmbedderWithCustomWordsBase.process_textscstjrddl}|j|S|\}}i}tdd|D}g}t|D]fdd|D}dd|D} dd|D} dd|Dj_ jj D]} | D]\} } | || j <qq | | }| |qLtj rd|rdg}|D]<\}} | j}|sq|d d d d }| |d |q|rdjjd rR| jjd d|jjd <tjddrt||djfSt|SdS)ac Accepts an array of texts; Passes texts through transformers network to create a tensor with numerical representation of those texts. Returns a tensor with shape of (B, T, C), where B is length of the array; T is length, in tokens, of texts (including padding) - T will be a multiple of 77; and C is dimensionality of each token - for SD1 it's 768, for SD2 it's 1024, and for SDXL it's 1280. An example shape returned by this function can be: (2, 77, 768). For SDXL, instead of returning one tensor avobe, it returns a tuple with two: the other one with shape (B, 1280) with pooled values. Webui usually sends just one text at a time through this function - the only time when texts is an array with more than one elemenet is when you do prompt editing: "a picture of a [cat:dog:0.4] eating ice cream" rNcSsg|] }t|qSr )r?r9xr r rr<r=zAFrozenCLIPEmbedderWithCustomWordsBase.forward..cs(g|] }t|kr|nqSr )r?r+)r9rDir r rr<r=cSsg|] }|jqSr )r r\r r rr<r=cSsg|] }|jqSr )r r\r r rr<r=cSsg|] }|jqSr )r r\r r rr<r=:,z: z TI hashesz, return_pooledF)ruse_old_emphasis_implementationZmodules.sd_hijack_clip_oldZsd_hijack_clip_oldZ forward_oldr[r.rangerr nameprocess_tokensr@(textual_inversion_add_hashes_to_infotextitems shorthashreplaceextra_generation_paramsgetjoinr rtorchhstackpooled)r r3modulesrZr/Zused_embeddingsZ chunk_countzsZ batch_chunkr r r _positionrzhashesrfrjr r^rforwards@      z-FrozenCLIPEmbedderWithCustomWordsBase.forwardc Cst|tj}|j|jkr\tt|D]0}|| |j}|j|||d|j df<q*| |}t |dd}t|tj}| }|||j d|j }| } ||| }|dur||_|S)a sends one single prompt chunk to be encoded by transformers neural network. remade_batch_tokens is a batch of tokens - a list, where every element is a list of tokens; usually there are exactly 77 tokens in the list. batch_multipliers is the same but for multipliers instead of tokens. Multipliers are used to give more or less weight to the outputs of transformers network. Each multiplier corresponds to one token. r%rqN)r%)roasarraytordevicer)id_padrer?indexshaper5r meanreshapeexpandrq) r Zremade_batch_tokensZbatch_multipliersr Z batch_posr|rurqZ original_meanZnew_meanr r rrgs    z4FrozenCLIPEmbedderWithCustomWordsBase.process_tokens)rrrrrr+r0r4r5r8rXr[rwrg __classcell__r r r#rrs  _7rcs4eZdZfddZddZddZddZZS) !FrozenCLIPEmbedderWithCustomWordsc st|||j|_|j}|dd|_i|_dd|D}|D]h\}}d}|D]D}|dkrn|d}|dkr~|d9}|dkr|d9}|d krZ|d}qZ|dkrJ||j|<qJ|jjj |_ |jjj |_ |j |_ dS) Nz,cSs8g|]0\}}d|vs,d|vs,d|vs,d|vr||fqS)()[]r )r9kvr r rr<,r=z>FrozenCLIPEmbedderWithCustomWords.__init__..r&rg?rrr)rr tokenizer get_vocabrmrKZ token_multsrir bos_token_idr( eos_token_idr)r{) r rrvocabZtokens_with_parensr:identmultcr#r rr#s,      z*FrozenCLIPEmbedderWithCustomWords.__init__cCs|jj|dddd}|S)NF) truncationadd_special_tokens input_ids)rr)r r3rSr r rr4@sz*FrozenCLIPEmbedderWithCustomWords.tokenizecCsH|jj|tj d}tjdkr>|jtj }|jjj|}n|j}|S)Nroutput_hidden_statesr%)r transformerrCLIP_stop_at_last_layers hidden_states text_modelfinal_layer_normlast_hidden_stater r outputsrur r rr5Es  z:FrozenCLIPEmbedderWithCustomWords.encode_with_transformerscCsH|jjjj}|jj||dddd}|j||jjjj d}|S)NptF) max_lengthreturn_tensorsrrr) rrr embeddingsrtoken_embeddingryrTrzsqueeze)r r6r7Zembedding_layeridsembeddedr r rr8Ps  z|jj||jjdkd}|jjdkr,|j}n|j|jj}|S)Nhiddenrlast)rrlayerrr layer_idxrr r rr5\s  zAFrozenCLIPEmbedderForSDXLWithCustomWords.encode_with_transformers)rrrrr5rr r r#rrXs r)r, collectionsrrorrrrrmodules.sharedrrrnnModulerrrr r r rs  6