a Ad@sddlmZddlZddlmZddlmZddlZddlmZm Z m Z ee e e edddZ Gd d d e Z Gd d d ZGd ddZeeeee dddZeeeee edddZeeeeedddZGddde Zdeeeee ee dddZdS))partialN)Tensor) checkpoint)Optional NamedTupleList)inputdimstartlengthreturncCs0t||||j|||kr |n |j||SN)torchnarrowshape)rr r r rG/var/www/html/stable-diffusion-webui/modules/sub_quadratic_attention.py narrow_truncsrc@s&eZdZUeed<eed<eed<dS) AttnChunk exp_valuesZexp_weights_sum max_scoreN)__name__ __module__ __qualname__r__annotations__rrrrrs rc@s$eZdZeeeeedddZdS)SummarizeChunkquerykeyvaluer cCsdSr rrrrrrr__call__%szSummarizeChunk.__call__N)rrr staticmethodrrr!rrrrr$s rc@s$eZdZeeeeedddZdS)ComputeQueryChunkAttnrcCsdSr rr rrrr!.szComputeQueryChunkAttn.__call__N)rrrr"rr!rrrrr#-s r#)rrrscaler c Cstjtjddd|j|jd||dd|dd}tj|ddd\}}|}t||}|jj d krrt ||nt || |j |j}| d}t ||jdd |S) NdevicedtyperalphabetaTkeepdimmpsr )rbaddbmmzerosr'r( transposemaxdetachexptypebmmtosqueezersum) rrrr$ attn_weightsr_Z exp_weightsrrrr_summarize_chunk6s 4 r?)rrrsummarize_chunk kv_chunk_sizer csj\}}}j\}}} ttdfdd fddtd|D} tttjt| } | \} } }tj|ddd\}}t ||}| t |d 9} | |9} | j dd }t | d j dd }||S) N chunk_idxr cs(td|}td|}||SNr%)r)rCZ key_chunkZ value_chunk)rrArr@rrr chunk_scannerUsz-_query_chunk_attention..chunk_scannercsg|] }|qSrr).0chunk)rErr dsz*_query_chunk_attention..rTr.r-r1) rintrrarangemapstackzipr5r7 unsqueezer<)rrrr@rA batch_x_headsk_tokensZk_channels_per_headr>Zv_channels_per_headchunksZ acc_chunkZ chunk_valuesZ chunk_weightsZ chunk_maxZ global_maxZ max_diffs all_values all_weightsr)rErrArr@rr_query_chunk_attentionKs      rTcCsvtjtjddd|j|jd||dd|dd}|jdd}~|jjdkrVt||nt|| |j |j}|S) Nr%r&r)rr*r-r1r0) rr2r3r'r(r4softmaxr8r9r:)rrrr$ attn_scores attn_probsZhidden_states_slicerrr$_get_attention_scores_no_kv_chunkingus  4rXc@seZdZUeed<eed<dS) ScannedChunkrCZ attn_chunkN)rrrrIrrrrrrrYs rYT)rrrrAkv_chunk_size_mincs j\}}|j\} } } |d} t|p2tt| | }|durJt||}ttdfdd } tt| d} |r|tt | n| } | |krtt | dn tt || d}kr|||dSt }ttD]F}|| |||d}||dd|||jd ddf<q|S) aComputes efficient dot-product attention given query, key, and value. This is efficient version of attention presented in https://arxiv.org/abs/2112.05682v2 which comes with O(sqrt(n)) memory requirements. Args: query: queries for calculating attention with shape of `[batch * num_heads, tokens, channels_per_head]`. key: keys for calculating attention with shape of `[batch * num_heads, tokens, channels_per_head]`. value: values to be used in attention with shape of `[batch * num_heads, tokens, channels_per_head]`. query_chunk_size: int: query chunks size kv_chunk_size: Optional[int]: key/value chunks size. if None: defaults to sqrt(key_tokens) kv_chunk_size_min: Optional[int]: key/value minimum chunk size. only considered when kv_chunk_size is None. changes `sqrt(key_tokens)` into `max(sqrt(key_tokens), kv_chunk_size_min)`, to ensure our chunk sizes don't get too small (smaller chunks = more chunks = less concurrent work done). use_checkpoint: bool: whether to use checkpointing (recommended True for training, False for inference) Returns: Output of shape `[batch * num_heads, query_tokens, channels_per_head]`. gNrBcstd|tSrD)rmin)rCq_tokensrquery_chunk_sizerrget_query_chunks z8efficient_dot_product_attention..get_query_chunk)r$)rAr@r r%)rr\rImathsqrtr5rrr?rrXrTr zeros_likerangeceil)rrrr_rAr[use_checkpointrOZq_channels_per_headr>rPr$r`r@Zcompute_query_chunk_attnresirVrr]refficient_dot_product_attentionsF       .ri)rZNNT) functoolsrrrtorch.utils.checkpointrratypingrrrrIrrrr#floatr?rTrXrYrirrrr sZ        +