a  d)@sdZddlmZddlZddlmZddlmmZddlm Z ddl m Z ddl m Z eed d d ZGd d d ejZGdddejZdS)a] Halo Self Attention Paper: `Scaling Local Self-Attention for Parameter Efficient Visual Backbones` - https://arxiv.org/abs/2103.12731 @misc{2103.12731, Author = {Ashish Vaswani and Prajit Ramachandran and Aravind Srinivas and Niki Parmar and Blake Hechtman and Jonathon Shlens}, Title = {Scaling Local Self-Attention for Parameter Efficient Visual Backbones}, Year = {2021}, } Status: This impl is a WIP, there is no official ref impl and some details in paper weren't clear to me. The attention mechanism works but it's slow as implemented. Hacked together by / Copyright 2021 Ross Wightman )ListN)nn)make_divisible) trunc_normal_)_assert permute_maskc Cs|j\}}}}|jd}|dd}||dd} | d||} t| ddgd} t| d||g} | d|d|} | ddd||ddf} | ||d||dd|dd} | |S)a~ Compute relative logits along one dimension As per: https://gist.github.com/aravindsrinivas/56359b79f0ce4449bcb04ab4b56a57a2 Originally from: `Attention Augmented Convolutional Networks` - https://arxiv.org/abs/1904.09925 Args: q: (batch, height, width, dim) rel_k: (2 * window - 1, dim) permute_mask: permute output dim according to this rrN)shape transposereshapeFpadflattenexpandpermute) qZrel_kr BHWdimZrel_sizewin_sizexZx_padr^/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/timm/layers/halo_attn.py rel_logits_1ds    rcs(eZdZdZfddZddZZS) PosEmbedRelz Relative Position Embedding As per: https://gist.github.com/aravindsrinivas/56359b79f0ce4449bcb04ab4b56a57a2 Originally from: `Attention Augmented Convolutional Networks` - https://arxiv.org/abs/1904.09925 csZt||_||_tt|dd|||_tt|dd|||_ dS)z Args: block_size (int): block size win_size (int): neighbourhood window size dim_head (int): attention head dim scale (float): scale factor (for init) r rN) super__init__ block_sizedim_headr ParametertorchZrandn height_rel width_rel)selfr"rr#scale __class__rrr!Cs   zPosEmbedRel.__init__c Csl|j\}}}}|d|j|j|j}t||jdd}|dd}t||jdd}||}||||d}|S)Nr )rrr rrr )rr,rr-r )r rr"r#rr'rr&) r(rrBBZHW_Z rel_logits_wZ rel_logits_hZ rel_logitsrrrforwardQs zPosEmbedRel.forward)__name__ __module__ __qualname____doc__r!r0 __classcell__rrr*rr=s rc s2eZdZdZdfdd Zd d Zd d ZZS)HaloAttna Halo Attention Paper: `Scaling Local Self-Attention for Parameter Efficient Visual Backbones` - https://arxiv.org/abs/2103.12731 The internal dimensions of the attention module are controlled by the interaction of several arguments. * the output dimension of the module is specified by dim_out, which falls back to input dim if not set * the value (v) dimension is set to dim_out // num_heads, the v projection determines the output dim * the query and key (qk) dimensions are determined by * num_heads * dim_head if dim_head is not None * num_heads * (dim_out * attn_ratio // num_heads) if dim_head is None * as seen above, attn_ratio determines the ratio of q and k relative to the output if dim_head not used Args: dim (int): input dimension to the module dim_out (int): output dimension of the module, same as dim if not set feat_size (Tuple[int, int]): size of input feature_map (not used, for arg compat with bottle/lambda) stride: output stride of the module, query downscaled if > 1 (default: 1). num_heads: parallel attention heads (default: 8). dim_head: dimension of query and key heads, calculated from dim_out * attn_ratio // num_heads if not set block_size (int): size of blocks. (default: 8) halo_size (int): size of halo overlap. (default: 3) qk_ratio (float): ratio of q and k dimensions to output dimension when dim_head not set. (default: 1.0) qkv_bias (bool) : add bias to q, k, and v projections avg_down (bool): use average pool downsample instead of strided query blocks scale_pos_embed (bool): scale the position embedding as well as Q @ K Nrr,?Fc sZt|p|}||dks"J|dvs.J||_|pJt|| dd||_||j|_||j|_||j|_|jd|_| |_ ||_ |_ ||_ ||d|_ d|_d} |dkr| p||dk} | rdn||_|j |j|_ tj||jd|j| d |_tj||j|jd| d |_t|j |j |j|jd |_| rDtddnt|_|dS) Nr)rr r7)Zdivisorr rF)stridebias)r;)r"rr#r))r r! num_headsr dim_head_qk dim_head_vZ dim_out_qk dim_out_vr)scale_pos_embedr" block_size_ds halo_sizer block_striderZConv2drkvr pos_embedZ AvgPool2dZIdentitypoolreset_parameters)r(rZdim_outZ feat_sizer:r<r#r"rBZqk_ratioZqkv_biasZavg_downr@Z use_avg_poolr*rrr!}s6       zHaloAttn.__init__cCsZ|jjjdd}t|jj|dt|jj|dt|jj|jdt|jj|jddS)Nrr9)std) rZweightr rrDrEr&r)r')r(rHrrrrGs zHaloAttn.reset_parameterscCs|j\}}}}t||jdkdt||jdkd||j}||j}||}||} | d|j||j||jdddddd} | ||j|jd| dd} | |} t | |j |j |j |j g} | d|j|jd|j|j||j|j|j|ddddd} tj| |j|jgdd \} } |jrX| | dd || |j} n | | dd |j|| } | jdd } | |  dd}|d|j|j||}|ddddd||j||j||j}||}|S) Nrr rr,r r-)rr )r rr"rrr=rArr<rrDrrrBZunfoldrr>r%splitr@rEr)Zsoftmax contiguousviewr?rCrF)r(rrCrrZ num_h_blocksZ num_w_blocksZ num_blocksrrDkvZattnoutrrrr0sB        "   zHaloAttn.forward) NNrr7Nr7r,r8FFF)r1r2r3r4r!rGr0r5rrr*rr6as %r6)r4typingrr%rZtorch.nn.functionalZ functionalrZhelpersrZ weight_initrZ trace_utilsrintrModulerr6rrrrs     $t