a  d@sdZddlmZddlZddlmZddlmmZddl m Z m Z ddl m Z ddlmZeedd d ZGd d d ejZGd ddejZdS)aP Bottleneck Self Attention (Bottleneck Transformers) Paper: `Bottleneck Transformers for Visual Recognition` - https://arxiv.org/abs/2101.11605 @misc{2101.11605, Author = {Aravind Srinivas and Tsung-Yi Lin and Niki Parmar and Jonathon Shlens and Pieter Abbeel and Ashish Vaswani}, Title = {Bottleneck Transformers for Visual Recognition}, Year = {2021}, } Based on ref gist at: https://gist.github.com/aravindsrinivas/56359b79f0ce4449bcb04ab4b56a57a2 This impl is a WIP but given that it is based on the ref gist likely not too far off. Hacked together by / Copyright 2021 Ross Wightman )ListN) to_2tuplemake_divisible) trunc_normal_)_assert permute_maskc Cs|j\}}}}||dd}|d|d|d}t|ddgd}t|d|dg}|d|dd|d}|ddd||ddf}|||d||dd|dd}||S)a Compute relative logits along one dimension As per: https://gist.github.com/aravindsrinivas/56359b79f0ce4449bcb04ab4b56a57a2 Originally from: `Attention Augmented Convolutional Networks` - https://arxiv.org/abs/1904.09925 Args: q: (batch, heads, height, width, dim) rel_k: (2 * width - 1, dim) permute_mask: permute output dim according to this rrN)shape transposereshapeFpadflattenexpandZpermute) qZrel_kr BHWdimxZx_padrd/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/timm/layers/bottleneck_attn.py rel_logits_1ds  rcs(eZdZdZfddZddZZS) PosEmbedRelz Relative Position Embedding As per: https://gist.github.com/aravindsrinivas/56359b79f0ce4449bcb04ab4b56a57a2 Originally from: `Attention Augmented Convolutional Networks` - https://arxiv.org/abs/1904.09925 cshtt|\|_|_||_tt |jdd|||_ tt |jdd|||_ dS)Nr r) super__init__rheightwidthdim_headnn ParametertorchZrandn height_rel width_rel)self feat_sizer"scale __class__rrr=s  "zPosEmbedRel.__init__cCsf|j\}}}|||j|jd}t||jdd}|dd}t||jdd}||}||||}|S)Nr )rrr rrr )rr-rr.r )r rr r!rr'rr&)r(rrZHW_Z rel_logits_wZ rel_logits_hZ rel_logitsrrrforwardDs  zPosEmbedRel.forward)__name__ __module__ __qualname____doc__rr0 __classcell__rrr+rr8s rcs2eZdZdZd fdd Zd d Zd d ZZS)BottleneckAttna Bottleneck Attention Paper: `Bottleneck Transformers for Visual Recognition` - https://arxiv.org/abs/2101.11605 The internal dimensions of the attention module are controlled by the interaction of several arguments. * the output dimension of the module is specified by dim_out, which falls back to input dim if not set * the value (v) dimension is set to dim_out // num_heads, the v projection determines the output dim * the query and key (qk) dimensions are determined by * num_heads * dim_head if dim_head is not None * num_heads * (dim_out * attn_ratio // num_heads) if dim_head is None * as seen above, attn_ratio determines the ratio of q and k relative to the output if dim_head not used Args: dim (int): input dimension to the module dim_out (int): output dimension of the module, same as dim if not set stride (int): output stride of the module, avg pool used if stride == 2 (default: 1). num_heads (int): parallel attention heads (default: 4) dim_head (int): dimension of query and key heads, calculated from dim_out * attn_ratio // num_heads if not set qk_ratio (float): ratio of q and k dimensions to output dimension when dim_head not set. (default: 1.0) qkv_bias (bool): add bias to q, k, and v projections scale_pos_embed (bool): scale the position embedding as well as Q @ K Nrr.?Fc st|dusJd|p |}||dks2J||_|pNt||dd||_||j|_||j|_||j|_|jd|_| |_ t j ||jd|jd|d|_ t ||j|jd |_|dkrt ddnt |_|dS) NzBA concrete feature size matching expected input (H, W) is requiredr)Zdivisorr r)Zbias)r"r*)rr num_headsr dim_head_qk dim_head_v dim_out_qk dim_out_vr*scale_pos_embedr#ZConv2dqkvr pos_embedZ AvgPool2dZIdentitypoolreset_parameters) r(rZdim_outr)Zstrider:r"Zqk_ratioZqkv_biasr?r+rrrjs      zBottleneckAttn.__init__cCsFt|jj|jjjdddt|jj|jdt|jj|jddS)Nrr9)Zstd)rr@Zweightr rAr&r*r')r(rrrrCszBottleneckAttn.reset_parametersc Cs |j\}}}}t||jjkdt||jjkd||}tj||j|j|j gdd\}}}| ||j |j d dd}| ||j |j d}| ||j |jd dd}|jr|||||j} n|||j||} | jdd} | | dd ||j ||} || } | S)Nr)rr r )r rrAr r!r@r%splitr=r>rr:r;rr<r?r*ZsoftmaxrB) r(rrCrrrkvZattnoutrrrr0s "  zBottleneckAttn.forward)NNrr.Nr7FF)r1r2r3r4rrCr0r5rrr+rr6Ts r6)r4typingrr%Ztorch.nnr#Ztorch.nn.functionalZ functionalrZhelpersrrZ weight_initrZ trace_utilsrintrModulerr6rrrrs