a  dF@sxdZddlmZmZddlZddlmZddlmZddl m Z m Z ddl m Z Gdd d ejZGd d d ejZdS) a: Attention Pool 2D Implementations of 2D spatial feature pooling using multi-head attention instead of average pool. Based on idea in CLIP by OpenAI, licensed Apache 2.0 https://github.com/openai/CLIP/blob/3b473b0e682c091a9e53623eebc1ca1657385717/clip/model.py Hacked together by / Copyright 2021 Ross Wightman )UnionTupleN) to_2tuple)apply_rot_embedRotaryEmbedding) trunc_normal_cs8eZdZdZd eeeeedfdd Zdd ZZS) RotAttentionPool2daB Attention based 2D feature pooling w/ rotary (relative) pos embedding. This is a multi-head attention based replacement for (spatial) average pooling in NN architectures. Adapted from the AttentionPool2d in CLIP w/ rotary embedding instead of learned embed. https://github.com/openai/CLIP/blob/3b473b0e682c091a9e53623eebc1ca1657385717/clip/model.py NOTE: While this impl does not require a fixed feature size, performance at differeing resolutions from train varies widely and falls off dramatically. I'm not sure if there is a way around this... -RW NT) in_features out_features embed_dim num_headsqkv_biascst|p|}|p|}tj||d|d|_t|||_||_||dksTJ|||_|jd|_t |j|_ t |jj |ddtj |jjdS)Nbiasr࿩Zstd)super__init__nnLinearqkvprojrhead_dimscaler pos_embedrweightinitzeros_r)selfr r r rr __class__e/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/timm/layers/attention_pool2d.pyrs    zRotAttentionPool2d.__init__cCs|j\}}}}||}||d|ddd}tj|jddd|gdd}||||dd|j|jddddd }|d|d|d}}} |ddddddf|ddddddf} }|j ||f\} } t || | }tj| |gdd}|ddddddf|ddddddf} }t || | }tj| |gdd}|| d d|j }|jdd}||  dd||dd}||}|dddfS) NrrTZkeepdimZdimrr )shapereshapepermutetorchcatmeanrrrrZ get_embedr transposersoftmaxr)r!xB_HWNqkvZqcZsin_embZcos_embZkcattnr$r$r%forward4s$.6 6   zRotAttentionPool2d.forward)NNr T) __name__ __module__ __qualname____doc__intboolrr= __classcell__r$r$r"r%r s r csJeZdZdZd eeeeeeffeeeedfdd Zdd Z Z S) AttentionPool2da Attention based 2D feature pooling w/ learned (absolute) pos embedding. This is a multi-head attention based replacement for (spatial) average pooling in NN architectures. It was based on impl in CLIP by OpenAI https://github.com/openai/CLIP/blob/3b473b0e682c091a9e53623eebc1ca1657385717/clip/model.py NOTE: This requires feature size upon construction and well prevent adaptive sizing of the network. Nr T)r feat_sizer r rrcst|p|}|p|}||dks*Jt||_tj||d|d|_t|||_||_|||_ |j d|_ |jd|jd}t t |d||_t|j|ddt|jj|ddtj|jjdS)Nrrrrrr)rrrrFrrrrrrr Parameterr.zerosrrrrr r)r!r rFr r rrZ spatial_dimr"r$r%rXs    zAttentionPool2d.__init__c Cs*|j\}}}}||}|jd|ks(J|jd|ks:J||d|ddd}tj|jddd|gdd}||jd |j }| |||dd|j |j ddddd }|d|d|d}}} ||d d|j} | jdd} | | dd||dd}||}|dddfS) Nrrr&r'Tr(r)rr r*)r+rFr,r-r.r/r0rZ unsqueezetoZdtyperrrr1rr2r) r!r3r4r5r6r7r8r9r:r;r<r$r$r%r=ss.  zAttentionPool2d.forward)NNr T) r>r?r@rArBrrrCrr=rDr$r$r"r%rEOs rE)rAtypingrrr.Ztorch.nnrZhelpersrZpos_embed_sincosrrZ weight_initrModuler rEr$r$r$r%s    ;