a d@s>ddlZddlmZddlmZddlmZddlZddl Z ddl m Z ddl m mZddlmZddlmZddlmZmZmZmZmZmZmZdd lmZmZd d Zd d Z Gddde j!Z"Gddde j!Z#Gddde j$e#Z%Gddde j!Z&Gddde j!Z'Gddde j!Z(Gddde#Z)Gddde j!Z*ddZ+Gd d!d!e j!Z,Gd"d#d#e j!Z-Gd$d%d%e j!Z.Gd&d'd'e j!Z/Gd(d)d)e/Z0Gd*d+d+e j!Z1e2d,kr:Gd-d.d.e j!Z3e/d/d0d1d1d2d1d3gd3gd4d0d5d/d6d5d7 4Z5e 6d8d1d0d04Z7e j8dd9d:d;d<Z9e5e7e9Z:e;d=dS)>N)abstractmethod)partial)Iterable) rearrange)SpatialTransformer) avg_pool_nd checkpointconv_ndlinear normalizationtimestep_embedding zero_module)defaultexistscCsdSNxrro/var/www/html/stable-diffusion-webui/repositories/generative-models/sgm/modules/diffusionmodules/openaimodel.pyconvert_module_to_f16srcCsdSrrrrrrconvert_module_to_f32srcs6eZdZdZdeeeedfdd ZddZZS) AttentionPool2dzS Adapted from CLIP: https://github.com/openai/CLIP/blob/main/clip/model.py N) spacial_dim embed_dimnum_heads_channels output_dimcspttt||dd|d|_td|d|d|_td||pN|d|_ |||_ t |j |_ dS)Ng?r) super__init__nn Parameterthrandnpositional_embeddingr qkv_projc_proj num_heads QKVAttention attention)selfrrrr __class__rrr (s  zAttentionPool2d.__init__cCs|j^}}}|||d}tj|jddd|gdd}||jdddddf|j}||}| |}| |}|dddddfS)NT)dimkeepdimr/r) shapereshaper#catmeanr%todtyper&r*r')r+rbcZ_spatialrrrforward8s $   zAttentionPool2d.forward)N)__name__ __module__ __qualname____doc__intr r: __classcell__rrr,rr#s rc@seZdZdZeddZdS) TimestepBlockzT Any module where forward() takes timestep embeddings as a second argument. cCsdS)zJ Apply the module to `x` given `emb` timestep embeddings. Nrr+rembrrrr:HszTimestepBlock.forwardN)r;r<r=r>rr:rrrrrACsrAc@seZdZdZdddZdS)TimestepEmbedSequentialzt A sequential module that passes timestep embeddings to the children that support it as an extra input. NFc CsB|D]8} t| tr| ||}qt| tr4| ||}q| |}q|Sr) isinstancerAr) r+rrCcontextZ skip_time_mixZ time_contextZnum_video_framesZtime_context_catZ*use_crossframe_attention_in_spatial_layerslayerrrrr:Us      zTimestepEmbedSequential.forward)NFNNNF)r;r<r=r>r:rrrrrDOs rDcs*eZdZdZd fdd Zdd ZZS) UpsampleaA An upsampling layer with an optional convolution. :param channels: channels in the inputs and outputs. :param use_conv: a bool determining if a convolution is applied. :param dims: determines if the signal is 1D, 2D, or 3D. If 3D, then upsampling occurs in the inner-two dimensions. rNrFcsLt||_|p||_||_||_||_|rHt||j|jd|d|_dS)Nrpadding) rr channels out_channelsuse_convdimsthird_upr conv)r+rKrMrNrLrJrOr,rrr ss  zUpsample.__init__cCs|jd|jksJ|jdkrb|js(dnd}tj|||jd|jdd|jddfdd}ntj|ddd}|jr||}|S)Nrrrnearest)mode) scale_factorrS)r2rKrNrOF interpolaterMrP)r+rZt_factorrrrr:s & zUpsample.forward)rNrFr;r<r=r>r r:r@rrr,rrHjs rHcs*eZdZdZdfdd ZddZZS) TransposedUpsamplez%Learned 2x upsampling without paddingNcs6t||_|p||_tj|j|j|dd|_dS)Nr kernel_sizestride)rr rKrLr!ConvTranspose2dup)r+rKrLksr,rrr s    zTransposedUpsample.__init__cCs ||Sr)r^r+rrrrr:szTransposedUpsample.forward)NrYrWrrr,rrXs rXcs*eZdZdZd fdd Zdd ZZS) DownsampleaD A downsampling layer with an optional convolution. :param channels: channels in the inputs and outputs. :param use_conv: a bool determining if a convolution is applied. :param dims: determines if the signal is 1D, 2D, or 3D. If 3D, then downsampling occurs in the inner-two dimensions. rNrFc st||_|p||_||_||_|dkr2dn |s:dnd}|rtd|dtd|jd|jd |d ||dkrtd |t||j|jd||d |_n |j|jksJt |||d |_dS)Nrr)rrr)rrrz!Building a Downsample layer with z dims.z --> settings are: in-chn: z , out-chn: z, kernel-size: 3, stride: z , padding: z& --> Downsampling third axis (time): )r\rJrZ) rr rKrLrMrNprintr opr)r+rKrMrNrLrJZ third_downr\r,rrr s6   zDownsample.__init__cCs|jd|jksJ||S)Nr)r2rKrcr`rrrr:szDownsample.forward)rNrFrWrrr,rras rac s2eZdZdZd fdd Zdd Zd d ZZS) ResBlocka A residual block that can optionally change the number of channels. :param channels: the number of input channels. :param emb_channels: the number of timestep embedding channels. :param dropout: the rate of dropout. :param out_channels: if specified, the number of out channels. :param use_conv: if True and out_channels is specified, use a spatial convolution instead of a smaller 1x1 convolution to change the channels in the skip connection. :param dims: determines if the signal is 1D, 2D, or 3D. :param use_checkpoint: if True, use gradient checkpointing on this module. :param up: if True, use this block for upsampling. :param down: if True, use this block for downsampling. NFrrc st||_||_||_|p"||_||_||_||_| |_ t | t rXdd| D}n| d}t t|t t|||j| |d|_| p| |_| rt|d||_t|d||_n2| rt|d||_t|d||_nt |_|_| |_|rd|jn|j|_|jr6td|jj|jr(Jd|_d|_ nt t t||j|_t t|jt t j|dt t||j|j| |d|_!|j|krt |_"n0|rt|||j| |d|_"nt|||jd|_"dS) NcSsg|] }|dqS)rr).0krrr z%ResBlock.__init__..rrIFzSkipping timestep embedding in )pr)#rr rK emb_channelsdropoutrLrMuse_checkpointuse_scale_shift_normexchange_temb_dimsrErr! Sequentialr SiLUr in_layersupdownrHh_updx_updraIdentity skip_t_embZemb_out_channelsrbr-r; emb_layersr Dropoutr out_layersskip_connection)r+rKrjrkrLrMrmrNrlr^downr[rnrvrJr,rrr sz         zResBlock.__init__cCst|j||f||jS)a Apply the block to a Tensor, conditioned on a timestep embedding. :param x: an [N x C x ...] Tensor of features. :param emb: an [N x emb_channels] Tensor of timestep embeddings. :return: an [N x C x ...] Tensor of outputs. )r _forward parametersrlrBrrrr:6szResBlock.forwardc Cs|jrF|jdd|jd}}||}||}||}||}n ||}|jrbt|}n|||j }t |j t |j kr|d}qt|j r|j d|j dd}}tj|ddd\} } ||d| | }||}n"|jrt|d}||}| |}|||S)Nr.).Nrrrr1zb t c ... -> b c t ...)rrrqrsrtrvr# zeros_likerwtyper7lenr2rmrychunkrnrrz) r+rrCZin_restZin_convhemb_outout_normZout_restscaleshiftrrrr|As,         zResBlock._forward) NFFrFFFrFFr;r<r=r>r r:r|r@rrr,rrdsZ rdcs2eZdZdZd fdd ZddZd d ZZS) AttentionBlocka An attention block that allows spatial positions to attend to each other. Originally ported from here, but adapted to the N-d case. https://github.com/hojonathanho/diffusion/blob/1e0dceb3b3495bbe19116a5e1b3596cd0706c543/diffusion_tf/models/unet.py#L66. rr.Fcst||_|dkr ||_n*||dks@Jd|d||||_||_t||_td||dd|_|rt |j|_ n t |j|_ t td||d|_ dS)Nr.rzq,k,v channels z' is not divisible by num_head_channels rr)rr rKr(rlr normr qkvr)r*QKVAttentionLegacyrproj_out)r+rKr(num_head_channelsrluse_new_attention_orderr,rrr es      zAttentionBlock.__init__cKst|j|f|dS)NT)r r|r})r+rkwargsrrrr:szAttentionBlock.forwardcCsV|j^}}}|||d}|||}||}||}||j||g|RS)Nr.)r2r3rrr*r)r+rr8r9spatialrrrrrr|s    zAttentionBlock._forward)rr.FFrrrr,rr^s rcCsL|dj^}}}tt|}d||d|}|jt|g7_dS)a( A counter for the `thop` package to count the operations in an attention operation. Meant to be used like: macs, params = thop.profile( model, inputs=(inputs, timestamps), custom_ops={QKVAttention: QKVAttention.count_flops}, ) rrN)r2r?npprodZ total_opsr# DoubleTensor)model_xyr8r9rZ num_spatialZ matmul_opsrrrcount_flops_attns rcs4eZdZdZfddZddZeddZZS)rzh A module which performs QKV attention. Matches legacy QKVAttention + input/ouput heads shaping cst||_dSrrr n_headsr+rr,rrr s zQKVAttentionLegacy.__init__c Cs|j\}}}|d|jdks"J|d|j}|||j|d|j|dd\}}}dtt|} td|| || } tj| dd | j } td| |} | |d|S)z Apply QKV attention. :param qkv: an [N x (H * 3 * C) x T] tensor of Qs, Ks, and Vs. :return: an [N x (H * C) x T] tensor after attention. rrrr1 bct,bcs->btsr. bts,bcs->bct) r2rr3splitmathsqrtr#einsumsoftmaxfloatrr7 r+rbswidthlengthchqrfvrweightarrrr:s (zQKVAttentionLegacy.forwardcCs t|||Srrrrrrrr count_flopsszQKVAttentionLegacy.count_flops r;r<r=r>r r: staticmethodrr@rrr,rrs  rcs4eZdZdZfddZddZeddZZS)r)zP A module which performs QKV attention and splits in a different order. cst||_dSrrrr,rrr s zQKVAttention.__init__c Cs|j\}}}|d|jdks"J|d|j}|jddd\}}}dtt|} td|| ||j|||| ||j||} tj| dd | j } td| | ||j||} | |d|S)z Apply QKV attention. :param qkv: an [N x (3 * H * C) x T] tensor of Qs, Ks, and Vs. :return: an [N x (H * C) x T] tensor after attention. rrrr1rr.r) r2rrrrr#rviewrrrr7r3rrrrr:s zQKVAttention.forwardcCs t|||SrrrrrrrszQKVAttention.count_flopsrrrr,rr)s  r)cs$eZdZfddZddZZS)Timestepcst||_dSr)rr r/)r+r/r,rrr s zTimestep.__init__cCs t||jSr)r r/)r+trrrr:szTimestep.forward)r;r<r=r r:r@rrr,rrs rcs<eZdZdZdfd d Zd dZddZdddZZS) UNetModela The full UNet model with attention and timestep embedding. :param in_channels: channels in the input Tensor. :param model_channels: base channel count for the model. :param out_channels: channels in the output Tensor. :param num_res_blocks: number of residual blocks per downsample. :param attention_resolutions: a collection of downsample rates at which attention will take place. May be a set, list, or tuple. For example, if this contains 4, then at 4x downsampling, attention will be used. :param dropout: the dropout probability. :param channel_mult: channel multiplier for each level of the UNet. :param conv_resample: if True, use learned convolutions for upsampling and downsampling. :param dims: determines if the signal is 1D, 2D, or 3D. :param num_classes: if specified (as an int), then this model will be class-conditional with `num_classes` classes. :param use_checkpoint: use gradient checkpointing to reduce memory usage. :param num_heads: the number of attention heads in each attention layer. :param num_heads_channels: if specified, ignore num_heads and instead use a fixed channel width per attention head. :param num_heads_upsample: works with num_heads to set a different number of heads for upsampling. Deprecated. :param use_scale_shift_norm: use a FiLM-like conditioning mechanism. :param resblock_updown: use residual blocks for up/downsampling. :param use_new_attention_order: use a different attention pattern for potentially increased efficiency. rrrrQTrNFr.rrc!0stddlm}!|r*|dus*Jd|durR|s>Jdt||!krRt|}|dkr^| }| dkrv|dksvJd|dkr| dksJd|_|_|_t |t rt ||g}nt ||!rt|}t | |d} t |t rt ||g_ n t |t |krtd|_ |dur6t |t |ks6Jdurt t j ksXJttfdd tt s~Jtd d |d |_|_|_|_| _| _| rtd | _|_|_|du_|| ks| s|rJd_jrtt|dndd }"|d}#|"t !t"||#t #t"|#|#_$jdurt jt rtt %| |#_&njdkrtdt 'd|#_&njdkr|"t !t(|t !t"||#t #t"|#|#_&nHjdkr|dusJt !t !t"||#t #t"|#|#_&ntt )t*t+| ||dddg_,|_-|g}$|}%d}&t.|D]\}'}(tj |'D] })|"t/|%|#||(|| | |dg}*|(|}%|&|vrV|dkr|%| }+n |%|} |}+|r|r|%| n|}+t0|r||'},nd},t0r|)|'krV|*1|s2|"t2|%| | |+|dn |"t3|%| |+||'||,||| d j,1t*|*j-|%7_-|$1|%qr|'t |dkrZ|%}-j,1t*|r|"t/|%|#||-| | |ddnt4|%|| |-d|-}%|$1|%|&d9}&j-|%7_-qZ|dkr|%| }+n |%|} |}+|r6|r2|%| n|}+t*|"t/|%|#|| | |d |sj|"t2|%| | |+|dn|"t3|%| |+| ||||| d |"t/|%|#|| | |d _5j-|%7_-t )g_6tt.|dddD]\}'}(tj |'dD]h}.|$7}/|"t/|%|/|#|||(| | |dg}*||(}%|&|vr|dkrF|%| }+n |%|} |}+|rj|rf|%| n|}+t0|r~||'},nd},t0r|.|'kr|*1|s|"t2|%| ||+|dn |"t3|%| |+||'||,||| d |'r8|.j |'kr8|%}-|*1|r|"t/|%|#||-| | |dd!nt8|%|| |-d|&d}&j61t*|*j-|%7_-qq|"t !t9|%t #t:t+| ||ddd_;jr|"t !t9|%t+| ||d_jrhz$UNetModel.__init__..z7Constructor of UNetModel received num_attention_blocks=z;. This option has LESS priority than attention_resolutions zz, i.e., in cases where num_attention_blocks[i] > 0 but 2**i not in attention_resolutions, attention will still not be set.z8WARNING: use_fp16 was dropped and has no effect anymore.F)offload_to_cpucSs|SrrrrrrrrhrQ continuousz'setting up linear c_adm embedding layerrtimestep sequentialrrIrLrNrlrmrlr(rr)depth context_dimdisable_self_attn use_linear attn_typerlTrLrNrlrmr{rNrLrrNrlrm)rLrNrlrmr^)=rr Zomegaconf.listconfigrrlist in_channelsmodel_channelsrLrEr?rrr ValueErrorallmaprangerbattention_resolutionsrk channel_mult conv_resample num_classesrlr(rnum_heads_upsamplepredict_codebook_idsuse_fairscale_checkpointrcheckpoint_wrapperr!ror rp time_embed Embedding label_embLinearr ModuleListrDr input_blocks _feature_size enumeraterdrappendrrra middle_block output_blockspoprHr routZ id_predictor)0r+rrrLrrrkrrrNrrluse_fp16r(rrrmresblock_updownruse_spatial_transformertransformer_depthrZn_embedlegacyZdisable_self_attentionsrZdisable_middle_self_attnuse_linear_in_transformerZspatial_transformer_attn_typeZadm_in_channelsrrZtransformer_depth_middlercheckpoint_wrapper_fntime_embed_diminput_block_chansrdslevelmultnrlayersdim_headZ disabled_saout_chrZichr,rrr s#                                  -               zUNetModel.__init__cCs(|jt|jt|jtdSz< Convert the torso of the model to float16. N)rapplyrrrr+rrrconvert_to_fp16s  zUNetModel.convert_to_fp16cCs(|jt|jt|jtdSz< Convert the torso of the model to float32. N)rrrrrrrrrconvert_to_fp32s  zUNetModel.convert_to_fp32c Ks|du|jduksJdg}t||jdd}||}|jdurh|jd|jdksZJ|||}|} |jD]} | | ||} || qr|| ||} |j D]&} t j | | gdd} | | ||} q| |j} |jrdsJdn || SdS) aQ Apply the model to an input batch. :param x: an [N x C x ...] Tensor of inputs. :param timesteps: a 1-D batch of timesteps. :param context: conditioning plugged in via crossattn :param y: an [N] Tensor of labels, if class-conditional. :return: an [N x C x ...] Tensor of outputs. Nzr rrr:r@rrr,rrsF$.rcseZdZdfdd ZZS)NoTimeUNetModelNc s$t|}tj||||fi|Sr)r#r~rr:)r+rrrFrrr,rrr:s zNoTimeUNetModel.forward)NNN)r;r<r=r:r@rrr,rrsrc s:eZdZdZdfd d Zd d ZddZddZZS)EncoderUNetModelz] The half UNet model with attention and timestep embedding. For usage, see UNet. rrTrFrr.adaptivecst|dkr| }||_||_||_||_||_||_||_| |_ | |_ | rVt j nt j |_| |_||_||_|d}tt||tt|||_ttt| ||dddg|_||_|g}|}d}t|D]\}}t|D]r}t|||||| | |dg}||}||vr.|t || | ||d|jt||j|7_||q|t!|dkr|}|jt|rt||||| | |dd nt"|| | |d |}|||d 9}|j|7_qtt|||| | |d t || | ||dt|||| | |d |_#|j|7_||_$|d kr\tt%|tt&dt't| ||dt(|_)n|dkr|dkstJtt%|tt*||||||_)n~|dkrtt+|jdt,t+d|j|_)nJ|dkr tt+|jdt%dtt+d|j|_)nt-d|ddS)Nr.rQrrrIrrTrrrrr)rrr*riZ spatial_v2z Unexpected z pooling).rr rrrLrrrkrrrlr#float16float32r7r(rrr!ror rprrrDr rrrrrdrrrrarpoolr AdaptiveAvgPool2drFlattenrrrReLUNotImplementedError)r+ image_sizerrrLrrrkrrrNrlrr(rrrmrrrargsrrrrrrr_rrr,rrr s                  zEncoderUNetModel.__init__cCs|jt|jtdSr)rrrrrrrrrs z EncoderUNetModel.convert_to_fp16cCs|jt|jtdSr)rrrrrrrrrs z EncoderUNetModel.convert_to_fp32cCs|t||j}g}|}|jD]4}|||}|jdr |||jj ddq | ||}|jdr|||jj ddt j |dd}| |S||j}| |SdS)z Apply the model to an input batch. :param x: an [N x C x ...] Tensor of inputs. :param timesteps: a 1-D batch of timesteps. :return: an [N x K] Tensor of outputs. r)rrr1r.)axisN)rr rrr startswithrrr7r5rr#r4r)r+rrrCresultsrrrrrr:s       zEncoderUNetModel.forward) rrTrFFrr.r.FFFrrrrr,rrs& .r__main__cseZdZdfdd ZZS)Dummyr@c s.ttttd||dddg|_dS)NrrrrI)rr r!rrDr r)r+rrr,rrr s zDummy.__init__)rr)r;r<r=r r@rrr,rrsrTrrQr)rrrQFr) rlr rrLrrrrrrrrr )rcuda)lowhighsizedevicezdone.)sn      $   '+4 " \