a Ad @sddlZddlZddlZddlmZddlmmZddl m m Z ddl m Z mZmZGdddejZddZddZGd d d ejZGd d d ejZGd ddejZGdddejZGdddejZGdddejZGdddejZGdddejZGdddejZGdddejZGdddejZe dkrd Z!d!Z"d"e!e"d#e"Z#d$e!e"d#e"Z$ed%e#e$fe"d&gd'd(gd'd%d)d* Z%e&e%e&e#e$e%'d+e(d#d,e#e$fZ)e%e)Z)e&e)j*dS)-N)DropPath to_2tuple trunc_normal_cs0eZdZddejdffdd ZddZZS)MlpNcsNt|p|}|p|}t|||_||_t|||_t||_dSN) super__init__nnLinearfc1actfc2Dropoutdrop)self in_featureshidden_features out_features act_layerr __class__V/var/www/html/stable-diffusion-webui/extensions-builtin/SwinIR/swinir_model_arch_v2.pyr s z Mlp.__init__cCs6||}||}||}||}||}|Sr)r r rr)rxrrrforwards      z Mlp.forward)__name__ __module__ __qualname__r GELUr r __classcell__rrrrrs rcCsR|j\}}}}|||||||||}|ddddddd|||}|S)z Args: x: (B, H, W, C) window_size (int): window size Returns: windows: (num_windows*B, window_size, window_size, C) r)shapeviewpermute contiguous)r window_sizeBHWCwindowsrrrwindow_partition"s$r1cCsbt|jd||||}||||||||d}|dddddd|||d}|S)z Args: windows: (num_windows*B, window_size, window_size, C) window_size (int): Window size H (int): Height of image W (int): Width of image Returns: x: (B, H, W, C) rr&r!r"r#r$r%)intr'r(r)r*)r0r+r-r.r,rrrrwindow_reverse0s $r3csBeZdZdZdfdd Zddd Zed d d Zd dZZ S)WindowAttentiona Window based multi-head self attention (W-MSA) module with relative position bias. It supports both of shifted and non-shifted window. Args: dim (int): Number of input channels. window_size (tuple[int]): The height and width of the window. num_heads (int): Number of attention heads. qkv_bias (bool, optional): If True, add a learnable bias to query, key, value. Default: True attn_drop (float, optional): Dropout ratio of attention weight. Default: 0.0 proj_drop (float, optional): Dropout ratio of output. Default: 0.0 pretrained_window_size (tuple[int]): The height and width of the window in pre-training. Trrrc s~t||_||_||_||_tjt dt |ddfdd|_ t tj ddddtjddtj d|d d|_tj|jd d |jd tjd }tj|jdd |jdtjd } tt|| gddd d } |d d krP| ddddddd f|d d<| dddddddf|dd<n\| ddddddd f|jd d<| dddddddf|jdd<| d 9} t| tt| d td } |d| t|jd } t|jd} tt| | g} t| d}|dddddf|dddddf}|ddd }|ddddd f|jd d7<|dddddf|jdd7<|ddddd fd|jdd9<|d}|d|tj ||dd d|_|r:tt||_ tt||_!n d|_ d|_!t"||_#t |||_$t"||_%tj&dd|_'dS)N r!T requires_gradr#ibiasinplaceFr)dtype?relative_coords_tabler&relative_position_indexr"dim)(rr rCr+pretrained_window_size num_headsr Parametertorchlogones logit_scale Sequentialr ReLUcpb_mlparangefloat32stackmeshgridr)r* unsqueezesignlog2absnpregister_bufferflattensumqkvzerosq_biasv_biasr attn_dropproj proj_dropSoftmaxsoftmax)rrCr+rEqkv_biasr^r`rDrelative_coords_hrelative_coords_wr@coords_hcoords_wcoordscoords_flattenrelative_coordsrArrrr Lsl & $$,...    ,((,    zWindowAttention.__init__NcCs|j\}}}d}|jdur:t|jtj|jdd|jf}tj||jj |d}| ||d|j d dddd d }|d|d |d}} } tj |dd tj | dd d d} tj|jttd |jjd} | | } ||jd|j } | |jd|jd|jd |jd|jd d}| ddd }dt|}| |d} |dur|jd}| ||||j |||d d} | d|j ||} || } n || } || } | | d d |||}| |}|!|}|S)z Args: x: input features with shape of (num_windows*B, N, C) mask: (0/-inf) mask with shape of (num_windows, Wh*Ww, Wh*Ww) or None NFr7)inputweightr:r"r&r#rr!r$rBgY@)max)"r'r\rGcat zeros_liker]FlinearrZrlreshaperEr) normalize transposeclamprJrHtensortodeviceexprMr@r(rAr+r*sigmoidrRrbr^r_r`)rrmaskB_Nr/rcrZqkvattnrJrelative_position_bias_tablerelative_position_biasnWrrrrs8   "$*&  (     zWindowAttention.forwardreturncCs$d|jd|jd|jd|jS)Ndim=, window_size=z, pretrained_window_size= , num_heads=)rCr+rDrErrrr extra_reprs zWindowAttention.extra_reprcCspd}|||jd|j7}||j||j|j|7}||j|||j|j7}|||j|j7}|S)Nrr")rCrE)rrflopsrrrrs zWindowAttention.flops)Trrr5)N) rrr__doc__r rstrrrr rrrrr4?s ; )r4c s`eZdZdZdddddddejejdf fdd Zd d Zd d Z e d ddZ ddZ Z S)SwinTransformerBlockaw Swin Transformer Block. Args: dim (int): Number of input channels. input_resolution (tuple[int]): Input resulotion. num_heads (int): Number of attention heads. window_size (int): Window size. shift_size (int): Shift size for SW-MSA. mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. qkv_bias (bool, optional): If True, add a learnable bias to query, key, value. Default: True drop (float, optional): Dropout rate. Default: 0.0 attn_drop (float, optional): Attention dropout rate. Default: 0.0 drop_path (float, optional): Stochastic depth rate. Default: 0.0 act_layer (nn.Module, optional): Activation layer. Default: nn.GELU norm_layer (nn.Module, optional): Normalization layer. Default: nn.LayerNorm pretrained_window_size (int): Window size in pre-training. r@Trc s t||_||_||_||_||_||_t|j|jkrPd|_t|j|_d|jkrh|jksrnJd| ||_ t |t |j||| |t | d|_ | dkrt | nt|_| ||_t||}t||| |d|_|jdkr||j}nd}|d|dS)Nrz shift_size must in 0-window_size)r+rErcr^r`rDr)rrrr attn_mask)rr rCinput_resolutionrEr+ shift_size mlp_ratiominnorm1r4rrrr Identity drop_pathnorm2r2rmlpcalculate_maskrW)rrCrrEr+rrrcrr^rr norm_layerrDmlp_hidden_dimrrrrr s2  "     zSwinTransformerBlock.__init__c Cs|\}}td||df}td|j t|j |j t|j df}td|j t|j |j t|j df}d}|D].}|D]$} ||dd|| ddf<|d7}qqzt||j} | d|j|j} | d| d} | | dkt d| dkt d} | S)Nr!rr&r#gYr) rGr[slicer+rr1r(rR masked_fillfloat) rx_sizer-r.img_maskh_slicesw_slicescnthw mask_windowsrrrrrs(      $z#SwinTransformerBlock.calculate_maskc Cs>|\}}|j\}}}|}|||||}|jdkrPtj||j |j fdd} n|} t| |j} | d|j|j|} |j|kr|j| |j d} n|j| | | |j d} | d|j|j|} t | |j||} |jdkrtj| |j|jfdd}n| }|||||}||||}|||||}|S)Nr)r!r#)shiftsdimsr&)r})r'r(rrGrollr1r+rrrrryrzr3rrrr) rrrr-r.r,Lr/shortcut shifted_x x_windows attn_windowsrrrr s*     zSwinTransformerBlock.forwardrc Cs4d|jd|jd|jd|jd|jd|j S)Nr, input_resolution=rrz , shift_size=z , mlp_ratio=)rCrrEr+rrrrrrr5szSwinTransformerBlock.extra_reprcCsd}|j\}}||j||7}|||j|j}|||j|j|j7}|d|||j|j|j7}||j||7}|S)Nrr#)rrCr+rrr)rrr-r.rrrrr9s "zSwinTransformerBlock.flops)rrrrr r LayerNormr rrrrrr rrrrrs  ")rcsDeZdZdZejffdd ZddZedddZ d d Z Z S) PatchMergingz Patch Merging Layer. Args: input_resolution (tuple[int]): Resolution of input feature. dim (int): Number of input channels. norm_layer (nn.Module, optional): Normalization layer. Default: nn.LayerNorm csBt||_||_tjd|d|dd|_|d||_dS)Nr$r#Fr9)rr rrCr r reductionnorm)rrrCrrrrr Os  zPatchMerging.__init__c Cs6|j\}}|j\}}}|||ks*Jd|ddkrB|ddksXJd|d|d|||||}|ddddddddddf}|ddddddddddf}|ddddddddddf} |ddddddddddf} t||| | gd }||d d |}||}||}|S) z x: B, H*W, C zinput feature has wrong sizer#rzx size (*z) are not even.Nr!r&r$)rr'r(rGrprr) rrr-r.r,rr/x0x1x2x3rrrrVs  .$$$$  zPatchMerging.forwardrcCsd|jd|jS)Nzinput_resolution=z, dim=rrCrrrrrmszPatchMerging.extra_reprcCsH|j\}}|d|dd|jd|j}||||jd7}|S)Nr#r$rrr-r.rrrrrps $zPatchMerging.flops) rrrrr rr rrrrr rrrrrGs rc s\eZdZdZdddddejdddf fdd Zd d Zed d dZ ddZ ddZ Z S) BasicLayera A basic Swin Transformer layer for one stage. Args: dim (int): Number of input channels. input_resolution (tuple[int]): Input resolution. depth (int): Number of blocks. num_heads (int): Number of attention heads. window_size (int): Local window size. mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. qkv_bias (bool, optional): If True, add a learnable bias to query, key, value. Default: True drop (float, optional): Dropout rate. Default: 0.0 attn_drop (float, optional): Attention dropout rate. Default: 0.0 drop_path (float | tuple[float], optional): Stochastic depth rate. Default: 0.0 norm_layer (nn.Module, optional): Normalization layer. Default: nn.LayerNorm downsample (nn.Module | None, optional): Downsample layer at the end of the layer. Default: None use_checkpoint (bool): Whether to use checkpointing to save memory. Default: False. pretrained_window_size (int): Local window size in pre-training. rTrNFrc sxt|_|_||_| |_t f ddt|D|_ | durn| d|_ nd|_ dS)NcsRg|]J}t |ddkr"dn d ttrB|nd qS)r#r) rCrrEr+rrrcrr^rrrD)r isinstancelist).0i r^rCrrrrrrErDrcr+rr s z'BasicLayer.__init__..)rCr) rr rCrdepthuse_checkpointr ModuleListrangeblocks downsample)rrCrrrEr+rrcrr^rrrrrDrrrr s "  zBasicLayer.__init__cCsD|jD]$}|jr t|||}q|||}q|jdur@||}|Sr)rr checkpointr)rrrblkrrrrs    zBasicLayer.forwardrcCsd|jd|jd|jS)Nrrz, depth=)rCrrrrrrrszBasicLayer.extra_reprcCs8d}|jD]}||7}q |jdur4||j7}|SNr)rrr)rrrrrrrs   zBasicLayer.flopscCsX|jD]L}tj|jjdtj|jjdtj|jjdtj|jjdqdSr)rr init constant_rr:rlr)rrrrr_init_respostnorms  zBasicLayer._init_respostnorm) rrrrr rr rrrrrr rrrrrvs  rcs2eZdZdZd fdd Zd d Zd d ZZS) PatchEmbedaw Image to Patch Embedding Args: img_size (int): Image size. Default: 224. patch_size (int): Patch token size. Default: 4. in_chans (int): Number of input image channels. Default: 3. embed_dim (int): Number of linear projection output channels. Default: 96. norm_layer (nn.Module, optional): Normalization layer. Default: None r$r"`Ncstt|}t|}|d|d|d|dg}||_||_||_|d|d|_||_||_t j ||||d|_ |dur|||_ nd|_ dS)Nrr!) kernel_sizestride) rr rimg_size patch_sizepatches_resolution num_patchesin_chans embed_dimr Conv2dr_rrrrrrrrrrrr s   zPatchEmbed.__init__cCs>|j\}}}}||ddd}|jdur:||}|S)Nr#r!)r'r_rXrvr)rrr,r/r-r.rrrrs   zPatchEmbed.forwardcCsR|j\}}|||j|j|jd|jd}|jdurN||||j7}|SNrr!)rrrrr)rZHoZWorrrrrs  ( zPatchEmbed.flops)rr$r"rNrrrrr rrr rrrrrs  rc sJeZdZdZdddddejddddd f fd d Zd d ZddZZ S)RSTBaXResidual Swin Transformer Block (RSTB). Args: dim (int): Number of input channels. input_resolution (tuple[int]): Input resolution. depth (int): Number of blocks. num_heads (int): Number of attention heads. window_size (int): Local window size. mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. qkv_bias (bool, optional): If True, add a learnable bias to query, key, value. Default: True drop (float, optional): Dropout rate. Default: 0.0 attn_drop (float, optional): Attention dropout rate. Default: 0.0 drop_path (float | tuple[float], optional): Stochastic depth rate. Default: 0.0 norm_layer (nn.Module, optional): Normalization layer. Default: nn.LayerNorm downsample (nn.Module | None, optional): Downsample layer at the end of the layer. Default: None use_checkpoint (bool): Whether to use checkpointing to save memory. Default: False. img_size: Input image size. patch_size: Patch size. resi_connection: The convolutional block before residual connection. rTrNFrr$1convcstt|||_||_t||||||||| | | | | d |_|dkr\t||ddd|_ nj|dkrt t||ddddtj ddd t|d|dddd tj ddd t|d|ddd|_ t ||||dd |_ t||||dd |_dS) N) rCrrrEr+rrcrr^rrrrrr"r!3convr$皙?Tnegative_sloper<rrrrrr)rrr rCrrresidual_groupr rconvrK LeakyReLUr patch_embed PatchUnEmbed patch_unembed)rrCrrrEr+rrcrr^rrrrrrresi_connectionrrrr s@ $ z RSTB.__init__c Cs$||||||||Sr)rrrr)rrrrrrr.sz RSTB.forwardcCsXd}||j7}|j\}}||||j|jd7}||j7}||j7}|S)Nr )rrrrCrr)rrr-r.rrrr1s z RSTB.flops) rrrrr rr rrr rrrrrs 'rcs2eZdZdZd fdd Zd d Zd d ZZS)raz Image to Patch Unembedding Args: img_size (int): Image size. Default: 224. patch_size (int): Patch token size. Default: 4. in_chans (int): Number of input image channels. Default: 3. embed_dim (int): Number of linear projection output channels. Default: 96. norm_layer (nn.Module, optional): Normalization layer. Default: None rr$r"rNcsntt|}t|}|d|d|d|dg}||_||_||_|d|d|_||_||_dSr) rr rrrrrrrrrrrr Fs  zPatchUnEmbed.__init__cCs2|j\}}}|dd||j|d|d}|S)Nr!r#r)r'rvr(r)rrrr,HWr/rrrrSs "zPatchUnEmbed.forwardcCsd}|Srr)rrrrrrXszPatchUnEmbed.flops)rr$r"rNrrrrrr;s  rcs eZdZdZfddZZS)UpsampleUpsample module. Args: scale (int): Scale factor. Supported scales: 2^n and 3. num_feat (int): Channel number of intermediate features. c sg}||d@dkr\ttt|dD]0}|t|d|ddd|tdq(nF|dkr|t|d|ddd|tdntd|dt t |j |dS Nr!rr#r$r"rzscale z/ is not supported. Supported scales: 2^n and 3.) rr2mathrHappendr r PixelShuffle ValueErrorrrr rscalenum_featm_rrrr eszUpsample.__init__rrrrr r rrrrr]srcs eZdZdZfddZZS) Upsample_hfrc sg}||d@dkr\ttt|dD]0}|t|d|ddd|tdq(nF|dkr|t|d|ddd|tdntd|dt t |j |dSr) rr2rrHrr rrrrrr rrrrr zszUpsample_hf.__init__rrrrrrrsrcs*eZdZdZdfdd ZddZZS)UpsampleOneStepa*UpsampleOneStep module (the difference with Upsample is that it always only has 1conv + 1pixelshuffle) Used in lightweight SR to save parameters. Args: scale (int): Scale factor. Supported scales: 2^n and 3. num_feat (int): Channel number of intermediate features. Nc sT||_||_g}|t||d|ddd|t|tt|j|dS)Nr#r"r!) rrrr rrrrr )rrr num_out_chrrrrrr s  zUpsampleOneStep.__init__cCs$|j\}}|||jdd}|S)Nr"r)rrrrrrrs zUpsampleOneStep.flops)N)rrrrr rr rrrrrs rcseZdZdZddddddddd d d d ejd d d d dddffdd ZddZej j ddZ ej j ddZ ddZ ddZddZdd Zd!d"ZZS)#Swin2SRa Swin2SR A PyTorch impl of : `Swin2SR: SwinV2 Transformer for Compressed Image Super-Resolution and Restoration`. Args: img_size (int | tuple(int)): Input image size. Default 64 patch_size (int | tuple(int)): Patch size. Default: 1 in_chans (int): Number of input image channels. Default: 3 embed_dim (int): Patch embedding dimension. Default: 96 depths (tuple(int)): Depth of each Swin Transformer layer. num_heads (tuple(int)): Number of attention heads in different layers. window_size (int): Window size. Default: 7 mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. Default: 4 qkv_bias (bool): If True, add a learnable bias to query, key, value. Default: True drop_rate (float): Dropout rate. Default: 0 attn_drop_rate (float): Attention dropout rate. Default: 0 drop_path_rate (float): Stochastic depth rate. Default: 0.1 norm_layer (nn.Module): Normalization layer. Default: nn.LayerNorm. ape (bool): If True, add absolute position embedding to the patch embedding. Default: False patch_norm (bool): If True, add normalization after patch embedding. Default: True use_checkpoint (bool): Whether to use checkpointing to save memory. Default: False upscale: Upscale factor. 2/3/4/8 for image SR, 1 for denoising and compress artifact reduction img_range: Image range. 1. or 255. upsampler: The reconstruction reconstruction module. 'pixelshuffle'/'pixelshuffledirect'/'nearest+conv'/None resi_connection: The convolutional block before residual connection. '1conv'/'3conv' @r!r"rrrrrrTrg?Fr#r?rc stt||}|}d}||_|dkrFd}t|dddd|_ntdddd|_||_ ||_ ||_ t ||ddd|_t||_||_||_||_||_||_t|||||jr| ndd|_|jj}|jj}||_t|||||jr| ndd|_|jr$t td|||_t|jddt j| d|_ d d t!d | t"|D}t #|_$t%|jD]t}t&||d |df||||||j| | | |t"|d|t"|d|d| d||||d }|j$'|qb|j d krnt #|_(t%|jD]t}t&||d |df||||||j| | | |t"|d|t"|d|d| d||||d }|j('|q| |j|_)|dkrt ||ddd|_*nl|dkrt +t ||ddddt j,dddt |d|dddd t j,dddt |d|ddd|_*|j dkrZt +t ||dddt j,dd|_-t.|||_/t ||ddd|_0nR|j dkrt ||ddd|_1t +t ||dddt j,dd|_-t ||ddd|_2t +t d|dddt j,dd|_3t.|||_/t ||ddd|_0n|j d krt +t ||dddt j,dd|_-t.|||_/t4|||_5t ||ddd|_0t +t ||dddt j,dd|_6t ||ddd|_7t +t ||dddt j,dd|_8t ||ddd|_9n|j dkrt:||||d |df|_/n|j dkr|j dksJdt +t ||dddt j,dd|_-t ||ddd|_;t ||ddd|_nt ||ddd|_0|?|j@dS)Nrr")gw#?g8EGr?gB`"?r!r{Gz?std)pcSsg|] }|qSr)item)rrrrrrz$Swin2SR.__init__..r)rCrrrEr+rrcrr^rrrrrrrpixelshuffle_hfrrr$rTr pixelshuffler;pixelshuffle_auxpixelshuffledirect nearest+convzonly support x4 now.)Arrr img_rangerGTensorr(meanr[upscale upsamplerr+r r conv_firstlen num_layersrape patch_norm num_featuresrrrrrrrrFabsolute_pos_embedrrpos_droplinspacerYrlayersrrr layers_hfrconv_after_bodyrKrconv_before_upsamplerupsample conv_last conv_bicubicconv_auxconv_after_auxr upsample_hf conv_first_hfconv_after_body_hfconv_before_upsample_hf conv_last_hfrconv_up1conv_up2conv_hrlreluapply _init_weights)rrrrrdepthsrEr+rrc drop_rateattn_drop_ratedrop_path_raterrrrrrrrkwargs num_in_chrrrgb_meanrrdpri_layerlayerrrrr s    &  &                     zSwin2SR.__init__cCsrt|tjrBt|jddt|tjrn|jdurntj|jdn,t|tjrntj|jdtj|jddS)Nrrrr?) rr r rrlr:rrr)rrrrrr3fs  zSwin2SR._init_weightscCsdhS)Nrrrrrrno_weight_decayoszSwin2SR.no_weight_decaycCsdhS)Nrrrrrrno_weight_decay_keywordsssz Swin2SR.no_weight_decay_keywordscCsV|\}}}}|j||j|j}|j||j|j}t|d|d|fd}|S)Nrreflect)sizer+rrpad)rrrrr mod_pad_h mod_pad_wrrrcheck_image_sizews zSwin2SR.check_image_sizecCsh|jd|jdf}||}|jr.||j}||}|jD]}|||}q>||}|||}|SNr#r")r'rrrrr rrrrrr=rrrforward_features~s       zSwin2SR.forward_featurescCsh|jd|jdf}||}|jr.||j}||}|jD]}|||}q>||}|||}|SrF)r'rrrrr!rrrGrrrforward_features_hfs       zSwin2SR.forward_features_hfc Cs|jdd\}}||}|j||_||j|j}|jdkr||}||||}| |}| | |}nt|jdkrpt j |||j||jfddd}||}||}||||}| |}||}||}| |ddddd||jd||jf|ddddd||jd||jf}| |}||j|j}n|jdkr||}||||}| |}| | |}||}||||}||}|||}||}||j|j}n|jdkr<||}||||}| |}n|jd kr||}||||}| |}||tjjj |dd d }||tjjj |dd d }| |||}n,||} ||| | } || | }||j|j}|jdkr@|ddddd||jd||jf|fS|jdkr||j|j}|ddddd||jd||jf|ddddd||jd||jf|ddddd||jd||jffS|ddddd||jd||jfSdS) Nr#rrbicubicF)rAmode align_cornersr rrnearest) scale_factorrK)r'rErtype_asrrrr"rHr#r%r$rr interpolaterr&r'r(r*r+rIr,r-r)r1r.rGr functionalr/r0) rrr-r.rJauxZx_beforex_outZx_hfx_firstresrrrrsh     "     ^                0 zSwin2SR.forwardcCs|d}|j\}}|||d|jd7}||j7}|jD]}||7}q<|||d|j|j7}||j7}|S)Nrr"r)rrrrr r$)rrr-r.r=rrrrs  z Swin2SR.flops)rrrrr rr r3rGjitignorer>r?rErHrIrrr rrrrrs& +   Dr__main__r$r>ir!ir#r?r<r) rrr+rr4rrErrgeAr")+rnumpyrVrGtorch.nnr Ztorch.nn.functionalrQrrtorch.utils.checkpointutilsrtimm.models.layersrrrModulerr1r3r4rrrrrrrKrrrrrrr+heightwidthmodelprintrrandnrr'rrrrsL /M.J"M