a  þdt?ã@s”ddlZddlmZddlmZddlmZddlmZm Z m Z Gdd„dej ƒZ Gd d „d ej ƒZ Gd d „d ej ƒZe ¡Gd d„dej ƒƒZdS)éN)Únn)Ú functional)Ú ARCH_REGISTRYé)Ú DCNv2PackÚResidualBlockNoBNÚ make_layercs*eZdZdZd‡fdd„ Zdd„Z‡ZS) Ú PCDAlignmentaVAlignment module using Pyramid, Cascading and Deformable convolution (PCD). It is used in EDVR. Ref: EDVR: Video Restoration with Enhanced Deformable Convolutional Networks Args: num_feat (int): Channel number of middle features. Default: 64. deformable_groups (int): Deformable groups. Defaults: 8. é@écsptt|ƒ ¡t ¡|_t ¡|_t ¡|_t ¡|_t ¡|_ t dddƒD]¼}d|›}t  |d|ddd¡|j|<|dkr˜t  ||ddd¡|j|<n4t  |d|ddd¡|j|<t  ||ddd¡|j|<t ||dd|d|j|<|dkrLt  |d|ddd¡|j |<qLt  |d|ddd¡|_ t  ||ddd¡|_t ||dd|d|_tjddd d |_tjd d d |_dS)NéréÿÿÿÿÚlér)ÚpaddingÚdeformable_groupsÚbilinearF©Z scale_factorÚmodeZ align_cornersçš™™™™™¹?T©Znegative_slopeZinplace)Úsuperr Ú__init__rZ ModuleDictÚ offset_conv1Ú offset_conv2Ú offset_conv3Údcn_packÚ feat_convÚrangeÚConv2drÚcas_offset_conv1Úcas_offset_conv2Ú cas_dcnpackÚUpsampleÚupsampleÚ LeakyReLUÚlrelu)ÚselfÚnum_featrÚiÚlevel©Ú __class__©ú`/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/basicsr/archs/edvr_arch.pyrs*      zPCDAlignment.__init__c Csfd\}}tdddƒD]}d|›}tj||d||dgdd}| |j||ƒ¡}|dkrx| |j||ƒ¡}n6| |j|tj||gddƒ¡}| |j||ƒ¡}|j|||d|ƒ}|dkrê|j|tj||gddƒ}|dkrü| |¡}|dkr|  |¡d}|  |¡}qtj||dgdd}| |  | |  |¡¡¡¡}| |  ||¡¡}|S) aëAlign neighboring frame features to the reference frame features. Args: nbr_feat_l (list[Tensor]): Neighboring feature list. It contains three pyramid levels (L1, L2, L3), each with shape (b, c, h, w). ref_feat_l (list[Tensor]): Reference feature list. It contains three pyramid levels (L1, L2, L3), each with shape (b, c, h, w). Returns: Tensor: Aligned features. )NNr rr rr©Zdimr) rÚtorchÚcatr&rrrrrr$r!r r") r'Ú nbr_feat_lÚ ref_feat_lZupsampled_offsetZupsampled_featr)r*ÚoffsetÚfeatr-r-r.Úforward8s* ""  zPCDAlignment.forward)r r ©Ú__name__Ú __module__Ú __qualname__Ú__doc__rr6Ú __classcell__r-r-r+r.r s #r cs*eZdZdZd ‡fdd„ Zdd„Z‡ZS) Ú TSAFusionaTemporal Spatial Attention (TSA) fusion module. Temporal: Calculate the correlation between center frame and neighboring frames; Spatial: It has 3 pyramid levels, the attention is similar to SFT. (SFT: Recovering realistic texture in image super-resolution by deep spatial feature transform.) Args: num_feat (int): Channel number of middle features. Default: 64. num_frame (int): Number of frames. Default: 5. center_frame_idx (int): The index of center frame. Default: 2. r ércsXtt|ƒ ¡||_t ||ddd¡|_t ||ddd¡|_t |||dd¡|_tj dddd|_ tj dddd|_ t |||d¡|_ t |d|d¡|_t ||ddd¡|_t ||d¡|_t ||ddd¡|_t ||d¡|_t |d|ddd¡|_t ||ddd¡|_t ||d¡|_t ||d¡|_tjddd|_tjddd d |_dS) Nr rr)ZstriderrTrrFr)rr=rÚcenter_frame_idxrrÚtemporal_attn1Útemporal_attn2Ú feat_fusionZ MaxPool2dÚmax_poolZ AvgPool2dÚavg_poolÚ spatial_attn1Ú spatial_attn2Ú spatial_attn3Ú spatial_attn4Ú spatial_attn5Úspatial_attn_l1Úspatial_attn_l2Úspatial_attn_l3Úspatial_attn_add1Úspatial_attn_add2r%r&r#r$)r'r(Ú num_framer?r+r-r.rts&zTSAFusion.__init__c Cs*| ¡\}}}}}| |dd…|jdd…dd…dd…f ¡¡}| | d|||¡¡}| ||d||¡}g} t|ƒD]F} |dd…| dd…dd…dd…f} t | |d¡} |   |   d¡¡qtt  tj | dd¡} |   d¡  |||||¡} |  ¡ |d||¡} | |d||¡| }| | |¡¡}| | |¡¡}| |¡}| |¡}| | tj ||gdd¡¡}| | |¡¡}| |¡}| |¡}| | tj ||gdd¡¡}| | |¡¡}| |¡}| | |¡¡|}| | |¡¡}| |¡}| |¡}| | | |¡¡¡}t  |¡}||d|}|S)z½ Args: aligned_feat (Tensor): Aligned features with shape (b, t, c, h, w). Returns: Tensor: Features after TSA with the shape (b, c, h, w). Nr rr/r)Úsizer@r?ÚclonerAÚviewrr0ÚsumÚappendZ unsqueezeZsigmoidr1ÚexpandÚ contiguousr&rBrErCrDrFrJrKrLr$rGrHrIrNrM)r'Ú aligned_featÚbÚtÚcÚhÚwZ embedding_refZ embeddingZcorr_lr)Z emb_neighborZcorrZ corr_probr5ZattnZattn_maxZattn_avgZ attn_levelZattn_addr-r-r.r6s@. "        zTSAFusion.forward)r r>rr7r-r-r+r.r=esr=cs*eZdZdZd ‡fdd„ Zdd„Z‡ZS) ÚPredeblurModuleaPre-dublur module. Args: num_in_ch (int): Channel number of input image. Default: 3. num_feat (int): Channel number of intermediate features. Default: 64. hr_in (bool): Whether the input has high resolution. Default: False. r r Fcsætt|ƒ ¡||_t |ˆddd¡|_|jrVt ˆˆddd¡|_t ˆˆddd¡|_t ˆˆddd¡|_ t ˆˆddd¡|_ t ˆd|_ t ˆd|_ t ˆd|_t ‡fdd„tdƒDƒ¡|_tjddd d |_tjd d d |_dS)Nr rr©r(csg|]}tˆd‘qS)r^)r)Ú.0r)r^r-r.Ú Ûóz,PredeblurModule.__init__..r>rFrrTr)rr]rÚhr_inrrÚ conv_firstÚstride_conv_hr1Ústride_conv_hr2Ústride_conv_l2Ústride_conv_l3rÚ resblock_l3Ú resblock_l2_1Ú resblock_l2_2Z ModuleListrÚ resblock_l1r#r$r%r&)r'Ú num_in_chr(rbr+r^r.rÊs   zPredeblurModule.__init__cCsÊ| | |¡¡}|jr6| | |¡¡}| | |¡¡}| | |¡¡}| | |¡¡}| | |¡¡}|  |¡|}| |  |¡¡}t dƒD]}|j ||ƒ}qŒ||}t ddƒD]}|j ||ƒ}q²|S)Nrr>) r&rcrbrdrerfrgr$rhrirjrrk)r'ÚxÚfeat_l1Úfeat_l2Úfeat_l3r)r-r-r.r6às zPredeblurModule.forward)r r Fr7r-r-r+r.r]Ásr]c s*eZdZdZd‡fd d „ Zd d „Z‡ZS)ÚEDVRaIEDVR network structure for video super-resolution. Now only support X4 upsampling factor. Paper: EDVR: Video Restoration with Enhanced Deformable Convolutional Networks Args: num_in_ch (int): Channel number of input image. Default: 3. num_out_ch (int): Channel number of output image. Default: 3. num_feat (int): Channel number of intermediate features. Default: 64. num_frame (int): Number of input frames. Default: 5. deformable_groups (int): Deformable groups. Defaults: 8. num_extract_block (int): Number of blocks for feature extraction. Default: 5. num_reconstruct_block (int): Number of blocks for reconstruction. Default: 10. center_frame_idx (int): The index of center frame. Frame counting from 0. Default: Middle of input frames. hr_in (bool): Whether the input has high resolution. Default: False. with_predeblur (bool): Whether has predeblur module. Default: False. with_tsa (bool): Whether has TSA module. Default: True. r r r>r é NFTc sœtt|ƒ ¡|dur"|d|_n||_| |_| |_| |_|jrdt||jd|_t   ||dd¡|_ nt   ||ddd¡|_ t t||d|_t   ||ddd¡|_t   ||ddd¡|_t   ||ddd¡|_t   ||ddd¡|_t||d|_|jrt|||jd|_nt   |||dd¡|_t t||d|_t   ||dddd¡|_t   |d ddd¡|_t  d¡|_t   d d ddd¡|_t   d dddd¡|_t jd d d |_ dS)Nr)r(rbrr r^)r(r)r(rOr?éér rTr)!rrqrr?rbÚwith_predeblurÚwith_tsar]Ú predeblurrrÚconv_1x1rcrrÚfeature_extractionÚ conv_l2_1Ú conv_l2_2Ú conv_l3_1Ú conv_l3_2r Ú pcd_alignr=ÚfusionÚreconstructionÚupconv1Úupconv2Z PixelShuffleÚ pixel_shuffleÚconv_hrÚ conv_lastr%r&) r'rlZ num_out_chr(rOrZnum_extract_blockZnum_reconstruct_blockr?rbrurvr+r-r.rs6   z EDVR.__init__c Cs"| ¡\}}}}}|jr:|ddkr0|ddksZJdƒ‚n |ddkrR|ddksZJdƒ‚|dd…|jdd…dd…dd…f ¡}|jr¾| | | d|||¡¡¡}|jrÚ|d|d}}n| |  | d|||¡¡¡}|  |¡}| |  |¡¡} | |  | ¡¡} | |  | ¡¡} | | | ¡¡} | ||d||¡}|  ||d|d|d¡} |  ||d|d|d¡} |dd…|jdd…dd…dd…f ¡| dd…|jdd…dd…dd…f ¡| dd…|jdd…dd…dd…f ¡g} g} t|ƒD]ˆ} |dd…| dd…dd…dd…f ¡| dd…| dd…dd…dd…f ¡| dd…| dd…dd…dd…f ¡g}|  | || ¡¡qìtj| dd } |jsœ|  |d||¡} | | ¡}| |¡}| | | |¡¡¡}| | | |¡¡¡}| | |¡¡}| |¡}|jr|}ntj|dd d d }||7}|S) Nérz,The height and width must be multiple of 16.rsz+The height and width must be multiple of 4.r rrr/rFr)rPrbr?rVrurxrwrRr&rcryrzr{r|r}rQrrTr~r0Ústackrvrr€rƒrr‚r„r…ÚFZ interpolate)r'rmrXrYrZr[r\Zx_centerrnrorpr3rWr)r2r5ÚoutÚbaser-r-r.r6FsP" ( L&þ lÿ   z EDVR.forward) r r r r>r r>rrNFFTr7r-r-r+r.rqösõ6rq)r0rZtorch.nnrrˆZbasicsr.utils.registryrZ arch_utilrrrÚModuler r=r]Úregisterrqr-r-r-r.Ús   \\5