a di<@s:UddlmZmZmZddlZddlmZddlmmZ ddlm Z ddl m Z ddl mZddlmZddlmZmZmZiZeeefed <d ed <dd d ddddddddddddddddddddddddd dddd!d d"d#Zeed$<Gd%d&d&ejZGd'd(d(ejZe ee d)d*d+Zd,d-ZdS).)DictOptionalTupleN)Tensor)normalize_pixel_coordinates)KORNIA_CHECK_SHAPE)SOLD2Net)LineSegmentDetectionModuleline_map_to_segmentsprob_to_junctionsurlsz>> images = torch.rand(2, 1, 512, 512) >>> sold2 = SOLD2() >>> outputs = sold2(images) >>> line_seg1 = outputs["line_segments"][0] >>> line_seg2 = outputs["line_segments"][1] >>> desc1 = outputs["dense_desc"][0] >>> desc2 = outputs["dense_desc"][1] >>> matches = sold2.match(line_seg1, line_seg2, desc1[None], desc2[None]) TN) pretrainedconfigcst|durtn||_|jd|_|jdd|_|jdd|_t|j|_ |rt j j t dddd }||d }|j |||jd |_tfi|jd |_tfi|jd |_dS) Nr r!g ?r"rrcSs|SN)Zstoragelocr*r*c/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/kornia/feature/sold2/sold2.pyZz SOLD2.__init__..)Z map_locationZmodel_state_dictr#r$)super__init__r%r(r getjunc_detect_threshr"r modeltorchZhubZload_state_dict_from_urlr adapt_state_dictZload_state_dictevalr#r line_detectorWunschLineMatcher line_matcher)selfr'r(Zpretrained_dict state_dict __class__r*r,r0Ns       zSOLD2.__init__)imgreturnc Cst|gdi}||}|d|d<|d|d<|d|d<g}t|d|dD]@\}}t||j|j|j}|j||\}}} | t ||qV||d<|S) a Args: img: batched images with shape :math:`(B, 1, H, W)`. Return: - ``line_segments``: list of N line segments in each of the B images :math:`List[(N, 2, 2)]`. - ``junction_heatmap``: raw junction heatmap of shape :math:`(B, H, W)`. - ``line_heatmap``: raw line heatmap of shape :math:`(B, H, W)`. - ``dense_desc``: the semi-dense descriptor map of shape :math:`(B, 128, H/4, W/4)`. )B1HW junctionsZjunction_heatmapheatmapZ line_heatmapZ descriptorsZ dense_descZ line_segments) rr3zipr r r2r"r7detectappendr ) r:r>outputsZ net_outputslinesZ junc_probrErDZline_map_r*r*r,forwardgs     z SOLD2.forward line_seg1 line_seg2desc1desc2r?cCs|||||S)aFind the best matches between two sets of line segments and their corresponding descriptors. Args: line_seg1, line_seg2: list of line segments in two images, with shape [num_lines, 2, 2]. desc1, desc2: semi-dense descriptor maps of the images, with shape [1, 128, H/4, W/4]. Returns: A np.array of size [num_lines1] indicating the index in line_seg2 of the matched line, for each line in line_seg1. -1 means that the line is not matched. )r9)r:rNrOrPrQr*r*r,matchs z SOLD2.matchcCs:|d=|d=|d=|d|d<|d|d<|d=|d=|S)NZw_juncZ w_heatmapZw_descz'heatmap_decoder.conv_block_lst.2.weightz)heatmap_decoder.conv_block_lst.2.0.weightz%heatmap_decoder.conv_block_lst.2.biasz'heatmap_decoder.conv_block_lst.2.0.biasr*)r:r;r*r*r,r5s  zSOLD2.adapt_state_dict)TN) __name__ __module__ __qualname____doc__boolrrr0rrLrRr5 __classcell__r*r*r<r,r&4s ! r&csxeZdZdZdeeeeeedfdd Zeeeeed d d Zee d d dZ eedddZ eedddZ Z S)r8zClass matching two sets of line segments with the Needleman-Wunsch algorithm. TODO: move it later in kornia.feature.matching TrrF)rrrrr line_scorecs2t||_||_||_||_||_||_dSr))r/r0rrrrr rY)r:rrrrr rYr<r*r,r0s zWunschLineMatcher.__init__rMcCst|gdt|gdt|gdt|gd|j}|jd|j|jd|jf}|jd|j|jd|jf}t|dkrtjdtj|dSt|dkrtjt|tj|d S| |\}} | |\} } | dd}| dd} t ||} t | |} t j t j|| dd dd d d d dfdd }t j t j|| dd dd d d d dfdd }||}d|| <d|d d | f<| t||jt||j}|ddd d}||}|jr ||d ddd}||tjt||d k}d||<|S)z\Find the best matches between two sets of line segments and their corresponding descriptors.N2r\)r@DrBrBrrdtypedeviceF)Z align_cornersNdimr)ra)rrashaper lenr4emptyintonessample_line_pointsreshapekeypoints_to_gridF normalizeZ grid_sampletflattenrZpermutefilter_and_match_linesrarange)r:rNrOrPrQraZ img_size1Z img_size2Z line_points1Z valid_points1Z line_points2Z valid_points2Zgrid1Zgrid2scoresmatchesZmatches2Zmutualr*r*r,rLs<        ..   zWunschLineMatcher.forward)line_segr?c Cs~t|gdt|}tj|dddf|dddfdd}ttj||jddd|j}tj ||jdftj d }tj ||jftj d }t d|jdD]}||k}||} t | ddddf| ddddf|d d} t | ddddf| ddddf|d d} tj| | gd d} t| ddd|j|f} tjt| |jtj d } d | dd|df<| ||<| ||<q||fS) aERegularly sample points along each line segments, with a minimal distance between each point. Pad the remaining points. Inputs: line_seg: an Nx2x2 Tensor. Outputs: line_points: an N x num_samples x 2 Tensor. valid_points: a boolean N x num_samples Tensor. rZNrrrcfloor)Z rounding_moderr`rbF)rrfr4ZnormclampdivrrrhrgfloatrWrangebatched_linspacestackrmpadri)r:ruZ num_linesZ line_lengthsZnum_samples_lstZ line_pointsZ valid_pointsZn_sampZcur_maskZ cur_line_segZ line_points_xZ line_points_yZcur_line_pointsZcur_valid_pointsr*r*r,rjs( *,, z$WunschLineMatcher.sample_line_points)rsr?cCsBt|gd|dd}|dk}||d|d}|dd}|dk}||d|d}||d}tj|dddd|j df}tj||ddddddfdd}tj|tj|dgd gdd}|j \} } } } | | | | | f}| |} | | | } t tj | dd| d}|t| |f}|S) a~Use the scores to keep the top k best lines, compute the Needleman- Wunsch algorithm on each candidate pairs, and keep the highest score. Inputs: scores: a (N, M, n, n) Tensor containing the pairwise scores of the elements to match. Outputs: matches: a (N) Tensor containing the indices of the best match )Mr[nrr^rrbrrrcN)dims)rmaxsumr4ZargsortrZtake_along_dimcatZfliprerkneedleman_wunsch remainderZargmaxrr)r:rsZ line_scores1Z valid_scores1Z line_scores2Z valid_scores2Z line_scoresZ topk_linesZ top_scoresZn_lines1Ztop2krm nw_scoresrtr*r*r,rq s$  "$  z(WunschLineMatcher.filter_and_match_linesc Cst|gd|j\}}}d}||}tj||d|dtjd}t|D]}t|D]r} tt|dd|d| f|dd|| df|dd|| f|dd|| f|dd|d| df<qVqJ|ddddfS)a1Batched implementation of the Needleman-Wunsch algorithm. The cost of the InDel operation is set to 0 by subtracting the gap penalty to the scores. Inputs: scores: a (B, N, M) Tensor containing the pairwise scores of the elements to match. )r@r[rg?rrwNrb)rrer4zerosrzr{maximum) r:rsbrrZgaprZnw_gridijr*r*r,r2s    Pz"WunschLineMatcher.needleman_wunsch)TrrrrF)rSrTrUrVrWrhr0rrLrrjrqrrXr*r*r<r,r8s&0'%r8) keypointsimg_sizer?cCsNt|ddgt|}t|ddddgf|d|d}|d|dd}|S)uConvert a list of keypoints into a grid in [-1, 1]² that can be used in torch.nn.functional.interpolate. Args: keypoints: a tensor [N, 2] of N keypoints (ij coordinates convention). img_size: the original image size (H, W) r[r\Nrrrbr)rrfrview)rrZn_pointsZ grid_pointsr*r*r,rlMs $rlcCs^|||d|}dgt|j}|||<tj|tj|jd|}||||}|S)zsV     "k/