a  diB@sddlZddlmZmZddlmZmZmZmZm Z dZ dZ e dZ eGdddeeZGd d d Ze eeeeefe eeefeeeeeefd d d Zeeeee dddZeedfeedfdddZeedfeee e fee ee fdddZdS)N)Enumunique)DictListSequenceTupleUnioni'g7yACc@s$eZdZdZdZdZdZdZdZdS)_EditOperationsz1Enumerations for the Levenhstein edit operations.insertdelete substituteZnothingZ undefinedN) __name__ __module__ __qualname____doc__ OP_INSERT OP_DELETE OP_SUBSTITUTE OP_NOTHING OP_UNDEFINEDrrl/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/torchmetrics/functional/text/helper.pyr +s r c @sNeZdZdZdeeeeeddddZeeeeee dffdd d Z eeeeeeee feeeeeee fee dffd d d Z eeeeee fee dfdddZ eeeeeee fddddZ eeeeeeeee ffdddZeeeee fdddZeeeee fdddZdS)_LevenshteinEditDistancea-A convenience class for calculating the Levenshtein edit distance. Class will cache some intermediate values to hasten the calculation. The implementation follows the implemenation from https://github.com/mjpost/sacrebleu/blob/master/sacrebleu/metrics/lib_ter.py, where the most of this implementation is adapted and copied from. Args: reference_tokens: list of reference tokens op_insert: cost of insertion operation op_delete: cost of deletion operation op_substitute: cost of substitution operation N)reference_tokens op_insert op_delete op_substitutereturncCs>||_t||_i|_d|_||_||_||_d|_t |_ dS)Nr) rlen reference_lencache cache_sizerrr op_nothing _INT_INFINITY op_undefined)selfrrrrrrr__init__Es z!_LevenshteinEditDistance.__init__.)prediction_tokensrcCs6||\}}||||\}}}|||||fS)a(Calculate edit distance between self._words_ref and the hypothesis. Uses cache to skip some computations. Args: prediction_tokens: A tokenized predicted sentence. Return: A tuple of a calculated edit distance and a trace of executed operations. ) _find_cache_levenshtein_edit_distance _add_cache)r'r)start_positionZcached_edit_distanceZedit_distance_int edit_distancetracerrr__call__Ts   z!_LevenshteinEditDistance.__call__)r)prediction_startr"rcst|}fddt||D}||}|r8j|nd}|dtkrZt|dtnt}t|d|dD]T} t| |} td| |} | |krjdntjd| |} t| | D]} | dkr|| d| dj t j f|| | <q|| dj | dkr$j }t j}n j}t j}|| d| dd||f|| d| dj t j f|| | ddjt jff}|D]0\}}|| | d|kr||f|| | <qqqp||}|ddd|t|d|fS) aDynamic programming algorithm to compute the Levenhstein edit distance. Args: prediction_tokens: A tokenized predicted sentence. prediction_start: An index where a predicted sentence to be considered from. cache: A cached Levenshtein edit distance. Returns: Edit distance between the predicted sentence and the reference sentence csg|]}tjqSr)list_get_empty_rowr!.0_r'rr |szG_LevenshteinEditDistance._levenshtein_edit_distance..g?rrN)r ranger! _BEAM_WIDTHmathceilfloormaxminrr rrr$rrrrr _get_trace)r'r)r1r"prediction_lenZ empty_rowsr.Z length_ratioZ beam_widthiZ pseudo_diagZmin_jZmax_jjZcost_substituteZoperation_substitute operationsZoperation_costZoperation_namer/rr7rr+is>  "$  z3_LevenshteinEditDistance._levenshtein_edit_distance)rCr.rcCsd}|}|j}|dks|dkr|||d}|g|R}|tjtjfvr\|d8}|d8}q|tjkrp|d8}q|tjkr|d8}qtd|q|S)a@Get a trace of executed operations from the edit distance matrix. Args: prediction_len: A length of a tokenized predicted sentence. edit_distance: A matrix of the Levenshtedin edit distance. The element part of the matrix is a tuple of an edit operation cost and an edit operation itself. Return: A trace of executed operations returned as a tuple of `_EDIT_OPERATIONS` enumerates. Raises: ValueError: If an unknown operation has been applied. rrrUnknown operation )r!r rrrr ValueError)r'rCr.r/rDrE operationrrrrBs      z#_LevenshteinEditDistance._get_trace)r)r.rc Cs|jtkrdS|j}t|t|}t|D]}|||d}q,t||d|D]>\}}||vrit|f||<|jd7_||}|d}qTdS)a,Add newly computed rows to cache. Since edit distance is only calculated on the hypothesis suffix that was not in cache, the number of rows in `edit_distance` matrx may be shorter than hypothesis length. In that case we skip over these initial words. Args: prediction_tokens: A tokenized predicted sentence. edit_distance: A matrix of the Levenshtedin edit distance. The element part of the matrix is a tuple of an edit operation cost and an edit operation itself. Nrr)r#_MAX_CACHE_SIZEr"r r;ziptuple) r'r)r.nodeZskip_numrDwordrowvaluerrrr,s  z#_LevenshteinEditDistance._add_cachecCsV|j}d}||jg}|D]0}||vrH|d7}||\}}||qqNq||fS)a+Find the already calculated rows of the Levenshtein edit distance matric. Args: prediction_tokens: A tokenized predicted sentence. Return: A tuple of a start hypothesis position and `edit_distance` matrix. prediction_start: An index where a predicted sentence to be considered from. edit_distance: A matrix of the cached Levenshtedin edit distance. The element part of the matrix is a tuple of an edit operation cost and an edit operation itself. rr)r"_get_initial_rowr!append)r'r)rMr-r.rNrOrrrr*s  z$_LevenshteinEditDistance._find_cache)lengthrcCst|jtjfg|dS)aPrecomputed empty matrix row for Levenhstein edit distance. Args: length: A length of a tokenized sentence. Return: A list of tuples containing infinite edit operation costs and yet undefined edit operations. r)intr&r rr'rSrrrr3s z'_LevenshteinEditDistance._get_empty_rowcsfddt|dDS)a0First row corresponds to insertion operations of the reference, so we do 1 edit operation per reference word. Args: length: A length of a tokenized sentence. Return: A list of tuples containing edit operation costs of insert and insert edit operations. csg|]}|jtjfqSr)rr r)r5rDr7rrr8&z=_LevenshteinEditDistance._get_initial_row..r)r;rUrr7rrQs z)_LevenshteinEditDistance._get_initial_row)rrr)rrrrrstrrTr(rr r0r+rBr,r*r3rQrrrrr6s&  $$ E &&!, r) ref_corpushypothesis_corpusrcCst|tr|g}tdd|DrBt|dkr4|gn dd|D}|rtdd|Drt|t|krtdt|dt|||fS) aCheck and update (if needed) the format of reference and hypothesis corpora for various text evaluation metrics. Args: ref_corpus: An iterable of iterables of reference corpus. hypothesis_corpus: An iterable of hypothesis corpus. Return: ref_corpus: An iterable of iterables of reference corpus. hypothesis_corpus: An iterable of hypothesis corpus. Raises: ValueError: If length of `ref_corpus` and `hypothesis_corpus` differs. css|]}t|tVqdSN) isinstancerWr5refrrr @rVz#_validate_inputs..rcSsg|] }|gqSrrr\rrrr8ArVz$_validate_inputs..css|] }|VqdSrZrr\rrrr^CrVzCorpus has different size z != )r[rWallr rH)rXrYrrr_validate_inputs)s  &r`)r)rrcsfddtt|dD}tt|dD]}|||d<q.ttdD]}||d|<qPtdt|dD]}tdtdD]z}||d|dkr||d|d|||<qt||d||||d||d|dd|||<qqt|ddS)aDynamic programming algorithm to compute the edit distance. Args: prediction_tokens: A tokenized predicted sentence reference_tokens: A tokenized reference sentence Returns: Edit distance between the predicted sentence and the reference sentence csg|]}dgtdqS)rr)r r4rrrr8SrVz"_edit_distance..rrr:)r;r rA)r)rZdprDrErrar_edit_distanceIs Drb.)r/rcsFtjtjtjtjittttftdddtfdd|DS)aFlip the trace of edit operations. Instead of rewriting a->b, get a recipe for rewriting b->a. Simply flips insertions and deletions. Args: trace: A tuple of edit operations. Return: inverted_trace: A tuple of inverted edit operations. )rI_flip_operationsrcSs||vr||S|SrZ)get)rIrcrrr_replace_operation_or_retainss z1_flip_trace.._replace_operation_or_retainc3s|]}|VqdSrZr)r5rIrcrerrr^zrVz_flip_trace..)r rrrrL)r/rrfr _flip_traceas  rgcCsd}}g}g}i}|D]}|tjkrT|d7}|d7}|||<|d|dq|tjkr|d7}|d7}|||<|d|dq|tjkr|d7}|dq|tjkr|d7}|||<|dqtd|dq|||fS)aTransform trace of edit operations into an alignment of the sequences. Args: trace: A trace of edit operations as a tuple of `_EDIT_OPERATIONS` enumerates. Return: alignments: A dictionary mapping aligned positions between a reference and a hypothesis. reference_errors: A list of error positions in a reference. hypothesis_errors: A list of error positions in a hypothesis. Raises: ValueError: If an unknown operation is r:rrrG.)r rrRrrrrH)r/Zreference_positionZhypothesis_positionZreference_errorsZhypothesis_errorsZ alignmentsrIrrr_trace_to_alignment}s4          ri)r=enumrrtypingrrrrrr<rJrTr%rWr rr`rbrgrirrrrs  u