a  dU@s0UddlZddlmZddlmZmZmZmZmZm Z m Z m Z ddl Z ddl m Z mZddlmZddlmZddgiZd d d d d ddddddd Zeee eeffed<dZddddZee edddZeeeeee fdddZdAe ee eee ee e efd!d"d#Ze e ee ee ee ed$d%d&Ze e ee ee ed'd(d)ZdBeeeeeegefeeege efe ed*d+d,Z e ee eeeee fd-d.d/Z!e ee eeee fd0d1d2Z"e e ee e eeee fd0d3d4Z#dCe ee e eee eefeeeeeegefeeege efee eefeeee ffd5d6d7Z$eeee feee fd8d9d:Z%dDe ee efe ee ee e efedeeeegefeeege efe ee ed=ffeee fd>d?d@Z&dS)EN)Counter)AnyCallableDictListOptionalSequenceTupleUnion)Tensortensor)Literal)_NLTK_AVAILABLE) rouge_score_rouge_score_updatenltk LLsum) rouge1rouge2Zrouge3Zrouge4Zrouge5Zrouge6Zrouge7Zrouge8Zrouge9rougeL rougeLsumALLOWED_ROUGE_KEYS)avgbest)returnc Csxddl}z|jdWnZtyrz|jddddddWn.tyl}ztd|WYd}~n d}~00Yn0dS) zyCheck whether `nltk` `punkt` is downloaded. If not, try to download if a machine is connected to the internet. rNztokenizers/punktZpunktTF)quietforceZ halt_on_errorZraise_on_errorzz`nltk` resource `punkt` is not available on a disk and cannot be downloaded as a machine is not connected to the internet.)rdatafind LookupErrordownload ValueErrorOSError)rerrr.k/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/torchmetrics/functional/text/rouge.py _ensure_nltk_punkt_is_downloaded*s r0)xr$cCs2ts tdddl}ttdd|||S)z]Split sentence to get rougeLsum scores matching published rougeL scores for BART and PEGASUS.zQROUGE-Lsum calculation requires that `nltk` is installed. Use `pip install nltk`.rNz)rModuleNotFoundErrorrr0resubZ sent_tokenize)r1rr.r.r/_split_sentence>s r6) hits_or_lcspred_len target_lenr$cCsl||}||}||kr$dkr@nntdtdtddSd||||}t|t|t|dS)aCompute overall metrics. This function computes precision, recall and F1 score based on hits/lcs, the length of lists of tokenizer predicted and target sentences. Args: hits_or_lcs: A number of matches or a length of the longest common subsequence. pred_len: A length of a tokenized predicted sentence. target_len: A length of a tokenized target sentence.  precisionrecallfmeasurer)r )r7r8r9r<r=r>r.r.r/_compute_metricsJs r?F) pred_tokens target_tokensreturn_full_tabler$csfddtt|dD}tdt|dD]}tdtdD]h}||d|dkr||d|dd|||<qFt||d||||d|||<qFq0|r|S|ddS)a0DP algorithm to compute the length of the longest common subsequence. Args: pred_tokens: A tokenized predicted sentence. target_tokens: A tokenized target sentence. return_full_table: If the full table of logest common subsequence should be returned or just the largest csg|]}dgtdqS)rr)len).0_r@r.r/ jz_lcs..r)rangerCmax)r@rArBlcsijr.rFr/_lcs_s ".rO) lcs_tabler@rAr$cCst|}t|}g}|dkr|dkr||d||dkr^|d|d|d8}|d8}q|||d||d|kr|d8}q|d8}q|S)zBacktrack LCS table. Args: lcs_table: A table containing information for the calculation of the longest common subsequence. pred_tokens: A tokenized predicted sentence. target_tokens: A tokenized target sentence. rr)rCinsert)rPr@rArMrNZbacktracked_lcsr.r.r/_backtracked_lcsvs     rR)pred_tokens_listrAr$csdttttttdddtttttddd}fdd|D}fd d||DS) zFind union LCS between a target sentence and iterable of predicted tokens. Args: pred_tokens_list: A tokenized predicted sentence split by ``'\n'``. target_tokens: A tokenized single part of target sentence split by ``'\n'``. )r@rAr$cSst||dd}t|||S)zRReturn one of the longest of longest common subsequence via backtracked lcs table.T)rB)rOrR)r@rArPr.r.r/lcs_indsz_union_lcs..lcs_ind) lcs_tablesr$cSsttj|S)z#Find union LCS given a list of LCS.)sortedsetunion)rUr.r.r/ find_unionsz_union_lcs..find_unioncsg|]}|qSr.r.)rDr@rTrAr.r/rGrHz_union_lcs..csg|] }|qSr.r.rDrM)rAr.r/rGrH)rstrint)rSrArYrUr.rZr/ _union_lcss r^)textstemmer normalizer tokenizerr$csbt|r||ntdd|}t|r2||n td|}rTfdd|D}dd|DS)aRouge score should be calculated only over lowercased words and digits. Optionally, Porter stemmer can be used to strip word suffixes to improve matching. The text normalization follows the implemantion from `Rouge score_Text Normalizition`_. Args: text: An input sentence. stemmer: Porter stemmer instance to strip word suffixes to improve matching. normalizer: A user's own normalizer function. If this is ``None``, replacing any non-alpha-numeric characters with spaces is default. This function must take a ``str`` and return a ``str``. tokenizer: A user's own tokenizer function. If this is ``None``, splitting by spaces is default This function must take a ``str`` and return ``Sequence[str]`` z [^a-z0-9]+ z\s+cs&g|]}t|dkr|n|qS)r)rCstemrDr1r`r.r/rGrHz0_normalize_and_tokenize_text..cSs&g|]}t|trt|dkr|qS)r) isinstancer\rCrer.r.r/rGrH)callabler4r5lowersplit)r_r`rarbtokensr.rfr/_normalize_and_tokenize_texts "rl)predtargetn_gramr$csttttddd}||||||tt}}d||fvrjtdtdtddStfddtD}t|t |d t |d S) zCompute precision, recall and F1 score for the Rouge-N metric. Args: pred: A predicted sentence. target: A target sentence. n_gram: N-gram overlap. )rknr$csDt}fddttdDD]}||d7<q*|S)Nc3s"|]}t||VqdSN)tupler[rprkr.r/ rHz9_rouge_n_score.._create_ngrams..r)rrJrC)rkrpngramsZngramr.rsr/_create_ngramss(z&_rouge_n_score.._create_ngramsrr:r;c3s |]}t||VqdSrq)min)rDwZ pred_ngramsZ target_ngramsr.r/rtrHz!_rouge_n_score..r) rr\r]rsumvaluesr rWr?rK)rmrnrorvr8r9hitsr.ryr/_rouge_n_scores  r})rmrnr$cCsLt|t|}}d||fvr6tdtdtddSt||}t|||S)zCompute precision, recall and F1 score for the Rouge-L metric. Args: pred: A predicted sentence. target: A target sentence. rr:r;)rCr rOr?)rmrnr8r9rLr.r.r/_rouge_l_scores   r~c Csttt|}ttt|}d||fvr@tdtdtddSttttddd}||}||}d}|D]X}t||} | D]D} || dkr|| dkr|d7}|| d8<|| d8<qqpt|||S)a>Compute precision, recall and F1 score for the Rouge-LSum metric. More information can be found in Section 3.2 of the referenced paper [1]. This implementation follow the official implementation from: https://github.com/google-research/google-research/blob/master/rouge/rouge_scorer.py. Args: pred: An iterable of predicted sentence split by '\n'. target: An iterable target sentence split by '\n'. References: [1] ROUGE: A Package for Automatic Evaluation of Summaries by Chin-Yew Lin. https://aclanthology.org/W04-1013/ rr:r;) sentencesr$cSst}|D]}||q |Srq)rupdate)rruZsentencer.r.r/_get_token_counts s z,_rouge_lsum_score.._get_token_countsr) rzmaprCr rr\rr^r?) rmrnr8r9rZpred_tokens_countZtarget_tokens_countr|tgtrLtokenr.r.r/_rouge_lsum_scores   r)predsrnrouge_keys_values accumulater`rarbr$cs dd|D}t||D]\}} dd|D} dd|D} g} t|} d|vrrfddt|D}| D]}t|}d|vrfddt|D}|D]X}t|trt| ||}n&|d krt| |}n|dkrt||}|| |<| ||q| | qv|d krx|d t fd d| D}tt | }|D]}||| ||qXq|d krdd|D}| D]d\}}i|D]:}|D]*\}}|vrg|<||qqfddD||<q|D]}||||qq|S)a Update the rouge score with the current set of predicted and target sentences. Args: preds: An iterable of predicted sentences. target: An iterable of iterable of target sentences. rouge_keys_values: List of N-grams/'L'/'Lsum' arguments. accumulate: Useful incase of multi-reference rouge score. ``avg`` takes the avg of all references with respect to predictions ``best`` takes the best fmeasure score obtained between prediction and multiple corresponding references. Allowed values are ``avg`` and ``best``. stemmer: Porter stemmer instance to strip word suffixes to improve matching. normalizer: A user's own normalizer function. If this is ``None``, replacing any non-alpha-numeric characters with spaces is default. This function must take a `str` and return a `str`. tokenizer: A user's own tokenizer function. If this is ``None``, spliting by spaces is default This function must take a `str` and return `Sequence[str]` Example: >>> preds = "My name is John".split() >>> target = "Is your name John".split() >>> from pprint import pprint >>> score = _rouge_score_update(preds, target, rouge_keys_values=[1, 2, 3, 'L'], accumulate='best') >>> pprint(score) {1: [{'fmeasure': tensor(0.), 'precision': tensor(0.), 'recall': tensor(0.)}, {'fmeasure': tensor(0.), 'precision': tensor(0.), 'recall': tensor(0.)}, {'fmeasure': tensor(0.), 'precision': tensor(0.), 'recall': tensor(0.)}, {'fmeasure': tensor(0.), 'precision': tensor(0.), 'recall': tensor(0.)}], 2: [{'fmeasure': tensor(0.), 'precision': tensor(0.), 'recall': tensor(0.)}, {'fmeasure': tensor(0.), 'precision': tensor(0.), 'recall': tensor(0.)}, {'fmeasure': tensor(0.), 'precision': tensor(0.), 'recall': tensor(0.)}, {'fmeasure': tensor(0.), 'precision': tensor(0.), 'recall': tensor(0.)}], 3: [{'fmeasure': tensor(0.), 'precision': tensor(0.), 'recall': tensor(0.)}, {'fmeasure': tensor(0.), 'precision': tensor(0.), 'recall': tensor(0.)}, {'fmeasure': tensor(0.), 'precision': tensor(0.), 'recall': tensor(0.)}, {'fmeasure': tensor(0.), 'precision': tensor(0.), 'recall': tensor(0.)}], 'L': [{'fmeasure': tensor(0.), 'precision': tensor(0.), 'recall': tensor(0.)}, {'fmeasure': tensor(0.), 'precision': tensor(0.), 'recall': tensor(0.)}, {'fmeasure': tensor(0.), 'precision': tensor(0.), 'recall': tensor(0.)}, {'fmeasure': tensor(0.), 'precision': tensor(0.), 'recall': tensor(0.)}]} cSsi|] }|gqSr.r.rD rouge_keyr.r.r/ SrHz'_rouge_score_update..cSsi|] }|iqSr.r.rr.r.r/rVrHcSsi|] }|gqSr.r.rr.r.r/rWrHrcsg|]}t|qSr.rl)rDZ pred_sentencerar`rbr.r/rG[sz'_rouge_score_update..csg|]}t|qSr.r)rDZ tgt_sentencerr.r/rGdsrr#rcsg|]}|dqS)r>r.)rDv)key_currr.r/rGvrHr"cSsi|] }|iqSr.r.rr.r.r/r}scs i|]}|t|qSr.)torchr mean)rD_type)_dict_metric_score_batchr.r/rs)ziprlr6rgr]r}r~rappendcopyrr Zargmaxitemitems)rrnrrr`rarbresultsZpred_rawZ target_rawZ result_innerZ result_avgZ list_resultsrmZ pred_lsumZtarget_raw_innerrZ target_lsumrZscoreZ all_fmeasureZ highest_idxZnew_result_avgmetricsmetricrvaluer.)rrrar`rbr/rsb4      r)sentence_resultsr$cCs8i}|ikr|S|D]\}}t|||<q|S)zCompute the combined ROUGE metric for all the input set of predicted and target sentences. Args: sentence_results: Rouge-N/Rouge-L/Rouge-LSum metrics calculated for single sentence. )rrr r)rrrZscoresr.r.r/_rouge_score_computes rr#rrrr .)rrnr use_stemmerrarb rouge_keysr$c CsR|rtstdddl}|r(|jjnd}t|ts<|f}|D],} | t vr@t d| dt t q@dd|D} t|t rt dd |Drt|t r|gn d d|D}t|t r|g}t|t r|gg}t||| ||||d } d d | D} | D]D\} }|D]4}|D]$\}}| d| d||qqqt| S)ax Calculate `Calculate Rouge Score`_ , used for automatic summarization. Args: preds: An iterable of predicted sentences or a single predicted sentence. target: An iterable of iterables of target sentences or an iterable of target sentences or a single target sentence. accumulate: Useful incase of multi-reference rouge score. - ``avg`` takes the avg of all references with respect to predictions - ``best`` takes the best fmeasure score obtained between prediction and multiple corresponding references. use_stemmer: Use Porter stemmer to strip word suffixes to improve matching. normalizer: A user's own normalizer function. If this is ``None``, replacing any non-alpha-numeric characters with spaces is default. This function must take a ``str`` and return a ``str``. tokenizer: A user's own tokenizer function. If this is ``None``, spliting by spaces is default This function must take a ``str`` and return ``Sequence[str]`` rouge_keys: A list of rouge types to calculate. Keys that are allowed are ``rougeL``, ``rougeLsum``, and ``rouge1`` through ``rouge9``. Return: Python dictionary of rouge scores for each input rouge key. Example: >>> from torchmetrics.functional.text.rouge import rouge_score >>> preds = "My name is John" >>> target = "Is your name John" >>> from pprint import pprint >>> pprint(rouge_score(preds, target)) {'rouge1_fmeasure': tensor(0.7500), 'rouge1_precision': tensor(0.7500), 'rouge1_recall': tensor(0.7500), 'rouge2_fmeasure': tensor(0.), 'rouge2_precision': tensor(0.), 'rouge2_recall': tensor(0.), 'rougeL_fmeasure': tensor(0.5000), 'rougeL_precision': tensor(0.5000), 'rougeL_recall': tensor(0.5000), 'rougeLsum_fmeasure': tensor(0.5000), 'rougeLsum_precision': tensor(0.5000), 'rougeLsum_recall': tensor(0.5000)} Raises: ModuleNotFoundError: If the python package ``nltk`` is not installed. ValueError: If any of the ``rouge_keys`` does not belong to the allowed set of keys. References: [1] ROUGE: A Package for Automatic Evaluation of Summaries by Chin-Yew Lin. https://aclanthology.org/W04-1013/ zBStemmer requires that `nltk` is installed. Use `pip install nltk`.rNzGot unknown rouge key z. Expected to be one of cSsg|] }t|qSr.)r!)rDkeyr.r.r/rGrHzrouge_score..css|]}t|tVqdSrq)rgr\rDrr.r.r/rtrHzrouge_score..cSsg|] }|gqSr.r.rr.r.r/rGrH)r`rarbrcSs(i|] }dD]}d|d|gq qS))r>r<r=rougerEr.)rDrtpr.r.r/rszrouge_score..rrE)rr3rrdZporterZ PorterStemmerrgrrr!keysr+listallr\rrrr)rrnrrrarbrrr`rrroutputrrrrrr.r.r/rsD?     &r)F)NNN)NNN)r#FNNr)'r4 collectionsrtypingrrrrrrr r rr r Ztyping_extensionsr Ztorchmetrics.utilities.importsrZ__doctest_requires__r!r\r]__annotations__ZALLOWED_ACCUMULATE_VALUESr0r6r?boolrOrRr^rlr}r~rrrrr.r.r.r/s  (       " $$"*0  s$