a  þdšã @sddlmZddlmZmZmZmZmZddlZddlm Z m Z ee e edœdd„Z e ee dœd d „Zd efee eee e e e e e ee gee fee e fd œ d d„Ze e e e e eeee dœdd„Zdee ee feee ee fe eeeee dœdd„ZdS)é)ÚCounter)ÚCallableÚOptionalÚSequenceÚTupleÚUnionN)ÚTensorÚtensor)Úngram_input_listÚn_gramÚreturncCs\tƒ}td|dƒD]B}tt|ƒ|dƒD](}t||||…ƒ}||d7<q,q|S)a Count how many times each word appears in a given text with ngram. Args: ngram_input_list: A list of translated text or reference texts n_gram: gram value ranged 1 to 4 Return: ngram_counter: a collections.Counter object of ngram é)rÚrangeÚlenÚtuple)r r Z ngram_counterÚiÚjZ ngram_key©rúj/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/torchmetrics/functional/text/bleu.pyÚ _count_ngrams r)Úsentencer cCs| ¡S)z‘Tokenizes sentence into list of words. Args: sentence: A sentence separated by white space. Return: List of words )Úsplit)rrrrÚ _tokenize_fn/s ré) ÚpredsÚtargetÚ numeratorÚ denominatorÚ preds_lenÚ target_lenr Ú tokenizerr csþ‡fdd„|Dƒ}‡fdd„|Dƒ} t| |ƒD]Æ\‰} |tˆƒ7}dd„| Dƒ} ‡fdd„| Dƒ} || |  t| ƒ¡7}tˆ|ƒ} tƒ}| D]}|t||ƒO}qŒ| |@}|D] }|t|ƒd||7<q¬| D] }|t|ƒd| |7<qÒq.||fS)a€Update and returns variables required to compute the BLEU score. Args: preds: An iterable of machine translated corpus target: An iterable of iterables of reference corpus numerator: Numerator of precision score (true positives) denominator: Denominator of precision score (true positives + false positives) preds_len: count of words in a candidate prediction target_len: count of words in a reference translation target: count of words in a reference translation n_gram: gram value ranged 1 to 4 tokenizer: A function that turns sentence into list of words csg|]}‡fdd„|Dƒ‘qS)csg|]}|rˆ|ƒng‘qSrr©Ú.0Úline©r rrÚ Tóz1_bleu_score_update...r)r"Útr$rrr%Tr&z&_bleu_score_update..csg|]}|rˆ|ƒng‘qSrrr!r$rrr%Ur&cSsg|] }t|ƒ‘qSr)r©r"Útgtrrrr%Yr&csg|]}ttˆƒ|ƒ‘qSr)Úabsr)r"Úx)Úpredrrr%Zr&r )ÚziprÚindexÚminrr)rrrrrrr r Útarget_Úpreds_ÚtargetsZtarget_len_listZtarget_len_diffZ preds_counterZtarget_counterr)Zngram_counter_clipZ counter_clipÚcounterr)r,r rÚ_bleu_score_update<s"   r4)rrrrr ÚweightsÚsmoothr c CsÄ|j}t|ƒdkrtd|dS|rht t |tj||d¡t |tj||d¡¡}|d|d|d<n||}t||dt |¡} t t  | ¡¡} ||krªtd|dnt d||¡} | | S)aâCompute the BLEU score. Args: preds_len: count of words in a candidate translation target_len: count of words in a reference translation numerator: Numerator of precision score (true positives) denominator: Denominator of precision score (true positives + false positives) n_gram: gram value ranged 1 to 4 weights: Weights used for unigrams, bigrams, etc. to calculate BLEU score. smooth: Whether to apply smoothing ç)Údevicerçð?r ) r8r/r ÚtorchÚdivÚaddZonesÚlogÚexpÚsum) rrrrr r5r6r8Zprecision_scoresZlog_precision_scoresZgeometric_meanZbrevity_penaltyrrrÚ_bleu_score_computems  þ&r@F)rrr r6r5r c Csât|tƒr|gn|}dd„|Dƒ}t|ƒt|ƒkrNtdt|ƒ›dt|ƒ›ƒ‚|durzt|ƒ|krztdt|ƒ›d|›ƒ‚|durd|g|}t |¡}t |¡}tdƒ} tdƒ} t||||| | |tƒ\} } t | | |||||ƒS) a3Calculate `BLEU score`_ of machine translated text with one or more references. Args: preds: An iterable of machine translated corpus target: An iterable of iterables of reference corpus n_gram: Gram value ranged from 1 to 4 smooth: Whether to apply smoothing - see [2] weights: Weights used for unigrams, bigrams, etc. to calculate BLEU score. If not provided, uniform weights are used. Return: Tensor with BLEU Score Raises: ValueError: If ``preds`` and ``target`` corpus have different lengths. ValueError: If a length of a list of weights is not ``None`` and not equal to ``n_gram``. Example: >>> from torchmetrics.functional.text import bleu_score >>> preds = ['the cat is on the mat'] >>> target = [['there is a cat on the mat', 'a cat is on the mat']] >>> bleu_score(preds, target) tensor(0.7598) References: [1] BLEU: a Method for Automatic Evaluation of Machine Translation by Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu `BLEU`_ [2] Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics by Chin-Yew Lin and Franz Josef Och `Machine Translation Evolution`_ cSs g|]}t|tƒr|gn|‘qSr)Ú isinstanceÚstrr(rrrr%¾r&zbleu_score..zCorpus has different size z != Nz5List of weights has different weights than `n_gram`: r9r7) rArBrÚ ValueErrorr:Úzerosr r4rr@) rrr r6r5r1r0rrrrrrrÚ bleu_score•s (  ÿrE)rFN)Ú collectionsrÚtypingrrrrrr:rr rBÚintrrr4ÚfloatÚboolr@rErrrrÚsL ø  ÷ 2ø +û ú