a  dG3 @sddlZddlmZddlmZmZmZddlZddlmZm Z ddl m Z ddl m Z mZddlmZdZd ZGd d d Zdeeeeeeee deeeeedddZdS)N)partial)ClassVarOptionalSequence)Tensortensor)Literal)_bleu_score_compute_bleu_score_update)_REGEX_AVAILABLEnone13azhintlchar))u㐀u䶵)u一u龥)u龦u龻)u豈u鶴)u侮u頻)u並u龎)u 0u⩭6)u⾀0u⾡d)u＀u￯)u⺀u⻿)u　u〿)u㇀u㇯)u⼀u⿟)u⿰u⿿)u㄀uㄯ)uㆠuㆿ)u︐u︟)u︰u﹏)u☀u⛿)u✀u➿)u㈀u㋿)u㌀u㏿c@seZdZUdZeddfeddfeddfeddffZervd d lZed dfed dfed dffZ ddddddZ e e e d<d1eded dddZeeedddZed2eedeeedddZeeeddd Zeeed!d"d#Zeeedd$d%Zeeedd&d'Zeeedd(d)Zeeedd*d+Zeeedd,d-Zeeeed.d/d0Zd S)3_SacreBLEUTokenizerzTokenizer used for SacreBLEU calculation. Source: https://github.com/mjpost/sacrebleu/tree/master/sacrebleu/tokenizers z([\{-\~\[-\` -\&\(-\+\:-\@\/])z \1 z([^0-9])([\.,])z\1 \2 z([\.,])([^0-9])z \1 \2z ([0-9])(-)rNz(\P{N})(\p{P})z(\p{P})(\P{N})z(\p{S})_tokenize_base _tokenize_13a _tokenize_zh_tokenize_international_tokenize_charr _TOKENIZE_FNF)tokenize lowercasereturncCst||j||_||_dSN)getattrr tokenize_fnr)selfrrr p/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/torchmetrics/functional/text/sacre_bleu.py__init__ysz_SacreBLEUTokenizer.__init__)linercCs||}|||jSr)r_lowerrsplit)rr#tokenized_liner r r!__call__}s z_SacreBLEUTokenizer.__call__)r#rrrcCs(t||j|}||}|||Sr)rrr$r%)clsr#rrrr&r r r!rsz_SacreBLEUTokenizer.tokenizecCs*|jD]\}}|||}qd|S)zPost-processing tokenizer for `13a` and `zh` tokenizers. Args: line: a segment to tokenize Return: the tokenized line  )_REGEXsubjoinr%r(r#_rereplr r r!_tokenize_regexs z#_SacreBLEUTokenizer._tokenize_regex)ucharrcstfddtDS)zCheck if character is chinese. Args: uchar: input char in unicode. Return: whether the input char is a Chinese character. c3s*|]"\}}|ko|knVqdSrr ).0startendr1r r! z7_SacreBLEUTokenizer._is_chinese_char..)any _UCODE_RANGESr5r r5r!_is_chinese_chars z$_SacreBLEUTokenizer._is_chinese_charcCs|S)zTokenizes an input line with the tokenizer. Args: line: a segment to tokenize Return: the tokenized line r r(r#r r r!rs z"_SacreBLEUTokenizer._tokenize_basecCsn|dd}|dd}|dd}d|vr\|dd}|d d}|d d }|d d }|d|dS)zTokenizes a line using a relatively minimal tokenization that is equivalent to mteval-v13a, used by WMT. Args: line: input sentence Return: tokenized sentence z z-  r)&z""z&z<)replacer0r;r r r!rs       z!_SacreBLEUTokenizer._tokenize_13acCsL|}d}|D]0}||r8|d7}||7}|d7}q||7}q||S)apTokenization of Chinese text. This is done in two steps: separate each Chinese characters (by utf-8 encoding) and afterwards tokenize the Chinese part (following the `13a` i.e. mteval tokenizer). Author: Shujian Huang huangsj@nju.edu.cn. Args: line: input sentence Return: tokenized sentence r<r))stripr:r0)r(r#Z line_in_charsrr r r!rs   z _SacreBLEUTokenizer._tokenize_zhcCs*|jD]\}}|||}qd|S)aTokenizes a string following the official BLEU implementation. See github.com/moses-smt/mosesdecoder/blob/master/scripts/generic/mteval-v14.pl#L954-L983 In our case, the input string is expected to be just one line. We just tokenize on punctuation and symbols, except when a punctuation is preceded and followed by a digit (e.g. a comma/dot as a thousand/decimal separator). We do not recover escaped forms of punctuations such as ' or > as these should never appear in MT system outputs (see issue #138) Note that a number (e.g., a year) followed by a dot at the end of sentence is NOT tokenized, i.e. the dot stays with the number because `s/(\\p{P})(\\P{N})/ $1 $2/g` does not match this case (unless we add a space after each sentence). However, this error is already in the original mteval-v14.pl and we want to be consistent with it. The error is not present in the non-international version, which uses `$norm_text = " $norm_text "`. Args: line: the input string to tokenize. Return: The tokenized string. r)) _INT_REGEXr+r,r%r-r r r!rsz+_SacreBLEUTokenizer._tokenize_internationalcCsddd|DS)zTokenizes all the characters in the input line. Args: line: a segment to tokenize Return: the tokenized line r)css|] }|VqdSrr )r2rr r r!r6r7z5_SacreBLEUTokenizer._tokenize_char..)r,r;r r r!rs z"_SacreBLEUTokenizer._tokenize_char)r#rrcCs|r |S|Sr)lower)r#rr r r!r$sz_SacreBLEUTokenizer._lower)F)F)__name__ __module__ __qualname____doc__recompiler*r regexrDrrdict__annotations__rboolr"strrr' classmethodrr0 staticmethodr:rrrrrr$r r r r!rPsT              rFr)predstargetn_gramsmoothrrweightsrc Cs(|tvrtdtd|d|tjvrDtdttjt|t|krptdt|dt||dkrtstd|d urt||krtd t|d||d urd |g|}t |}t |}t d } t d } t tj ||d } t||||| | || \} } t| | |||||S)aCalculate `BLEU score`_ [1] of machine translated text with one or more references. This implementation follows the behaviour of SacreBLEU [2] implementation from https://github.com/mjpost/sacrebleu. Args: preds: An iterable of machine translated corpus target: An iterable of iterables of reference corpus n_gram: Gram value ranged from 1 to 4 smooth: Whether to apply smoothing - see [2] tokenize: Tokenization technique to be used. Supported tokenization: ['none', '13a', 'zh', 'intl', 'char'] lowercase: If ``True``, BLEU score over lowercased text is calculated. weights: Weights used for unigrams, bigrams, etc. to calculate BLEU score. If not provided, uniform weights are used. Return: Tensor with BLEU Score Raises: ValueError: If ``preds`` and ``target`` corpus have different lengths. ValueError: If a length of a list of weights is not ``None`` and not equal to ``n_gram``. Example: >>> from torchmetrics.functional.text import sacre_bleu_score >>> preds = ['the cat is on the mat'] >>> target = [['there is a cat on the mat', 'a cat is on the mat']] >>> sacre_bleu_score(preds, target) tensor(0.7598) References: [1] BLEU: a Method for Automatic Evaluation of Machine Translation by Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu `BLEU`_ [2] A Call for Clarity in Reporting BLEU Scores by Matt Post. [3] Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics by Chin-Yew Lin and Franz Josef Och `Machine Translation Evolution`_ z*Argument `tokenize` expected to be one of z but got .z6Unsupported tokenizer selected. Please, choose one of zCorpus has different size z != rzv`'intl'` tokenization requires that `regex` is installed. Use `pip install regex` or `pip install torchmetrics[text]`.Nz5List of weights has different weights than `n_gram`: g?g)rr)AVAILABLE_TOKENIZERS ValueErrorrrkeyslistlenr ModuleNotFoundErrortorchzerosrrrr r ) rTrUrVrWrrrX numerator denominatorZ preds_lenZ target_lenrr r r!sacre_bleu_scores@1    rd)rSFrFN)rJ functoolsrtypingrrrr`rrZtyping_extensionsrZ!torchmetrics.functional.text.bleur r Ztorchmetrics.utilities.importsr rZr9rrPintrOfloatrdr r r r!(s4   P