a  d'@sddlZddlmZmZmZmZmZmZmZddl Z ddl m Z ddl m Z ddl mZmZmZmZmZmZddlmZddlmZddlmZmZdd lmZmZesd gZesd d gZGd d d eZdS) N)AnyDictListOptionalSequenceTupleUnion)Tensor)_load_tokenizer_and_model)$_ALLOWED_INFORMATION_MEASURE_LITERAL_get_dataloader_get_special_tokens_map_infolm_compute_infolm_update_InformationMeasure)Metric) dim_zero_cat)_MATPLOTLIB_AVAILABLE_TRANSFORMERS_GREATER_EQUAL_4_4)_AX_TYPE_PLOT_OUT_TYPE InfoLM.plotInfoLMcseZdZUdZdZdZeeed<eeed<eeed<eeed<de e e j fe eeee ee ee e ejfeeeeeeee efd dfdd Ze e ee fe e ee fd dddZe eeeeffdddZdee eeefeeedddZZS)ru Calculate `InfoLM`_. InfoLM measures a distance/divergence between predicted and reference sentence discrete distribution using one of the following information measures: - `KL divergence`_ - `alpha divergence`_ - `beta divergence`_ - `AB divergence`_ - `Rényi divergence`_ - L1 distance - L2 distance - L-infinity distance - `Fisher-Rao distance`_ `InfoLM`_ is a family of untrained embedding-based metrics which addresses some famous flaws of standard string-based metrics thanks to the usage of pre-trained masked language models. This family of metrics is mainly designed for summarization and data-to-text tasks. The implementation of this metric is fully based HuggingFace ``transformers``' package. As input to ``forward`` and ``update`` the metric accepts the following input: - ``preds`` (:class:`~Sequence`): An iterable of hypothesis corpus - ``target`` (:class:`~Sequence`): An iterable of reference corpus As output of ``forward`` and ``compute`` the metric returns the following output: - ``infolm`` (:class:`~torch.Tensor`): If `return_sentence_level_score=True` return a tuple with a tensor with the corpus-level InfoLM score and a list of sentence-level InfoLM scores, else return a corpus-level InfoLM score Args: model_name_or_path: A name or a model path used to load ``transformers`` pretrained model. By default the `"bert-base-uncased"` model is used. temperature: A temperature for calibrating language modelling. For more information, please reference `InfoLM`_ paper. information_measure: A name of information measure to be used. Please use one of: ['kl_divergence', 'alpha_divergence', 'beta_divergence', 'ab_divergence', 'renyi_divergence', 'l1_distance', 'l2_distance', 'l_infinity_distance', 'fisher_rao_distance'] idf: An indication of whether normalization using inverse document frequencies should be used. alpha: Alpha parameter of the divergence used for alpha, AB and Rényi divergence measures. beta: Beta parameter of the divergence used for beta and AB divergence measures. device: A device to be used for calculation. max_length: A maximum length of input sequences. Sequences longer than ``max_length`` are to be trimmed. batch_size: A batch size used for model processing. num_threads: A number of threads to use for a dataloader. verbose: An indication of whether a progress bar to be displayed during the embeddings calculation. return_sentence_level_score: An indication whether a sentence-level InfoLM score to be returned. Example: >>> from torchmetrics.text.infolm import InfoLM >>> preds = ['he read the book because he was interested in world history'] >>> target = ['he was interested in world history because he read the book'] >>> infolm = InfoLM('google/bert_uncased_L-2_H-128_A-2', idf=False) >>> infolm(preds, target) tensor(-0.1784) FTpreds_input_idspreds_attention_masktarget_input_idstarget_attention_maskbert-base-uncased? kl_divergenceN@r)model_name_or_path temperatureinformation_measureidfalphabetadevice max_length batch_size num_threadsverbosereturn_sentence_level_scorekwargsreturnc  stjfi| ||_||_||_||_||_||_t |p@d|_ | |_ | |_ | |_ | |_t||\|_|_t||||_|p|jjj|_t|j|_|jdgdd|jdgdd|jdgdd|jdgdddS)Ncpurcat)Zdist_reduce_fxrrr)super__init__r!r"r#r$r%r&torchr'Z_devicer)r*r+r,r tokenizermodelrinformation_measure_clsconfigr(r special_tokens_mapZ add_state)selfr!r"r#r$r%r&r'r(r)r*r+r,r- __class__a/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/torchmetrics/text/infolm.pyr2xs( zInfoLM.__init__)predstargetr.cCsNt|||j|j\}}}}|j||j||j||j|dS)z*Update state with predictions and targets.N)rr4r(rappendrrr)r9r>r?rrrrr<r<r=updates    z InfoLM.update)r.c Cstt|jt|j|j|j|jd}tt|jt|j|j|j|jd}t |j |||j |j|j |j |j}|jr|||fS|S)zLCalculate selected information measure using the pre-trained language model.)Z input_idsZattention_maskr$r)Z num_workers)r rrrr$r)r*rrrr5r"r6r8r+r,mean)r9Zpreds_dataloaderZtarget_dataloaderZ info_lm_scorer<r<r=computes6  zInfoLM.compute)valaxr.cCs |||S)a1Plot a single or multiple values from the metric. Args: val: Either a single result from calling `metric.forward` or `metric.compute` or a list of these results. If no value is provided, will automatically call `metric.compute` and plot that result. ax: An matplotlib axis object. If provided will add plot to that axis Returns: Figure and Axes object Raises: ModuleNotFoundError: If `matplotlib` is not installed .. plot:: :scale: 75 >>> # Example plotting a single value >>> from torchmetrics.text.infolm import InfoLM >>> metric = InfoLM('google/bert_uncased_L-2_H-128_A-2', idf=False) >>> preds = ['he read the book because he was interested in world history'] >>> target = ['he was interested in world history because he read the book'] >>> metric.update(preds, target) >>> fig_, ax_ = metric.plot() .. plot:: :scale: 75 >>> # Example plotting multiple values >>> from torchmetrics.text.infolm import InfoLM >>> metric = InfoLM('google/bert_uncased_L-2_H-128_A-2', idf=False) >>> preds = ["this is the prediction", "there is an other sample"] >>> target = ["this is the reference", "there is another one"] >>> values = [ ] >>> for _ in range(10): ... values.append(metric(preds, target)) >>> fig_, ax_ = metric.plot(values) )Z_plot)r9rDrEr<r<r=plots*r) rrrTNNNNr rTF)NN)__name__ __module__ __qualname____doc__Zis_differentiableZhigher_is_betterrr __annotations__rstrosPathLikefloatr boolrr3r'intrrr2rrArrCrrrF __classcell__r<r<r:r=r)sT G      '* ") rMtypingrrrrrrrr3r Z4torchmetrics.functional.text.helper_embedding_metricr Z#torchmetrics.functional.text.infolmr r r rrrZtorchmetrics.metricrZtorchmetrics.utilities.datarZtorchmetrics.utilities.importsrrZtorchmetrics.utilities.plotrrZ__doctest_skip__rr<r<r<r=s$