a  d@sddlmZmZmZmZmZddlZddlmZddlm Z ddl m Z m Z m Z mZmZddlmZddlmZmZes~dgZGd d d e ZdS) )AnyDictOptionalSequenceUnionN)Tensor)Metric) PREDS_TYPE TARGETS_TYPE_squad_compute_squad_input_check _squad_update)_MATPLOTLIB_AVAILABLE)_AX_TYPE_PLOT_OUT_TYPE SQuAD.plotcseZdZUdZdZeed<dZeed<dZeed<dZ e ed<d Z e ed <e ed <e ed <e ed <e ddfdd ZeeddddZeee fdddZdeee ee feeedddZZS)SQuADaCalculate `SQuAD Metric`_ which is a metric for evaluating question answering models. This metric corresponds to the scoring script for version 1 of the Stanford Question Answering Dataset (SQuAD). As input to ``forward`` and ``update`` the metric accepts the following input: - ``preds`` (:class:`~Dict`): A Dictionary or List of Dictionary-s that map ``id`` and ``prediction_text`` to the respective values Example ``prediction``: .. code-block:: python {"prediction_text": "TorchMetrics is awesome", "id": "123"} - ``target`` (:class:`~Dict`): A Dictionary or List of Dictionary-s that contain the ``answers`` and ``id`` in the SQuAD Format. Example ``target``: .. code-block:: python { 'answers': [{'answer_start': [1], 'text': ['This is a test answer']}], 'id': '1', } Reference SQuAD Format: .. code-block:: python { 'answers': {'answer_start': [1], 'text': ['This is a test text']}, 'context': 'This is a test context.', 'id': '1', 'question': 'Is this a test?', 'title': 'train test' } As output of ``forward`` and ``compute`` the metric returns the following output: - ``squad`` (:class:`~Dict`): A dictionary containing the F1 score (key: "f1"), and Exact match score (key: "exact_match") for the batch. Args: kwargs: Additional keyword arguments, see :ref:`Metric kwargs` for more info. Example: >>> from torchmetrics.text import SQuAD >>> preds = [{"prediction_text": "1976", "id": "56e10a3be3433e1400422b22"}] >>> target = [{"answers": {"answer_start": [97], "text": ["1976"]}, "id": "56e10a3be3433e1400422b22"}] >>> squad = SQuAD() >>> squad(preds, target) {'exact_match': tensor(100.), 'f1': tensor(100.)} Fis_differentiableThigher_is_betterfull_state_updategplot_lower_boundgY@plot_upper_boundf1_score exact_matchtotalN)kwargsreturnc sjtjfi||jdtjdtjddd|jdtjdtjddd|jdtjdtjddddS)Nrr)Zdtypesum)namedefaultZdist_reduce_fxrr)super__init__Z add_statetorchZtensorfloatint)selfr __class__`/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/torchmetrics/text/squad.pyr!gszSQuAD.__init__)predstargetrcCsLt||\}}t||\}}}|j|7_|j|7_|j|7_dS)z*Update state with predictions and targets.N)r r rrr)r%r*r+Z preds_dict target_dictrrrr(r(r)updateqs z SQuAD.update)rcCst|j|j|jS)z5Aggregate the F1 Score and Exact match for the batch.)r rrr)r%r(r(r)computeysz SQuAD.compute)valaxrcCs |||S)aPlot a single or multiple values from the metric. Args: val: Either a single result from calling `metric.forward` or `metric.compute` or a list of these results. If no value is provided, will automatically call `metric.compute` and plot that result. ax: An matplotlib axis object. If provided will add plot to that axis Returns: Figure and Axes object Raises: ModuleNotFoundError: If `matplotlib` is not installed .. plot:: :scale: 75 >>> # Example plotting a single value >>> from torchmetrics.text import SQuAD >>> metric = SQuAD() >>> preds = [{"prediction_text": "1976", "id": "56e10a3be3433e1400422b22"}] >>> target = [{"answers": {"answer_start": [97], "text": ["1976"]}, "id": "56e10a3be3433e1400422b22"}] >>> metric.update(preds, target) >>> fig_, ax_ = metric.plot() .. plot:: :scale: 75 >>> # Example plotting multiple values >>> from torchmetrics.text import SQuAD >>> metric = SQuAD() >>> preds = [{"prediction_text": "1976", "id": "56e10a3be3433e1400422b22"}] >>> target = [{"answers": {"answer_start": [97], "text": ["1976"]}, "id": "56e10a3be3433e1400422b22"}] >>> values = [ ] >>> for _ in range(10): ... values.append(metric(preds, target)) >>> fig_, ax_ = metric.plot(values) )Z_plot)r%r/r0r(r(r)plot}s*r)NN)__name__ __module__ __qualname____doc__rbool__annotations__rrrr#rrrr!r r r-rstrr.rrrrrr1 __classcell__r(r(r&r)r"s& :      r)typingrrrrrr"rZ torchmetricsrZ"torchmetrics.functional.text.squadr r r r r Ztorchmetrics.utilities.importsrZtorchmetrics.utilities.plotrrZ__doctest_skip__rr(r(r(r)s