a  dR@sddlZddlZddlmZmZmZmZmZmZm Z m Z ddl Z ddl m Z ddl mZddlmZddlmZmZmZmZmZmZmZddlmZddlmZmZdd lmZm Z d Z!e rdd l"m#Z#m$Z$dd d dZ%eree%sdgZ&ndgZ&d.ee'eee e(e j)fee'e*e*e*eeeee(e fge fe e e fd ddZ+e e(e e dddZ,e e e e e e e e fdddZ-d/ee(ee'e*e(dddZ.e(e dddZ/e(e d d!d"Z0d0e(ee(ee(ee(ee d$d%d&Z1d1e e e e ee'e*e e e e fd'd(d)Z2d2e e(ee(ee(e ffe e(ee(ee(e ffee(ee'e*eeeeeeee(e fge fe*e*ee e(e j)fe'e'e'e*e(e*ee(ee(ee(e e ee3e(ffd,d-dZ4dS)3N)AnyCallableDictListOptionalSequenceTupleUnion)Tensor)Module) DataLoader) TextDatasetTokenizedDataset_check_shape_of_model_output_get_progress_bar_input_data_collator_output_data_collator*_process_attention_mask_for_special_tokens)rank_zero_warn)_SKIP_SLOW_DOCTEST_try_proceed_with_timeout)_TQDM_AVAILABLE_TRANSFORMERS_GREATER_EQUAL_4_4z roberta-large) AutoModel AutoTokenizer)returncCsttttdS)zDownload intensive operations.N)rfrom_pretrained_DEFAULT_MODELrrrj/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/torchmetrics/functional/text/bert.py_download_model*s r bert_scoreF) dataloader target_lenmodeldevice num_layers all_layersidfverboseuser_forward_fnrc  Csg} g} t||D]X} tt| |} |s|sb|| d| ddd} | j|durZ|nd} n||| } t| | d| d} n<|rtd|| d| ddd} tjd d | jDdd } Wdn1s0Y| | j dd d} t | | d|\} } t | }t d | |} | | |r@| d |n || j}||jddd}| |qt| }t| }||fS)a<Calculate sentence embeddings and the inverse-document-frequency scaling factor. Args: dataloader: dataloader instance. target_len: A length of the longest sequence in the data. Used for padding the model output. model: BERT model. device: A device to be used for calculation. num_layers: The layer of representation to use. all_layers: An indication whether representation from all model layers should be used for BERTScore. idf: An Indication whether normalization using inverse document frequencies should be used. verbose: An indication of whether a progress bar to be displayed during the embeddings' calculation. user_forward_fn: A user's own forward function used in a combination with ``user_model``. This function must take ``user_model`` and a python dictionary of containing ``"input_ids"`` and ``"attention_mask"`` represented by :class:`~torch.Tensor` as an input and return the model's output represented by the single :class:`~torch.Tensor`. Return: A tuple of :class:`~torch.Tensor`s containing the model's embeddings and the normalized tokens IDF. When ``idf = False``, tokens IDF is not calculated, and a matrix of mean weights is returned instead. For a single sentence, ``mean_weight = 1/seq_len``, where ``seq_len`` is a sum over the corresponding ``attention_mask``. Raises: ValueError: If ``all_layers = True`` and a model, which is not from the ``transformers`` package, is used. input_idsattention_maskT)Zoutput_hidden_statesNzQThe option `all_layers=True` can be used only with default `transformers` models.cSsg|]}|dqS)r.) unsqueeze).0orrr pz1_get_embeddings_and_idf_scale..dimzblsd, bs -> blsd input_ids_idf)Zkeepdim)rtorchZno_gradrZ hidden_statesrr/ ValueErrorcatZnormrreinsumappendcputypeZdtypesum)r"r#r$r%r&r'r(r)r*Zembeddings_listZidf_scale_listbatchoutr,Zprocessed_attention_maskr6Z embeddings idf_scalerrr_get_embeddings_and_idf_scale5s<'    8  rB)cos_simmetricrArcCsB|dkr dnd}|j|dj}td||d}|ddS) zOCalculate precision or recall, transpose it and scale it with idf_scale factor. precisionr4zbls, bs -> blsr-rr.)maxvaluesr7r:r>Z transposeZsqueeze)rCrDrAr5resrrr_get_scaled_precision_or_recallsrK)preds_embeddingstarget_embeddingspreds_idf_scaletarget_idf_scalercCsVtd||}t|d|}t|d|}d||||}|t|d}|||fS)aCalculate precision, recall and F1 score over candidate and reference sentences. Args: preds_embeddings: Embeddings of candidate sentences. target_embeddings: Embeddings of reference sentences. preds_idf_scale: An IDF scale factor for candidate sentences. target_idf_scale: An IDF scale factor for reference sentences. Return: Tensors containing precision, recall and F1 score, respectively. zblpd, blrd -> blprrErecallrG)r7r:rKZ masked_fillisnan)rLrMrNrOrCrErPf1_scorerrr_get_precision_recall_f1s   rT)model_name_or_pathr&r(rcCs|d||rdndS)z,Compute `BERT_score`_ (copied and adjusted)._LZ_idfz_no-idfr)rUr&r(rrr _get_hashsrW) baseline_pathrcCs^t|,}t|}ddt|D}Wdn1s:0Yt|ddddfS)zqRead baseline from csv file from the local file. This method implemented to avoid `pandas` dependency. cSs&g|]\}}|dkrdd|DqS)rcSsg|] }t|qSrfloatr0itemrrrr2r3z8_read_csv_from_local_file...rr0idxrowrrrr2r3z-_read_csv_from_local_file..Nr.)opencsvreader enumerater7tensor)rXfnameZcsv_file baseline_listrrr_read_csv_from_local_files  0rg) baseline_urlrcCs\tj|<}ddt|D}t|ddddfWdS1sN0YdS)ziRead baseline from csv file from URL. This method is implemented to avoid `pandas` dependency. cSs6g|].\}}|dkrdd|ddDqS)rcSsg|] }t|qSrrYr[rrrr2r3z1_read_csv_from_url...zutf-8,)stripdecodesplitr]rrrr2sz&_read_csv_from_url..Nr.)urllibrequesturlopenrcr7rd)rh http_requestrfrrr_read_csv_from_urls rqen)langrUrXrhrcCsX|rt|}nF|rt|}n8|rH|rHd}|d|d|d}t|}n tddS|S)z>> from pprint import pprint >>> from torchmetrics.functional.text.bert import bert_score >>> preds = ["hello there", "general kenobi"] >>> target = ["hello there", "master kenobi"] >>> pprint(bert_score(preds, target)) {'f1': tensor([1.0000, 0.9961]), 'precision': tensor([1.0000, 0.9961]), 'recall': tensor([1.0000, 0.9961])} z=Number of predicted and reference sententes must be the same!zcAn argument `verbose = True` requires `tqdm` package be installed. Install with `pip install tqdm`.Nz`bert_score` metric with default models requires `transformers` package be installed. Either install with `pip install transformers>=4.4` or `pip install torchmetrics[text]`.zThe argument `model_name_or_path` was not specified while it is required when default `transformers` model are used.It is, therefore, used the default recommended model - .z num_layers=z is forbidden for z. Please use num_layers <= zXIt was not possible to retrieve the parameter `num_layers` from the model specification.css$|]}t|tot|dkVqdSrN) isinstancelistlenr0textrrr wr3zbert_score..css2|]*}t|to(t|dko(t|dtVqdSr)rrrstrrrrrrxscss&|]}t|tot|dtVqdS)r+N)rdictr rrrrr{sz%Predictions and references are empty.rQ)rErPf1hash)r()r( tokens_idfr(zInvalid input provided.)rZ num_workersr.rG)"rr8rrrrrModuleNotFoundErrorrrrrrrevaltoconfigZnum_hidden_layersAttributeErrorallupdaterWrvr rrr rBr~rTshapeZdatasetZsorting_indicesrx)$r{r|rUr&r'r$r}r*r)r(r%r~rrrrsrrXrh tokenizerZ_are_empty_listsZ_are_valid_listsZ_are_valid_tensorsZ output_dictruZtarget_datasetZ preds_datasetZ target_loaderZ preds_loaderrMrOrLrNrErPrSrrrr!s\      )NNFFFN)NNF)rrNNN)NF)NNFNNNFFNryrzrFrrFNN)5rarmtypingrrrrrrrr r7r Ztorch.nnr Ztorch.utils.datar Z4torchmetrics.functional.text.helper_embedding_metricr rrrrrrZtorchmetrics.utilitiesrZtorchmetrics.utilities.checksrrZtorchmetrics.utilities.importsrrrZ transformersrrr Z__doctest_skip__intrr%boolrBrKrTrWrgrqrvrxrZr!rrrrs(   $   Q