a
d4; @ s d Z ddlZddlZddlZddlmZ ddlmZmZm Z m
Z
m Z ddl Z
ddlmZ ddlmZ eeZdd iZdd
d d d
ddiZddddddZG dd deZdS )z! Tokenization class for model T5. N)copyfile)AnyDictListOptionalTuple )PreTrainedTokenizer)logging
vocab_filez spiece.modelz9https://huggingface.co/t5-small/resolve/main/spiece.modelz8https://huggingface.co/t5-base/resolve/main/spiece.modelz9https://huggingface.co/t5-large/resolve/main/spiece.modelz6https://huggingface.co/t5-3b/resolve/main/spiece.modelz7https://huggingface.co/t5-11b/resolve/main/spiece.model)zt5-smallzt5-basezt5-largezt5-3bzt5-11bi c s\ e Zd ZdZeZeZeZ ddgZ
d2e e e
ef dd fd
d
Zed d
Zedd Zdd Zd3ee e ee eee d fdd
Zdd Zdd Zee ee ddd Zd4ee e ee ee d d d Z d5ee e ee ee d d d!Z d"d# Z d$d% Z e
ee
d&d'd(Z d)d* Z!d+d, Z"d-d. Z#d6e
e e
e$e
d/d0d1Z% Z&S )7 T5Tokenizera
Construct a T5 tokenizer. Based on [SentencePiece](https://github.com/google/sentencepiece).
This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.
Args:
vocab_file (`str`):
[SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that
contains the vocabulary necessary to instantiate a tokenizer.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
extra_ids (`int`, *optional*, defaults to 100):
Add a number of extra ids added to the vocabulary for use as sentinels. These tokens are
accessible as "" where "{%d}" is a number between 0 and extra_ids-1. These tokens can be
retrieved by calling get_sentinel_tokens method and token ids can be by calling get_sentinel_token_ids
method
additional_special_tokens (`List[str]`, *optional*):
Additional special tokens used by the tokenizer.
sp_model_kwargs (`dict`, *optional*):
Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for
SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things,
to set:
- `enable_sampling`: Enable subword regularization.
- `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout.
- `nbest_size = {0,1}`: No sampling is performed.
- `nbest_size > 1`: samples from the nbest_size results.
- `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice)
using forward-filtering-and-backward-sampling algorithm.
- `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for
BPE-dropout.
Attributes:
sp_model (`SentencePieceProcessor`):
The *SentencePiece* processor that is used for every conversion (string, tokens and IDs).
Z input_idsZattention_maskd N)sp_model_kwargsreturnc
s |dkr$|d u r$dd t |D }nD|dkrh|d urhtttdd |} | |krhtd| d| d|d u rti n|| _t jf |||||| jd | || _|| _ t
j f i | j| _ | j
| d S )
Nr c S s g | ]}d | dqS )
.0ir r o/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/transformers/models/t5/tokenization_t5.py
z(T5Tokenizer.__init__..c S s t dt| v S )NZextra_id)boolstrxr r r r z&T5Tokenizer.__init__..zBoth extra_ids (z!) and additional_special_tokens (zk) are provided to T5Tokenizer. In this case the additional_special_tokens must include the extra_ids tokens) eos_token unk_token pad_token extra_idsadditional_special_tokensr )rangelensetfilter
ValueErrorr super__init__r
_extra_idsspmSentencePieceProcessorsp_modelLoad)
selfr r! r" r# r$ r% r kwargsZ extra_tokens __class__r r r, r s.
zT5Tokenizer.__init__c C sZ | t jv rVt j| }|d ur(||kr(|S |d u rVtd| d| d| d| d t |S )NzGThis tokenizer was incorrectly instantiated with a model max length of z which will be corrected in Transformers v5.
For now, this behavior is kept to avoid breaking backwards compatibility when padding/encoding with `truncation is True`.
- Be aware that you SHOULD NOT rely on z( automatically truncating your input to zM when padding/encoding.
- If you want to encode/pad to sequences longer than z you can either instantiate this tokenizer with `model_max_length` or pass `max_length` when encoding/padding.
- To avoid this warning, please instantiate this tokenizer with `model_max_length` set to your preferred value.)r max_model_input_sizeswarningswarn
FutureWarning)Z pretrained_model_name_or_pathZmax_model_lengthZinit_max_model_lengthZdeprecated_max_model_lengthr r r !_eventually_correct_t5_max_length s$
z-T5Tokenizer._eventually_correct_t5_max_lengthc C s | j | j S )N)r0 get_piece_sizer- r2 r r r
vocab_size s zT5Tokenizer.vocab_sizec s( fddt jD }| j |S )Nc s i | ]} ||qS r )Zconvert_ids_to_tokensr r< r r
r z)T5Tokenizer.get_vocab..)r& r= updateZadded_tokens_encoder)r2 Zvocabr r< r get_vocab s zT5Tokenizer.get_vocabF) token_ids_0 token_ids_1already_has_special_tokensr c sZ |rt j||ddS |du r2dgt| dg S dgt| dg dgt| dg S )a
Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer `prepare_for_model` method.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
already_has_special_tokens (`bool`, *optional*, defaults to `False`):
Whether or not the token list is already formatted with special tokens for the model.
Returns:
`List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
T)rA rB rC Nr )r+ get_special_tokens_maskr' )r2 rA rB rC r4 r r rE s z#T5Tokenizer.get_special_tokens_maskc C s t ttdd | jS )Nc S s t td| d uS )Nz)r researchr r r r r r z1T5Tokenizer.get_sentinel_tokens..)listr( r) r% r< r r r get_sentinel_tokens s z T5Tokenizer.get_sentinel_tokensc s fdd D S )Nc s g | ]} |qS r )_convert_token_to_id)r tokenr< r r r r z6T5Tokenizer.get_sentinel_token_ids..)rI r< r r< r get_sentinel_token_ids s z"T5Tokenizer.get_sentinel_token_ids) token_idsr c C sB t |dkr2|d | jkr2td| j d |S || jg S dS )z.Do not add eos again if user already added it.r zThis sequence already has zQ. In future versions this behavior may lead to duplicated eos tokens being added.N)r' eos_token_idr7 r8 r! )r2 rM r r r _add_eos_if_not_present s z#T5Tokenizer._add_eos_if_not_present)rA rB r c C s< | j g}|du r"t|| dg S t|| | | dg S )a
Create a mask from the two sequences passed to be used in a sequence-pair classification task. T5 does not make
use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of zeros.
Nr )rO r' )r2 rA rB Zeosr r r $create_token_type_ids_from_sequences s z0T5Tokenizer.create_token_type_ids_from_sequencesc C s, | |}|du r|S | |}|| S dS )a
Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
adding special tokens. A sequence has the following format:
- single sequence: `X `
- pair of sequences: `A B `
Args:
token_ids_0 (`List[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
N)rP )r2 rA rB r r r build_inputs_with_special_tokens s
z,T5Tokenizer.build_inputs_with_special_tokensc C s | j }d |d< |S )Nr0 )__dict__copy)r2 stater r r __getstate__ s
zT5Tokenizer.__getstate__c C s< || _ t| dsi | _tjf i | j| _| j| j d S )Nr )rS hasattrr r. r/ r0 r1 r )r2 dr r r __setstate__ s
zT5Tokenizer.__setstate__)textr c C s | j j|tdS )zPTake as input a string and return a list of strings (tokens) for words/sub-words)Zout_type)r0 encoder )r2 rZ r r r _tokenize) s zT5Tokenizer._tokenizec C s> | dr2td|}t|d}| j| d S | j|S )z0Converts a token (str) in an id using the vocab.r zrD )
startswithrF matchintgroupr= r0 Z piece_to_id)r2 rK r^ numr r r rJ - s
z T5Tokenizer._convert_token_to_idc C s6 || j k r | j |}nd| jd | d}|S )z=Converts an index (integer) in a token (str) using the vocab.r rD r )r0 r; Z IdToPiecer= )r2 indexrK r r r _convert_id_to_token5 s z T5Tokenizer._convert_id_to_tokenc C sp g }d}d}|D ]F}|| j v rH|s*|d7 }|| j|| 7 }d}g }q|| d}q|| j|7 }| S )z:Converts a sequence of tokens (string) in a single string. F T)Zall_special_tokensr0 decodeappendstrip)r2 tokensZcurrent_sub_tokensZ
out_stringZprev_is_specialrK r r r convert_tokens_to_string= s
z$T5Tokenizer.convert_tokens_to_string)save_directoryfilename_prefixr c C s t j|s"td| d d S t j||r6|d ndtd }t j| jt j|krzt j | jrzt
| j| nLt j | jst |d$}| j
}|| W d n1 s0 Y |fS )NzVocabulary path (z) should be a directory-rd r wb)ospathisdirloggererrorjoinVOCAB_FILES_NAMESabspathr isfiler openr0 Zserialized_model_protowrite)r2 rk rl Zout_vocab_filefiZcontent_spiece_modelr r r save_vocabularyP s (
(zT5Tokenizer.save_vocabulary)r
r r r NN)NF)N)N)N)'__name__
__module__ __qualname____doc__ru Zvocab_files_namesPRETRAINED_VOCAB_FILES_MAPZpretrained_vocab_files_map&PRETRAINED_POSITIONAL_EMBEDDINGS_SIZESr6 Zmodel_input_namesr r r r r, staticmethodr: propertyr= r@ r r_ r rE rI rL rP rQ rR rV rY r\ rJ rc rj r r{
__classcell__r r r4 r r 7 sZ 5 *
r )r ro rF r7 shutilr typingr r r r r Z
sentencepiecer. Ztokenization_utilsr utilsr
Z
get_loggerr| rr ru r r r r r r r s2