a
d) @ s^ d Z ddlZddlmZmZmZmZ ddlmZm Z ddl
m Z e e
ZG dd de ZdS ) z# Tokenization class for model ByT5. N)DictListOptionalTuple )
AddedTokenPreTrainedTokenizer)loggingc s e Zd ZdZddgZd&dd fd
d
Zed d
Zd'ee e
ee e ee d fdd
Z ee ee dddZ
d(ee e
ee ee dddZd)ee e
ee ee dddZeee ddd Zd d Zd d Zd!d" Zd*ee
e ee d#d$d%Z ZS )+
ByT5Tokenizera
Construct a ByT5 tokenizer. ByT5 simply uses raw bytes utf-8 encoding.
This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.
Args:
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
extra_ids (`int`, *optional*, defaults to 100):
Add a number of extra ids added to the end of the vocabulary for use as sentinels. These tokens are
accessible as "" where "{%d}" is a number between 0 and extra_ids-1. Extra tokens are
indexed from the end of the vocabulary up to beginning ("" is the last token in the vocabulary
like in ByT5 preprocessing see
[here](https://github.com/google-research/text-to-text-transfer-transformer/blob/9fd7b14a769417be33bc6c850f9598764913c833/t5/data/preprocessors.py#L2117)).
additional_special_tokens (`List[str]`, *optional*):
Additional special tokens used by the tokenizer.
Z input_idsZattention_mask} N)returnc sT |dkr$|d u r$dd t |D }nD|dkrh|d urhtttdd |}||krhtd| d| dt|trt|d d d
n|}t|trt|d d d
n|}t|trt|d d d
n|}t j f |||||d | || _
d | _ | j d| j
d
| jdi| _t| j| _t|}t|D ] \} }
| j| | | j|
< qdd | j D | _d S )Nr c S s g | ]}d | dqS )z
.0ir r s/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/transformers/models/byt5/tokenization_byt5.py
J z*ByT5Tokenizer.__init__..c S s t dt| v S )NZextra_id)boolstr)xr r r M r z(ByT5Tokenizer.__init__..zBoth extra_ids (z!) and additional_special_tokens (zm) are provided to ByT5Tokenizer. In this case the additional_special_tokens must include the extra_ids tokensF)lstriprstrip) eos_token unk_token pad_token extra_idsadditional_special_tokens c S s i | ]\}}||qS r r )r kvr r r
p r z*ByT5Tokenizer.__init__..)rangelensetfilter
ValueError
isinstancer r super__init__
_extra_ids_utf_vocab_sizer r r special_tokens_encoder_num_special_tokens enumerate
vocab_sizeitemsspecial_tokens_decoder) selfr r r r! r" kwargsZ extra_tokensnr token __class__r r r0 ? s>
zByT5Tokenizer.__init__c C s | j | j | j S )N)r2 r4 r1 )r9 r r r r6 r s zByT5Tokenizer.vocab_sizeF) token_ids_0 token_ids_1already_has_special_tokensr c sZ |rt j||ddS |du r2dgt| dg S dgt| dg dgt| dg S )a
Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer `prepare_for_model` method.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
already_has_special_tokens (`bool`, *optional*, defaults to `False`):
Whether or not the token list is already formatted with special tokens for the model.
Returns:
`List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
T)r? r@ rA Nr r$ )r/ get_special_tokens_maskr* )r9 r? r@ rA r= r r rB v s z%ByT5Tokenizer.get_special_tokens_mask) token_idsr c C sB t |dkr2|d | jkr2td| j d |S || jg S dS )z.Do not add eos again if user already added it.r zThis sequence already has zQ. In future versions this behavior may lead to duplicated eos tokens being added.N)r* eos_token_idwarningswarnr )r9 rC r r r _add_eos_if_not_present s z%ByT5Tokenizer._add_eos_if_not_present)r? r@ r c C s< | j g}|du r"t|| dg S t|| | | dg S )a
Create a mask from the two sequences passed to be used in a sequence-pair classification task. ByT5 does not
make use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of zeros.
Nr )rE r* )r9 r? r@ Zeosr r r $create_token_type_ids_from_sequences s z2ByT5Tokenizer.create_token_type_ids_from_sequencesc C s, | |}|du r|S | |}|| S dS )a
Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
adding special tokens. A sequence has the following format:
- single sequence: `X `
- pair of sequences: `A B `
Args:
token_ids_0 (`List[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
N)rH )r9 r? r@ r r r build_inputs_with_special_tokens s
z.ByT5Tokenizer.build_inputs_with_special_tokens)textr c C s dd | dD }|S )zPTake as input a string and return a list of strings (tokens) for words/sub-wordsc S s g | ] }t |qS r )chrr r r r r r z+ByT5Tokenizer._tokenize..utf-8)encode)r9 rK tokensr r r _tokenize s zByT5Tokenizer._tokenizec C sR || j v r| j | }n8|| jv r,| j| }n"t|dkr@| j}nt|| j }|S )z0Converts a token (str) in an id using the vocab.r$ )r3 added_tokens_encoderr* Z unk_token_idordr4 )r9 r< Ztoken_idr r r _convert_token_to_id s
z"ByT5Tokenizer._convert_token_to_idc C s( || j v r| j | }nt|| j }|S )z=Converts an index (integer) in a token (str) using the vocab.)r8 rL r4 )r9 indexr< r r r _convert_id_to_token s
z"ByT5Tokenizer._convert_id_to_tokenc C s d}|D ]~}|| j v r(| j | d}nV|| jv rD| j | d}n:|| jv rZ|d}n$|| jv rp|d}ntt|g}||7 }q|jddd}|S )z:Converts a sequence of tokens (string) in a single string.r rM ignore)errors)r8 rN Zadded_tokens_decoderr3 rQ bytesrR decode)r9 rO bstringr< Z
tok_stringstringr r r convert_tokens_to_string s
z&ByT5Tokenizer.convert_tokens_to_string)save_directoryfilename_prefixr c C s dS )Nr r )r9 r] r^ r r r save_vocabulary s z ByT5Tokenizer.save_vocabulary)r r r
r N)NF)N)N)N)__name__
__module__ __qualname____doc__Zmodel_input_namesr0 propertyr6 r intr r rB rH rI rJ r rP rS rU r\ r r_
__classcell__r r r= r r
sB 3
r
)rc rF typingr r r r Ztokenization_utilsr r utilsr Z
get_loggerr` loggerr
r r r r s