a
d" @ sV d Z ddlmZmZmZmZ ddlmZmZ ddl m
Z
e
e Z
G dd deZdS ) z" Tokenization class for Perceiver. )DictListOptionalTuple )
AddedTokenPreTrainedTokenizer)loggingc s e Zd ZdZddgZd&d d fd
d
Zeeef d ddZ e
dd Z d'e e e
e e ee e d fdd
Zd(e e e
e e e e dddZee e ddd Zd d Zd d Zd!d" Zd)ee
e ee d#d$d%Z ZS )*PerceiverTokenizeraS
Construct a Perceiver tokenizer. The Perceiver simply uses raw bytes utf-8 encoding.
This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.
Args:
pad_token (`str`, *optional*, defaults to `"[PAD]"`):
The token used for padding, for example when batching sequences of different lengths.
bos_token (`str`, *optional*, defaults to `"[BOS]"`):
The BOS token (reserved in the vocab, but not actually used).
eos_token (`str`, *optional*, defaults to `"[EOS]"`):
The end of sequence token (reserved in the vocab, but not actually used).
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
mask_token (`str`, *optional*, defaults to `"[MASK]"`):
The MASK token, useful for masked language modeling.
cls_token (`str`, *optional*, defaults to `"[CLS]"`):
The CLS token (reserved in the vocab, but not actually used).
sep_token (`str`, *optional*, defaults to `"[SEP]"`):
The separator token, which is used when building a sequence from two sequences.
Z input_idsZattention_mask[PAD][BOS][EOS][MASK][CLS][SEP] N)returnc s t |trt|dddn|}t |tr4t|dddn|}t |trPt|dddn|}t |trlt|dddn|}t |trt|dddn|}t |trt|dddn|}t jf |||||||d| d| _| jd| jd| jd| j d| j
d | j d
i| _ t
| j | _d d | j D | _d S )
NF)lstriprstrip) pad_token bos_token eos_token
mask_token cls_token sep_tokenmodel_max_length r r c S s i | ]\}}||qS r! ).0kvr! r! }/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/transformers/models/perceiver/tokenization_perceiver.py
e z/PerceiverTokenizer.__init__..)
isinstancestrr super__init___utf_vocab_sizer r r r r r special_tokens_encoderlen_num_special_tokensitemsspecial_tokens_decoder) selfr r r r r r r kwargs __class__r! r% r+ < s6 zPerceiverTokenizer.__init__c C sD | j }|| j t| jD ] }t|}|t| j ||< q |S N)r- copyupdateadded_tokens_encoderranger, chrr. )r2 Zvocabitokenr! r! r% get_vocabg s
z PerceiverTokenizer.get_vocabc C s | j | j S r6 )r, r/ )r2 r! r! r%
vocab_sizeo s z PerceiverTokenizer.vocab_sizeF) token_ids_0 token_ids_1already_has_special_tokensr c sf |rt j||ddS |du r8dgdgt| dg S dgdgt| dg dgt| dg S )a
Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer `prepare_for_model` method.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
already_has_special_tokens (`bool`, *optional*, defaults to `False`):
Whether or not the token list is already formatted with special tokens for the model.
Returns:
`List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
T)r@ rA rB Nr r )r* get_special_tokens_maskr. )r2 r@ rA rB r4 r! r% rC s s z*PerceiverTokenizer.get_special_tokens_mask)r@ rA r c C s@ |du r | j g| | jg S | j g| | jg | | jg S dS )af
Build model inputs from a sequence or a pair of sequence for sequence classification tasks. A sequence has the
following format:
- single sequence: `[CLS] X [SEP]`
- pair of sequences: `[CLS] A [SEP] B [SEP]`
Args:
token_ids_0 (`List[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
N)Z cls_token_idZ sep_token_id)r2 r@ rA r! r! r% build_inputs_with_special_tokens s z3PerceiverTokenizer.build_inputs_with_special_tokens)textr c C s dd | dD }|S )zPTake as input a string and return a list of strings (tokens) for words/sub-wordsc S s g | ] }t |qS r! )r; )r" r< r! r! r%
r' z0PerceiverTokenizer._tokenize..utf-8)encode)r2 rE tokensr! r! r% _tokenize s z PerceiverTokenizer._tokenizec C sR || j v r| j | }n8|| jv r,| j| }n"t|dkr@| j}nt|| j }|S )z0Converts a token (str) in an id using the vocab.r )r- r9 r. Z unk_token_idordr/ )r2 r= Ztoken_idr! r! r% _convert_token_to_id s
z'PerceiverTokenizer._convert_token_to_idc C s> || j v r| j | }n$|| jv r,| j| }nt|| j }|S )z=Converts an index (integer) in a token (str) using the vocab.)r1 added_tokens_decoderr; r/ )r2 indexr= r! r! r% _convert_id_to_token s
z'PerceiverTokenizer._convert_id_to_tokenc C s d}|D ]~}|| j v r(| j | d}nV|| jv rD| j | d}n:|| jv rZ|d}n$|| jv rp|d}ntt|g}||7 }q|jddd}|S )z:Converts a sequence of tokens (string) in a single string.r' rG replace)errors)r1 rH rM r- r9 bytesrK decode)r2 rI bstringr= Z
tok_stringstringr! r! r% convert_tokens_to_string s
z+PerceiverTokenizer.convert_tokens_to_string)save_directoryfilename_prefixr c C s dS )Nr! r! )r2 rW rX r! r! r% save_vocabulary s z"PerceiverTokenizer.save_vocabulary)r r r
r r r r )NF)N)N)__name__
__module__ __qualname____doc__Zmodel_input_namesr+ r r) intr> propertyr? r r boolrC rD rJ rL rO rV r rY
__classcell__r! r! r4 r% r
s<
+
r
N)r] typingr r r r Ztokenization_utilsr r utilsr Z
get_loggerrZ loggerr
r! r! r! r% s