a
d: @ s d Z ddlZddlZddlZddlmZ ddlmZ ddlm Z m
Z
ddl m Z ddl
mZ dd lmZ dd
lmZmZmZ d d lmZ d d
lmZmZmZmZ d dl m Z d dl m Z d dl m!Z! e"e#Z$eg dZ%eee%Z&e'dddZ(G dd dZ)dS )z AutoProcessor class. N) OrderedDict )PretrainedConfig) get_class_from_dynamic_moduleresolve_trust_remote_code)FeatureExtractionMixin)ImageProcessingMixin)TOKENIZER_CONFIG_FILE)FEATURE_EXTRACTOR_NAMEget_file_from_repologging )_LazyAutoMapping)CONFIG_MAPPING_NAMES
AutoConfigmodel_type_to_module_name!replace_list_option_in_docstrings)AutoFeatureExtractor)AutoImageProcessor)
AutoTokenizer)&)ZalignZAlignProcessor)ZaltclipZAltCLIPProcessor)ZblipZ
BlipProcessor)zblip-2ZBlip2Processor)Z bridgetowerZBridgeTowerProcessor)Z chinese_clipZChineseCLIPProcessor)ZclapZ
ClapProcessor)Zclip
CLIPProcessor)ZclipsegZCLIPSegProcessor)ZflavaZFlavaProcessor)gitZ GitProcessor)Zgroupvitr )ZhubertWav2Vec2Processor)Z
layoutlmv2ZLayoutLMv2Processor)Z
layoutlmv3ZLayoutLMv3Processor)ZmarkuplmZMarkupLMProcessor)ZmctctZMCTCTProcessor)zmgp-strZMgpstrProcessor)Z oneformerZOneFormerProcessor)ZowlvitZOwlViTProcessor)Z
pix2structZPix2StructProcessor)ZsamZ SamProcessor)Zsewr )zsew-dr )Zspeech_to_textZSpeech2TextProcessor)Zspeech_to_text_2ZSpeech2Text2Processor)Zspeecht5ZSpeechT5Processor)ZtrocrZTrOCRProcessor)ZtvltZ
TvltProcessor)Z unispeechr )z
unispeech-satr )ZviltZ
ViltProcessor)zvision-text-dual-encoderZ VisionTextDualEncoderProcessor)Zwav2vec2r )zwav2vec2-conformerr )Zwavlmr )ZwhisperZWhisperProcessor)ZxclipZXCLIPProcessor)
class_namec C s t D ]R\}}| |v rt|}td| d}zt|| W S tyX Y qY q0 qtj D ] }t|dd | krf| S qftd}t
|| rt|| S d S )N.ztransformers.models__name__Z transformers) PROCESSOR_MAPPING_NAMESitemsr importlib
import_modulegetattrAttributeErrorPROCESSOR_MAPPINGZ_extra_contentvalueshasattr)r module_nameZ
processorsmodule processorZ main_module r( q/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/transformers/models/auto/processing_auto.pyprocessor_class_from_nameX s
r* c @ s8 e Zd ZdZdd Zeeedd Ze dd Z
dS )
AutoProcessora
This is a generic processor class that will be instantiated as one of the processor classes of the library when
created with the [`AutoProcessor.from_pretrained`] class method.
This class cannot be instantiated directly using `__init__()` (throws an error).
c C s t dd S )Nz}AutoProcessor is designed to be instantiated using the `AutoProcessor.from_pretrained(pretrained_model_name_or_path)` method.)EnvironmentError)selfr( r( r) __init__x s zAutoProcessor.__init__c s: dd} dd}d d< d}d} fddttj D }t|tfi |}|durtj|fi \} }
| dd}d | d
i v r| d
d }|dur|du rt
j |fi \} }
| dd}d | d
i v r| d
d }|du rtt|t fi |} | durtt
| d d } t| } W d n1 s@0 Y | dd}d | d
i v rt| d
d }|du rt|tstj|fd|i }t|dd}t|d
rd |jv r|jd }|durt|}|du}
|dupt|tv }t||||
}|
rL|rLt||fi } d
d}
|j|fd|i S |durl|j|fd|i S t|tv rtt| j|fi S zt j|fd|i W S t y$ z t j|fd|i W Y S t y Y n0 z t j|fd|i W Y S t y Y n0 Y n0 t d| ddS )a
Instantiate one of the processor classes of the library from a pretrained model vocabulary.
The processor class to instantiate is selected based on the `model_type` property of the config object (either
passed as an argument or loaded from `pretrained_model_name_or_path` if possible):
List options
Params:
pretrained_model_name_or_path (`str` or `os.PathLike`):
This can be either:
- a string, the *model id* of a pretrained feature_extractor hosted inside a model repo on
huggingface.co. Valid model ids can be located at the root-level, like `bert-base-uncased`, or
namespaced under a user or organization name, like `dbmdz/bert-base-german-cased`.
- a path to a *directory* containing a processor files saved using the `save_pretrained()` method,
e.g., `./my_model_directory/`.
cache_dir (`str` or `os.PathLike`, *optional*):
Path to a directory in which a downloaded pretrained model feature extractor should be cached if the
standard cache should not be used.
force_download (`bool`, *optional*, defaults to `False`):
Whether or not to force to (re-)download the feature extractor files and override the cached versions
if they exist.
resume_download (`bool`, *optional*, defaults to `False`):
Whether or not to delete incompletely received file. Attempts to resume the download if such a file
exists.
proxies (`Dict[str, str]`, *optional*):
A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128',
'http://hostname': 'foo.bar:4012'}.` The proxies are used on each request.
use_auth_token (`str` or *bool*, *optional*):
The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated
when running `huggingface-cli login` (stored in `~/.huggingface`).
revision (`str`, *optional*, defaults to `"main"`):
The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a
git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any
identifier allowed by git.
return_unused_kwargs (`bool`, *optional*, defaults to `False`):
If `False`, then this function returns just the final feature extractor object. If `True`, then this
functions returns a `Tuple(feature_extractor, unused_kwargs)` where *unused_kwargs* is a dictionary
consisting of the key/value pairs whose keys are not feature extractor attributes: i.e., the part of
`kwargs` which has not been used to update `feature_extractor` and is otherwise ignored.
trust_remote_code (`bool`, *optional*, defaults to `False`):
Whether or not to allow for custom models defined on the Hub in their own modeling files. This option
should only be set to `True` for repositories you trust and in which you have read the code, as it will
execute code present on the Hub on your local machine.
kwargs (`Dict[str, Any]`, *optional*):
The values in kwargs of any keys which are feature extractor attributes will be used to override the
loaded values. Behavior concerning key/value pairs whose keys are *not* feature extractor attributes is
controlled by the `return_unused_kwargs` keyword parameter.
Passing `use_auth_token=True` is required when you want to use a private model.
Examples:
```python
>>> from transformers import AutoProcessor
>>> # Download processor from huggingface.co and cache.
>>> processor = AutoProcessor.from_pretrained("facebook/wav2vec2-base-960h")
>>> # If processor files are in a directory (e.g. processor was saved using *save_pretrained('./test/saved_model/')*)
>>> # processor = AutoProcessor.from_pretrained("./test/saved_model/")
```configNtrust_remote_codeTZ
_from_autoc s i | ]}| v r| | qS r( r( ).0keykwargsr( r)
s z1AutoProcessor.from_pretrained..processor_classr+ auto_mapzutf-8)encodingZ
code_revisionz!Unrecognized processing class in z. Can't instantiate a processor, a tokenizer, an image processor or a feature extractor for this model. Make sure the repository containsthe files of at least one of those processing classes.)!popinspect signaturer
parameterskeysr
r Zget_image_processor_dictgetr Zget_feature_extractor_dictr openjsonload
isinstancer r from_pretrainedr r$ r7 r* typer" r r r Exceptionr r
ValueError)clsZ pretrained_model_name_or_pathr4 r/ r0 r6 Zprocessor_auto_mapZget_file_from_repo_kwargsZpreprocessor_config_fileZ config_dict_Ztokenizer_config_filereaderZhas_remote_codeZhas_local_coder( r3 r) rC ~ s F
*
z AutoProcessor.from_pretrainedc C s t | | dS )a
Register a new processor for this class.
Args:
config_class ([`PretrainedConfig`]):
The configuration corresponding to the model to register.
processor_class ([`FeatureExtractorMixin`]): The processor to register.
N)r" register)Z config_classr6 r( r( r) rJ . s
zAutoProcessor.registerN) r
__module__ __qualname____doc__r. classmethodr r rC staticmethodrJ r( r( r( r) r+ p s /r+ )*rM r r: r@ collectionsr Zconfiguration_utilsr Zdynamic_module_utilsr r Zfeature_extraction_utilsr Zimage_processing_utilsr Ztokenization_utilsr utilsr
r r Z auto_factoryr Zconfiguration_autor r r r Zfeature_extraction_autor Zimage_processing_autor Ztokenization_autor Z
get_loggerr loggerr r" strr* r+ r( r( r( r) s,
+