a  d9*@sdZddlZddlmZddlmZddlmZddlm Z m Z m Z m Z e eZe eejZdd d d ZGd d d e Ze eje_ejjdurejjjddddej_dS)z9 Processing saving/loading class for common processors. N)Path)custom_object_save)PreTrainedTokenizerBase)PushToHubMixin copy_funcdirect_transformers_importloggingrZFeatureExtractionMixinZImageProcessingMixin)Z AutoTokenizerZAutoFeatureExtractorZAutoImageProcessorc@sveZdZdZddgZdZdZdZddZddZ de d d d Z e d dZ e dddZe ddZeddZdS)ProcessorMixinza This is a mixin used to provide saving/loading functionality for all processor classes. Zfeature_extractor tokenizerNc Os0|D]}||jvrtd|dqt||jD]*\}}||vrRtd|dq0|||<q0t|t|jkrtdt|jdd|jdt|d|D]\}}t||d }t ||}t |t rt d d |D}n tt |}t ||std t |jd |d|dt|||qdS)NzUnexpected keyword argument .z!Got multiple values for argument zThis processor requires z arguments: z, z. Got z arguments instead._classcss |]}|durtt|VqdSNgetattrtransformers_module.0nrf/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/transformers/processing_utils.py Kz*ProcessorMixin.__init__..z Received a z for argument z, but a z was expected.) attributes TypeErrorziplen ValueErrorjoinitemsrAUTO_TO_BASE_CLASS_MAPPINGget isinstancetuplertype__name__setattr)selfargskwargskeyargattribute_name class_nameZ proper_classrrr__init__4s0      zProcessorMixin.__init__cs0fddjD}d|}jjd|S)Nc s&g|]}d|dtt|qS)z- z: )reprr)rnamer'rr Wrz+ProcessorMixin.__repr__.. z: )rr __class__r%)r'Zattributes_reprrr1r__repr__Vs zProcessorMixin.__repr__F) push_to_hubc  stj|dd|rT|dd}|d|tjjd}j|fi|}|}jdurfddj D}d d|D}t ||d j D]0} t | } t | d r| jj| |qjdurj D] } t | } t| tr| jd =q|rj|||||d ddS)a Saves the attributes of this processor (feature extractor, tokenizer...) in the specified directory so that it can be reloaded using the [`~ProcessorMixin.from_pretrained`] method. This class method is simply calling [`~feature_extraction_utils.FeatureExtractionMixin.save_pretrained`] and [`~tokenization_utils_base.PreTrainedTokenizerBase.save_pretrained`]. Please refer to the docstrings of the methods above for more information. Args: save_directory (`str` or `os.PathLike`): Directory where the feature extractor JSON file and the tokenizer files will be saved (directory will be created if it does not exist). push_to_hub (`bool`, *optional*, defaults to `False`): Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the repository you want to push to with `repo_id` (will default to the name of `save_directory` in your namespace). kwargs: Additional key word arguments passed along to the [`~utils.PushToHubMixin.push_to_hub`] method. T)exist_okcommit_messageNrepo_idcsg|]}t|qSr)r)rr,r1rrr2}rz2ProcessorMixin.save_pretrained..cSs g|]}t|tr|jn|qSr)r"r init_kwargs)rarrrr2~r)config_set_processor_classZauto_mapZuse_auth_token)r8token)osmakedirspopsplitpathsepZ _create_repoZ_get_files_timestamps _auto_classrrrhasattrr>r4r%save_pretrainedr"rr;Z_upload_modified_filesr!) r'Zsave_directoryr6r)r8r9Zfiles_timestampsattrsZconfigsr, attributerr1rrH[s8            zProcessorMixin.save_pretrainedcKs|j|fi|}||S)a Instantiate a processor associated with a pretrained model. This class method is simply calling the feature extractor [`~feature_extraction_utils.FeatureExtractionMixin.from_pretrained`], image processor [`~image_processing_utils.ImageProcessingMixin`] and the tokenizer [`~tokenization_utils_base.PreTrainedTokenizer.from_pretrained`] methods. Please refer to the docstrings of the methods above for more information. Args: pretrained_model_name_or_path (`str` or `os.PathLike`): This can be either: - a string, the *model id* of a pretrained feature_extractor hosted inside a model repo on huggingface.co. Valid model ids can be located at the root-level, like `bert-base-uncased`, or namespaced under a user or organization name, like `dbmdz/bert-base-german-cased`. - a path to a *directory* containing a feature extractor file saved using the [`~SequenceFeatureExtractor.save_pretrained`] method, e.g., `./my_model_directory/`. - a path or url to a saved feature extractor JSON *file*, e.g., `./my_model_directory/preprocessor_config.json`. **kwargs Additional keyword arguments passed along to both [`~feature_extraction_utils.FeatureExtractionMixin.from_pretrained`] and [`~tokenization_utils_base.PreTrainedTokenizer.from_pretrained`]. )_get_arguments_from_pretrained)clspretrained_model_name_or_pathr)r(rrrfrom_pretrainedszProcessorMixin.from_pretrained AutoProcessorcCsDt|ts|j}ddlmm}t||s:t|d||_dS)a Register this class with a given auto class. This should only be used for custom feature extractors as the ones in the library are already mapped with `AutoProcessor`. This API is experimental and may have some slight breaking changes in the next releases. Args: auto_class (`str` or `type`, *optional*, defaults to `"AutoProcessor"`): The auto class to register this new feature extractor with. rNz is not a valid auto class.) r"strr%Ztransformers.models.automodelsautorGrrF)rLZ auto_classZ auto_modulerrrregister_for_auto_classs   z&ProcessorMixin.register_for_auto_classc Ksg}|jD]}t||d}t|trjtdd|D}|dd}|r`|ddur`|d}qt|d}n tt|}||j|fi|q |S)Nr css$|]}|durtt|ndVqdSrrrrrrrrz@ProcessorMixin._get_arguments_from_pretrained..use_fastTrr)rrr"r#r!rappendrN) rLrMr)r(r,r-classesrTZattribute_classrrrrKs      z-ProcessorMixin._get_arguments_from_pretrainedcCst||jd}t|ddS)Nrmodel_input_names)rr)r'Zfirst_attributerrrrWsz ProcessorMixin.model_input_names)F)rO)r% __module__ __qualname____doc__rZfeature_extractor_classZtokenizer_classrFr.r5boolrH classmethodrNrSrKpropertyrWrrrrr (s "> !  r processorrOzprocessor files)objectZ object_classZ object_files)rZr@pathlibrZdynamic_module_utilsrZtokenization_utils_baserutilsrrrr Z get_loggerr%logger__file__parentrr r r6formatrrrrs$    F