a  d_ @sdZddlZddlZddlZddlZddlZddlZddlZddlm Z ddl m Z m Z m Z ddlmZmZmZmZmZmZmZeeZddZe eejfd d d Zd d ZddZddZddZ ddZ!d$e eejfee e eejfe"e"e e eefe e e"efe ee"e ee ed ddZ#d%ee eejfe e eejfe"e"e e eefe e e"efe ee"e ee ed ddZ$d&ddZ%dd Z&d!Z'd"d#Z(dS)'z3Utilities to dynamically load objects from the Hub.N)Path)DictOptionalUnion)HF_MODULES_CACHE TRANSFORMERS_DYNAMIC_MODULE_NAME cached_fileextract_commit_hashis_offline_modeloggingtry_to_load_from_cachecCsPttjvrdStjttjtddttd}|sL|t dS)z_ Creates the cache directory for modules with an init, and adds it to the Python path. NTexist_ok __init__.py) rsyspathappendosmakedirsrexiststouch importlibinvalidate_caches) init_pathrj/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/transformers/dynamic_module_utils.pyinit_hf_modules(s   r)namecCsXttt|}|js&t|jtj|dd|d}|sT|t dS)zF Creates a dynamic module in the cache directory for modules. TrrN) rrrparentrcreate_dynamic_modulerrrrr)rZdynamic_module_pathrrrrr 8s   r cCsjt|ddd}|}Wdn1s,0Ytjd|tjd}|tjd|tjd7}tt|S)z Get the list of modules that are relatively imported in a module file. Args: module_file (`str` or `os.PathLike`): The module file to inspect. rutf-8encodingNz^\s*import\s+\.(\S+)\s*$flagsz^\s*from\s+\.(\S+)\s+import)openreadrefindall MULTILINElistset) module_filefcontentZrelative_importsrrrget_relative_importsHs &r1csd}|g}g|sg}|D]}|t|qt|jfdd|D}fdd|D}dd|D}t|dk}|qS)a) Get the list of all files that are needed for a given module. Note that this function recurses through the relative imports (if a imports b and b imports c, it will return module files for b and c). Args: module_file (`str` or `os.PathLike`): The module file to inspect. Fcsg|]}t|qSr)str).0m) module_pathrr mz-get_relative_import_files..csg|]}|vr|qSrrr3r/)all_relative_importsrrr6nr7cSsg|]}|dqS).pyrr8rrrr6or7r)extendr1rrlen)r.Z no_changeZfiles_to_checkZ new_importsr/Znew_import_filesr)r9r5rget_relative_import_filesZs   r=cCst|ddd}|}Wdn1s,0Ytjdd|tjtjBd}tjd|tjd}|tjd |tjd7}d d |D}tt|S) zA Extracts all the libraries that are imported in a file. r!r"r#Nz \s*try\s*:\s*.*?\s*except\s*.*?:r%z^\s*import\s+(\S+)\s*$z^\s*from\s+(\S+)\s+importcSs$g|]}|ds|ddqS).r) startswithsplit)r3imprrrr6r7zget_imports..) r'r(r)subr+DOTALLr*r,r-)filenamer/r0importsrrr get_importsws&rGc Csxt|}g}|D]0}zt|Wqty>||Yq0qt|dkrptdd|dd|dt|S)zi Check if the current Python environment contains all the libraries that are imported in a file. rz\This modeling file requires the following packages that were not found in your environment: z, z. Run `pip install  `)rGr import_module ImportErrorrr<joinr1)rErFZmissing_packagesrBrrr check_importss   rMcCs$|tjjd}t|}t||S)zY Import a module on the cache directory for modules and extract a class from it. r?)replacerrseprrJgetattr) class_namer5modulerrrget_class_in_modules rSF) pretrained_model_name_or_pathr. cache_dirforce_downloadresume_downloadproxiesuse_auth_tokenrevisionlocal_files_only repo_type _commit_hashc  Cstr|stdd}t|}tj|} | rD|tjjd} n"| dtjj} t |||| | d} g}z8t |||||||||| | d }| s| |kr| |Wn,t ytd|d|d Yn0t|}ttjj| }t|tt|}| |tjjdkr||r8t|t||sPt|||t|D]Z}|d }tj||}||rt|t||sTt|||tqTnt|| }||}|tjj|}t|||st|||t|D]L}||d st||d ||||||||d | |d qt|d kr|d urddd|D}| d urdn| d}d||}td|d|dtj||S)a Prepares Downloads a module from a local folder or a distant repo and returns its path inside the cached Transformers module. Args: pretrained_model_name_or_path (`str` or `os.PathLike`): This can be either: - a string, the *model id* of a pretrained model configuration hosted inside a model repo on huggingface.co. Valid model ids can be located at the root-level, like `bert-base-uncased`, or namespaced under a user or organization name, like `dbmdz/bert-base-german-cased`. - a path to a *directory* containing a configuration file saved using the [`~PreTrainedTokenizer.save_pretrained`] method, e.g., `./my_model_directory/`. module_file (`str`): The name of the module file containing the class to look for. cache_dir (`str` or `os.PathLike`, *optional*): Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force to (re-)download the configuration files and override the cached versions if they exist. resume_download (`bool`, *optional*, defaults to `False`): Whether or not to delete incompletely received file. Attempts to resume the download if such a file exists. proxies (`Dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}.` The proxies are used on each request. use_auth_token (`str` or *bool*, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated when running `huggingface-cli login` (stored in `~/.huggingface`). revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. local_files_only (`bool`, *optional*, defaults to `False`): If `True`, will only try to load the tokenizer configuration from local files. repo_type (`str`, *optional*): Specify the repo type (useful when downloading from a space for instance). Passing `use_auth_token=True` is required when you want to use a private model. Returns: `str`: The path to the module inside the cache. z+Offline mode: forcing local_files_only=TrueT/)rUrZr\) rUrVrXrWr[rYrZr\r]zCould not locate the z inside r?r:)rUrVrWrXrYrZr[r]rN cSsg|]}d|qS)z- rr8rrrr6Ar7z*get_cached_module_file..r>zs/zhttps://huggingface.co/z9A new version of the following files was downloaded from z: z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.) r loggerinfor2rrisdirrArOrNr r rEnvironmentErrorerrorrMrr rrrfilecmpcmpshutilcopyrrrLr get_cached_module_filer<warning)rTr.rUrVrWrXrYrZr[r\r]is_local submoduleZ cached_moduleZ new_filesZresolved_module_fileZmodules_neededZfull_submoduleZsubmodule_pathZ module_neededZmodule_needed_fileZ commit_hashZ repo_type_strurlrrrrjs=            rj) class_referencerTrUrVrWrXrYrZr[r\ code_revisionc  Kspd|vr|d\} }n|} |d\} }| dur>|| kr>|} t| | d|||||| || d }t||ddS)a  Extracts a class from a module file, present in the local folder or repository of a model. Calling this function will execute the code in the module file found locally or downloaded from the Hub. It should therefore only be called on trusted repos. Args: class_reference (`str`): The full name of the class to load, including its module and optionally its repo. pretrained_model_name_or_path (`str` or `os.PathLike`): This can be either: - a string, the *model id* of a pretrained model configuration hosted inside a model repo on huggingface.co. Valid model ids can be located at the root-level, like `bert-base-uncased`, or namespaced under a user or organization name, like `dbmdz/bert-base-german-cased`. - a path to a *directory* containing a configuration file saved using the [`~PreTrainedTokenizer.save_pretrained`] method, e.g., `./my_model_directory/`. This is used when `class_reference` does not specify another repo. module_file (`str`): The name of the module file containing the class to look for. class_name (`str`): The name of the class to import in the module. cache_dir (`str` or `os.PathLike`, *optional*): Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force to (re-)download the configuration files and override the cached versions if they exist. resume_download (`bool`, *optional*, defaults to `False`): Whether or not to delete incompletely received file. Attempts to resume the download if such a file exists. proxies (`Dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}.` The proxies are used on each request. use_auth_token (`str` or `bool`, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated when running `huggingface-cli login` (stored in `~/.huggingface`). revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. local_files_only (`bool`, *optional*, defaults to `False`): If `True`, will only try to load the tokenizer configuration from local files. repo_type (`str`, *optional*): Specify the repo type (useful when downloading from a space for instance). code_revision (`str`, *optional*, defaults to `"main"`): The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. Passing `use_auth_token=True` is required when you want to use a private model. Returns: `type`: The class, dynamically imported from the module. Examples: ```python # Download module `modeling.py` from huggingface.co and cache then extract the class `MyBertModel` from this # module. cls = get_class_from_dynamic_module("modeling.MyBertModel", "sgugger/my-bert-model") # Download module `modeling.py` from a given repo and cache then extract the class `MyBertModel` from this # module. cls = get_class_from_dynamic_module("sgugger/my-bert-model--modeling.MyBertModel", "sgugger/another-bert-model") ```z--r?Nr:)rUrVrWrXrYrZr[r\r>)rArjrSrN)rorTrUrVrWrXrYrZr[r\rpkwargsZrepo_idr.rQZ final_modulerrrget_class_from_dynamic_moduleMs&Y rrc sjdkr&tdd|ddSfdd}t|ttfrT|D] }||qDn|durd||g}tjjj}t |t |j }t ||| |t|D],}t |t |j }t ||| |q|S)a Save the modeling files corresponding to a custom model/configuration/tokenizer etc. in a given folder. Optionally adds the proper fields in a config. Args: obj (`Any`): The object for which to save the module files. folder (`str` or `os.PathLike`): The folder where to save. config (`PretrainedConfig` or dictionary, `optional`): A config in which to register the auto_map corresponding to this custom object. __main__z We can't save the code defining z in z as it's been defined in __main__. You should put this code in a separate module so we can include it in the saved folder and make it easier to share via the Hub.Nc s jj}|dd}|djj}d|vrd}d}jjdr|djj}tdddurtd}|j}|dd}|d|j}n|djj}||f}t|tr|di} || j <| |d<n*t|dddur||j j <n j |i|_ dS)Nr?r^ TokenizerZFastslow_tokenizer_classauto_map) __class__ __module__rA__name__endswithrP isinstancedictgetZ _auto_classrv) _config module_nameZ last_moduleZ full_nameruZfast_tokenizer_classZslow_tokenizerZslow_tok_module_nameZlast_slow_tok_modulervobjrr_set_auto_map_in_configs,     z3custom_object_save.._set_auto_map_in_config)rxrarkr{r,tuplermodules__file__rrrhrirr=) rfolderconfigrcfgresultZ object_file dest_fileZ needed_filerrrcustom_object_saves*        rcCs tddS)NaLoading this model requires you to execute the configuration file in that repo on your local machine. We asked if it was okay but did not get an answer. Make sure you have read the code there to avoid malicious use, then set the option `trust_remote_code=True` to remove this error.) ValueError)signumframerrr_raise_timeout_errorsrcCs|dur|rd}n|rtdkrttjttt|durxtd|d|d}|dvrfd}q6|dvr6d}q6tdn|rtdd|r|s|std|d |S) NFrzLoading zl requires to execute some code in that repo, you can inspect the content of the repository at https://hf.co/zX. You can dismiss this prompt by passing `trust_remote_code=True`. Do you accept? [y/N] )yesy1T)non0r>z requires you to execute the configuration file in that repo on your local machine. Make sure you have read the code there to avoid malicious use, then set the option `trust_remote_code=True` to remove this error.)TIME_OUT_REMOTE_CODEsignalSIGALRMralarminputlowerr)Ztrust_remote_codeZ model_nameZhas_local_codeZhas_remote_codeZanswerrrrresolve_trust_remote_codes0        r) NFFNNNFNN) NFFNNNFNN)N))__doc__rfrrr)rhrrpathlibrtypingrrrutilsrrr r r r r Z get_loggerryrarr2PathLiker r1r=rGrMrSboolrjrrrrrrrrrrs $    )  q I