a  ds @sddlZddlZddlZddlZddlZddlZddlZddlmZm Z m Z m Z m Z ddl mZmZmZmZddlmZmZmZddlmZmZmZddlmZddlmZdd lmZm Z m!Z!m"Z"m#Z#m$Z$e$%e&Z'e"rddl(Z(e!rdd l)m*Z*d Z+d0d d Z,dZ-GdddZ.Gddde.Z/Gddde.Z0e.dddZ1ddZ2ddddddd d!d"d#d$ Z3d%d&Z4d'd(Z5d1d*d+Z6d,d-Z7Gd.d/d/Z8dS)2N)AnyDictListOptionalUnion) create_repohf_hub_downloadmetadata_update upload_folder)RepositoryNotFoundErrorbuild_hf_headers get_session)custom_object_saveget_class_from_dynamic_module get_imports) is_pil_image) AutoProcessor) CONFIG_NAME cached_fileis_accelerate_availableis_torch_availableis_vision_availablelogging)send_to_devicetool_config.jsonc Ks|dur |Szt|tfddi|WdStyzt|tfddi|WYdStyvtd|dYntyYYdS0YntyYdS0dS)N repo_typespacemodel`z9` does not seem to be a valid repo identifier on the Hub.)rTOOL_CONFIG_FILEr EnvironmentError Exception)repo_idr hub_kwargsr%`/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/transformers/tools/base.py get_repo_type6s    r'zufrom transformers import launch_gradio_demo from {module_name} import {class_name} launch_gradio_demo({class_name}) c@seZdZUdZdZeed<dZeed<eeed<eeed<dd Z d d Z d d Z ddZ e deeeeeedddZdeeeeeeeefeedddZeddZdS)Toola A base class for the functions used by the agent. Subclass this and implement the `__call__` method as well as the following class attributes: - **description** (`str`) -- A short description of what your tool does, the inputs it expects and the output(s) it will return. For instance 'This is a tool that downloads a file from a `url`. It takes the `url` as input, and returns the text contained in the file'. - **name** (`str`) -- A performative name that will be used for your tool in the prompt to the agent. For instance `"text-classifier"` or `"image_generator"`. - **inputs** (`List[str]`) -- The list of modalities expected for the inputs (in the same order as in the call). Modalitiies should be `"text"`, `"image"` or `"audio"`. This is only used by `launch_gradio_demo` or to make a nice space from your tool. - **outputs** (`List[str]`) -- The list of modalities returned but the tool (in the same order as the return of the call method). Modalitiies should be `"text"`, `"image"` or `"audio"`. This is only used by `launch_gradio_demo` or to make a nice space from your tool. You can also override the method [`~Tool.setup`] if your tool as an expensive operation to perform before being usable (such as loading a model). [`~Tool.setup`] will be called the first time you use your tool, but not at instantiation. zThis is a tool that ... descriptionnameinputsoutputscOs d|_dS)NFis_initializedselfargskwargsr%r%r&__init__lsz Tool.__init__cOstdS)Nz-Write this method in your subclass of `Tool`.)NotImplementedr0r%r%r&__call__osz Tool.__call__cCs d|_dS)z Overwrite this method here for any operation that is expensive and needs to be executed before you start using your tool. Such as loading a big model. TNr.r1r%r%r&setuprsz Tool.setupc Cstj|dd|jdkr.td|d|dt||}|jj}|dd}|d|jj}tj |d }tj |rt |d d d }t |}Wd q1s0Yni}||j|jd}t |dd d *}|t j|ddddWd n1s0Ytj |d} t | dd d (}|tj||jjdWd n1s^0Ytj |d} g} |D]} | t| q~tt| } t | dd d $}|d | dWd n1s0Yd S)a Saves the relevant code files for your tool so it can be pushed to the Hub. This will copy the code of your tool in `output_dir` as well as autogenerate: - a config file named `tool_config.json` - an `app.py` file so that your tool can be converted to a space - a `requirements.txt` containing the names of the module used by your tool (as detected when inspecting its code) You should only use this method to save tools that are defined in a separate module (not `__main__`). Args: output_dir (`str`): The folder in which you want to save your tool. T)exist_ok__main__z We can't save the code defining z in z{ as it's been defined in __main__. You have to put this code in a separate module so we can include it in the saved folder..rrutf-8encodingN) tool_classr)r+wr)indent sort_keys zapp.py) module_name class_namezrequirements.txt)osmakedirs __module__ ValueErrorr __class__split__name__pathjoinisfileopenjsonloadr)r+writedumpsAPP_FILE_TEMPLATEformatextendrlistset) r1 output_dirZ module_filesrFZ last_moduleZ full_name config_filefZ tool_configZapp_fileZrequirements_fileZimportsmoduler%r%r&saveys6   *:8 z Tool.saveNF)r# model_repo_idtokenremotec s|r2|dur2t}||vr*td|d||}gdfdd|D}t|fi||d<t|tfd|i|d d d }|du} |durt|tfd|i|d d d }|durt|d t|d d } t | } Wdn1s0Y| s(d| vrt|d| d} n| } | d} t | |fd|i|} t | j dkrd| d| _ | j | dkrt| jd| d| _ t | jdkr| d| _| j| dkrt| jd| d| _|rt||| dS| |fd|i|S)aw Loads a tool defined on the Hub. Args: repo_id (`str`): The name of the repo on the Hub where your tool is defined. model_repo_id (`str`, *optional*): If your tool uses a model and you want to use a different model than the default, you can pass a second repo ID or an endpoint url to this argument. token (`str`, *optional*): The token to identify you on hf.co. If unset, will use the token generated when running `huggingface-cli login` (stored in `~/.huggingface`). remote (`bool`, *optional*, defaults to `False`): Whether to use your tool by downloading the model or (if it is available) with an inference endpoint. kwargs (additional keyword arguments, *optional*): Additional keyword arguments that will be split in two: all arguments relevant to the Hub (such as `cache_dir`, `revision`, `subfolder`) will be used when downloading the files for your tool, and the others will be passed along to its init. N'Could not infer a default endpoint for :, you need to pass one using the `model_repo_id` argument.) cache_dirZforce_downloadZresume_downloadproxiesrevisionrZ subfolderZlocal_files_onlycsi|]\}}|vr||qSr%r%).0kvZhub_kwargs_namesr%r& z!Tool.from_hub..ruse_auth_tokenF)Z%_raise_exceptions_for_missing_entriesZ'_raise_exceptions_for_connection_errorszY does not appear to provide a valid configuration in `tool_config.json` or `config.json`.r>r? custom_toolzO does not provide a mapping to custom tools in its configuration `config.json`.rArr+z_ implements a different name in its configuration and class. Using the tool configuration name.r)zm implements a different description in its configuration and class. Using the tool configuration description.rbrArb)get_default_endpointsrKitemsr'rr rr!rRrSrTrlenr+loggerwarnrNr) RemoteTool)clsr#rarbrcr3 endpointsr$Zresolved_config_fileZis_tool_configreaderconfigrprAr%rlr&from_hubs     (        z Tool.from_hub Upload tool)r#commit_messageprivaterb create_prreturnc Cst|||dddd}|j}t|ddgiddtP}||td|d d t |t |||||dd Wd S1s0Yd S) a Upload the tool to the Hub. Parameters: repo_id (`str`): The name of the repository you want to push your tool to. It should contain your organization name when pushing to a given organization. commit_message (`str`, *optional*, defaults to `"Upload tool"`): Message to commit while pushing. private (`bool`, *optional*): Whether or not the repository created should be private. token (`bool` or `str`, *optional*): The token to use as HTTP bearer authorization for remote files. If unset, will use the token generated when running `huggingface-cli login` (stored in `~/.huggingface`). create_pr (`bool`, *optional*, defaults to `False`): Whether or not to create a PR with the uploaded files or directly commit. Trgradio)r#rbrr9rZ space_sdktagstoolrz!Uploading the following files to z: ,)r#r~Z folder_pathrbrrN) rr#r tempfileTemporaryDirectoryr`ruinforPrHlistdirr )r1r#r~rrbrrepo_urlZwork_dirr%r%r& push_to_hubs    "zTool.push_to_hubcCs Gdddt}|j|_||S)z8 Creates a [`Tool`] from a gradio tool. cseZdZfddZZS)z+Tool.from_gradio..GradioToolWrappercst|j|_|j|_dSN)superr4r+r))r1Z _gradio_toolrLr%r&r4Ps z4Tool.from_gradio..GradioToolWrapper.__init__)rNrJ __qualname__r4 __classcell__r%r%rr&GradioToolWrapperOsr)r(runr6)Z gradio_toolrr%r%r& from_gradioIszTool.from_gradio)NNF)r}NNF)rNrJr__doc__r)str__annotations__r+rr4r6r8r` classmethodrboolr|rr staticmethodrr%r%r%r&r(PsB     6p ,r(c@s2eZdZdZd ddZddZddZd d ZdS) rwah A [`Tool`] that will make requests to an inference endpoint. Args: endpoint_url (`str`): The url of the endpoint to use. token (`str`, *optional*): The token to use as HTTP bearer authorization for remote files. If unset, will use the token generated when running `huggingface-cli login` (stored in `~/.huggingface`). tool_class (`type`, *optional*): The corresponding `tool_class` if this is a remote version of an existing tool. Will help determine when the output should be converted to another type (like images). NcCs||_t||d|_||_dS)Nrb) endpoint_urlEndpointClientclientrA)r1rrbrAr%r%r&r4hszRemoteTool.__init__c OsN|}t|dkr|jdurt|jtr6|jj}n|jj}t|j }dd| D}|ddkrt|dd}t|t|krt |jdt|dt|d t ||D]\}}|||<qnRt|dkrt d n~sz-RemoteTool.prepare_inputs..r1z only accepts z arguments but z were given.z4A `RemoteTool` can only accept one positional input.r,)copyrtrA issubclass PipelineToolencoder6r signature parametersrsrKziprr encode_image) r1r2r3r,Z call_methodrrargr+keyvaluer%r%r&prepare_inputsms8           zRemoteTool.prepare_inputscCs|S)z You can override this method in your custom class of [`RemoteTool`] to apply some custom post-processing of the outputs of the endpoint. r%r1r-r%r%r&extract_outputsszRemoteTool.extract_outputscOs|jduo|jjdgk}|j|i|}t|trL|jfi|d|i}n|j||d}t|trt|dkrt|dtr|d}||S)Nimage output_image)rrr) rAr-r isinstancedictrrZrtr)r1r2r3rr,r-r%r%r&r6s $zRemoteTool.__call__)NNN)rNrJrrr4rrr6r%r%r%r&rwYs  +rwcs^eZdZdZeZdZeZdZdfdd Z fddZ ddZ d d Z d d Z d dZZS)ra0 A [`Tool`] tailored towards Transformer models. On top of the class attributes of the base class [`Tool`], you will need to specify: - **model_class** (`type`) -- The class to use to load the model in this tool. - **default_checkpoint** (`str`) -- The default checkpoint that should be used when the user doesn't specify one. - **pre_processor_class** (`type`, *optional*, defaults to [`AutoProcessor`]) -- The class to use to load the pre-processor - **post_processor_class** (`type`, *optional*, defaults to [`AutoProcessor`]) -- The class to use to load the post-processor (when different from the pre-processor). Args: model (`str` or [`PreTrainedModel`], *optional*): The name of the checkpoint to use for the model, or the instantiated model. If unset, will default to the value of the class attribute `default_checkpoint`. pre_processor (`str` or `Any`, *optional*): The name of the checkpoint to use for the pre-processor, or the instantiated pre-processor (can be a tokenizer, an image processor, a feature extractor or a processor). Will default to the value of `model` if unset. post_processor (`str` or `Any`, *optional*): The name of the checkpoint to use for the post-processor, or the instantiated pre-processor (can be a tokenizer, an image processor, a feature extractor or a processor). Will default to the `pre_processor` if unset. device (`int`, `str` or `torch.device`, *optional*): The device on which to execute the model. Will default to any accelerator available (GPU, MPS etc...), the CPU otherwise. device_map (`str` or `dict`, *optional*): If passed along, will be used to instantiate the model. model_kwargs (`dict`, *optional*): Any keyword argument to send to the model instantiation. token (`str`, *optional*): The token to use as HTTP bearer authorization for remote files. If unset, will use the token generated when running `huggingface-cli login` (stored in `~/.huggingface`). hub_kwargs (additional keyword arguments, *optional*): Any additional keyword argument to send to the methods that will load the data from the Hub. Nc  ststdtstd|dur<|jdur6td|j}|durH|}||_||_||_||_||_ |durrin||_ |dur||j d<||_ ||j d<t dS)N/Please install torch in order to use this tool.z4Please install accelerate in order to use this tool.zHThis tool does not implement a default checkpoint, you need to pass one. device_mapro)r ImportErrorrdefault_checkpointrKr pre_processorpost_processordevicer model_kwargsr$rr4) r1rrrrrrrbr$rr%r&r4s*    zPipelineTool.__init__cst|jtr&|jj|jfi|j|_t|jtrR|jj|jfi|j|j|_|j durf|j|_ n&t|j tr|j j|j fi|j|_ |j dur|j durt |jjd|_ nt|_ |j dur|j|j tdS)z^ Instantiates the `pre_processor`, `model` and `post_processor` if necessary. Nr)rrrpre_processor_classZfrom_pretrainedr$r model_classrrpost_processor_classrrrZZ hf_device_mapvaluesget_default_devicetorr8r7rr%r&r8s         zPipelineTool.setupcCs ||S)zQ Uses the `pre_processor` to prepare the inputs for the `model`. )r)r1Z raw_inputsr%r%r&rszPipelineTool.encodecCs<t |jfi|WdS1s.0YdS)z7 Sends the inputs through the `model`. N)torchZno_gradr)r1r,r%r%r&forwards zPipelineTool.forwardcCs ||S)zG Uses the `post_processor` to decode the model output. )rrr%r%r&decode"szPipelineTool.decodecOsH|js||j|i|}t||j}||}t|d}||S)Ncpu)r/r8rrrrr)r1r2r3Zencoded_inputsr-r%r%r&r6(s   zPipelineTool.__call__)NNNNNNN)rNrJrrrrrrrr4r8rrrr6rr%r%rr&rs$%% r)rAcs^z ddl}Wnty&tdYn0|fdd}|j||j|j|jjddS)z Launches a gradio demo for a tool. The corresponding tool class needs to properly implement the class attributes `inputs` and `outputs`. Args: tool_class (`type`): The class of the tool for which to launch the demo. rNz.fn)rr,r-titlearticle)rrZ Interfacer,r-rNr)Zlaunch)rAZgrrr%rr&launch_gradio_demo3s   rcCsRtstdtjjr0tjjr0tdStjrDtdStdSdS)Nrmpscudar) rrrbackendsrZ is_availableZis_builtrrr%r%r%r&rOs   rZDocumentQuestionAnsweringToolZImageCaptioningToolZImageQuestionAnsweringToolZImageSegmentationToolZSpeechToTextToolZTextSummarizationToolZTextClassificationToolZTextQuestionAnsweringToolZTextToSpeechToolZTranslationTool) zdocument-question-answeringzimage-captioningzimage-question-answeringzimage-segmentationzspeech-to-textZ summarizationztext-classificationztext-question-answeringztext-to-speech translationcCsJtdddd}t|ddd}t|}Wdn1s<0Y|S)Nz#huggingface-tools/default-endpointszdefault_endpoints.jsonZdatasetrr=r>r?)rrRrSrT)Zendpoints_filer^ryr%r%r&rris(rrcCst}||vSr)rr)task_or_repo_idryr%r%r&supports_remotepsrFc Ks|tvrt|}td}|j}t||}|rj|dur\t} || vrTtd|d| |}t|||dS||fd|i|Sntj |f|||d|SdS)a Main function to quickly load a tool, be it on the Hub or in the Transformers library. Args: task_or_repo_id (`str`): The task for which to load the tool or a repo ID of a tool on the Hub. Tasks implemented in Transformers are: - `"document-question-answering"` - `"image-captioning"` - `"image-question-answering"` - `"image-segmentation"` - `"speech-to-text"` - `"summarization"` - `"text-classification"` - `"text-question-answering"` - `"text-to-speech"` - `"translation"` model_repo_id (`str`, *optional*): Use this argument to use a different model than the default one for the tool you selected. remote (`bool`, *optional*, defaults to `False`): Whether to use your tool by downloading the model or (if it is available) with an inference endpoint. token (`str`, *optional*): The token to identify you on hf.co. If unset, will use the token generated when running `huggingface-cli login` (stored in `~/.huggingface`). kwargs (additional keyword arguments, *optional*): Additional keyword arguments that will be split in two: all arguments relevant to the Hub (such as `cache_dir`, `revision`, `subfolder`) will be used when downloading the files for your tool, and the others will be passed along to its init. Z transformersNrdrerqrb)rarbrc) TASK_MAPPING importlib import_moduleZtoolsgetattrrrrKrwr(r|) rrarcrbr3Ztool_class_nameZ main_moduleZ tools_modulerAryr%r%r& load_toolus    rcsfdd}|S)z< A decorator that adds a description to a function. cs|_|j|_|Sr)r)rNr+)funcr)r%r&innerszadd_description..innerr%)r)rr%rr&add_descriptions rc @sveZdZd eeedddZeddZeddZdee ee e ee e efee ee e ed d d ZdS)rN)rrbcCs"it|dddi|_||_dS)Nrz Content-Typezapplication/json)r headersr)r1rrbr%r%r&r4szEndpointClient.__init__cCs.t}|j|ddt|}|dS)NZPNG)rXr>)ioBytesIOr`base64 b64encodegetvaluer)r_bytesb64r%r%r&rszEndpointClient.encode_imagecCs8tstdddlm}t|}t|}||S)NzbThis tool returned an image but Pillow is not installed. Please install it (`pip install Pillow`).r)Image) rrZPILrr b64decoderrrR)Z raw_imagerrrr%r%r& decode_images   zEndpointClient.decode_imageF)r,paramsdatarrcCsPi}|r||d<|r||d<tj|j|j||d}|rD||jS|SdS)Nr,r)rrSr)r postrrrcontentrS)r1r,rrrpayloadresponser%r%r&r6s zEndpointClient.__call__)N)NNNF)rNrJrrrr4rrrrrrbytesrrr6r%r%r%r&rs   r)N)NFN)9rrrrrSrHrtypingrrrrrZhuggingface_hubrrr r Zhuggingface_hub.utilsr r r Zdynamic_module_utilsrrrZ image_utilsrZ models.autorutilsrrrrrrZ get_loggerrNrurZaccelerate.utilsrr r'rWr(rwrrrrrrrrrrr%r%r%r&sZ      R   6