a  d~@s@dZddlZddlZddlmZddlmZGdddeZdS)z Processor class for Donut. N)contextmanager)ProcessorMixincsxeZdZdZddgZdZdZdfdd Zd d Zd d Z d dZ e ddZ dddZ eddZeddZZS)DonutProcessora Constructs a Donut processor which wraps a Donut image processor and an XLMRoBERTa tokenizer into a single processor. [`DonutProcessor`] offers all the functionalities of [`DonutImageProcessor`] and [`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`]. See the [`~DonutProcessor.__call__`] and [`~DonutProcessor.decode`] for more information. Args: image_processor ([`DonutImageProcessor`]): An instance of [`DonutImageProcessor`]. The image processor is a required input. tokenizer ([`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`]): An instance of [`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`]. The tokenizer is a required input. image_processor tokenizerZAutoImageProcessorZ AutoTokenizerNc snd|vrtdt|d}|dur*|n|}|dur>td|durNtdt|||j|_d|_ dS)Nfeature_extractorzhThe `feature_extractor` argument is deprecated and will be removed in v5, use `image_processor` instead.z)You need to specify an `image_processor`.z"You need to specify a `tokenizer`.F) warningswarn FutureWarningpop ValueErrorsuper__init__rcurrent_processor_in_target_context_manager)selfrrkwargsr __class__s/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/transformers/models/donut/processing_donut.pyr,s zDonutProcessor.__init__cOs|jr|j|i|S|dd}|dd}t|dkrN|d}|dd}|durf|durftd|dur|j|g|Ri|}|dur|j|fi|}|dur|S|dur|S|d|d<|SdS) a When used in normal mode, this method forwards all its arguments to AutoImageProcessor's [`~AutoImageProcessor.__call__`] and returns its output. If used in the context [`~DonutProcessor.as_target_processor`] this method forwards all its arguments to DonutTokenizer's [`~DonutTokenizer.__call__`]. Please refer to the doctsring of the above two methods for more information. imagesNtextrzBYou need to specify either an `images` or `text` input to process.Z input_idslabels)rrr lenr rr)rargsrrrinputs encodingsrrr__call__?s&     zDonutProcessor.__call__cOs|jj|i|S)z This method forwards all its arguments to DonutTokenizer's [`~PreTrainedTokenizer.batch_decode`]. Please refer to the docstring of this method for more information. )r batch_decoderrrrrrr!`szDonutProcessor.batch_decodecOs|jj|i|S)z This method forwards all its arguments to DonutTokenizer's [`~PreTrainedTokenizer.decode`]. Please refer to the docstring of this method for more information. )rdecoder"rrrr#gszDonutProcessor.decodeccs0tdd|_|j|_dV|j|_d|_dS)z Temporarily sets the tokenizer for processing the input. Useful for encoding the labels when fine-tuning TrOCR. z`as_target_processor` is deprecated and will be removed in v5 of Transformers. You can process your labels by using the argument `text` of the regular `__call__` method (either in the same call as your images inputs, or in a separate call.TNF)r r rrrrrrrras_target_processornsz"DonutProcessor.as_target_processorFc Cs|dur|j}i}|rtd|tj}|dur8q|d}td|d|tj}|}|durx||d}q|}t|}t|} t|d| |tj} | dur| d} d| vrd| vr|j | d |d } | rt | dkr| d } | ||<ng||<| d D]R} | } | |vrd| d d krd| dddkrd| dd} || | q"t ||dkr||d ||<|| |t |d}|ddd kr|g|j |ddd |d Sqt |r|r|gS|S|r gSd|iSdS)zS Convert a (generated) token sequence into an ordered JSON format. Nz rzz(.*?)zZ text_sequence)rZget_added_vocabresearch IGNORECASEgroupreplaceescapestrip token2jsonrsplitappendfind) rtokensr(r)outputZ start_tokenkeyZ end_tokenZstart_token_escapedZend_token_escapedcontentvalueZleafrrrr4~sL      *   zDonutProcessor.token2jsoncCstdt|jS)Nzg`feature_extractor_class` is deprecated and will be removed in v5. Use `image_processor_class` instead.)r r r image_processor_classr$rrrfeature_extractor_classs z&DonutProcessor.feature_extractor_classcCstdt|jS)Nz[`feature_extractor` is deprecated and will be removed in v5. Use `image_processor` instead.)r r r rr$rrrrs z DonutProcessor.feature_extractor)NN)FN)__name__ __module__ __qualname____doc__ attributesr=Ztokenizer_classrr r!r#rr%r4propertyr>r __classcell__rrrrrs!  2 r)rBr-r contextlibrZprocessing_utilsrrrrrrs