a  d^@sdZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl m Z m Z m Z mZmZmZmZddlZddlmZmZmZmZmZmZmZerddlZerddlZddZe dd d Z!e dd d Z"Gd ddZ#GdddeZ$GdddeZ%GdddeZ&dZ'e(de'dZ)ddZ*GdddeZ+GdddeZ,GdddeZ-Gd d!d!eZ.e e/e0fe0d"d#d$Z1e e/e0fd%d&d'Z2e e/e0fd%d(d)Z3d*d+Z4e e/e0fd%d,d-Z5Gd.d/d/eZ6e6j7e2e6j8e3e6j9e4e6j:e5iZ;d0d1ZGd6d7d7eZ?Gd8d9d9Z@d:d;ZAdd?ZCGd@dAdAeZDdLeEe eFdDdEdFZGGdGdHdHeZHGdIdJdJZIdS)Mzc Utilities for the Trainer and TFTrainer class. Should be independent from PyTorch and TensorFlow. N)AnyDictList NamedTupleOptionalTupleUnion) ExplicitEnumis_psutil_availableis_tf_availableis_torch_availableis_torch_cuda_availableis_torch_tpu_availablerequires_backendscCstd}t|dS)zN Helper function to set worker seed during Dataloader initialization. lN)torchZ initial_seedset_seed)_Z worker_seedrc/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/transformers/trainer_utils.py seed_worker2s rseedcCsVt|tr@dtjd<dtjd<tddtjj_dtjj_ t rRt j j dS)a  Helper function for reproducible behavior during distributed training. See - https://pytorch.org/docs/stable/notes/randomness.html for pytorch - https://www.tensorflow.org/api_docs/python/tf/config/experimental/enable_op_determinism for tensorflow 1ZCUDA_LAUNCH_BLOCKINGz:16:8ZCUBLAS_WORKSPACE_CONFIGTFN)rr osenvironrZuse_deterministic_algorithmsbackendsZcudnnZ deterministicZ benchmarkr tfconfigZ experimentalZenable_op_determinismrrrrenable_full_determinism:s     rcCsHt|tj|tr2t|tj|trDt j |dS)z Helper function for reproducible behavior to set the seed in `random`, `numpy`, `torch` and/or `tf` (if installed). Args: seed (`int`): The seed to set. N) randomrnpr rZ manual_seedcudaZmanual_seed_allr rrrrrrrSs    rc@sheZdZdZd eejeejfeejeejfeeejeejfdddZ ddZ dd Z dS) EvalPredictiona Evaluation output (always contains labels), to be used to compute metrics. Parameters: predictions (`np.ndarray`): Predictions of the model. label_ids (`np.ndarray`): Targets to be matched. inputs (`np.ndarray`, *optional*) N predictions label_idsinputscCs||_||_||_dSNr$)selfr%r&r'rrr__init__nszEvalPrediction.__init__cCs2|jdurt|j|j|jfSt|j|jfSdSr()r'iterr%r&r)rrr__iter__xs zEvalPrediction.__iter__cCs`|dks|dkrtd|dkr2|jdur2td|dkr@|jS|dkrN|jS|dkr\|jSdS)Nrztuple index out of ranger ) IndexErrorr'r%r&)r)idxrrr __getitem__~szEvalPrediction.__getitem__)N) __name__ __module__ __qualname____doc__rr!ndarrayrrr*r-r1rrrrr#ds  r#c@sbeZdZUeejeejfed<eeejeejfed<ee e e fed<ee ed<dS)EvalLoopOutputr%r&metrics num_samplesN) r2r3r4rr!r6r__annotations__rrstrfloatintrrrrr7s r7c@sVeZdZUeejeejfed<eeejeejfed<ee e e fed<dS)PredictionOutputr%r&r8N) r2r3r4rr!r6rr:rrr;r<rrrrr>s r>c@s.eZdZUeed<eed<eeefed<dS) TrainOutputZ global_stepZ training_lossr8N)r2r3r4r=r:r<rr;rrrrr?s r? checkpoint^z\-(\d+)$csFt}fdd|D}t|dkr,dStjt|dddS)Ncs4g|],}t|durtjtj|r|qSr()_re_checkpointsearchrpathisdirjoin).0rDfolderrr s$z'get_last_checkpoint..rcSstt|dS)Nr)r=rBrCgroups)xrrrz%get_last_checkpoint..)key)rlistdirlenrDrFmax)rIcontentZ checkpointsrrHrget_last_checkpoints   rTc@seZdZdZdZdZdS)IntervalStrategynostepsepochNr2r3r4NOZSTEPSEPOCHrrrrrUsrUc@seZdZdZdZdZdS)EvaluationStrategyrVrWrXNrYrrrrr\sr\c@seZdZdZdZdZdZdS) HubStrategyendZ every_saver@Zall_checkpointsN)r2r3r4ZENDZ EVERY_SAVE CHECKPOINTZALL_CHECKPOINTSrrrrr]sr]c@sBeZdZUdZeed<eed<eeefed<dZ e eed<dS)BestRunac The best run found by a hyperparameter search (see [`~Trainer.hyperparameter_search`]). Parameters: run_id (`str`): The id of the best run (if models were saved, the corresponding checkpoint will be in the folder ending with run-{run_id}). objective (`float`): The objective that was obtained for this run. hyperparameters (`Dict[str, Any]`): The hyperparameters picked to get this run. run_summary (`Optional[Any]`): A summary of tuning experiments. `ray.tune.ExperimentAnalysis` object for Ray backend. Zrun_id objectiveZhyperparametersN run_summary) r2r3r4r5r;r:r<rrrbrrrrrr`s r`)r8returncCsft|}|dd}|dd}dd|D}|D]}||d}q8t|dkrZ|St|S)aj The default objective to maximize/minimize when doing an hyperparameter search. It is the evaluation loss if no metrics are provided to the [`Trainer`], the sum of all metrics otherwise. Args: metrics (`Dict[str, float]`): The metrics returned by the evaluate method. Return: `float`: The objective to minimize or maximize Z eval_lossNrXcSs.g|]&}|ds&|ds&|dr|qS)_runtimeZ _per_secondZ_compilation_time)endswithrGmrrrrJsz-default_compute_objective..r)copydeepcopypopkeysrQsumvalues)r8Zlossr speed_metricssmrrrdefault_compute_objectives   rp)rccCsVddlm}|sJd|jddddd|d dd |d dd |d gddS)Nr )is_optuna_availablez:This function needs Optuna installed: `pip install optuna` learning_rateư>-C6?T)lognum_train_epochsr(per_device_train_batch_size @rrrvrry) integrationsrqZ suggest_floatZ suggest_intZsuggest_categorical)trialrqrrrdefault_hp_space_optunas   rcCs^ddlm}|sJdddlm}|dd|ttdd|dd |gd d S) Nr )is_ray_tune_availablez:This function needs ray installed: `pip install ray[tune]`r)tunersrtrxrzr) rrrayrZ loguniformchoicelistrangeuniform)rrrrrrdefault_hp_space_rays     rcCsDddddddddd dd d d dd ddd d gddddgS)Nrsrt)minrRrrdoubleru)boundsnametypeZtransformamtionr rrvr=)rrrrxr)48Z16Z32Z64ryZ categorical)Zcategorical_valuesrrr)rrrrdefault_hp_space_sigopt srcCsVddlm}|stddddddd d d d dd d d ddd dgdiddS)Nr )is_wandb_availablez8This function needs wandb installed: `pip install wandb`r raZminimize)rZgoalrrsrt) distributionrrRZ int_uniformrrxrmrzr)methodZmetric parameters)rr ImportError)rrrrrdefault_hp_space_wandbs     rc@seZdZdZdZdZdZdS)HPSearchBackendZoptunarZsigoptZwandbN)r2r3r4OPTUNARAYSIGOPTWANDBrrrrr)srcCs0tddr(ddlmm}|dkS|dvS)z Whether or not the current process is the local process, based on `xm.get_ordinal()` (for TPUs) first, then on `local_rank`. TZ check_devicerN)r)rtorch_xla.core.xla_modelcore xla_modelZ get_ordinal) local_rankxmrrris_main_process8s  rcCsHtddr$ddlmm}|S|dkrDtrDddl}|jSdS)zg Return the number of processes launched in parallel. Works with `torch.distributed` and TPUs. TrrNrr ) rrrrZxrt_world_sizer rZ distributedZget_world_size)rrrrrrtotal_processes_numberDs  rcCsxt|}|dt|di}|dkr,|S|durP||}t|d||d<|durt||}t|d||d<|S)a Measure and return speed performance metrics. This function requires a time snapshot `start_time` before the operation to be measured starts and this function should be run immediately after the operation to be measured has completed. Args: - split: name to prefix metric (like train, eval, test...) - start_time: operation start time - num_samples: number of samples processed rdr{rNZ_samples_per_secondZ_steps_per_second)timeround)split start_timer9Z num_stepsZruntimeresultZsamples_per_secondZsteps_per_secondrrrrnSs rnc@s,eZdZdZdZdZdZdZdZdZ dZ d S) SchedulerTypeZlinearZcosineZcosine_with_restartsZ polynomialZconstantZconstant_with_warmupZ inverse_sqrtZreduce_lr_on_plateauN) r2r3r4ZLINEARZCOSINEZCOSINE_WITH_RESTARTSZ POLYNOMIALZCONSTANTZCONSTANT_WITH_WARMUPZ INVERSE_SQRTZREDUCE_ON_PLATEAUrrrrrlsrc@sdeZdZdZddddddZddd Zd d Zd d ZddZddZ ddZ ddZ dddZ dS)TrainerMemoryTrackera A helper class that tracks cpu and gpu memory. This class will silently skip unless `psutil` is available. Install with `pip install psutil`. When a stage completes, it can pass metrics dict to update with the memory metrics gathered during this stage. Example : ```python self._memory_tracker = TrainerMemoryTracker(self.args.skip_memory_metrics) self._memory_tracker.start() # code ... metrics = {"train_runtime": 10.5} self._memory_tracker.stop_and_update_metrics(metrics) ``` At the moment GPU tracking is only for `pytorch`, but can be extended to support `tensorflow`. To understand this class' intricacies please read the documentation of [`~Trainer.log_metrics`]. inittrainevaltest)r*rZ_inner_training_loopevaluateZpredictFcCsf||_tsd|_|jrdSddl}tr@ddl}||_i|_nd|_||_d|_i|_ d|_ dS)NTrF) skip_memory_metricsr psutilrrgpuProcessprocess cur_stagecpu init_reported)r)rrrrrrr*s zTrainerMemoryTracker.__init__cCsBtjjjj}||jvr$|j|Std|d|jdS)z+derives the stage/caller name automaticallyzwas called from z+, but only expect to be called from one of N)inspect currentframef_backf_codeco_namestages ValueErrorrk)r)Zcallerrrr derive_stages   z!TrainerMemoryTracker.derive_stagecCs |jjS)z4get resident set size memory for the current process)rZ memory_infoZrssr,rrr cpu_mem_usedsz!TrainerMemoryTracker.cpu_mem_usedcCs&d|_t||j|_|jsq"qdS)Nr)cpu_mem_used_peakrRrpeak_monitoringr,rrrpeak_monitor_funcsz&TrainerMemoryTracker.peak_monitor_funccCs|jr dS|}|jdur*|j|kr*dS||_t|jdurZ|jj|jj|jdurr|jj |_ | |_ d|_ tj|jd}d|_|dS)z%start tracking for the caller's stageNT)target)rrrgccollectrr"Zreset_peak_memory_stats empty_cachememory_allocatedgpu_mem_used_at_startrcpu_mem_used_at_startr threadingThreadrdaemonstart)r)stageZpeak_monitor_threadrrrrs"     zTrainerMemoryTracker.startcCs|jdur|j|krdSd|_t|jdur<|jj|jdur|jj|_|jj |_ |j |j|j|j t d|j |jd|j |j<||_|j|j|j|jt d|j|jd|j|j<d|_dS)z"stop tracking for the passed stageNFr)beginr^allocpeaked)rrrrrr"rrZgpu_mem_used_nowZmax_memory_allocatedZgpu_mem_used_peakrrRrrZcpu_mem_used_nowrrr)r)rrrrstops*      zTrainerMemoryTracker.stopcCs |jr dS|jdur"|j|kr"dS|g}|js@|ddd|_|D]}dD]z}||jvr||j|vr|j||||d|d<|jdurL||jvrL||j|vrL|j||||d|d<qLqD|ddkr|jdd |d <|jdur|jdd |d <dS) zupdates the metricsNrrT)rrZ _mem_cpu_Z_deltaZ _mem_gpu_rZbefore_init_mem_cpuZbefore_init_mem_gpu)rrrinsertrrr)r)rr8rtrrrupdate_metricss$ ""  z#TrainerMemoryTracker.update_metricsNcCs4|jr dS|}|||dur0|||dS)zs  z,TrainerMemoryTracker.stop_and_update_metrics)F)N) r2r3r4r5rr*rrrrrrrrrrrrws   ,)rcCs(zt|duWSty"YdS0dS)zR Checks if the dataset implements __len__() and it doesn't raise an error NF)rQ TypeError)Zdatasetrrr has_lengthKs rcCst|ttfr$t|dd|DSt|trHt|dd|DSt|tjr\|St rt|t j r| dkr|S|S)zM Recursively calls `.item()` on the element of the dictionary passed css|]}t|VqdSr(denumpify_detensorizerfrrr [rNz(denumpify_detensorize..cSsi|]\}}|t|qSrrrGkvrrr ]rNz)denumpify_detensorize..r ) isinstancertuplerdictitemsr!Zgenericitemr rZTensorZnumel)r8rrrrVs  rcCsFt|tjr6tt|jj}|t|jt|j Stt|jS)za Return the number of arguments of the passed function, even if it's a partial function. ) r functoolspartialrQr signaturefuncrargskeywords)rZ total_argsrrrnumber_of_argumentses rc@s eZdZdZdZdZdZdZdS)ShardedDDPOptionsimpleZ zero_dp_2Z zero_dp_3offload auto_wrapN)r2r3r4ZSIMPLEZ ZERO_DP_2Z ZERO_DP_3OFFLOAD AUTO_WRAPrrrrros rF)functionstarting_batch_sizeauto_find_batch_sizecCsL|durtjt||dS|r>ttdddlm}|||dStj||dS)a$ Args: A basic decorator that will try to execute `function`. If it fails from exceptions related to out-of-memory or CUDNN, the batch size is cut in half and passed to `function` `function` must take in a `batch_size` parameter as its first argument. function (`callable`, *optional*) A function to wrap starting_batch_size (`int`, *optional*) The batch size to try and fit into memory auto_find_batch_size (`bool`, *optional*) If False, will just execute `function` N)rrZ accelerater)find_executable_batch_size)rr)Z batch_size)rrrrZaccelerate.utils)rrrZ%accelerate_find_executable_batch_sizerrrrws   rc@s eZdZdZdZdZdZdZdS) FSDPOptionZ full_shardZ shard_grad_opZno_shardrrN)r2r3r4Z FULL_SHARDZ SHARD_GRAD_OPZNO_SHARDrrrrrrrs rc@sLeZdZdZd eeeedddZeedddZe ed d d Z dS) RemoveColumnsCollatorzWWrap the data collator to remove unused columns before they are passed to the collator.N) model_name descriptioncCs(||_||_||_||_||_d|_dS)NF) data_collatorsignature_columnsloggerrrmessage_logged)r)rrrrrrrrr*s zRemoveColumnsCollator.__init__)featurercc st|ts|Sjsjrjrtt|tj}t |dkrj durTdn dj d}j d|djdd |d d |d jd d _fd d| DS)Nrzin the z setzThe following columns z) don't have a corresponding argument in `z!.forward` and have been ignored: z, z. If z are not expected by `z/.forward`, you can safely ignore this message.Tcs i|]\}}|jvr||qSr)rrr,rrrrNz9RemoveColumnsCollator._remove_columns..)rrrrrrsetrkrrQrinforFr)r)rZignored_columnsZdset_descriptionrr,r_remove_columnss&  z%RemoveColumnsCollator._remove_columns)featurescsfdd|D}|S)Ncsg|]}|qSr)r )rGrr,rrrJrNz2RemoveColumnsCollator.__call__..)r)r)r rr,r__call__szRemoveColumnsCollator.__call__)NNN) r2r3r4r5rr;r*rr rr rrrrrs r)NN)NrF)Jr5rhrrrrr rerrtypingrrrrrrrnumpyr!utilsr r r r rrrrZ tensorflowrrr=rrr#r7r>r?ZPREFIX_CHECKPOINT_DIRcompilerBrTrUr\r]r`r;r<rprrrrrrrrrZdefault_hp_spacerrrnrrrrrrcallableboolrrrrrrrsr$$ '      U