a  dFB@sdZddlZddlZddlmZddlmZddlm Z m Z m Z e rNddl Z e eZddZe rzerzdd lmZn dd lmZGd d d eZGd ddeZdaddZddZddZddZddZdddZddZdS)z Integration with Deepspeed N) partialmethod)dep_version_check)is_accelerate_availableis_torch_availableloggingcCstjdduS)N deepspeed) importlibutil find_specr r _/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/transformers/deepspeed.pyis_deepspeed_available sr)HfDeepSpeedConfig)objectcs eZdZdZfddZZS)raJ This object contains a DeepSpeed configuration dictionary and can be quickly queried for things like zero stage. A `weakref` of this object is stored in the module's globals to be able to access the config from areas where things like the Trainer object is not available (e.g. `from_pretrained` and `_get_resized_embeddings`). Therefore it's important that this object remains alive while the program is still running. [`Trainer`] uses the `HfTrainerDeepSpeedConfig` subclass instead. That subclass has logic to sync the configuration with values of [`TrainingArguments`] by replacing special placeholder values: `"auto"`. Without this special logic the DeepSpeed configuration is not modified in any way. Args: config_file_or_dict (`Union[str, Dict]`): path to DeepSpeed config file or dict. cs(t|tdtdt|dS)NZ accelerater)set_hf_deepspeed_configrsuper__init__selfZconfig_file_or_dict __class__r r r=szHfDeepSpeedConfig.__init__)__name__ __module__ __qualname____doc__r __classcell__r r rr r,srcsVeZdZdZfddZddZddZdd d Zeed d Z ddZ ddZ Z S)HfTrainerDeepSpeedConfigz The `HfTrainerDeepSpeedConfig` object is meant to be created during `TrainingArguments` object creation and has the same lifespan as the latter. cst|d|_g|_dSN)rr_dtype mismatchesrrr r rKs z!HfTrainerDeepSpeedConfig.__init__cCs|jdurtd|jS)Nz8trainer_config_process() wasn't called yet to tell dtype)r ValueErrorrr r r dtypePs zHfTrainerDeepSpeedConfig.dtypecCs"||}|durdS|dkSdS)NFauto) get_value)r ds_key_longvalr r r is_autoUs z HfTrainerDeepSpeedConfig.is_autoNTc Cs~||\}}|durdS||dkr4|||<dS|szDHfTrainerDeepSpeedConfig.trainer_config_finalize..r hidden_size hidden_sizeszThe model's config file has neither `hidden_size` nor `hidden_sizes` entry, therefore it's not possible to automatically fill out the following `auto` entries in the DeepSpeed config file: zb. You can fix that by replacing `auto` values for these keys with an integer value of your choice.r@rAg?rB z scheduler.params.total_num_stepsznum_training_steps (calculated)z!scheduler.params.warmup_num_stepsZ warmup_steps z]Please correct the following DeepSpeed config values that mismatch TrainingArguments values: zF The easiest method is to set these DeepSpeed config values to 'auto'.N) lenhasattrr-rGmaxrHr!r9is_zero3r.Zget_warmup_stepsr join)rr>modelnum_training_stepsZhidden_size_based_keysZhidden_size_auto_keysrGr r r"r trainer_config_finalizes2      z0HfTrainerDeepSpeedConfig.trainer_config_finalize)NT) rrrrrr#r(r.rr9r?rRrr r rr rEs   :rcCst|adSr)weakrefref_hf_deepspeed_config_weak_ref)Zhf_deepspeed_config_objr r r rsrcCsdadSr)rUr r r r unset_hf_deepspeed_configsrVcCs$tdurtdurtSdSdS)NF)rUrNr r r r is_deepspeed_zero3_enableds rWcCs"tdurtdurtjSdSdSr)rUr-r r r r deepspeed_configsrXc Csddlm}m}|j}d}d|vr<|jr0td||d}n"|rNtd| }d|d <d} d |vrt||} n t ||rtd |j ||d } || fS) zY A convenience wrapper that deals with optimizer and lr scheduler configuration. r) DummyOptimDummySchedulerN optimizerz|--adafactor was passed, but also found `optimizer` configured in the DeepSpeed config. Only one optimizer can be configured.)paramszDetected ZeRO Offload and non-DeepSpeed optimizers: This combination should work as long as the custom optimizer has both CPU and GPU implementation (except LAMB)TZzero_allow_untested_optimizerZ schedulerzFound `optimizer` configured in the DeepSpeed config, but no `scheduler`. Please configure a scheduler in the DeepSpeed config.)rQr[) Zaccelerate.utilsrYrZr-Z adafactorr!Z is_offloadloggerinfoZcreate_optimizer isinstanceZcreate_scheduler) trainerhf_deepspeed_configr>rQmodel_parametersrYrZr-r[ lr_schedulerr r r deepspeed_optim_scheds0   rdFc Csddlm}|j}|j}|jjjj}||||| | |rv| sTt d| d| dd\}}d} n0d|_ttdd |} t||||| \}}||fS) aj Init DeepSpeed, after updating the DeepSpeed configuration with any relevant Trainer's args. If `resume_from_checkpoint` was passed then an attempt to resume from a previously saved checkpoint will be made. Args: trainer: Trainer object num_training_steps: per single gpu resume_from_checkpoint: path to a checkpoint if to resume from after normal DeepSpeedEngine load inference: launch in inference mode (no optimizer and no lr scheduler) Returns: optimizer, lr_scheduler We may use `deepspeed_init` more than once during the life of Trainer, when we do - it's a temp hack based on: https://github.com/microsoft/DeepSpeed/issues/1394#issuecomment-937405374 until Deepspeed fixes a bug where it can't resume from a checkpoint after it did some stepping https://github.com/microsoft/DeepSpeed/issues/1612 r)r]zMZeRO inference only makes sense with ZeRO Stage 3 - please adjust your configr[rc)NNNcSs|jSr)Z requires_grad)pr r r hrFz deepspeed_init..)Zdeepspeed.utilsr]rPr>Z acceleratorstateZdeepspeed_pluginZ hf_ds_configrRsetLevelZget_process_log_levelrNr!Zdel_config_sub_treer[listfilter parametersrd) r`rQZ inferenceZ ds_loggerrPr>rar[rcrbr r r deepspeed_init<s&     rlcCsvddl}t||d}t|dkrdtd||j|ddd\}}|durrtd|ntd|dS)Nrz /global_step*zAttempting to resume from T)Zload_optimizer_statesZload_lr_scheduler_statesz-[deepspeed] failed to resume from checkpoint z!Can't find a valid checkpoint at )globsortedrKr]r^Zload_checkpointr!)Zdeepspeed_engineZcheckpoint_pathrmZdeepspeed_checkpoint_dirsZ load_path_r r r deepspeed_load_checkpointss  rp)F)rZimportlib.utilr rS functoolsrZdependency_versions_checkrutilsrrrr;Z get_loggerrr]rZaccelerate.utils.deepspeedrZDeepSpeedConfigbuiltinsrrrUrrVrWrXrdrlrpr r r r s.     : 7