a  dv;@svddlZddlZddlZddlmZddlmZddlmZm Z ddl m Z ddl m Z ddlmZGdd d ZdS) N) OrderedDict)deepcopy) DataParallelDistributedDataParallel) lr_scheduler)get_root_logger) master_onlyc@seZdZdZddZddZddZdd Zd d Zd:d dZ ddZ ddZ d;ddZ ddZ ddZddZddZddZed d!Zd"d#Zd$d%Zdd/d0Zd?d1d2Zed3d4Zd5d6Zd7d8Zd9S)@ BaseModelz Base model.cCs<||_t|ddkrdnd|_|d|_g|_g|_dS)Nnum_gpurcudacpuis_train)opttorchdevicer schedulers optimizers)selfrrb/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/basicsr/models/base_model.py__init__s  zBaseModel.__init__cCsdSNr)rdatarrr feed_dataszBaseModel.feed_datacCsdSrrrrrroptimize_parametersszBaseModel.optimize_parameterscCsdSrrrrrrget_current_visualsszBaseModel.get_current_visualscCsdS)z!Save networks and training state.Nr)repoch current_iterrrrsave szBaseModel.saveFcCs0|jdr|||||n|||||dS)a1Validation function. Args: dataloader (torch.utils.data.DataLoader): Validation dataloader. current_iter (int): Current iteration. tb_logger (tensorboard logger): Tensorboard logger. save_img (bool): Whether to save images. Default: False. distN)rZdist_validationZnondist_validation)rZ dataloaderrZ tb_loggerZsave_imgrrr validation$s zBaseModel.validationcCst|dr||jvrdSt|ds*t|_t}|jddD]>\}}|dd}|dkrftdntd}t||d d ||<qB||j|<dS) zZInitialize the best metric results dict for recording the best metric value and iteration.best_metric_resultsNvalZmetricsbetterhigherz-infinf)r$r#iter)hasattrr"dictritemsgetfloat)r dataset_namerecordmetriccontentr$Zinit_valrrr_initialize_best_metric_results2s  z)BaseModel._initialize_best_metric_resultscCs|j||ddkrR||j||dkr||j||d<||j||d<n:||j||dkr||j||d<||j||d<dS)Nr$r%r#r()r")rr.r0r#rrrr_update_best_metric_resultAsz$BaseModel._update_best_metric_result+?cCs\||j}t|}t|j}|D](}||j|j||jd|dq.dS)N)alpha) get_bare_modelnet_gr*Znamed_parametersZ net_g_emakeysrZmul_Zadd_)rZdecayr8Z net_g_paramsZnet_g_ema_paramskrrr model_emaKs    zBaseModel.model_emacCs|jSr)log_dictrrrrget_current_logTszBaseModel.get_current_logcCsV||j}|jdr<|jdd}t|tjg|d}n|jddkrRt|}|S)zModel to device. It also warps models with DistributedDataParallel or DataParallel. Args: net (nn.Module) r find_unused_parametersF)Z device_idsr>r r5) torrr,rrr Zcurrent_devicer)rnetr>rrrmodel_to_deviceWs  zBaseModel.model_to_devicecKs4|dkr tjj||fi|}ntd|d|S)NAdamz optimizer z is not supperted yet.)rZoptimrBNotImplementedError)rZ optim_typeparamslrkwargs optimizerrrr get_optimizergszBaseModel.get_optimizercCs|jd}|dd}|dvrL|jD]"}|jtj|fi|dq&nD|dkr|jD]"}|jtj|fi|dqZntd|ddS) zSet up schedulers.Ztrain schedulertype)Z MultiStepLRMultiStepRestartLRCosineAnnealingRestartLRz Scheduler z is not implemented yet.N) rpoprrappendrrKrLrC)rZ train_optZscheduler_typerGrrrsetup_schedulersns  " "zBaseModel.setup_schedulerscCst|ttfr|j}|S)zhGet bare model, especially under wrapping with DistributedDataParallel or DataParallel. ) isinstancerrmodule)rr@rrrr7{szBaseModel.get_bare_modelcCst|ttfr(|jjd|jjj}n |jj}||}t|}tt dd| }t }| d|d|d| |dS)zdPrint the str and parameter number of a network. Args: net (nn.Module) z - cSs|Sr)Znumel)xrrrz)BaseModel.print_network..z Network: z, with parameters: z,dN) rPrr __class____name__rQr7strsummap parametersrinfo)rr@Z net_cls_strZnet_strZ net_paramsloggerrrr print_networks  zBaseModel.print_networkcCs8t|j|D]&\}}t|j|D]\}}||d<q q dS)zSet learning rate for warm-up. Args: lr_groups_l (list): List for lr_groups, each for an optimizer. rEN)zipr param_groups)rZ lr_groups_lrGZ lr_groups param_grouprErrr_set_lrszBaseModel._set_lrcCs*g}|jD]}|dd|jDq |S)z;Get the initial lr, which is set by the scheduler. cSsg|] }|dqS)Z initial_lrr.0vrrr rTz*BaseModel._get_init_lr..)rrNr_)rZinit_lr_groups_lrGrrr _get_init_lrs zBaseModel._get_init_lrr'csbdkr|jD] }|qkr^|}g}|D]}|fdd|Dq4||dS)uUpdate learning rate. Args: current_iter (int): Current iteration. warmup_iter (int)： Warm-up iter numbers. -1 for no warm-up. Default： -1. r5csg|]}|qSrrrbr warmup_iterrrrerTz2BaseModel.update_learning_rate..N)rsteprfrNra)rrrhrIZ init_lr_g_lZ warm_up_lr_lZ init_lr_grrgrupdate_learning_rates  zBaseModel.update_learning_ratecCsdd|jdjDS)NcSsg|] }|dqS)rEr)rcr`rrrrerTz7BaseModel.get_current_learning_rate..r)rr_rrrrget_current_learning_ratesz#BaseModel.get_current_learning_raterDc Cs|dkr d}|d|d}tj|jdd|}t|trB|n|g}t|trV|n|g}t|t|kstJdi}t||D]V\}} ||}| } | D]*\} } | dr| d d } | | | <q| || <qd } | d krrz|zt ||WnPtyJ}z6t}|d |d| dtdWYd }~nd }~00W| d8} qrW| d8} q| d8} 0q| d kr|d|dd S)a@Save networks. Args: net (nn.Module | list[nn.Module]): Network(s) to be saved. net_label (str): Network label. current_iter (int): Current iter number. param_key (str | list[str]): The parameter key(s) to save network. Default: 'params'. r'Zlatest_z.pthpathmodelsz4The lengths of net and param_key should be the same.module.NrzSave model error: , remaining retry times: r5Still cannot save . Just ignore it.)osrmjoinrrPlistlenr^r7 state_dictr+ startswithr rr Exceptionrwarningtimesleep)rr@Z net_labelr param_key save_filename save_path save_dictZnet_Z param_key_rykeyparamretryer\rrr save_networks<        zBaseModel.save_networkTc Cs||}|}t|}t|}t}||kr|dtt||D]}|d|qR|dtt||D]}|d|q|s||@}|D]V} || || kr|d| d|| j d|| j | | || d<qdS) aPrint keys with different name or different size when loading models. 1. Print keys with different names. 2. If strict=False, print the same key but with different tensor size. It also ignore these keys with different sizes (not load). Args: crt_net (torch model): Current network. load_net (dict): Loaded network. strict (bool): Whether strictly loaded. Default: True. zCurrent net - loaded net:z zLoaded net - current net:zSize different, ignore [z ]: crt_net: z ; load_net: z.ignoreN) r7rysetr9rr|sortedrwsizeshaperM) rZcrt_netload_netstrictZ crt_net_keysZ load_net_keysr\rdZ common_keysr:rrr_print_different_keys_loadings,      z'BaseModel._print_different_keys_loadingc Cst}||}tj|ddd}|durP||vrHd|vrHd}|d||}|d|jjd|d |d t|D],\}}| d r~|||d d<| |q~| ||||j ||d dS)aYLoad network. Args: load_path (str): The path of networks to be loaded. net (nn.Module): Network. strict (bool): Whether strictly loaded. param_key (str): The parameter key of loaded network. If set to None, use the root 'path'. Default: 'params'. cSs|Srr)ZstoragelocrrrrS"rTz(BaseModel.load_network..)Z map_locationNrDz/Loading: params_ema does not exist, use params.zLoading z model from z, with param key: [z].rorp)r) rr7rloadr[rUrVrr+rzrMrload_state_dict) rr@Z load_pathrrr\rr:rdrrr load_networks   "  zBaseModel.load_networkc Cs,|dkr(||ggd}|jD]}|d|q|jD]}|d|q<|d}tj|jdd|}d}|d kr zzzt ||WnNt y} z6t } | d | d |d t d WYd } ~ nd } ~ 00W|d 8}q W|d 8}qz|d 8}0qz|d kr(| d|dd S)zSave training states during training, which will be used for resuming. Args: epoch (int): Current epoch. current_iter (int): Current iteration. r')rr(rrrrz.statermZtraining_statesrqrzSave training state error: rrr5Nrsrt)rrNryrrurmrvrrrr{rr|r}r~) rrrstateosrrrrr\rrrsave_training_state1s,       zBaseModel.save_training_statecCs|d}|d}t|t|jks*Jdt|t|jksDJdt|D]\}}|j||qLt|D]\}}|j||qndS)zReload the optimizers and schedulers for resumed training. Args: resume_state (dict): Resume state. rrzWrong lengths of optimizerszWrong lengths of schedulersN)rxrr enumerater)rZ resume_stateZresume_optimizersZresume_schedulersirrrrrresume_trainingTszBaseModel.resume_trainingcCst|jdrg}g}|D]\}}||||q$t|d}tjj|dd|jddkrz||jd}ddt||D}t }|D]\}}| ||<q|WdS1s0YdS) zreduce loss dict. In distributed training, it averages the losses among different GPUs . Args: loss_dict (OrderedDict): Loss dict. r r)dstZrankZ world_sizecSsi|]\}}||qSrr)rcrZlossrrr vrTz.BaseModel.reduce_loss_dict..N) rZno_gradrr+rNstackZ distributedreducer^rmeanitem)rZ loss_dictr9Zlossesnamevaluer<rrrreduce_loss_dictcs      zBaseModel.reduce_loss_dictN)F)r4)r')rD)T)TrD)rV __module__ __qualname____doc__rrrrrr!r2r3r;r=rArHrOr7rr]rarfrjrkrrrrrrrrrrr s:       . #  "r )rur}r collectionsrcopyrZtorch.nn.parallelrrZbasicsr.modelsrZ basicsr.utilsrZbasicsr.utils.dist_utilrr rrrrs