a  d&@sfdZddlZddlmZddlmZddlZddlmZe e Z GdddZ Gdddej ZdS) zf Exponential Moving Average (EMA) of model updates Hacked together by / Copyright 2020 Ross Wightman N) OrderedDict)deepcopyc@s*eZdZdZd ddZddZdd Zd S) ModelEmaa Model Exponential Moving Average (DEPRECATED) Keep a moving average of everything in the model state_dict (parameters and buffers). This version is deprecated, it does not work with scripted models. Will be removed eventually. This is intended to allow functionality like https://www.tensorflow.org/api_docs/python/tf/train/ExponentialMovingAverage A smoothed version of the weights is necessary for some training schemes to perform well. E.g. Google's hyper-params for training MNASNet, MobileNet-V3, EfficientNet, etc that use RMSprop with a short 2.4-3 epoch decay period and slow LR decay rate of .96-.99 requires EMA smoothing of weights to match results. Pay attention to the decay constant you are using relative to your update count per epoch. To keep EMA from using GPU resources, set device='cpu'. This will save a bit of memory but disable validation of the EMA weights. Validation will have to be done manually in a separate process, or after the training stops converging. This class is sensitive where it is initialized in the sequence of model init, GPU assignment and distributed training wrappers. H.?cCslt||_|j||_||_|r2|jj|dt|jd|_|rN|||j D]}| dqXdS)NdevicemoduleF) remaevaldecayrtohasattrema_has_module_load_checkpoint parametersZrequires_grad_)selfmodelr rresumepr]/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/timm/utils/model_ema.py__init__%s   zModelEma.__init__cCstj|dd}t|tsJd|vrt}|dD]2\}}|jr\|dsVd|n|}n|}|||<q6|j |t dn t ddS)Ncpu)Z map_locationZstate_dict_emar module.zLoaded state_dict_emazAFailed to find state_dict_ema, starting from loaded model weights) torchload isinstancedictritemsr startswithr Zload_state_dict_loggerinfowarning)rZcheckpoint_path checkpointZnew_state_dictkvnamerrrr3s   zModelEma._load_checkpointcCst|do|j }tz|}|jD]R\}}|rFd|}||}|jrf|j |jd}| ||j d|j |q2Wdn1s0YdS)Nr rr?) rrrno_grad state_dictr rdetachrr copy_r )rrZ needs_moduleZmsdr%ema_vmodel_vrrrupdateDs  zModelEma.updateN)rrr)__name__ __module__ __qualname____doc__rrr/rrrrrs rcs:eZdZdZd fdd ZddZdd Zd d ZZS) ModelEmaV2a Model Exponential Moving Average V2 Keep a moving average of everything in the model state_dict (parameters and buffers). V2 of this module is simpler, it does not match params/buffers based on name but simply iterates in order. It works with torchscript (JIT of full model). This is intended to allow functionality like https://www.tensorflow.org/api_docs/python/tf/train/ExponentialMovingAverage A smoothed version of the weights is necessary for some training schemes to perform well. E.g. Google's hyper-params for training MNASNet, MobileNet-V3, EfficientNet, etc that use RMSprop with a short 2.4-3 epoch decay period and slow LR decay rate of .96-.99 requires EMA smoothing of weights to match results. Pay attention to the decay constant you are using relative to your update count per epoch. To keep EMA from using GPU resources, set device='cpu'. This will save a bit of memory but disable validation of the EMA weights. Validation will have to be done manually in a separate process, or after the training stops converging. This class is sensitive where it is initialized in the sequence of model init, GPU assignment and distributed training wrappers. rNcsJtt|t||_|j||_||_|jdurF|jj|ddSNr) superr4rrr r r rr )rrr r __class__rrris   zModelEmaV2.__init__cCszt^t|j|D]0\}}|jdurF|j|jd}||||q&Wdn1sl0YdSr5) rr)zipr r*valuesrr r,)rr update_fnr-r.rrr_updatess  $ zModelEmaV2._updatecsj|fddddS)Ncsj|dj|S)Nr()r emrrr{z#ModelEmaV2.update..r;r<rrrr@rr/zszModelEmaV2.updatecCs|j|ddddS)NcSs|S)Nrr=rrrrA~rBz ModelEmaV2.set..rCrDrErrrset}szModelEmaV2.set)rN) r0r1r2r3rr<r/rF __classcell__rrr7rr4Rs  r4)r3logging collectionsrcopyrrZtorch.nnnn getLoggerr0r!rModuler4rrrrs    C