a  d; @sdZddlZddlmZddlmZmZmZddlZddl m Z ddl m Z ddl mZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlm Z ddl!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,e-e.Z/dhZ0d2e j1dddZ2ddZ3d3dd Z4d4e j1e5ee6e5ee5e7d$d%d&Z8d'd(Z9d5d*d+Z:d6e6ee5e5e5ee7e7ee5eed/d0d1Z;dS)7z\ Optimizer Factory w/ Custom Weight Decay Hacked together by / Copyright 2021 Ross Wightman N)islice)OptionalCallableTuple)group_parameters) AdaBelief) Adafactor) Adahessian)AdamP)Adan)Lamb)Lars)Lion) Lookahead)MADGRAD)Nadam) NvNovoGrad)RAdam) RMSpropTF)SGDPlionh㈵>)modelcCspt|}g}g}|D]B\}}|js(q|jdksD|dsD||vrP||q||q|dd||dgS)Nrz.bias)params weight_decay)setnamed_parameters requires_gradndimendswithappend)rrno_weight_decay_listdecayno_decaynameparamrra/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/timm/optim/optim_factory.pyparam_groups_weight_decay(s  r*csttfdddS)NcsttSN)tuplerritsizerr)@z_group..r)iterr-rr-r)_group>sr3 c sdd}t|didd}g}g}|D]&\}}|||rH||n||q,t|} |durp| |  }tt||}t|ddt|D} | fdd|D| S)Ncs:|sdSt|ttfr,tfdd|DS|SdS)NTcsg|]}|qSr) startswith).0Zhpinrr) Hr1z0_layer_map.._in_head..) isinstancer,listanyr5)r8hprr7r)_in_headDs z_layer_map.._in_headZpretrained_cfg classifiercSs i|]\}}|D] }||qqSrr)r6ilr8rrr) Yr1z_layer_map..csi|] }|qSrr)r6r8Znum_trunk_groupsrr)rBZr1) getattrgetrr#lenr;r3 enumerateupdate) rZlayers_per_groupZ num_groupsr>Z head_prefixZ names_trunkZ names_headr8_Znum_trunk_layers layer_maprrCr) _layer_mapCs  rK皙??F)rrr$ layer_decayend_layer_decayverbosecsJt|}i}i}t|dr2t||jdddd}nt|}t|d} | dtfddt| D} | D]\} } | j sqv| j dks| |vrd } d }nd } |}| | }d || f}||vr| |}||gd ||<||gd||<||d | ||d | qv|r>ddl}td|j|ddt|S)z Parameter groups for layer-wise lr decay & weight decay Based on BEiT: https://github.com/microsoft/unilm/blob/master/beit/optim_factory.py#L58 group_matcherF)ZcoarseT)reverserc3s|]}|VqdSr+r)r6r@rNZ layer_maxrr) vr1z+param_groups_layer_decay..r&rr%z layer_%d_%s)lr_scaler param_names)rUrrrVrrNzparameter groups: %s)indent)rhasattrrrQrKmaxvaluesr;rangerr r!rEr#json_loggerinfodumps)rrr$rNrOrPZparam_group_namesZ param_groupsrJZ num_layersZ layer_scalesr'r(Zg_decayZ this_decayZlayer_idZ group_nameZ this_scaler]rrSr)param_groups_layer_decay^sF      racCst|j|j|j|jd}t|dddur2|j|d<t|dddurL|j|d<t|dddurf|j|d<t|dddur| |j t|d ddur|j |d <|S) z cfg/argparse to kwargs helper Convert optimizer args in argparse args or cfg like object to keyword args for updated create fn. )optlrrmomentumopt_epsNeps opt_betasbetasrNopt_args opt_foreachforeach) dictrbrcrrdrDrergrNrHrirj)cfgkwargsrrr)optimizer_kwargss"     roTcCst|fit|dd|iS)zk Legacy optimizer factory for backwards compatibility. NOTE: Use create_optimizer_v2 for new code. )rmfilter_bias_and_bn)create_optimizer_v2ro)argsrrprrr)create_optimizersrssgdr?)rbrcrrdrkrprNparam_group_fnc Kst|tjrri} t|dr"|} |r0||} qv|durNt|||| d} d}qv|rh|rht||| } d}qv|} n|} |} | d} | d} | drz dd l m }m }m}m}d }Wntyd }Yn0|rtjsJd | d r8zddl}d }Wntyd }Yn0|r0tjs8Jdtfd|i| }|dur`|d||dur|tvr|dd n||d<| dks| dkr|ddtj| f|d d|}n| dkr|ddtj| f|d d|}n| dkrt| f|d d|}n| dkrkr|dd|| f|d d|}n| d?kr|dd|| f|d d|}n| d@kr@|| fdAd i|}n| dBkrb|| fdAd i|}nl| dCkr|| fi|}nN| dDkr|dEdF|| fi|}n$| dGkr|dd|jj| f|d d|}n| dHkr|dd|jj0| f|d d|}n| dIkrD|dd|jj| fd|i|}n| dJkrv|dd|jj0| fd|i|}nX| dKkr|jj| fi|}n6| dLkr|jj1| fi|}n| dMkr|jj| fi|}n| dNkr|jj2| fi|}n| dOkr|jj3| fi|}n| dPkr:|jj4| fi|}n| dQkrZ|jj5| fi|}nt| dRkrz|jj4| fi|}nT| dSkr|jj.| fi|}n4| dTkr|jj6| fi|}nd rdUsJt7t8| dVkr| ddWkrt9|}|S)Xa Create an optimizer. TODO currently the model is passed in and all parameters are selected for optimization. For more general use an interface that allows selection of parameters to optimize and lr groups, one of: * a filter fn interface that further breaks params into groups in a weight_decay compatible fashion * expose the parameters interface and leave it up to caller Args: model_or_params (nn.Module): model containing parameters to optimize opt: name of optimizer to create lr: initial learning rate weight_decay: weight decay to apply in optimizer momentum: momentum for momentum based optimizers (others may use betas via kwargs) foreach: Enable / disable foreach (multi-tensor) operation if True / False. Choose safe default if None filter_bias_and_bn: filter out bias, bn and other 1d params from weight decay **kwargs: extra optimizer specific kwargs to pass through Returns: Optimizer no_weight_decayN)rrNr$rrIZfusedr) FusedNovoGrad FusedAdam FusedLAMBFusedSGDTFz+APEX and CUDA required for fused optimizersbnbz1bitsandbytes and CUDA required for bnb optimizersrrcrkrtnesterovrf)rdr~rdsgdpZadamZadamwadampg{Gz?)Zwd_ratior~nadamradamZadamax adabeliefZrectifyZ radabeliefZadadeltaZadagradg:0yE> adafactorZadanpZno_proxZadanwlambZlambc trust_clipZlarc)rdrlarsZnlarc)rdrr~ZnlarsmadgradZmadgradw)rdZdecoupled_decayZnovograd nvnovogradZrmspropru)alphardZ rmsproptfr adahessianZfusedsgdZ fusedmomentumZ fusedadamZ adam_w_modeZ fusedadamwZ fusedlambZ fusednovogradrh)gffffff?g\(\?ZbnbsgdZ bnbsgd8bitZ bnbmomentumZbnbmomentum8bitZbnbadamZ bnbadam8bitZbnbadamwZ bnbadamw8bitZbnblambZ bnblamb8bitZbnblarsZ bnblarsb8bitZbnblionZ bnblion8bitzInvalid optimizerr lookahead):r:nnModulerYrwrar* parameterslowersplitr5Zapex.optimizersryrzr{r| ImportErrortorchZcudaZ is_availableZ bitsandbytesrl setdefault_DEFAULT_FOREACHpopoptimZSGDrZAdamZAdamWr rAttributeErrorrZAdamaxrZAdadeltaZAdagradr r r rrrZRMSproprrr ZSGD8bitZAdam8bitZ AdamW8bitZLAMBZLAMB8bitZLARSZLion8bit ValueErrorrFr)Zmodel_or_paramsrbrcrrdrkrprNrvrnrwrZ opt_lowerZ opt_splitryrzr{r|Zhas_apexr}Zhas_bnbriZ optimizerrrr)rqs@                                                                        rq)rr)r4N)rLrrMNF)T)rtNrruNTNN)<__doc__logging itertoolsrtypingrrrrZtorch.nnrZ torch.optimrZ timm.modelsrrrrr rr rr Zadanr rr rrrrrrrrrrrrrrZ rmsprop_tfrrr getLogger__name__r^rrr*r3rKfloatstrboolrarorsrqrrrr)s                       @