a  d@s,dZddlZddlmZGdddeZdS)an PyTorch LARS / LARC Optimizer An implementation of LARS (SGD) + LARC in PyTorch Based on: * PyTorch SGD: https://github.com/pytorch/pytorch/blob/1.7/torch/optim/sgd.py#L100 * NVIDIA APEX LARC: https://github.com/NVIDIA/apex/blob/master/apex/parallel/LARC.py Additional cleanup and modifications to properly support PyTorch XLA. Copyright 2021 Ross Wightman N) Optimizerc s@eZdZdZdfdd Zfd d Zedd d ZZ S)Larsa LARS for PyTorch Paper: `Large batch training of Convolutional Networks` - https://arxiv.org/pdf/1708.03888.pdf Args: params (iterable): iterable of parameters to optimize or dicts defining parameter groups. lr (float, optional): learning rate (default: 1.0). momentum (float, optional): momentum factor (default: 0) weight_decay (float, optional): weight decay (L2 penalty) (default: 0) dampening (float, optional): dampening for momentum (default: 0) nesterov (bool, optional): enables Nesterov momentum (default: False) trust_coeff (float): trust coefficient for computing adaptive lr / trust_ratio (default: 0.001) eps (float): eps for division denominator (default: 1e-8) trust_clip (bool): enable LARC trust ratio clipping (default: False) always_adapt (bool): always apply LARS LR adapt, otherwise only when group weight_decay != 0 (default: False) ?rFMbP?:0yE>c s|dkrtd||dkr,td||dkrBtd||r^|dksV|dkr^tdt|||||||| | d } t|| dS)NgzInvalid learning rate: zInvalid momentum value: zInvalid weight_decay value: rz8Nesterov momentum requires a momentum and zero dampening) lrmomentum dampening weight_decaynesterov trust_coeffeps trust_clip always_adapt) ValueErrordictsuper__init__) selfparamsrrr r r r r rrdefaults __class__X/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/timm/optim/lars.pyr#s(  z Lars.__init__cs(t||jD]}|ddqdS)Nr F)r __setstate__ param_groups setdefault)rstategrouprrrrFs  zLars.__setstate__Nc Csd}|dur:t|}Wdn1s00Y|jdddj}tjd|d}|jD]n}|d}|d}|d}|d } |d } |d } |dD].} | jdurq| j} |dks|d rL| d }| d }| ||||| }t|dkt|dk|||}|dr4t||d|}| j | |d| ||dkr|j | }d|vrt |  }|d<n |d}| |j | d|d| r| j||d} n|} | j | |d dqqb|S)zPerforms a single optimization step. Args: closure (callable, optional): A closure that reevaluates the model and returns the loss. Nrrr)devicer rr r r r rg@rr)alphaZmomentum_buffer)torchZ enable_gradrr ZtensorgradZnormwhereZminimumZadd_Zmul_rclonedetachadd)rclosureZlossr Z one_tensorrr rr r r r pr#Zw_normZg_normZ trust_ratioZ param_statebufrrrstepKsR $         z Lars.step) rrrrFrrFF)N) __name__ __module__ __qualname____doc__rrr"Zno_gradr+ __classcell__rrrrrs# r)r/r"Ztorch.optim.optimizerrrrrrrs