a d@s4ddlZddlmZGdddZGdddZdS)N)nnc@s0eZdZddZddZeddZddZd S) DDPGradientStatsHookcCs:z|||jWnty,tdYn0|dS)Nz=DDPGradientStatsHook does not support non-DDP wrapped modules)register_comm_hook_hook_fnAttributeError ValueError _clear_state)selfZ ddp_moduler P/var/www/html/stable-diffusion-webui/repositories/k-diffusion/k_diffusion/gns.py__init__s  zDDPGradientStatsHook.__init__cCsg|_g|_dS)N)bucket_sq_norms_small_batchbucket_sq_norms_large_batchr r r r r sz!DDPGradientStatsHook._clear_statecsP|}j|dtjj|tjjj dd }fdd}| |S)NT)opasync_opcs&|d}j|d|S)Nrr)valuerappendpowsum)futbufrr r callbacks z/DDPGradientStatsHook._hook_fn..callback) bufferr rrrtorch distributed all_reduceReduceOpAVG get_futurethen)r bucketrrrr rr rs  zDDPGradientStatsHook._hook_fncCsXt|j}t|j}|t||g}tjj|tjjj d|d |d fS)N)rr) rr rrrstackrrrritem)r sq_norm_small_batchsq_norm_large_batchstatsr r r get_statss   zDDPGradientStatsHook.get_statsN)__name__ __module__ __qualname__r r staticmethodrr)r r r r rs  rc@sBeZdZdZdddZddZdd Zd d Zd d ZddZ dS)GradientNoiseScaleaCalculates the gradient noise scale (1 / SNR), or critical batch size, from _An Empirical Model of Large-Batch Training_, https://arxiv.org/abs/1812.06162). Args: beta (float): The decay factor for the exponential moving averages used to calculate the gradient noise scale. Default: 0.9998 eps (float): Added for numerical stability. Default: 1e-8 cCs,||_||_d|_d|_d|_td|_dS)Ngg?nan)betaeps ema_sq_normema_var beta_cumprodfloatgradient_noise_scale)r r2r3r r r r 2s zGradientNoiseScale.__init__cCst|jS)z3Returns the state of the object as a :class:`dict`.)dict__dict__itemsrr r r state_dict:szGradientNoiseScale.state_dictcCs|j|dS)zLoads the object's state. Args: state_dict (dict): object state. Should be an object returned from a call to :meth:`state_dict`. N)r:update)r r<r r r load_state_dict>sz"GradientNoiseScale.load_state_dictcCs||||||}||d|d|}|j|jd|j||_|j|jd|j||_|j|j9_t|j|jt|j|j|_|jS)aUpdates the state with a new batch's gradient statistics, and returns the current gradient noise scale. Args: sq_norm_small_batch (float): The mean of the squared 2-norms of microbatch or per sample gradients. sq_norm_large_batch (float): The squared 2-norm of the mean of the microbatch or per sample gradients. n_small_batch (int): The batch size of the individual microbatch or per sample gradients (1 if per sample). n_large_batch (int): The total batch size of the mean of the microbatch or per sample gradients. r#)r2r4r5r6maxr3r8)r r&r'Z n_small_batchZ n_large_batchZ est_sq_normZest_varr r r r=FszGradientNoiseScale.updatecCs|jS)z)Returns the current gradient noise scale.)r8rr r r get_gns\szGradientNoiseScale.get_gnscCs |jd|j|jd|jfS)zdReturns the current (debiased) estimates of the squared mean gradient and gradient variance.r#)r4r6r5rr r r r)`szGradientNoiseScale.get_statsN)r/r0) r*r+r,__doc__r r<r>r=r@r)r r r r r.%s r.)rrrr.r r r r s