a d2@sddlmZddlZddlZddlmZddlZddlZddlZddl m Z ddl Z ddl m Z m Z ddlmZddlmZdd Zd d Zd6d dZddZddZd7ddZed8ddZddZe ddZGdddZGddde jjZ Gd d!d!e jjZ!d"d#d$e j"fd%d&Z#d"d#d"e$d'd$e j"fd(d)Z%d$e j"fd*d+Z&d#d"e$d'd$e j"fd,d-Z'd$e j"fd.d/Z(Gd0d1d1ej)Z*Gd2d3d3Z+ed9d4d5Z,dS):)contextmanagerN)Path)Image)nnoptim)data) functionalcCs(t|}|jdkr|d}|ddS)z&Converts from a PIL image to a tensor.).N)TF to_tensorndimxrR/var/www/html/stable-diffusion-webui/repositories/k-diffusion/k_diffusion/utils.pyfrom_pil_images  rcCsT|jdkr$|jddksJ|d}|jddkr:|d}t|ddddS)z&Converts from a tensor to a PIL image.rr r )r shaper to_pil_imageclamprrrrrs  rRGBcs fdd||D}||iS)z4Apply passed in transforms for HuggingFace Datasets.csg|]}|qSr)convert).0imagemode transformrr $z+hf_datasets_augs_helper..r)examplesr image_keyrimagesrrrhf_datasets_augs_helper"sr$cCs:||j}|dkr*td|jd|d|dd|S)zNAppends dimensions to the end of a tensor until it has target_dims dimensions.rz input has z dims but target_dims is z, which is less).N)r ValueError)r target_dimsdims_to_appendrrr append_dims(s r)cCstdd|DS)z7Returns the number of trainable parameters in a module.css|]}|VqdSr%)numel)rprrr 2r zn_params..)sum parameters)modulerrrn_params0sr0c Cst|}|jjddd|stj|F}t|d}t ||Wdn1sZ0YWdn1sx0Y|durt t|d  }||krtd|d|d|S) zLDownloads a file if it does not exist, optionally checking its SHA-256 hash.T)parentsexist_okwbNrbzhash of z (url: z) failed to validate)rparentmkdirexistsurllibrequesturlopenopenshutil copyfileobjhashlibsha256read hexdigestOSError)pathurldigestresponsefZ file_digestrrr download_file5sHrHTc csjdd|D}z0||VWt|D]\}}|||_q.n"t|D]\}}|||_qP0dS)zdA context manager that places a model into training mode and restores the previous mode on exit.cSsg|] }|jqSr)training)rr/rrrrGr ztrain_mode..N)modulestrain enumeraterI)modelrmodesir/rrr train_modeCsrPcCs t|dS)zfA context manager that places a model into evaluation mode and restores the previous mode on exit.F)rP)rMrrr eval_modeOsrQc Cst|}t|}||ks,J|D]$\}}|||j|d|dq4t|}t|}||ksJ|D]\}} ||| qdS)zIncorporates updated model parameters into an exponential moving averaged version of a model. It should be called after each optimizer step.r )alphaN)dictnamed_parameterskeysitemsmul_add_ named_bufferscopy_) rMZaveraged_modeldecayZ model_paramsZaveraged_paramsnameparamZ model_buffersZaveraged_buffersbufrrr ema_updateUs    r_c@s:eZdZdZdddZddZd d Zd d Zd dZdS) EMAWarmupaYImplements an EMA warmup using an inverse decay schedule. If inv_gamma=1 and power=1, implements a simple average. inv_gamma=1, power=2/3 are good values for models you plan to train for a million or more steps (reaches decay factor 0.999 at 31.6K steps, 0.9999 at 1M steps), inv_gamma=1, power=3/4 for models you plan to train for less (reaches decay factor 0.999 at 10K steps, 0.9999 at 215.4k steps). Args: inv_gamma (float): Inverse multiplicative factor of EMA warmup. Default: 1. power (float): Exponential factor of EMA warmup. Default: 1. min_value (float): The minimum EMA decay rate. Default: 0. max_value (float): The maximum EMA decay rate. Default: 1. start_at (int): The epoch to start averaging at. Default: 0. last_epoch (int): The index of last epoch. Default: 0. ?rcCs(||_||_||_||_||_||_dSr%) inv_gammapower min_value max_valuestart_at last_epoch)selfrcrdrerfrgrhrrr__init__xs zEMAWarmup.__init__cCst|jS)z2Returns the state of the class as a :class:`dict`.)rS__dict__rVrirrr state_dictszEMAWarmup.state_dictcCs|j|dS)zLoads the class's state. Args: state_dict (dict): scaler state. Should be an object returned from a call to :meth:`state_dict`. N)rkupdate)rirmrrrload_state_dictszEMAWarmup.load_state_dictcCsLtd|j|j}dd||j|j }|dkr8dSt|jt|j|S)z Gets the current EMA decay rate.rr rb)maxrhrgrcrdminrfre)riepochvaluerrr get_valueszEMAWarmup.get_valuecCs|jd7_dS)zUpdates the step count.r N)rhrlrrrstepszEMAWarmup.stepN)rararbrarr) __name__ __module__ __qualname____doc__rjrmrortrurrrrr`hs r`cs2eZdZdZd fdd Zdd Zd d ZZS) InverseLRaMImplements an inverse decay learning rate schedule with an optional exponential warmup. When last_epoch=-1, sets initial lr as lr. inv_gamma is the number of steps/epochs required for the learning rate to decay to (1 / 2)**power of its original value. Args: optimizer (Optimizer): Wrapped optimizer. inv_gamma (float): Inverse multiplicative factor of learning rate decay. Default: 1. power (float): Exponential factor of learning rate decay. Default: 1. warmup (float): Exponential warmup factor (0 <= warmup < 1, 0 to disable) Default: 0. min_lr (float): The minimum learning rate. Default: 0. last_epoch (int): The index of last epoch. Default: -1. verbose (bool): If ``True``, prints a message to stdout for each update. Default: ``False``. rarbrFcsJ||_||_d|kr dks*ntd||_||_t|||dSNrbr zInvalid value for warmup)rcrdr&warmupmin_lrsuperrj)ri optimizerrcrdr|r}rhverbose __class__rrrjszInverseLR.__init__cCs|jstd|SNzTTo get the last learning rate computed by the scheduler, please use `get_last_lr()`._get_lr_called_within_stepwarningswarn_get_closed_form_lrrlrrrget_lrs zInverseLR.get_lrcsDdjjddjjj fddjDS)Nr cs g|]}tj|qSrrpr}rbase_lrlr_multrir|rrrsz1InverseLR._get_closed_form_lr..)r|rhrcrdbase_lrsrlrrrrs zInverseLR._get_closed_form_lr)rararbrbrFrvrwrxryrjrr __classcell__rrrrrzs  rzcs2eZdZdZd fdd Zdd Zd d ZZS) ExponentialLRaEImplements an exponential learning rate schedule with an optional exponential warmup. When last_epoch=-1, sets initial lr as lr. Decays the learning rate continuously by decay (default 0.5) every num_steps steps. Args: optimizer (Optimizer): Wrapped optimizer. num_steps (float): The number of steps to decay the learning rate by decay in. decay (float): The factor by which to decay the learning rate every num_steps steps. Default: 0.5. warmup (float): Exponential warmup factor (0 <= warmup < 1, 0 to disable) Default: 0. min_lr (float): The minimum learning rate. Default: 0. last_epoch (int): The index of last epoch. Default: -1. verbose (bool): If ``True``, prints a message to stdout for each update. Default: ``False``. ?rbrFcsJ||_||_d|kr dks*ntd||_||_t|||dSr{) num_stepsr[r&r|r}r~rj)rirrr[r|r}rhrrrrrjszExponentialLR.__init__cCs|jstd|Srrrlrrrrs zExponentialLR.get_lrcsBdjjdjdjjfddjDS)Nr cs g|]}tj|qSrrrrrrrsz5ExponentialLR._get_closed_form_lr..)r|rhr[rrrlrrrrs z!ExponentialLR._get_closed_form_lr)rrbrbrFrrrrrrs  rrbracpucCstj|||d||S)z-Draws samples from an lognormal distribution.devicedtype)torchrandnexp)rlocscalerrrrrrand_log_normalsrinfc Cstj||tjd}tj||tjd}|||}|||}tj||tjd|||} |  | | |S)zEDraws samples from an optionally truncated log-logistic distribution.r) r as_tensorfloat64logsubdivsigmoidrandlogitmuladdrto) rrrrerfrrmin_cdfmax_cdfurrrrand_log_logistics rcCs4t|}t|}tj|||d|||S)z/Draws samples from an log-uniform distribution.r)mathrrrr)rrerfrrrrrrand_log_uniforms  rc Csdt||dtj}t||dtj}tj|||d|||}t|tjd|S)zJDraws samples from a truncated v-diffusion training timestep distribution.r r)ratanpirrtan) r sigma_datarerfrrrrrrrrrand_v_diffusionsrc Cs`tj|||d}tj|||d}|| |}|||} |||} t|| k|| S)z2Draws samples from a split lognormal distribution.r)rrabsrwherer) rrscale_1scale_2rrnrZn_leftZn_rightratiorrrrand_split_log_normals   rcsBeZdZdZhdZd fdd ZddZdd Zd d ZZ S) FolderOfImageszURecursively finds all images in a directory. It does not support classes/targets.> z.ppmz.pngz.tiffz.webpz.jpegz.bmpz.pgmz.tifz.jpgNcsNtt|_|dur$tn|_tfddjdD_ dS)Nc3s"|]}|jjvr|VqdSr%)suffixlowerIMG_EXTENSIONS)rrCrlrrr,r z*FolderOfImages.__init__..*) r~rjrrootrIdentityrsortedrglobpaths)rirrrrlrrjs  zFolderOfImages.__init__cCsd|jdt|dS)NzFolderOfImages(root="z", len: ))rlenrlrrr__repr__szFolderOfImages.__repr__cCs t|jSr%)rrrlrrr__len__!szFolderOfImages.__len__cCsT|j|}t|d }t|d}Wdn1s:0Y||}|fS)Nr4r)rr;rrr)rikeyrCrGrrrr __getitem__$s   . zFolderOfImages.__getitem__)N) rvrwrxryrrjrrrrrrrrrs rc@seZdZddZddZdS) CSVLoggercCsHt||_||_|jr*t|jd|_nt|jd|_|j|jdS)Naw)rfilenamecolumnsr7r;filewrite)rirrrrrrj-s   zCSVLogger.__init__cGst|d|jdddS)N,T)seprflush)printr)riargsrrrr6szCSVLogger.writeN)rvrwrxrjrrrrrr,s rccstjjj}tjjjj}zV|dur*|tjj_|dur>|tjjj_dVW|durX|tjj_|dur|tjjj_n(|dur|tjj_|dur|tjjj_0dS)zGA context manager that sets whether TF32 is allowed on cuDNN or matmul.N)rbackendscudnn allow_tf32cudamatmul)rrZ cudnn_oldZ matmul_oldrrr tf32_mode:s       r)r)N)T)NN)- contextlibrr>rpathlibrr<r8rPILrrrrZ torch.utilsrZtorchvision.transformsrr rrr$r)r0rHrPrQno_gradr_r` lr_scheduler _LRSchedulerrzrfloat32rfloatrrrrDatasetrrrrrrrsD         0))