a d,\@stddlmZddlmZmZmZddlmZmZm Z m Z m Z m Z m Z mZmZmZddlZddlmZmZddlmZddlmZddlmZmZmZdd lmZdd lmZdd l m!Z!m"Z"m#Z#dd l$m%Z%dd l&m'Z'GdddZ(eGdddZ)GdddZ*GdddZ+GdddZ,GdddZ-ee efee ee.e/fgee.e/ffee.e/fdddZ0dS))Sized)asdict dataclassfield) AnyCallableDictIterableIteratorListMappingOptionalSequenceUnionN)apply_to_collectionapply_to_collections)Tensor)Dataset)_BaseDataLoaderIter_MultiProcessingDataLoaderIter DataLoader)IterableDataset)_distributed_available)_reload_dataloader_state_dictMergedIteratorStatepatch_dataloader_iterator)MisconfigurationException)_fault_tolerant_trainingc@seZdZdZedddZdeeddddZeed d d Z edd d dZ eed ddZ eed ddZ eed ddZ eeedddZdS)TensorRunningAccumaTracks a running accumulation values (min, max, mean) without graph references. Examples: >>> accum = TensorRunningAccum(5) >>> accum.last(), accum.mean() (None, None) >>> accum.append(torch.tensor(1.5)) >>> accum.last(), accum.mean() (tensor(1.5000), tensor(1.5000)) >>> accum.append(torch.tensor(2.5)) >>> accum.last(), accum.mean() (tensor(2.5000), tensor(2.)) >>> accum.reset() >>> _= [accum.append(torch.tensor(i)) for i in range(13)] >>> accum.last(), accum.mean(), accum.min(), accum.max() (tensor(12.), tensor(10.), tensor(8.), tensor(12.)) ) window_lengthcCs||_||dSN)rresetselfrr$m/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/pytorch_lightning/trainer/supporters.py__init__7szTensorRunningAccum.__init__N)rreturncCs*|dur||_d|_d|_d|_d|_dS)zEmpty the accumulator.NrF)rmemory current_idxlast_idxrotatedr"r$r$r%r!;s zTensorRunningAccum.resetr'cCs.|jdur*t|jtsJ|j|jSdS)zGet the last added element.N)r* isinstancer(rfloatr#r$r$r%lastDs zTensorRunningAccum.last)xr'cCs|jdur.tj|jg|jR|j|jd|_t$||j|j<|j|_ Wdn1s`0Y|jd7_|j|j|_|jdkrd|_ dS)z"Add an element to the accumulator.N)devicedtyperT) r(torchzerosrshaper2r3Zno_gradr)r*r+)r#r1r$r$r%appendJs $  & zTensorRunningAccum.appendcCs |dS)z$Get mean value from stored elements.mean _agg_memoryr/r$r$r%r9]szTensorRunningAccum.meancCs |dS)z'Get maximal value from stored elements.maxr:r/r$r$r%r<aszTensorRunningAccum.maxcCs |dS)z'Get minimal value from stored elements.minr:r/r$r$r%r=eszTensorRunningAccum.min)howr'cCsR|jdurNt|jtsJ|jr2t|j|St|jd|j|SdSr )r*r-r(rr+getattrr.r))r#r>r$r$r%r;is  zTensorRunningAccum._agg_memory)N)__name__ __module__ __qualname____doc__intr&r r!rr0r8r9r<r=strr;r$r$r$r%r$s rc@seZdZUdZdZeed<edddZe e ed<edddZ e e efed <d Zeed <d d ddZeed ddZd S)SharedCycleIteratorStateazA state shared between all CycleIterators in a CombinedLoader. With a shared state, the iterators can decide to terminate based on the state of all others. If the mode is *max_size_cycle*, all iterators need to have finished before the combined loading is considered finished, and otherwise any iterator finishing early will lead to all iterators ending early. max_size_cyclemodecCsgSr r$r$r$r$r%{z!SharedCycleIteratorState.)default_factory dataloaderscCsiSr r$r$r$r$r%rI|rJ has_finishedF has_resetNr,cCs$|jD]}d|jt|<qd|_dS)NFT)rLrMidrN)r# dataloaderr$r$r%r!s zSharedCycleIteratorState.resetcCs@|jstdt|jdkr dS|jdkr.tnt}||jS)Nz7Please call reset once all dataloaders have been added.r4FrG) rNrlenrLrHallanyrMvalues)r#Z decision_fnr$r$r%dones zSharedCycleIteratorState.done)r@rArBrCrHrE__annotations__rrLr rrMrrDboolrNr!propertyrUr$r$r$r%rFqs   rFc@sbeZdZdZd eeeeefe dddZ edddZ edd d Z eeefdd d Z dS) CycleIteratorz?Iterator for restarting a dataloader if it runs out of samples.N)loaderlengthstatecCsd|durtd}|s0t}|j||n |j|||_||_||_d|_d|_ ||_dS)a Args: loader: the loader to restart for cyclic (and optionally infinite) sampling length: the number of batches to sample (with restarted loaders if necessary) before raising StopIteration if None: infinite Ninfr) r.rFrLr8r!r\r[rZ _loader_itercounter)r#rZr[r\r$r$r%r&s   zCycleIterator.__init__r,cCs d|_|jt|j|_|S)zjCreates the internal iterator and returns self. Returns: CycleIterator: self r)r_r\r!iterrZr^r/r$r$r%__iter__s  zCycleIterator.__iter__c Cst|jtsJ|j|ks&|jjr*tzzt|jWW|jd7_Styd|jj t |j <|jjrttt |j |_t |j dd}|rt|j |j|t|jYW|jd7_S0W|jd7_n|jd7_0dS)z Fetches the next batch from internal dataloader and restarts it if necessary Returns: Any: the resulting batch Raises: StopIteration: if more then :attr:`length` batches have been returned r4TZ_lightning_fetcherN)r-r^r r___len__r\rU StopIterationnextrMrOrZr`r?r)r#Zfetcherr$r$r%__next__s&    zCycleIterator.__next__cCs|jSr r[r/r$r$r%rbszCycleIterator.__len__)NN)r@rArBrCrr rrDr.rFr&rarerbr$r$r$r%rYs   'rYc@seZdZdZeedZdeee fe dddZ e ee efddd Ze ee efdd d Zeee fe ee efd d dZeee eeefdddZeeee efdddZee efdddZdS)CombinedDatasetz7Combine multiple datasets and compute their statistics.min_sizerGri)datasetsrHcCs>||_||jvr4td|dt|jd||_dS)a- Args: datasets: a sequence/mapping datasets. Can be a collections of torch.utils.Dataset, Iterable or even None. mode: whether to use the minimum number of batches in all samples or the maximum number of batches in all samples. z$You have selected unsupported mode "z", please select one the: .N)rj COMPUTE_FUNCSkeysrlistrH)r#rjrHr$r$r%r&s zCombinedDataset.__init__r,cCs||jdS)NrG_calc_num_datarjr/r$r$r%max_lenszCombinedDataset.max_lencCs||jdS)Nriror/r$r$r%min_lenszCombinedDataset.min_len)rjrHr'cCsR||jvrtd|||}|j|}t|ttfrD|}n t||}|S)aCompute the length of `CombinedDataset` according to the `mode`. Args: datasets: a sequence/mapping datasets. Can be a collections of torch.utils.data.Dataset, Iterable or even None. mode: Determine `CombinedDataset`'s length is the maximum or minimum of the datasets. Returns: length: the length of `CombinedDataset` Invalid Mode: )rlrmr_get_len_recursiver-rDr._nested_calc_num_data)r#rjrH all_lengths compute_funcr[r$r$r%rps    zCombinedDataset._calc_num_data)datar'cst|tr t|tsJt|St|ttfr2|St|trjtdd|Drfdd| DSn6t|t rt |}tdd|Drfdd|DS |S)Ncss |]}t|ttttfVqdSr r-r rrr .0vr$r$r% !rJz5CombinedDataset._get_len_recursive..csi|]\}}||qSr$rt)r{kr|r/r$r% "rJz6CombinedDataset._get_len_recursive..css |]}t|ttttfVqdSr ryrzr$r$r%r}%rJcsg|]}|qSr$r~rzr/r$r% &rJz6CombinedDataset._get_len_recursive..) r-rrrQr.rDr rSrTitemsrrn_get_len)r#rxr$r/r%rts   z"CombinedDataset._get_len_recursive)datasetr'c Cs.z t|WSttfy(tdYS0dS)Nr])rQ TypeErrorNotImplementedErrorr.)rr$r$r%r*s zCombinedDataset._get_lencCs||j|jS)z*Return the minimum length of the datasets.)rprjrHr/r$r$r%rb1szCombinedDataset.__len__N)ri)r@rArBrCr=r<rlrrr rEr&rXrDr.rqrrrprr rrt staticmethodrrbr$r$r$r%rgs "rgc@s,eZdZdZdZd*eedddZee e e e ddd Z d+ee d d d Ze ddddZe ddddZeeeeefdddZeeeeefdddZedddZddddZedddZed,eeee efd d!d"Zee efdd#d$Zeedd%d&d'Z ddd(d)Z!dS)-CombinedLoaderaCombines different dataloaders and allows sampling in parallel. Supported modes are ``"min_size"``, which raises StopIteration after the shortest loader (the one with the lowest number of batches) is done, and ``"max_size_cycle"`` which raises StopIteration after the longest loader (the one with most batches) is done, while cycling through the shorter loaders. Examples: >>> loaders = {'a': torch.utils.data.DataLoader(range(6), batch_size=4), ... 'b': torch.utils.data.DataLoader(range(15), batch_size=5)} >>> combined_loader = CombinedLoader(loaders, 'max_size_cycle') >>> for item in combined_loader: ... print(item) {'a': tensor([0, 1, 2, 3]), 'b': tensor([0, 1, 2, 3, 4])} {'a': tensor([4, 5]), 'b': tensor([5, 6, 7, 8, 9])} {'a': tensor([0, 1, 2, 3]), 'b': tensor([10, 11, 12, 13, 14])} >>> combined_loader = CombinedLoader(loaders, 'min_size') >>> for item in combined_loader: ... print(item) {'a': tensor([0, 1, 2, 3]), 'b': tensor([0, 1, 2, 3, 4])} {'a': tensor([4, 5]), 'b': tensor([5, 6, 7, 8, 9])} rhri)loadersrHcCsl||jvrtd|||_t|jttddttfd}t|||_ ||_ |j dkr\| d|_ d|_ dS)aD Args: loaders: the loaders to sample from. Can be all kind of collection mode: the mode. Supported are 'min_size' which stops if the shortest loader is exhausted and 'max_size_cycle' which stops if the longest loader is exhausted and cycles through the smaller ones. rsrN wrong_dtyperG)SUPPORTED_MODESrrrr r?rr rgrrH_wrap_loaders_max_size_cycle_loaders_iter_state_dict _iterator)r#rrHrjr$r$r%r&Ns   zCombinedLoader.__init__)iterator has_completedr'cCs8t|tr|j}|rdnd}t||d}|r4t|SiS)Nr\previous_state)r-rYr^r?r)rrZ state_to_saver\r$r$r%_state_dict_fnhs   zCombinedLoader._state_dict_fnF)rr'cCs*tr|jduriSt|jjt|j|dS)axThe state dict includes all states from wrapped dataloaders and their samplers through the ``CaptureIterableDataset`` and fast-forward samplers. Args: has_completed: whether the current state of data fetching is considered completed or not. If it is, the current state gets returned, otherwise the previously cached state. N)r)rrr loader_itersr r)r#rr$r$r% state_dictwszCombinedLoader.state_dictN)rr'cCs ||_dSr )r)r#rr$r$r%load_state_dictszCombinedLoader.load_state_dict)rr'cCsV|js dStttddd}Gdddt}t|j|jt|f|tt fd|_ d|_dS)N)rPrr'cSsj|}t|tr|j}tr$tjnd}||}t||t|}t |}t|tr`||j _ n||_ |S)zOFunction used to reload the iterator state before once the workers are created.r) r-rYrZrr5Z distributedZget_rankrr`rZfrom_state_dictr^r\)rPrZdataloader_to_iter_onZrankitr\r$r$r%create_loader_iterss     z6CombinedLoader.on_restart..create_loader_itersc@s eZdZdS)z1CombinedLoader.on_restart..DataLoaderDictN)r@rArBr$r$r$r%DataLoaderDictsrr) rrrr dictrrr rr _loader_iters)r#rrrr$r$r% on_restartszCombinedLoader.on_restartr,cCst|jttftddS)z8Return a collections of samplers extracted from loaders.samplerNrrrrr?r/r$r$r%rszCombinedLoader.samplercCst|jttftddS)z>Return a collections of batch samplers extracted from loaders. batch_samplerNrr/r$r$r%rszCombinedLoader.batch_samplercCsjddlm}t|jt|ttfd}t|t}t |jttfrft }t|jtt ||ttfd|_| dS)zWraps all loaders to make sure they are cycled until the longest loader is exhausted. Returns: the wrapped loaders rget_lenr)r[r\rN) pytorch_lightning.utilities.datarrrr rr rur<r-rFrYr!)r#rrvr[r\r$r$r%rs  z+CombinedLoader._wrap_loaders_max_size_cyclecsbddlm|jdkrdSttdddd}t|jtfdd }t|t}t|jt||d dS) zWhen the model is `max_size_cycle`, compute the length across all ``CycleIterator`` and re-assign it to all dataloaders.rrrGN)cycle_iteratorr[r'cSs ||_dSr rf)rr[r$r$r%set_lensz.set_lencs |jSr )rZ)crr$r%rIrJz=CombinedLoader._apply_cycle_iterator_length..rf) rrrHrYrDrrrur<)r#rrv max_lengthr$rr%_apply_cycle_iterator_lengths   z+CombinedLoader._apply_cycle_iterator_lengthcCs4ttddd}|t_t|j}||||_|S)zQCreate and return an iterator, `CombinedLoaderIterator`, for the combined loader.)_r'cWsiSr r$)rr$r$r%__getstate__patch__sz4CombinedLoader.__iter__..__getstate__patch__)rrr __getstate__CombinedLoaderIteratorrrr)r#rrr$r$r%ras   zCombinedLoader.__iter__)rrHr'cCsHddlm}t|t|ttfd}t|ttfr2|St ||dkrBt nt S)aCompute the length (aka the number of batches) of `CombinedLoader`. Args: loaders: a collections of loaders. mode: Mode used by the CombinedDataloader Returns: length: the minimum length of loaders rrrrG) rrrr rr r-rDr.rur<r=)rrHrrvr$r$r%_calc_num_batchess z CombinedLoader._calc_num_batchescCs|j|j|jdS)N)rH)rrrHr/r$r$r%rb szCombinedLoader.__len__)rPr'cCs*t|dr t|jtr |jd|_dS)Nr)hasattrr-rrZ_shutdown_workers)rPr$r$r%$_shutdown_workers_and_reset_iterators z3CombinedLoader._shutdown_workers_and_reset_iteratorcCs2|jrd|j_|jdur(t|jt|jd|_dSr )rrrrrrr/r$r$r%r!s  zCombinedLoader.reset)ri)F)ri)"r@rArBrCrrrEr&rr r rDrrrWrrrrXrr rr rrrrrar.rrbrrr!r$r$r$r%r6s*-rc@seZdZdZedddZeedddZeddd Zedd d Z e e e e efed d dZe e ee e efe ee e efdddZdS)rzVCustom Iterator returning data from multiple loaders, and allows sampling in parallel.rcCs||_d|_dS)zf Args: loaders: the loaders to sample from. Can be all kind of collection N)rr)r#rr$r$r%r& szCombinedLoaderIterator.__init__r,cCs|jdur||j|_|jS)z5Get the `_loader_iters` and create one if it is None.N)rrrr/r$r$r%r(s z#CombinedLoaderIterator.loader_iterscCs|Sr r$r/r$r$r%ra0szCombinedLoaderIterator.__iter__cCs ||jS)zuFetches the next batch from multiple data loaders. Returns: a collections of batch data )request_next_batchrr/r$r$r%re3szCombinedLoaderIterator.__next__)rr'cCs t|ttS)zReturn the batch of data from multiple iterators. Args: loader_iters: a collections of iterators Returns Any: a collections of batch data )rr rd)rr$r$r%r;s z)CombinedLoaderIterator.request_next_batch)rr'cCst|ttttfdS)zCreate and return a collection of iterators from loaders. Args: loaders: a collections of loaders Returns a collections of iterators r)rr r`rr rr$r$r%rGs z*CombinedLoaderIterator.create_loader_itersN)r@rArBrCrr&rXrrarerrr rr rrr$r$r$r%rs r)rxrwr'cCst|ttfr|St|tr(t|}t|tsFtdt|j g}|D].}t|ttfrr| t ||qN| |qN||S)Nz6Expected data to be int, Sequence or Mapping, but got ) r-r.rDr rnrTrrtyper@r8ru)rxrwZnew_datar1r$r$r%ruWs    ru)1collections.abcrZ dataclassesrrrtypingrrrr r r r r rrr5Z#lightning_utilities.core.apply_funcrrrZtorch.utils.datarZtorch.utils.data.dataloaderrrrZtorch.utils.data.datasetrZ&lightning_fabric.utilities.distributedrZ(pytorch_lightning.utilities.auto_restartrrrZ&pytorch_lightning.utilities.exceptionsrZ#pytorch_lightning.utilities.importsrrrFrYrgrrrDr.rur$r$r$r%s. 0      MRVh;*