a dU@sdZddlZddlmZddlmZmZddlZddlm Z ddl m Z dd l m Z mZdd lmZmZed d rddlmmZerdd lmZddZddZddZddZddZed dddZdKddZddZdd Z d!d"Z!d#d$Z"d%d&Z#d'd(Z$e$Z%d)d*Z&ed+d,d-Z'e'Z(ed+d.d/Z)dLd0d1Z*dMd3d4Z+dNe,d5d6d7Z-dOe,d5d8d9Z.d:d;Z/dPdd?Z1dRdAdBZ2dCdDZ3GdEdFdFZ4dGdHZ5dIdJZ6dS)SzB A set of basic tensor ops compatible with tpu, gpu, and multigpu N)update_wrapper)AnyMapping) PartialState)CUDA_DISTRIBUTED_TYPES)DistributedTypeTensorInformation)is_torch_distributed_availableis_tpu_availableF)Z check_device)ReduceOpcCs t|tjSN) isinstancetorchTensortensorrd/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/accelerate/utils/operations.pyis_torch_tensor'src Cs2t|tjjtjjtjjtjjtjjtjjtjj Sr) rrZxpuZ FloatTensorZ ByteTensorZ IntTensorZ LongTensorZ HalfTensorZ DoubleTensorZBFloat16Tensorrrrris_torch_xpu_tensor+srcCs t|tSr)rr Z tensor_inforrris_tensor_information8srcCsVt|}|j}t|dks&|dtkr*dSt|dd}t|tsDdStdd|DS)z Checks if `x` is a `namedtuple` or not. Can have false positives, but only if a user is trying to mimic a `namedtuple` perfectly. rrF_fieldsNcss|]}t|tVqdSr)rstr).0memberrrr Hz is_namedtuple..)type __bases__lentuplegetattrrall)dataZ data_typebasesfieldsrrr is_namedtuple<s  r)cCs(t|rt|t|St||SdS)zO Cast a generator to the same type as obj (list, tuple, or namedtuple) N)r)r list)obj generatorrrr honor_typeKsr- test_typeerror_on_other_typecst|ttfr.t|fdd|DSt|tr^t|fdd|DS|r||gRiSrtdt|djdjd|S) a9 Recursively apply a function on a data structure that is a nested list/tuple/dictionary of a given base type. Args: func (`callable`): The function to recursively apply. data (nested list/tuple/dictionary of `main_type`): The data on which to apply `func` *args: Positional arguments that will be passed to `func` when applied on the unpacked data. main_type (`type`, *optional*, defaults to `torch.Tensor`): The base type of the objects to which apply `func`. error_on_other_type (`bool`, *optional*, defaults to `False`): Whether to return an error or not if after unpacking `data`, we get on an object that is not of type `main_type`. If `False`, the function will leave objects of types different than `main_type` unchanged. **kwargs: Keyword arguments that will be passed to `func` when applied on the unpacked data. Returns: The same data structure as `data` with `func` applied to every object of type `main_type`. c3s,|]$}t|gRdVqdS)r.Nrecursively_apply)roargsr0funckwargsr/rrrosz$recursively_apply..cs0i|](\}}|t|gRdqS)r.r1rkvr4rr xsz%recursively_apply..zUnsupported types (z ) passed to `z?`. Only nested list/tuple/dicts of objects that are valid for `z` should be passed.) rr#r*r-rr items TypeError__name__)r6r&r/r0r5r7rr4rr2Vs, r2cst|ttfr*t|fdd|DSt|trtttrFgn durRgt|fdd|DSt|drz|j dWSt y| YS0n|SdS)a Recursively sends the elements in a nested list/tuple/dictionary of tensors to a given device. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to send to a given device. device (`torch.device`): The device to send the data to. Returns: The same data structure as `tensor` with all tensors sent to the proper device. c3s|]}t|dVqdS) non_blocking skip_keysNsend_to_device)rtdevicer@rArrrrz!send_to_device..Nc s.i|]&\}}||vr|nt|dqS)r?rB)rr9rDrErrr;sz"send_to_device..to)r@) rr#r*r-rrr r<hasattrrGr=)rrFr@rArrErrCs(     rCcCsdd}t||S)aK Recursively gathers the information needed to rebuild a nested list/tuple/dictionary of tensors. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data to send to analyze. Returns: The same data structure as `data` with [`~utils.TensorInformation`] instead of tensors. cSst|j|jdS)N)shapedtype)r rIrJrrrr_get_data_structuresz/get_data_structure.._get_data_structurer1)r&rKrrrget_data_structures rLcCsdd}t||tdS)z Recursively initializes tensors from a nested list/tuple/dictionary of [`~utils.TensorInformation`]. Returns: The same data structure as `data` with tensors instead of [`~utils.TensorInformation`]. cSstj|jd|jiSNrJ)remptyrIrJrrrr_initialize_tensorsz.initialize_tensors.._initialize_tensorr/)r2r)Zdata_structurerOrrrinitialize_tensorssrQcCslt|ttfrt|dSt|trB|D]}t||Sn t|tjsbtdt |d|j dS)a Recursively finds the batch size in a nested list/tuple/dictionary of lists of tensors. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data from which to find the batch size. Returns: `int`: The batch size. rz0Can only find the batch size of tensors but got .) rr#r*find_batch_sizerkeysrrr=r rI)r&r9rrrrSs     rScCsdd}t||S)aS Recursively finds tensors in a nested list/tuple/dictionary and converts them to a list of numbers. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data from which to convert to regular numbers. Returns: The same data structure as `data` with lists of numbers instead of `torch.Tensor`. cSs,|}|jtjkr$|tj}|Sr)detachcpurJrbfloat16rGfloat32tolistrrrr_convert_to_lists   z!listify.._convert_to_listr1)r&rZrrrlistifys  r[cCs"dd}t||dd}t|S)NcSs |jdkr|d}t|S)Nr)ndimclonexm all_gatherrrrr_tpu_gather_ones  z$_tpu_gather.._tpu_gather_oneTr0)r2r^Z mark_step)rr`resrrr _tpu_gathersrccCsdd}t||ddS)NcsNjdkrdfddttjD}tj|tj|ddS)Nrcsg|]}tqSr)rZ empty_liker_rrr rz8_gpu_gather.._gpu_gather_one..dim)r\r]ranger distributedZget_world_sizer_cat)rZoutput_tensorsrrr_gpu_gather_ones   z$_gpu_gather.._gpu_gather_oneTrar1)rrlrrr _gpu_gathersrmcCsttjtjkrt|Stjtvr*t|Stjtjvr@t|StjtjvrVt|Stjtj krlt |S|SdS)a4 Recursively gather tensor in a nested list/tuple/dictionary of tensors from all devices. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to gather. Returns: The same data structure as `tensor` with all tensors sent to the proper device. N) rdistributed_typer TPUrcrrm MULTI_NPU MULTI_XPU MULTI_CPU _cpu_gatherrrrrgathers  rtobjectcCs2ddttjD}tj||dd|DS)NcSsg|]}dqSrrrdrrrrf-rz&_gpu_gather_object..cSsg|]}|D]}|q qSrr)ryxrrrrf0r)rir num_processesrrjZall_gather_object)rvZoutput_objectsrrr_gpu_gather_object,srzcCsvtjtjkrtdnZtjtvr,t|StjtjvrBt|StjtjvrXt|Stjtj krnt |S|SdS)a5 Recursively gather object in a nested list/tuple/dictionary of objects from all devices. Args: object (nested list/tuple/dictionary of picklable object): The data to gather. Returns: The same data structure as `object` with all the objects sent to every device. z&gather objects in TPU is not supportedN) rrnr roNotImplementedErrorrrzrprqrr_cpu_gather_objectrurrr gather_object6s   r}cCsddd}t||d|dS)NrcSstjj||d|S)Nsrc)rrj broadcast)rrrrr_gpu_broadcast_onePsz*_gpu_broadcast.._gpu_broadcast_oneT)r0r)rr1)r&rrrrr_gpu_broadcastOs rbroadcast tensorcsht|ttfr*t|fddt|DSt|trRt|fdd|DSt |fddS)Nc3s(|] \}}t|d|dVqdS)renameN_tpu_broadcast)rirDrrrrYrz!_tpu_broadcast..cs(i|] \}}|t|d|dqS)rerrr8rrrr;[rz"_tpu_broadcast..cs|Srrrxr~rr\rz _tpu_broadcast..) rr*r#r- enumeraterr r<r^ mesh_reduce)rrrr)rrrrWs  r from_processcCstjtjkrt||ddStjtvr4t||dStjtjvrJtt Stjtj vrdt||dStjtj kr~t||dS|SdS)a Recursively broadcast tensor in a nested list/tuple/dictionary of tensors to all devices. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to gather. from_process (`int`, *optional*, defaults to 0): The process from which to send the data Returns: The same data structure as `tensor` with all tensors broadcasted to the proper device. zaccelerate.utils.broadcast)rrr~N) rrnr rorrrrprzrvrqrr)rrrrrr_s     rcstjtjkrtjtj vrtj j |dntjtj krtj j |d|S)a Broadcast a list of picklable objects form one process to the others. Args: object_list (list of picklable objects): The list of objects to broadcast. This list will be modified inplace. from_process (`int`, *optional*, defaults to 0): The process from which to send the data. Returns: The same list containing the objects from process 0. z&accelerate.utils.broadcast_object_listcs|Srrrrrrrrz'broadcast_object_list..r~)rrnr rorr^rrrrjbroadcast_object_listrprqrr)Z object_listrrr+rrrrzs  rcCsdd}t|||S)aN Recursively takes a slice in a nested list/tuple/dictionary of tensors. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data to slice. tensor_slice (`slice`): The slice to take. Returns: The same data structure as `data` with all the tensors slices. cSs||Srr)r tensor_slicerrr _slice_tensorsz$slice_tensors.._slice_tensorr1)r&rrrrr slice_tensorssrcstdttfr..rgN concatenaterr&rhrrrrzconcatenate..cs(i|] tfddDdqS)csg|] }|qSrrrr9rrrfrz*concatenate...rgrrrrrr;rzconcatenate..z%Can only concatenate tensors but got rg) rr#r*r-rir"rr rTrrr=rkrrrrrs *(rcCsddd}t||d|||dS)a3 Recursively pad the tensors in a nested list/tuple/dictionary of tensors from all devices to the same size so they can safely be gathered. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to gather. dim (`int`, *optional*, defaults to 0): The dimension on which to pad. pad_index (`int`, *optional*, defaults to 0): The value with which to pad. pad_first (`bool`, *optional*, defaults to `False`): Whether to pad at the beginning or the end. rFc st|jkr|Stj|j|jdd}t|}tfdd|D|jkr\|S|jt}|<| t ||}|rt fddt t|D}n t fddt t|D}|||<|S)N)rFc3s|]}|VqdSrr)rsrgrrrrzFpad_across_processes.._pad_across_processes..c3s0|](}|kr tntdVqdSrslicerrrhmax_sizeold_sizerrrsc3s,|]$}|krtdntdVqdS)rNrr)rhrrrrr) r"rIrrrFrtrVmaxr*Z new_zerosr#ri) rrh pad_index pad_firstsizesizesZnew_sizeZ new_tensorindicesrrr_pad_across_processess$   z3pad_across_processes.._pad_across_processesT)r0rhrr)rrFr1)rrhrrrrrrpad_across_processess  rmeancCsddd}t||d|dS)a Recursively reduce the tensors in a nested list/tuple/dictionary of lists of tensors across all processes by the mean of a given operation. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to reduce. reduction (`str`, *optional*, defaults to `"mean"`): A reduction method. Can be of "mean", "sum", or "none" Returns: The same data structure as `data` with all the tensors reduced. rcSst}|}|jtjkr|S|jtjkr8td|nz|jjt vrVt j |t j n\|jjtjvrvt j |t j n<|jjtjvrt j |t j n|jtjkrt j |t j |dkr||j}|S)Nsumr)rr]rnr NOror^Z all_reducevaluerrrjr ZSUMrprqrrry)r reductionstateZ cloned_tensorrrr_reduce_across_processess"     z(reduce.._reduce_across_processesT)r0r)rr1)rrrrrrreduces rcCsdd}dd}t|||dS)av Recursively converts the elements nested list/tuple/dictionary of tensors in FP16/BF16 precision to FP32. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to convert from FP16/BF16 to FP32. Returns: The same data structure as `tensor` with all tensors that were in FP16/BF16 precision converted to FP32. cSs|Sr)floatrrrr_convert_to_fp32sz)convert_to_fp32.._convert_to_fp32cSst|do|jtjtjfvSrM)rHrJrfloat16rWrrrr_is_fp16_bf16_tensor!sz-convert_to_fp32.._is_fp16_bf16_tensorrPr1)rrrrrrconvert_to_fp32s rc@s(eZdZdZddZddZddZdS) ConvertOutputsToFp32ad Decorator to apply to a function outputing tensors (like a model forward pass) that ensures the outputs in FP16 precision will be convert back to FP32. Args: model_forward (`Callable`): The function which outputs we want to treat. Returns: The same function as `model_forward` but with converted outputs. cCs||_t||dSr) model_forwardr)selfrrrr__init__4szConvertOutputsToFp32.__init__cOst|j|i|Sr)rr)rr5r7rrr__call__8szConvertOutputsToFp32.__call__cCstddS)NzCannot pickle a prepared model with automatic mixed precision, please unwrap the model with `Accelerator.unwrap_model(model)` before pickling it.)pickle PicklingError)rrrr __getstate__;sz!ConvertOutputsToFp32.__getstate__N)r> __module__ __qualname____doc__rrrrrrrr's rcstfdd}|_|S)Ncs|i|Srr)r5r7rrrforwardDsz(convert_outputs_to_fp32..forward)r __wrapped__)rrrrrconvert_outputs_to_fp32As rcCszt|tr2|D]}t|}|dur|SqnDt|ttfrd|D]}t|}|durD|SqDnt|tjrv|jSdS)z Finds the device on which a nested dict/list/tuple of tensors lies (assuming they are all on the same device). Args: (nested list/tuple/dictionary of `torch.Tensor`): The data we want to know the device of. N) rrvalues find_devicer#r*rrrF)r&r+rFrrrrMs     r)FN)r)rr)r)r)r)rrF)r)7rr functoolsrtypingrrrrr constantsrZ dataclassesr r Zimportsr r Ztorch_xla.core.xla_modelcoreZ xla_modelr^Ztorch.distributedr rrrr)r-r2rCrLrQrSr[rcrmrsrtrzr|r}rrintrrrrrrrrrrrrrrsT       3 %      . %