a  d^1@sddlZddlmZddlmZddlmZddlmZm Z m Z e rbddl Z ddl Z ddlmZe rddlmZdd lmZeeZdd d Zdd dZdddZddZddZddZdS)N)deepcopy)version)logging)importlib_metadatais_accelerate_availableis_bitsandbytes_available)init_empty_weights)find_tied_parameterscCsd|vrV|d}|ddD].}t||}|durHt|d|d|}q|d}||jvr~||jvr~t|d|d||jv}t||} | jtdkr|dtdfvr|durt|d|dd} d} |stsd} d} n6tt j d o t |j|t j j } t |j|t j j } | s.| r4|j|} | jjd kr|dur\| |} n^t |tjr|d } |jtjkrttd td k}|stdntj|d d} | j}| rt j j | fddi||} n$| r t j j | fddi||} | |j|<|durt|jd||nj|durJ| |} n(t |tjrd||} ntj||d} |r| |j|<nt j| | jd} | |j|<dS)a A helper function to set a given tensor (parameter of buffer) of a module on a specific device (note that doing `param.to(device)` creates a new tensor not linked to the parameter, which is why we need this function). The function is adapted from `set_module_tensor_to_device` function from accelerate that is adapted to support the class `Int8Params` from `bitsandbytes`. Args: module (`torch.nn.Module`): The module in which the tensor we want to move lives. tensor_name (`str`): The full name of the parameter/buffer. device (`int`, `str` or `torch.device`): The device on which to set the tensor. value (`torch.Tensor`, *optional*): The value of the tensor (useful when going from the meta device to any other device). fp16_statistics (`torch.HalfTensor`, *optional*): The list of fp16 statistics to set on the module, used for serialization. .Nz has no attribute z- does not have a parameter or a buffer named metaz7 is on the meta device, we need a `value` to put in on F Params4bitZcudacpu bitsandbytesz0.37.2zDetected int8 weights but the version of bitsandbytes is not compatible with int8 serialization. Make sure to download the latest `bitsandbytes` version. `pip install --upgrade bitsandbytes`.)device requires_gradZSCB)r)splitgetattr ValueError _parameters_buffersrtorchr hasattrbnbnn isinstancerZ Int8ParamstypetoZTensorZdtypeZint8rparserZtensor__dict__setattrZweight Parameterr)moduleZ tensor_namervalueZfp16_statisticssplitsrZ new_moduleZ is_buffer old_valueZis_4bitZis_8bitparam new_valueZis_8bit_serializablekwargsr+h/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/transformers/utils/bitsandbytes.py%set_module_quantized_tensor_to_devicesl    * "            r-Fc sT|D]@\}}durg|t|tjr||vrtfdd|Dst|dkrtjj |j |j |j du|j |jd|j|<d}nH|jdur||jvrn2tjj|j |j |j du|j|j|jd|j|<d}|j|dWdn1s 0Ytt|d kr@t||||d \}}d q||fS) z Private method that wraps the recursion for module replacement. Returns the converted model and a boolean that indicates if the conversion has been successfull or not. Nc3s|]}|dvVqdS)r N)join).0keycurrent_key_namer+r, z+_replace_with_bnb_linear..Zllm_int8)Zhas_fp16_weights thresholdT)Zcompress_statisticsZ quant_typeFr)has_been_replacedr )Znamed_childrenappendrrZLinearanyr Zquantization_methodrZ Linear8bitLtZ in_featuresZ out_featuresZbiasZllm_int8_has_fp16_weightZllm_int8_thresholdZ_modulesZllm_int8_skip_modulesZ Linear4bitZbnb_4bit_compute_dtypeZbnb_4bit_use_double_quantZbnb_4bit_quant_typeZrequires_grad_lenlistchildren_replace_with_bnb_linearpop)modelmodules_to_not_convertr2quantization_configr6namer$_r+r1r,r<psR    0  r<cCs6|durdgn|}t||||\}}|s2td|S)a6 A helper function to replace all `torch.nn.Linear` modules by `bnb.nn.Linear8bit` modules from the `bitsandbytes` library. This will enable running your models using mixed int8 precision as described by the paper `LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale`. Make sure `bitsandbytes` compiled with the correct CUDA version of your hardware is installed before running this function. `pip install -i https://test.pypi.org/simple/ bitsandbytes` The function will be run recursively and replace all `torch.nn.Linear` modules except for the `lm_head` that should be kept as a `torch.nn.Linear` module. The replacement is done under `init_empty_weights` context manager so no CPU/GPU memory is required to run this function. Int8 mixed-precision matrix decomposition works by separating a matrix multiplication into two streams: (1) and systematic feature outlier stream matrix multiplied in fp16 (0.01%), (2) a regular stream of int8 matrix multiplication (99.9%). With this method, int8 inference with no predictive degradation is possible for very large models (>=176B parameters). Parameters: model (`torch.nn.Module`): Input model or `torch.nn.Module` as the function is run recursively. modules_to_not_convert (`List[`str`]`, *optional*, defaults to `["lm_head"]`): Names of the modules to not convert in `Linear8bitLt`. In practice we keep the `lm_head` in full precision for numerical stability reasons. current_key_name (`List[`str`]`, *optional*): An array to track the current key of the recursion. This is used to check whether the current key (part of it) is not in the list of modules to not convert (for instances modules that are offloaded to `cpu` or `disk`). NZlm_headaYou are loading your model in 8bit or 4bit but no linear modules were found in your model. this can happen for some architectures such as gpt2 that uses Conv1D instead of Linear layers. Please double check your model architecture, or submit an issue on github if you think this is a bug.)r<loggerwarning)r>r?r2r@r6r+r+r,replace_with_bnb_linearsrEcOstdtt|i|S)Nzj`replace_8bit_linear` will be deprecated in a future version, please use `replace_with_bnb_linear` instead)warningswarn FutureWarningrEargsr*r+r+r,replace_8bit_linears rKcOstdtt|i|S)Nz`set_module_8bit_tensor_to_device` will be deprecated in a future version, please use `set_module_quantized_tensor_to_device` instead)rFrGrHr-rIr+r+r, set_module_8bit_tensor_to_devices rLcCst|}|t|}t|tr0t|}ntdd|Dg}t|dk}t ||j  }|sj|rjgSt| }|ddg}t |t |}|t|} ddg} g} | D],} | D]} | | vr| | d} q| | q| S)a An utility function to get the key of the module to keep in full precision if any For example for CausalLM modules we may want to keep the lm_head in full precision for numerical stability reasons. For other architectures, we want to keep the tied weights of the model. The function will return a list of the keys of the modules to not convert in int8. Parameters: model (`torch.nn.Module`): Input model cSsg|]}|ddqS)rNr+)r/xr+r+r, r4z+get_keys_to_not_convert..rr z.weightz.bias)rZ tie_weightsr rdictr:valuessumr9rZbase_model_prefixZnamed_parameterssetreplacer7)r>Z tied_modelZ tied_paramsZ tied_keysZhas_tied_paramsZ is_base_modelZ list_modulesZlist_last_module intersectionZlist_untouchedZnames_to_removeZfiltered_module_namesrAZname_to_remover+r+r,get_keys_to_not_converts,      rV)NN)NNNF)NNN)rFcopyr packagingrutilsrZ import_utilsrrr rrrZtorch.nnrZ accelerater Zaccelerate.utilsr Z get_logger__name__rCr-r<rErKrLrVr+r+r+r,s&        [ 9 +