a ds@sjdZddlZddlZddlmZddlmZmZddlm Z m Z m Z m Z m Z mZddlZddlmZddlmZddlmZee eefe ee eeffe fZd e_e Zee e e e fe e e e e ffe fZd e_Gd d d Zejjejjejjejjejj ejj!ejj"ejj#ejj$ejj%ejj&ejj'ejj(ejj)ejj*hZ+ejj,j-ejj,j.ejj,j/ejj,j0ejj,j1ejj,j2ejj,j3ejj,j4ejj,j5ejj,j6ejj,j7ejj,j8ejj,j9ejj,j:ejj,j;ejj,jejj,j?ej@ejAej?ejBejCejDejEejFejGhZHejId d ddddddddddddddddddd d!d"d#hZJd$d%ZKd&d'ZLd(d)ZMd*d+ZNe eOe d,d-d.ZPd/d0ZQd1d2ZRd3d4ZSd5d6ZTd7d8ZUd9d:ZVd;d<ZWd=d>ZXd?d@ZYdAdBZZdCdDZ[e\dEdFdGZ]dHdIZ^dJdKZ_ej`ej`e\dLdMdNZaebebe\ejce\e ebebfdOdPdQZddRdSZedTdUZfejjgejjge eOdVdWdXZhe e eOe fe eOe fdYdZd[Zie d\d]d^d_Zje e eOe fd\d`dadbZkebebddcdddeZlejmfej`ej`ebebejcej`e\ejne ej`ej`fdf dgdhZoe ebd]didjZpejjge e dkfe eOe e dkffdldmdnZqdsejjre e e e e e e e e e ejsjjtjrdodpdqZugdrZvdS)tz? Utils shared by different modes of quantization (eager/graph) N) OrderedDict)getfullargspec signature)AnyCallableDictOptionalTupleUnion) QuantType)Node)is_parametrizedztorch.ao.quantization.utilsc@seZdZdZdS) MatchAllNodezo A node pattern that matches all nodes, used in defining fusion patterns in FX Graph Mode Quantization N)__name__ __module__ __qualname____doc__rrd/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/torch/ao/quantization/utils.pyr"srreluZrelu_ contiguousdetachZdetach_ hardsigmoidZ hardsigmoid_Zpermuterepeatrepeat_interleaveZreshapeZresize_shapesigmoidZsigmoid_sizesqueezeZsqueeze_tanhZtanh_ transpose unsqueezeZ unsqueeze_viewcCsR|jdko|jtv}|jdko&|jtv}|jdkoFt|t|jtv}|||fS)N call_functionZ call_methodZ call_module)optarget func_list method_listtypestrmodule_type_list)nodemodulesZis_call_functionZis_call_methodZis_call_modulerrr check_nodets r-cCs|}|||SN)copyupdate)Z default_dictZadditional_dictdrrrget_combined_dict{s r2cCs|tjkp|tjkSr.)torchper_tensor_affineper_tensor_symmetricqschemerrr is_per_tensors r8cCs|tjtjtjfvSr.)r3per_channel_affine per_channel_affine_float_qparamsper_channel_symmetricr6rrris_per_channelsr<)objfqnreturncCstt|d|S)zO Given an obj and a fqn such as "foo.bar.baz", returns gm.foo.bar.baz. .) functoolsreducegetattrsplit)r=r>rrrgetattr_from_fqnsrEc CsHtjtjtjtjtjtjtjtjtjtji}||vs@Jd|||S)NzUnsupported dtype: ) r3quint8uint8qint8Zint8qint32int32quint4x2Zquint2x4)ZqdtypeZ DTYPE_MAPPINGrrrto_underlying_dtypesrLcCst|dr|jnd}|j}||d}|s,|St|r|dus J|}|}t|dr,|jnd}|j|j|fS)zh returns the qconfig tuple for qconfig: (activation_dtype, weight_dtype, activation_is_dynamic) N is_dynamicF)rXrZrPrhrM)rVrXrZZact_is_dynamicrrrr` s  r`cCs|dus J|}|}tjtjtjtjg}|j|vrft|drP|j rPt j S|j|vr`t j St j S|jtjkrt|dr|j rt j S|jtjkrt j Std|j|jdS)NrhzKUnrecognized dtype combination in get_quant_type: activation({}),weight({}))rXrZr3rFrHrKrIrMrPrhr ZDYNAMICZSTATICZ WEIGHT_ONLYr\ ExceptionrU)rVrXrZZ static_dtypesrrrrSs$     rS)min_valmax_valr?cCs|dks|dkr&tddS|dks>|dkr~|tdkrd|tdkrdtddS||ksJd||nt||ksJd||dS)z Checks if the given minimum and maximum values are valid, meaning that they exist and the min value is less than the max value. rzMmust run observer before calling calculate_qparams. Returning default values.Finfz-infz!min {} should be less than max {}T)ZnumelwarningswarnZdimfloatrUr3all)rjrkrrrcheck_min_max_valid)s( rq) quant_min quant_maxhas_customized_qrangerM reduce_ranger?c Cs&|r|tjkrd\}}nd\}}||}}|durD|durD||}}||d} |tjkrzd| krndksnJdn(|tjkrd| krdksnJd |r|d |d }}nd|tjkr|rd \}}nd \}}nB|tjkr|rd \}}nd\}}n|tjkrd\}}nd\}}||fS)ztCalculates actual qmin and qmax based on the quantization range, observer datatype and if range is reduced. )r)rNrzRquantization range should be positive and not exceed the maximum bit range (=256).lzYquantization range should be positive and not exceed the maximum bit range (=4294967296).)i?)i)rr|)irv)r)r3rIrHrF) rrrsrtrMruZinitial_quant_minZinitial_quant_maxZcustom_quant_minZcustom_quant_maxZ qrange_lenrrrcalculate_qmin_qmaxHsL                r~cCs8|dd}t|dkr$d|dfS|d|dfSdS)z, Turn 'foo.bar' into ['foo', 'bar'] r@rxrN)rsplitlen)r%rrrr _parent_name}s   rcCs:t|jdkrdSt|r2t|jdko0d|jvSdSdS)z Checks if module._modules is empty or if module is a parametrization, checks that module._modules only has the 'parametrizations' module rTrxZparametrizationsFN)rZ_modulesr )modulerrr)has_no_children_ignoring_parametrizationss r)root submoduler?cCs&|D]\}}||ur|SqdS)aS Get the path (fully qualified name) of a submodule Example:: >> class M(torch.nn.Module): def __init__(self): self.linear = torch.nn.Linear(5, 5) def forward(self, x): return self.linear(x) >> m = M() >> l = m.linear >> _get_path_of_module(m, l) "linear" N)Z named_modules)rrnprrr_get_path_of_modules r)flocr?csfdd|DS)z Get local keyword arguments Example:: >> def f(self, a, b=9): pass >> loc = {"a": 6, "c": 7} >> _get_signature_locals(f, loc) {"a": 6} cs$i|]\}}|tjvr||qSr)r parameters).0kvrrr z)_get_signature_locals..)items)rrrrr_get_signature_localss rzOrderedDict[str, Any])rr?cCsfi}t|jD]J\}}|j|jur2|j||<q|j|jurHd||<q|j|juri||<qt|S)z Get all default keyword arguments from function signature Example:: >> def f(self, a, b=9): pass >> _get_default_kwargs(f) {"b": 9} r) rrrdefaultemptykindVAR_POSITIONAL VAR_KEYWORDr)rkwargsnameparamrrr_get_default_kwargss       r)funcrr?cCs@t|}t||}|}|D]\}}||vr"|||<q"|S)a) Given a function and local function arguments, normalize the keyword arguments by filling in default arguments from function signature Example:: >> def f(self, key1=3, key2=3): pass >> loc = {"key2": 6} >> _normalize_kwargs(f, loc) {"key1": 3, "key2": 6} )rrr/r)rrZdefault_kwargsZ local_kwargsnormalized_kwargsattrvalrrr_normalize_kwargss   r)rrrsr?cCs2|dkr|ksnJd||ks.JddS)aeValidates that the user-specified quantization range is properly initialized and within the given bound supported by the observer dtype. To accommodate lower-bit quantization with respect to the existing torch.qint8 and torch.quint8 datatypes, the user can choose to use dynamic quantization range by passing in a tuple of initial qmin and qmax values. One use case is these customized qmin and qmax values are used to calculate static estimates of the scale and zero point for aggressive lower-bit fake quantization. These estimates are compared against parameters learned through backpropagation. The related literatures for scale and zero point via backpropagation are as follows: Learned Step Size Quantization: https://openreview.net/pdf?id=rkgO66VKDS Trained Quantization Thresholds: https://arxiv.org/pdf/1903.08066.pdf rz1Used-specified quantization range must include 0.zKqmin must be strictly less than qmax for user-specified quantization range.Nr)rrrsrrrvalidate_qmin_qmaxs r) rjrkrrrsrMepsrtr7r?c Cst||s2tjdg|jjdtjdg|jjdfSt|t|}t|t|} |j} tj| tj | d} tj | tj | d} |tj ks|tjkrt| | } | t||d} t| |} |tjks|tjkr|r| | ||d} n| | d} n|tjkrZ||t||} t| |k| t| } d|| } nH| |t||} t| |} |t|| tj} t| ||} t| jdkrtjt| g| j| d} t| jdkrtjt| g| j| d} |tjkrtjt| g| j| d} | | fS)adCalculates the quantization parameters, given min and max value tensors. Works for both per tensor and per channel cases Args: min_val: Minimum values per channel max_val: Maximum values per channel Returns: scales: Scales tensor of shape (#channels,) zero_points: Zero points tensor of shape (#channels,) g?)devicer)rMrrz)rqr3Ztensorrr(minZ zeros_likemaxZonesrfloat32zerosint64r5r;rorGrFZnew_fullr:whereZ ones_likeroundtointclamprrrM) rjrkrrrsrMrrtr7Z min_val_negZ max_val_posrrNrOrrrdetermine_qparamssN (    rcCstt|jS)z Get number of positional args for a function Example:: >> def f(self, key1=3, key2=3): pass >> _get_num_pos_args(f) 3 )rrargsrrrr_get_num_pos_argsBs r.)modelexample_inputsr?csT|ifdd}tjjj|tjj_z||Wtjj_n tjj_0S)a Given a model and its example inputs, return a dictionary from fully qualified name of submodules to example_inputs for that submodule, e.g. {"linear1": (tensor1,), "linear2": (tensor2,), "sub": (tensor3,), "sub.linear1": (tensor4,), ...} Used to make quantizing submodules easier now that FX Graph Mode Quantization requries example inputs. Also works for keyword arguments with default values, we would flatten keyword arguments as positional arguments and fill in the missing keyword args with default values, e.g. if we have a forward function: def forward(self, x, key1=3, key2=3): ... and we call it with self.submodule(x, key2=6) we'll get example_inputs: (x, 3, 6) user can also override `key1` with positional arguments as well: for self.submodule(x, 5, key2=6) we'll get: (x, 5, 6) variable positional arguments and variable positional keyword arguments in forward function are not supported currently, so please make sure no submodules is using them. c st|}t|j|}t|jd}|t|}|rP|rP|jdd|d8}q2||t |}t |}|dur||<|g|Ri|S)NrxF)last) listr/rZforwardrrpopitemextendvaluestupler) selfrrZsubmodule_example_inputsrnum_argsZ num_to_popZsubmodule_example_inputs_tupler>Zfqn_to_example_inputsZorig_module_callrrr_patched_module_callns      z7get_fqn_to_example_inputs.._patched_module_call)r3nnModule__call__)rrrrrrget_fqn_to_example_inputsNs   r) float_lstmlinear_output_obs_ctrsigmoid_obs_ctr tanh_obs_ctrcell_state_obs_ctrhidden_state_obs_ctrr?c Csttjjjddd}tjjj|j|j |j |j |j |j |j}|jD]}|jg} |jrf| |j| D]} | j} |dur||} | | j_| | j_|dur||} | | j_| | j_| | j_|dur||| j_|dur||| j_|} t| drt| dr| j| jf| _| j | _!|durj||| j"_|} t| drZt| drZ| j| jf| _#| j | _$qjqFddl%m&}||tjj'tjj(gd |S) aJ Return an observed `torch.ao.nn.quantizable.LSTM` created from a `torch.nn.LSTM` with specific observers or fake quantizes assigned to the inner ops or submodules. In both eager and FX graph mode quantization, `torch.ao.nn.quantizable.LSTM` is used as an observed custom module, which is responsible for inserting its own observers. By default, all inner ops inherit the parent custom module's QConfig. Users who wish to override this behavior may extend `torch.ao.nn.quantizable.LSTM` and use this helper function to customize the observer insertion logic. Args: `float_lstm`: The float LSTM module `linear_output_obs_ctr`: observer or fake quantize for linear outputs Wx + b, where W is the weight matrix, b is the bias, and x is either the inputs or the hidden state from the previous layer (if any) `sigmoid_obs_ctr`: observer or fake quantize for sigmoid activations `tanh_obs_ctr`: observer or fake quantize for tanh activations `cell_state_obs_ctr`: observer or fake quantize for the cell state `hidden_state_obs_ctr`: observer or fake quantize for the hidden state and the output Return: A `torch.ao.nn.quantizable.LSTM` with the specified observers or fake quantizes attached to the inner submodules. )obs_ctrr?cSs:t|tjjjrtjjj}n tjjj}tjjj||dS)zP Make a QConfig with fixed qparams observers or fake quantizes. )rXrZ) isinstancer3ao quantizationZFakeQuantizeBaseZdefault_weight_fake_quantZdefault_weight_observerQConfig)rrZrrr make_qconfigs  z@_get_lstm_with_individually_observed_parts..make_qconfigNrNrOr)_add_observer_)Znon_leaf_module_list))rr3rrrr quantizableLSTMZ input_sizeZ hidden_sizeZ num_layersZbiasZ batch_firstdropout bidirectionalZlayersZlayer_fwappendZlayer_bwcellZigatesrVZhgatesZ input_gateZ forget_gateZ output_gateZ cell_gateZfgate_cx_igate_cgaterPrNrOZinitial_cell_state_qparamsrMZcell_state_dtypeZogate_cyZinitial_hidden_state_qparamsZhidden_state_dtypeZtorch.ao.quantization.quantizerSigmoidTanh)rrrrrrrZ observed_lstmZlayerZ inner_layersZ inner_layerrrVZobsrrrr*_get_lstm_with_individually_observed_partssP"          r) NodePatternPatternrr-r2r8r<rErRrWrYr[r_r]rcrdrerfrgr`rSrqr~rrrLrr)NNNNN)wrrArm collectionsrinspectrrtypingrrrrr r r3Z torch.ao.quantization.quant_typer Ztorch.fxr Ztorch.nn.utils.parametrizer rrZ QuantizerClsrrrZReLUZReLU6ZAdaptiveAvgPool1dZAdaptiveAvgPool2dZAdaptiveAvgPool3dZ AvgPool1dZ AvgPool2dZ AvgPool3dZ MaxPool1dZ MaxPool2dZ MaxPool3dZIdentityZ Hardsigmoidrrr*Z functionalZadaptive_avg_pool1dZadaptive_avg_pool2dZadaptive_avg_pool3dZeluZ hardswishZ instance_normZ layer_normZ leaky_reluZsiluZmishrZ max_pool1dZ max_pool2dZ max_pool3drZhardtanhZ hardtanh_rrr rrstacksumrr!catr&meanr'r-r2r8r<r)rErLrRrWrYr[r_r]rcrdrerfboolrgr`rSZTensorrqrrMr~rrrrrrrrr4r7rrrrrrr__all__rrrrs      &"       5  "    G   < Z