a d@sNddlZddlZddlZddlZddlmZmZmZmZddl Z ddl m Z ddl m Z ddl mZddlmZdd lmZd d lmZgd Zejejd Zd#e e ee dddZd$e e ee dddZGddde ZGdddeZGdddeZGdddeZGddde ZGdddeZ Gdd d eZ!Gd!d"d"eZ"dS)%N)ListTupleOptionaloverload)Tensor)Module) Parameter)PackedSequence)init)_VF)RNNBaseRNNLSTMGRU RNNCellBaseRNNCellLSTMCellGRUCell)RNN_TANHRNN_RELU)tensor permutationdimreturncCs |||SN)Z index_selectrrrr]/var/www/html/stable-diffusion-webui/venv/lib/python3.9/site-packages/torch/nn/modules/rnn.py_apply_permutationsr!cCstdt|||S)NzYapply_permutation is deprecated, please use tensor.index_select(dim, permutation) instead)warningswarnr!rrrr apply_permutations r$c seZdZUgdZdgZeed<eed<eed<eed<eed<eed<e ed <eed <eed <d;eeeeeee eedd fdd Z ddZ fddZ ddddZ fddZddddZeeedd d!d"Zeeeeeeefd d#d$Zd|d krd |}n,|d kr |}n|dkr|}n td|g|_g|_t|D]t| D]}| dkr^| n|}dkrp|n|| }ttj||ffi| }ttj||ffi| }ttj|fi| }ttj|fi| }d}|j dkr|r||||f}n||f}n:ttj| |ffi| }|r6|||||f}n |||f}|dkrNdndddg}|rl|ddg7}|j dkr|dg7}fdd|D}t||D]\}}t|||q|j||j|qJq<|| dS)Ndevicedtyper rrzbdropout should be a number in range [0, 1] representing the probability of an element being zeroedzdropout option adds dropout after all but last recurrent layer, so non-zero dropout expects num_layers greater than 1, but got dropout={} and num_layers={}zEproj_size should be a positive integer or zero to disable projectionsz,proj_size has to be smaller than hidden_sizerrr rrzUnrecognized RNN mode: r_reverseweight_ih_l{}{}weight_hh_l{}{} bias_ih_l{}{} bias_hh_l{}{}weight_hr_l{}{}csg|]}|qSrformat.0xlayersuffixrr {z$RNNBase.__init__..)!super__init__r%r&r'r(r)r*floatr+r,r-_flat_weight_refs isinstancenumbersNumberbool ValueErrorr"r#r<_flat_weights_names _all_weightsranger torchemptyzipsetattrextendappend_init_flat_weightsreset_parameters)selfr%r&r'r(r)r*r+r,r-r1r2factory_kwargsnum_directionsZ gate_size directionreal_hidden_sizeZlayer_input_sizeZw_ihZw_hhZb_ihZb_hhZ layer_paramsZw_hrZ param_namesnameparam __class__r@r rF/s    $            zRNNBase.__init__cs4fddjD_ddjD_dS)Ncs$g|]}t|rt|ndqSrhasattrgetattrr>ZwnrYrr rCsz.RNNBase._init_flat_weights..cSs"g|]}|durt|ndqSrweakrefrefr>wrrr rCs)rN _flat_weightsrHflatten_parametersrfrrfr rWs zRNNBase._init_flat_weightscs<t|dr*||jvr*|j|}||j|<t||dS)NrN)rcrNindexrlrE __setattr__)rYattrvalueidxr`rr ros  zRNNBase.__setattr__rcCst|jt|jkrdS|jD]}t|tsdSq|jd}|j}|jD]:}t|jtr~|jj|kr~|jjr~tj j |jsJdSqJdd|jD}t|t|jkrdStj |ddlm m m}tptr<|jrdnd}|jdkr|d7}t|j||j||j|j|j|j|jt|j Wdn1sR0YWdn1sr0YdS)zResets parameter data pointer so that they can use faster code paths. Right now, this works only if the module is on the GPU and cuDNN is enabled. Otherwise, it's a no-op. NrcSsh|] }|qSr)Zdata_ptr)r>prrr rDz-RNNBase.flatten_parameters..r3r r)lenrlrNrIrr2dataZis_cudarQbackendsZcudnnZ is_acceptableZcudaZ device_ofZtorch.backends.cudnn.rnnrnnZno_gradZ_use_cudnn_rnn_flatten_weightr)r-Z_cudnn_rnn_flatten_weightr&Zget_cudnn_moder%r'r(r*rLr,)rYrkZfirst_fwr2fwZunique_data_ptrsryZ num_weightsrrr rms<         zRNNBase.flatten_parameterscst|}||Sr)rE_applyrW)rYfnretr`rr r{s zRNNBase._applycCs@|jdkrdt|jnd}|D]}t|| |q&dSNrg?r'mathsqrt parametersr Zuniform_rYZstdvweightrrr rXs zRNNBase.reset_parametersinput batch_sizesrcCs\|dur dnd}||kr0td|||j|dkrXtd|j|ddS)Nr r z%input must have {} dimensions, got {}z?input.size(-1) must be equal to input_size. Expected {}, got {})r RuntimeErrorr<r&size)rYrrZexpected_input_dimrrr check_inputs  zRNNBase.check_inputcCsr|durt|d}n|jr&|dn|d}|jr:dnd}|jdkr\|j|||jf}n|j|||jf}|SNrrr )intr*rr,r-r(r'rYrrZ mini_batchr[expected_hidden_sizerrr get_expected_hidden_sizes z RNNBase.get_expected_hidden_sizeExpected hidden size {}, got {})hxrmsgrcCs(||kr$t||t|dSr)rrr<list)rYrrrrrr check_hidden_sizes zRNNBase.check_hidden_sizecCsZd}t|j|jD]B\}}t||r.t||nd}|dur|dur||urd}qVq|S)NFT)rSrHrNrcrd)rYZweights_changedrir^rrrr _weights_have_changedszRNNBase._weights_have_changedrhiddenrcCs(||||||}|||dSr)rrr)rYrrrrrrr check_forward_argss  zRNNBase.check_forward_args)rrcCs|dur |St||Srr!rYrrrrr permute_hiddenszRNNBase.permute_hiddencCsd}|jdkr|d7}|jdkr(|d7}|jdur:|d7}|jdurL|d 7}|jdkr^|d 7}|jdurp|d 7}|jfi|jS) N{input_size}, {hidden_size}rz, proj_size={proj_size}rz, num_layers={num_layers}T , bias={bias}Fz, batch_first={batch_first}z, dropout={dropout}z, bidirectional={bidirectional})r-r(r)r*r+r,r<__dict__rYsrrr extra_reprs      zRNNBase.extra_reprcCs|j}|d=|S)NrH)rcopy)rYstaterrr __getstate__s zRNNBase.__getstate__cst|d|vr|d_d|vr,d_tjddtsj}jrRdnd}g_g_t |D] t |D]}|dkrdndgd}fd d |D}j rjdkrڈj|g7_j |n,j|dd g7_j |dd qxjdkr^j|ddg|d dg7_j |dd|d dgqxj|ddg7_j |ddqxqjfd d jD_ dd j D_ dS)Nr.r-rr rr4r5)r6r7r8r9r:csg|]}|qSrr;r=r@rr rC2rDz(RNNBase.__setstate__..r3rcs$g|]}t|rt|ndqSrrbrerfrr rCAscSs"g|]}|durt|ndqSrrgrjrrr rCDs)rE __setstate__rOr-rIstrr(r,rNrPr)rUrlrH)rYdr(r[r\weightsr`)rArYrBr rs@    &$ zRNNBase.__setstate__csfddjDS)Ncsg|]}fdd|DqS)csg|]}t|qSr)rd)r>rrfrr rCIrDz2RNNBase.all_weights...r)r>rrfrr rCIrDz'RNNBase.all_weights..)rOrfrrfr r.GszRNNBase.all_weightscs.t}|jdd|_|jdd|_|Sr)rE_replicate_for_data_parallelrlrN)rYZreplicar`rr rKs z$RNNBase._replicate_for_data_parallel)rTFr/FrNN)r)"__name__ __module__ __qualname__ __constants__Z__jit_unused_properties__r__annotations__rrLrGrFrWrormr{rXrrrrrrrrrrrrpropertyrr r.r __classcell__rrr`r r sN W 0     )rcseZdZdZfddZeejjd e e e e e e fdddZ eejjd e e e e e e fdddZ d d dZ ZS) raApplies a multi-layer Elman RNN with :math:`\tanh` or :math:`\text{ReLU}` non-linearity to an input sequence. For each element in the input sequence, each layer computes the following function: .. math:: h_t = \tanh(x_t W_{ih}^T + b_{ih} + h_{t-1}W_{hh}^T + b_{hh}) where :math:`h_t` is the hidden state at time `t`, :math:`x_t` is the input at time `t`, and :math:`h_{(t-1)}` is the hidden state of the previous layer at time `t-1` or the initial hidden state at time `0`. If :attr:`nonlinearity` is ``'relu'``, then :math:`\text{ReLU}` is used instead of :math:`\tanh`. Args: input_size: The number of expected features in the input `x` hidden_size: The number of features in the hidden state `h` num_layers: Number of recurrent layers. E.g., setting ``num_layers=2`` would mean stacking two RNNs together to form a `stacked RNN`, with the second RNN taking in outputs of the first RNN and computing the final results. Default: 1 nonlinearity: The non-linearity to use. Can be either ``'tanh'`` or ``'relu'``. Default: ``'tanh'`` bias: If ``False``, then the layer does not use bias weights `b_ih` and `b_hh`. Default: ``True`` batch_first: If ``True``, then the input and output tensors are provided as `(batch, seq, feature)` instead of `(seq, batch, feature)`. Note that this does not apply to hidden or cell states. See the Inputs/Outputs sections below for details. Default: ``False`` dropout: If non-zero, introduces a `Dropout` layer on the outputs of each RNN layer except the last layer, with dropout probability equal to :attr:`dropout`. Default: 0 bidirectional: If ``True``, becomes a bidirectional RNN. Default: ``False`` Inputs: input, h_0 * **input**: tensor of shape :math:`(L, H_{in})` for unbatched input, :math:`(L, N, H_{in})` when ``batch_first=False`` or :math:`(N, L, H_{in})` when ``batch_first=True`` containing the features of the input sequence. The input can also be a packed variable length sequence. See :func:`torch.nn.utils.rnn.pack_padded_sequence` or :func:`torch.nn.utils.rnn.pack_sequence` for details. * **h_0**: tensor of shape :math:`(D * \text{num\_layers}, H_{out})` for unbatched input or :math:`(D * \text{num\_layers}, N, H_{out})` containing the initial hidden state for the input sequence batch. Defaults to zeros if not provided. where: .. math:: \begin{aligned} N ={} & \text{batch size} \\ L ={} & \text{sequence length} \\ D ={} & 2 \text{ if bidirectional=True otherwise } 1 \\ H_{in} ={} & \text{input\_size} \\ H_{out} ={} & \text{hidden\_size} \end{aligned} Outputs: output, h_n * **output**: tensor of shape :math:`(L, D * H_{out})` for unbatched input, :math:`(L, N, D * H_{out})` when ``batch_first=False`` or :math:`(N, L, D * H_{out})` when ``batch_first=True`` containing the output features `(h_t)` from the last layer of the RNN, for each `t`. If a :class:`torch.nn.utils.rnn.PackedSequence` has been given as the input, the output will also be a packed sequence. * **h_n**: tensor of shape :math:`(D * \text{num\_layers}, H_{out})` for unbatched input or :math:`(D * \text{num\_layers}, N, H_{out})` containing the final hidden state for each element in the batch. Attributes: weight_ih_l[k]: the learnable input-hidden weights of the k-th layer, of shape `(hidden_size, input_size)` for `k = 0`. Otherwise, the shape is `(hidden_size, num_directions * hidden_size)` weight_hh_l[k]: the learnable hidden-hidden weights of the k-th layer, of shape `(hidden_size, hidden_size)` bias_ih_l[k]: the learnable input-hidden bias of the k-th layer, of shape `(hidden_size)` bias_hh_l[k]: the learnable hidden-hidden bias of the k-th layer, of shape `(hidden_size)` .. note:: All the weights and biases are initialized from :math:`\mathcal{U}(-\sqrt{k}, \sqrt{k})` where :math:`k = \frac{1}{\text{hidden\_size}}` .. note:: For bidirectional RNNs, forward and backward are directions 0 and 1 respectively. Example of splitting the output layers when ``batch_first=False``: ``output.view(seq_len, batch, num_directions, hidden_size)``. .. note:: ``batch_first`` argument is ignored for unbatched inputs. .. include:: ../cudnn_rnn_determinism.rst .. include:: ../cudnn_persistent_rnn.rst Examples:: >>> rnn = nn.RNN(10, 20, 2) >>> input = torch.randn(5, 3, 10) >>> h0 = torch.randn(2, 3, 20) >>> output, hn = rnn(input, h0) csld|vrtd|dd|_|jdkr.d}n |jdkr>d}ntd|jtj|g|Ri|dS) Nr-=proj_size argument is only supported for LSTM, not RNN or GRU nonlinearitytanhrrelurzUnknown nonlinearity '{}')rMpoprr<rErF)rYargskwargsr%r`rr rFs  z RNN.__init__NrrrcCsdSrrrYrrrrr forwardsz RNN.forwardcCsdSrrrrrr rsc Cstjs|r||}t|trB|\}}}}t|d}nd}|dvsfJd|d|dk}|j r|dnd} |s| | }|dur|dkrt d|d| d}n(|dur|dkrt d |d|j r| dn| d}d}d}|durH|j r"dnd} tj|j| ||j|j|jd }n |||}|dusbJ|||||jd ks|jd ksJ|dur|jd krt|||j|j|j|j|j|j |j } n(t|||j|j|j|j|j|j |j } nZ|jd kr*t||||j|j|j|j|j|j } n&t||||j|j|j|j|j|j } | d} | d} t|trt| |||}||| |fS|s| | } | d} | || |fS) Nrr r z2RNN: Expected input to be 2-D or 3-D but received -D tensorr rr 7For unbatched 2-D input, hx should also be 2-D but got 5For batched 3-D input, hx should also be 3-D but got r2r1rr)rQjit is_scriptingrrWrIr rrr* unsqueezerrr,zerosr(r'r2r1rrr%rrnn_tanhrlr)r+trainingrnn_relusqueezerYrr orig_inputrsorted_indicesunsorted_indicesmax_batch_size is_batched batch_dimr[resultoutputr output_packedrrr rs                     )N)N)Nrrr__doc__rFrrQ _jit_internal_overload_methodrrrrr rrrr`r rTsf $$rc seZdZdZfddZeeeeeeefdddZ eeeefeeddd Z eeefeeeeefd d d Z e e jjdeeeeefeeeeeffdddZe e jjdeeeeefeeeeeffdddZdddZZS)raW$Applies a multi-layer long short-term memory (LSTM) RNN to an input sequence. For each element in the input sequence, each layer computes the following function: .. math:: \begin{array}{ll} \\ i_t = \sigma(W_{ii} x_t + b_{ii} + W_{hi} h_{t-1} + b_{hi}) \\ f_t = \sigma(W_{if} x_t + b_{if} + W_{hf} h_{t-1} + b_{hf}) \\ g_t = \tanh(W_{ig} x_t + b_{ig} + W_{hg} h_{t-1} + b_{hg}) \\ o_t = \sigma(W_{io} x_t + b_{io} + W_{ho} h_{t-1} + b_{ho}) \\ c_t = f_t \odot c_{t-1} + i_t \odot g_t \\ h_t = o_t \odot \tanh(c_t) \\ \end{array} where :math:`h_t` is the hidden state at time `t`, :math:`c_t` is the cell state at time `t`, :math:`x_t` is the input at time `t`, :math:`h_{t-1}` is the hidden state of the layer at time `t-1` or the initial hidden state at time `0`, and :math:`i_t`, :math:`f_t`, :math:`g_t`, :math:`o_t` are the input, forget, cell, and output gates, respectively. :math:`\sigma` is the sigmoid function, and :math:`\odot` is the Hadamard product. In a multilayer LSTM, the input :math:`x^{(l)}_t` of the :math:`l` -th layer (:math:`l >= 2`) is the hidden state :math:`h^{(l-1)}_t` of the previous layer multiplied by dropout :math:`\delta^{(l-1)}_t` where each :math:`\delta^{(l-1)}_t` is a Bernoulli random variable which is :math:`0` with probability :attr:`dropout`. If ``proj_size > 0`` is specified, LSTM with projections will be used. This changes the LSTM cell in the following way. First, the dimension of :math:`h_t` will be changed from ``hidden_size`` to ``proj_size`` (dimensions of :math:`W_{hi}` will be changed accordingly). Second, the output hidden state of each layer will be multiplied by a learnable projection matrix: :math:`h_t = W_{hr}h_t`. Note that as a consequence of this, the output of LSTM network will be of different shape as well. See Inputs/Outputs sections below for exact dimensions of all variables. You can find more details in https://arxiv.org/abs/1402.1128. Args: input_size: The number of expected features in the input `x` hidden_size: The number of features in the hidden state `h` num_layers: Number of recurrent layers. E.g., setting ``num_layers=2`` would mean stacking two LSTMs together to form a `stacked LSTM`, with the second LSTM taking in outputs of the first LSTM and computing the final results. Default: 1 bias: If ``False``, then the layer does not use bias weights `b_ih` and `b_hh`. Default: ``True`` batch_first: If ``True``, then the input and output tensors are provided as `(batch, seq, feature)` instead of `(seq, batch, feature)`. Note that this does not apply to hidden or cell states. See the Inputs/Outputs sections below for details. Default: ``False`` dropout: If non-zero, introduces a `Dropout` layer on the outputs of each LSTM layer except the last layer, with dropout probability equal to :attr:`dropout`. Default: 0 bidirectional: If ``True``, becomes a bidirectional LSTM. Default: ``False`` proj_size: If ``> 0``, will use LSTM with projections of corresponding size. Default: 0 Inputs: input, (h_0, c_0) * **input**: tensor of shape :math:`(L, H_{in})` for unbatched input, :math:`(L, N, H_{in})` when ``batch_first=False`` or :math:`(N, L, H_{in})` when ``batch_first=True`` containing the features of the input sequence. The input can also be a packed variable length sequence. See :func:`torch.nn.utils.rnn.pack_padded_sequence` or :func:`torch.nn.utils.rnn.pack_sequence` for details. * **h_0**: tensor of shape :math:`(D * \text{num\_layers}, H_{out})` for unbatched input or :math:`(D * \text{num\_layers}, N, H_{out})` containing the initial hidden state for each element in the input sequence. Defaults to zeros if (h_0, c_0) is not provided. * **c_0**: tensor of shape :math:`(D * \text{num\_layers}, H_{cell})` for unbatched input or :math:`(D * \text{num\_layers}, N, H_{cell})` containing the initial cell state for each element in the input sequence. Defaults to zeros if (h_0, c_0) is not provided. where: .. math:: \begin{aligned} N ={} & \text{batch size} \\ L ={} & \text{sequence length} \\ D ={} & 2 \text{ if bidirectional=True otherwise } 1 \\ H_{in} ={} & \text{input\_size} \\ H_{cell} ={} & \text{hidden\_size} \\ H_{out} ={} & \text{proj\_size if } \text{proj\_size}>0 \text{ otherwise hidden\_size} \\ \end{aligned} Outputs: output, (h_n, c_n) * **output**: tensor of shape :math:`(L, D * H_{out})` for unbatched input, :math:`(L, N, D * H_{out})` when ``batch_first=False`` or :math:`(N, L, D * H_{out})` when ``batch_first=True`` containing the output features `(h_t)` from the last layer of the LSTM, for each `t`. If a :class:`torch.nn.utils.rnn.PackedSequence` has been given as the input, the output will also be a packed sequence. When ``bidirectional=True``, `output` will contain a concatenation of the forward and reverse hidden states at each time step in the sequence. * **h_n**: tensor of shape :math:`(D * \text{num\_layers}, H_{out})` for unbatched input or :math:`(D * \text{num\_layers}, N, H_{out})` containing the final hidden state for each element in the sequence. When ``bidirectional=True``, `h_n` will contain a concatenation of the final forward and reverse hidden states, respectively. * **c_n**: tensor of shape :math:`(D * \text{num\_layers}, H_{cell})` for unbatched input or :math:`(D * \text{num\_layers}, N, H_{cell})` containing the final cell state for each element in the sequence. When ``bidirectional=True``, `c_n` will contain a concatenation of the final forward and reverse cell states, respectively. Attributes: weight_ih_l[k] : the learnable input-hidden weights of the :math:`\text{k}^{th}` layer `(W_ii|W_if|W_ig|W_io)`, of shape `(4*hidden_size, input_size)` for `k = 0`. Otherwise, the shape is `(4*hidden_size, num_directions * hidden_size)`. If ``proj_size > 0`` was specified, the shape will be `(4*hidden_size, num_directions * proj_size)` for `k > 0` weight_hh_l[k] : the learnable hidden-hidden weights of the :math:`\text{k}^{th}` layer `(W_hi|W_hf|W_hg|W_ho)`, of shape `(4*hidden_size, hidden_size)`. If ``proj_size > 0`` was specified, the shape will be `(4*hidden_size, proj_size)`. bias_ih_l[k] : the learnable input-hidden bias of the :math:`\text{k}^{th}` layer `(b_ii|b_if|b_ig|b_io)`, of shape `(4*hidden_size)` bias_hh_l[k] : the learnable hidden-hidden bias of the :math:`\text{k}^{th}` layer `(b_hi|b_hf|b_hg|b_ho)`, of shape `(4*hidden_size)` weight_hr_l[k] : the learnable projection weights of the :math:`\text{k}^{th}` layer of shape `(proj_size, hidden_size)`. Only present when ``proj_size > 0`` was specified. weight_ih_l[k]_reverse: Analogous to `weight_ih_l[k]` for the reverse direction. Only present when ``bidirectional=True``. weight_hh_l[k]_reverse: Analogous to `weight_hh_l[k]` for the reverse direction. Only present when ``bidirectional=True``. bias_ih_l[k]_reverse: Analogous to `bias_ih_l[k]` for the reverse direction. Only present when ``bidirectional=True``. bias_hh_l[k]_reverse: Analogous to `bias_hh_l[k]` for the reverse direction. Only present when ``bidirectional=True``. weight_hr_l[k]_reverse: Analogous to `weight_hr_l[k]` for the reverse direction. Only present when ``bidirectional=True`` and ``proj_size > 0`` was specified. .. note:: All the weights and biases are initialized from :math:`\mathcal{U}(-\sqrt{k}, \sqrt{k})` where :math:`k = \frac{1}{\text{hidden\_size}}` .. note:: For bidirectional LSTMs, forward and backward are directions 0 and 1 respectively. Example of splitting the output layers when ``batch_first=False``: ``output.view(seq_len, batch, num_directions, hidden_size)``. .. note:: For bidirectional LSTMs, `h_n` is not equivalent to the last element of `output`; the former contains the final forward and reverse hidden states, while the latter contains the final forward hidden state and the initial reverse hidden state. .. note:: ``batch_first`` argument is ignored for unbatched inputs. .. include:: ../cudnn_rnn_determinism.rst .. include:: ../cudnn_persistent_rnn.rst Examples:: >>> rnn = nn.LSTM(10, 20, 2) >>> input = torch.randn(5, 3, 10) >>> h0 = torch.randn(2, 3, 20) >>> c0 = torch.randn(2, 3, 20) >>> output, (hn, cn) = rnn(input, (h0, c0)) cstjdg|Ri|dS)NrrErFrYrrr`rr rFsz LSTM.__init__rcCsT|durt|d}n|jr&|dn|d}|jr:dnd}|j|||jf}|Sr)rr*rr,r(r'rrrr get_expected_cell_sizeszLSTM.get_expected_cell_sizercCsD|||||d|||d||d|||ddS)Nrz"Expected hidden[0] size {}, got {}rz"Expected hidden[1] size {}, got {})rrrr)rYrrrrrr rs zLSTM.check_forward_args)rrrcCs(|dur |St|d|t|d|fS)Nrrrrrrr rszLSTM.permute_hiddenNrcCsdSrrrrrr rsz LSTM.forwardcCsdSrrrrrr rsc Cstjs|r||}d}t|trJ|\}}}}|d}t|}nnd}|dvsnJd|d|dk}|j rdnd} |s| | }|j r| dn| d}d}d}|dur.|j rdnd} |j dkr|j n|j} tj|j| || |j|jd} tj|j| ||j|j|jd} | | f}n|dur|r|ddksb|ddkrd |dd |dd }t|nj|ddks|ddkrd |dd |dd }t||d d|d df}|||}|||||durFt|||j|j|j|j|j|j |j }n&t||||j|j|j|j|j|j }|d}|dd}t|trt||||}||||fS|s|| }|dd|ddf}||||fSdS) Nrrz3LSTM: Expected input to be 2-D or 3-D but received rr rr rz=For batched 3-D input, hx and cx should also be 3-D but got (z-D, z -D) tensorsz?For unbatched 2-D input, hx and cx should also be 2-D but got ()rQrrrrWrIr rrr*rrr,r-r'rr(r2r1rrrrZlstmrlr)r+rr)rYrrrrrrrrrr[r]Zh_zerosZc_zerosrrrrrrrr rs            $   $       )N)N)N)rrrrrFrrrrrrrrrQrrrr rrrr`r r's.     rcseZdZdZfddZeejjd e e e e e e fdddZ eejjd e e e e e e fdddZ d d dZ ZS) raApplies a multi-layer gated recurrent unit (GRU) RNN to an input sequence. For each element in the input sequence, each layer computes the following function: .. math:: \begin{array}{ll} r_t = \sigma(W_{ir} x_t + b_{ir} + W_{hr} h_{(t-1)} + b_{hr}) \\ z_t = \sigma(W_{iz} x_t + b_{iz} + W_{hz} h_{(t-1)} + b_{hz}) \\ n_t = \tanh(W_{in} x_t + b_{in} + r_t * (W_{hn} h_{(t-1)}+ b_{hn})) \\ h_t = (1 - z_t) * n_t + z_t * h_{(t-1)} \end{array} where :math:`h_t` is the hidden state at time `t`, :math:`x_t` is the input at time `t`, :math:`h_{(t-1)}` is the hidden state of the layer at time `t-1` or the initial hidden state at time `0`, and :math:`r_t`, :math:`z_t`, :math:`n_t` are the reset, update, and new gates, respectively. :math:`\sigma` is the sigmoid function, and :math:`*` is the Hadamard product. In a multilayer GRU, the input :math:`x^{(l)}_t` of the :math:`l` -th layer (:math:`l >= 2`) is the hidden state :math:`h^{(l-1)}_t` of the previous layer multiplied by dropout :math:`\delta^{(l-1)}_t` where each :math:`\delta^{(l-1)}_t` is a Bernoulli random variable which is :math:`0` with probability :attr:`dropout`. Args: input_size: The number of expected features in the input `x` hidden_size: The number of features in the hidden state `h` num_layers: Number of recurrent layers. E.g., setting ``num_layers=2`` would mean stacking two GRUs together to form a `stacked GRU`, with the second GRU taking in outputs of the first GRU and computing the final results. Default: 1 bias: If ``False``, then the layer does not use bias weights `b_ih` and `b_hh`. Default: ``True`` batch_first: If ``True``, then the input and output tensors are provided as `(batch, seq, feature)` instead of `(seq, batch, feature)`. Note that this does not apply to hidden or cell states. See the Inputs/Outputs sections below for details. Default: ``False`` dropout: If non-zero, introduces a `Dropout` layer on the outputs of each GRU layer except the last layer, with dropout probability equal to :attr:`dropout`. Default: 0 bidirectional: If ``True``, becomes a bidirectional GRU. Default: ``False`` Inputs: input, h_0 * **input**: tensor of shape :math:`(L, H_{in})` for unbatched input, :math:`(L, N, H_{in})` when ``batch_first=False`` or :math:`(N, L, H_{in})` when ``batch_first=True`` containing the features of the input sequence. The input can also be a packed variable length sequence. See :func:`torch.nn.utils.rnn.pack_padded_sequence` or :func:`torch.nn.utils.rnn.pack_sequence` for details. * **h_0**: tensor of shape :math:`(D * \text{num\_layers}, H_{out})` or :math:`(D * \text{num\_layers}, N, H_{out})` containing the initial hidden state for the input sequence. Defaults to zeros if not provided. where: .. math:: \begin{aligned} N ={} & \text{batch size} \\ L ={} & \text{sequence length} \\ D ={} & 2 \text{ if bidirectional=True otherwise } 1 \\ H_{in} ={} & \text{input\_size} \\ H_{out} ={} & \text{hidden\_size} \end{aligned} Outputs: output, h_n * **output**: tensor of shape :math:`(L, D * H_{out})` for unbatched input, :math:`(L, N, D * H_{out})` when ``batch_first=False`` or :math:`(N, L, D * H_{out})` when ``batch_first=True`` containing the output features `(h_t)` from the last layer of the GRU, for each `t`. If a :class:`torch.nn.utils.rnn.PackedSequence` has been given as the input, the output will also be a packed sequence. * **h_n**: tensor of shape :math:`(D * \text{num\_layers}, H_{out})` or :math:`(D * \text{num\_layers}, N, H_{out})` containing the final hidden state for the input sequence. Attributes: weight_ih_l[k] : the learnable input-hidden weights of the :math:`\text{k}^{th}` layer (W_ir|W_iz|W_in), of shape `(3*hidden_size, input_size)` for `k = 0`. Otherwise, the shape is `(3*hidden_size, num_directions * hidden_size)` weight_hh_l[k] : the learnable hidden-hidden weights of the :math:`\text{k}^{th}` layer (W_hr|W_hz|W_hn), of shape `(3*hidden_size, hidden_size)` bias_ih_l[k] : the learnable input-hidden bias of the :math:`\text{k}^{th}` layer (b_ir|b_iz|b_in), of shape `(3*hidden_size)` bias_hh_l[k] : the learnable hidden-hidden bias of the :math:`\text{k}^{th}` layer (b_hr|b_hz|b_hn), of shape `(3*hidden_size)` .. note:: All the weights and biases are initialized from :math:`\mathcal{U}(-\sqrt{k}, \sqrt{k})` where :math:`k = \frac{1}{\text{hidden\_size}}` .. note:: For bidirectional GRUs, forward and backward are directions 0 and 1 respectively. Example of splitting the output layers when ``batch_first=False``: ``output.view(seq_len, batch, num_directions, hidden_size)``. .. note:: ``batch_first`` argument is ignored for unbatched inputs. .. include:: ../cudnn_persistent_rnn.rst Examples:: >>> rnn = nn.GRU(10, 20, 2) >>> input = torch.randn(5, 3, 10) >>> h0 = torch.randn(2, 3, 20) >>> output, hn = rnn(input, h0) cs.d|vrtdtjdg|Ri|dS)Nr-rr)rMrErFrr`rr rFsz GRU.__init__NrcCsdSrrrrrr rsz GRU.forwardcCsdSrrrrrr rsc Cs(tjs|r||}t|trF|\}}}}|d}t|}nd}|dvsjJd|d|dk}|j rdnd} |s| | }|dur|dkrt d|d| d}n(|dur|dkrt d |d|j r| dn| d}d}d}|durL|j r&dnd} tj|j| ||j|j|jd }n |||}|||||durt|||j|j|j|j|j|j |j } n&t||||j|j|j|j|j|j } | d} | d} t|trt| |||}||| |fS|s| | } | d} | || |fSdS) Nrrz2GRU: Expected input to be 2-D or 3-D but received rr rr rrr)rQrrrrWrIr rrr*rrrr,rr(r'r2r1rrrZgrurlr)r+rrrrrr rsf                )N)N)Nrrrr`r r>sm $$rcszeZdZUgdZeed<eed<eed<eed<eed<deeeeddfd d Ze d d d Z dd ddZ Z S)r)r&r'r)r&r'r) weight_ih weight_hhN)r&r'r) num_chunksrcs||d}t||_||_||_ttj|||ffi||_ttj|||ffi||_ |rttj||fi||_ ttj||fi||_ n| dd| dd| dS)Nr0bias_ihbias_hh)rErFr&r'r)r rQrRrrrrZregister_parameterrX)rYr&r'r)rr1r2rZr`rr rFs      zRNNCellBase.__init__rscCsNd}d|jvr |jdur |d7}d|jvr<|jdkr<|d7}|jfi|jS)Nrr)Trrrz, nonlinearity={nonlinearity})rr)rr<rrrr rs zRNNCellBase.extra_reprcCs@|jdkrdt|jnd}|D]}t|| |q&dSr~rrrrr rXs zRNNCellBase.reset_parameters)NN) rrrrrrrLrrFrrrXrrrr`r rs  rcsZeZdZUdZgdZeed<d eeeeddfdd Z de e e e d d d Z Z S)rarAn Elman RNN cell with tanh or ReLU non-linearity. .. math:: h' = \tanh(W_{ih} x + b_{ih} + W_{hh} h + b_{hh}) If :attr:`nonlinearity` is `'relu'`, then ReLU is used in place of tanh. Args: input_size: The number of expected features in the input `x` hidden_size: The number of features in the hidden state `h` bias: If ``False``, then the layer does not use bias weights `b_ih` and `b_hh`. Default: ``True`` nonlinearity: The non-linearity to use. Can be either ``'tanh'`` or ``'relu'``. Default: ``'tanh'`` Inputs: input, hidden - **input**: tensor containing input features - **hidden**: tensor containing the initial hidden state Defaults to zero if not provided. Outputs: h' - **h'** of shape `(batch, hidden_size)`: tensor containing the next hidden state for each element in the batch Shape: - input: :math:`(N, H_{in})` or :math:`(H_{in})` tensor containing input features where :math:`H_{in}` = `input_size`. - hidden: :math:`(N, H_{out})` or :math:`(H_{out})` tensor containing the initial hidden state where :math:`H_{out}` = `hidden_size`. Defaults to zero if not provided. - output: :math:`(N, H_{out})` or :math:`(H_{out})` tensor containing the next hidden state. Attributes: weight_ih: the learnable input-hidden weights, of shape `(hidden_size, input_size)` weight_hh: the learnable hidden-hidden weights, of shape `(hidden_size, hidden_size)` bias_ih: the learnable input-hidden bias, of shape `(hidden_size)` bias_hh: the learnable hidden-hidden bias, of shape `(hidden_size)` .. note:: All the weights and biases are initialized from :math:`\mathcal{U}(-\sqrt{k}, \sqrt{k})` where :math:`k = \frac{1}{\text{hidden\_size}}` Examples:: >>> rnn = nn.RNNCell(10, 20) >>> input = torch.randn(6, 3, 10) >>> hx = torch.randn(3, 20) >>> output = [] >>> for i in range(6): ... hx = rnn(input[i], hx) ... output.append(hx) )r&r'r)rrTrN)r&r'r)rrcs0||d}tj|||fddi|||_dS)Nr0rr)rErFr)rYr&r'r)rr1r2rZr`rr rF^s zRNNCell.__init__rcCs|dvs Jd|d|dk}|s:|d}|durbtj|d|j|j|jd}n|sp|dn|}|jdkrt |||j |j |j |j}n<|jdkrt |||j |j |j |j}n|}td |j|s|d}|S) Nrr z6RNNCell: Expected input to be 1-D or 2-D but received rr rrrrzUnknown nonlinearity: {})rrrQrrr'r2r1rrZ rnn_tanh_cellrrrrZ rnn_relu_cellrr<rrYrrrr}rrr rds8       zRNNCell.forward)TrNN)N)rrrrrrrrrLrFrrrrrrr`r r%s 5 rcsVeZdZdZd eeeddfdd Zd eee eefe eefddd Z Z S) ra" A long short-term memory (LSTM) cell. .. math:: \begin{array}{ll} i = \sigma(W_{ii} x + b_{ii} + W_{hi} h + b_{hi}) \\ f = \sigma(W_{if} x + b_{if} + W_{hf} h + b_{hf}) \\ g = \tanh(W_{ig} x + b_{ig} + W_{hg} h + b_{hg}) \\ o = \sigma(W_{io} x + b_{io} + W_{ho} h + b_{ho}) \\ c' = f * c + i * g \\ h' = o * \tanh(c') \\ \end{array} where :math:`\sigma` is the sigmoid function, and :math:`*` is the Hadamard product. Args: input_size: The number of expected features in the input `x` hidden_size: The number of features in the hidden state `h` bias: If ``False``, then the layer does not use bias weights `b_ih` and `b_hh`. Default: ``True`` Inputs: input, (h_0, c_0) - **input** of shape `(batch, input_size)` or `(input_size)`: tensor containing input features - **h_0** of shape `(batch, hidden_size)` or `(hidden_size)`: tensor containing the initial hidden state - **c_0** of shape `(batch, hidden_size)` or `(hidden_size)`: tensor containing the initial cell state If `(h_0, c_0)` is not provided, both **h_0** and **c_0** default to zero. Outputs: (h_1, c_1) - **h_1** of shape `(batch, hidden_size)` or `(hidden_size)`: tensor containing the next hidden state - **c_1** of shape `(batch, hidden_size)` or `(hidden_size)`: tensor containing the next cell state Attributes: weight_ih: the learnable input-hidden weights, of shape `(4*hidden_size, input_size)` weight_hh: the learnable hidden-hidden weights, of shape `(4*hidden_size, hidden_size)` bias_ih: the learnable input-hidden bias, of shape `(4*hidden_size)` bias_hh: the learnable hidden-hidden bias, of shape `(4*hidden_size)` .. note:: All the weights and biases are initialized from :math:`\mathcal{U}(-\sqrt{k}, \sqrt{k})` where :math:`k = \frac{1}{\text{hidden\_size}}` On certain ROCm devices, when using float16 inputs this module will use :ref:`different precision` for backward. Examples:: >>> rnn = nn.LSTMCell(10, 20) # (input_size, hidden_size) >>> input = torch.randn(2, 3, 10) # (time_steps, batch, input_size) >>> hx = torch.randn(3, 20) # (batch, hidden_size) >>> cx = torch.randn(3, 20) >>> output = [] >>> for i in range(input.size()[0]): ... hx, cx = rnn(input[i], (hx, cx)) ... output.append(hx) >>> output = torch.stack(output, dim=0) TNr&r'r)rcs*||d}tj|||fddi|dS)Nr0rr3rrYr&r'r)r1r2rZr`rr rFs zLSTMCell.__init__rcCs|dvs Jd|d|dk}|s:|d}|durjtj|d|j|j|jd}||f}n$|s|dd|ddfn|}t |||j |j |j |j }|s|dd|ddf}|S)Nrz7LSTMCell: Expected input to be 1-D or 2-D but received rr rrr)rrrQrrr'r2r1rZ lstm_cellrrrrr)rYrrrrr}rrr rs$   $zLSTMCell.forward)TNN)N) rrrrrrLrFrrrrrrrr`r rs;rcsFeZdZdZd eeeddfdd Zd eeeeddd Z Z S) rai A gated recurrent unit (GRU) cell .. math:: \begin{array}{ll} r = \sigma(W_{ir} x + b_{ir} + W_{hr} h + b_{hr}) \\ z = \sigma(W_{iz} x + b_{iz} + W_{hz} h + b_{hz}) \\ n = \tanh(W_{in} x + b_{in} + r * (W_{hn} h + b_{hn})) \\ h' = (1 - z) * n + z * h \end{array} where :math:`\sigma` is the sigmoid function, and :math:`*` is the Hadamard product. Args: input_size: The number of expected features in the input `x` hidden_size: The number of features in the hidden state `h` bias: If ``False``, then the layer does not use bias weights `b_ih` and `b_hh`. Default: ``True`` Inputs: input, hidden - **input** : tensor containing input features - **hidden** : tensor containing the initial hidden state for each element in the batch. Defaults to zero if not provided. Outputs: h' - **h'** : tensor containing the next hidden state for each element in the batch Shape: - input: :math:`(N, H_{in})` or :math:`(H_{in})` tensor containing input features where :math:`H_{in}` = `input_size`. - hidden: :math:`(N, H_{out})` or :math:`(H_{out})` tensor containing the initial hidden state where :math:`H_{out}` = `hidden_size`. Defaults to zero if not provided. - output: :math:`(N, H_{out})` or :math:`(H_{out})` tensor containing the next hidden state. Attributes: weight_ih: the learnable input-hidden weights, of shape `(3*hidden_size, input_size)` weight_hh: the learnable hidden-hidden weights, of shape `(3*hidden_size, hidden_size)` bias_ih: the learnable input-hidden bias, of shape `(3*hidden_size)` bias_hh: the learnable hidden-hidden bias, of shape `(3*hidden_size)` .. note:: All the weights and biases are initialized from :math:`\mathcal{U}(-\sqrt{k}, \sqrt{k})` where :math:`k = \frac{1}{\text{hidden\_size}}` On certain ROCm devices, when using float16 inputs this module will use :ref:`different precision` for backward. Examples:: >>> rnn = nn.GRUCell(10, 20) >>> input = torch.randn(6, 3, 10) >>> hx = torch.randn(3, 20) >>> output = [] >>> for i in range(6): ... hx = rnn(input[i], hx) ... output.append(hx) TNrcs*||d}tj|||fddi|dS)Nr0rr rrr`rr rFs zGRUCell.__init__rcCs|dvs Jd|d|dk}|s:|d}|durbtj|d|j|j|jd}n|sp|dn|}t |||j |j |j |j }|s|d}|S)Nrz6GRUCell: Expected input to be 1-D or 2-D but received rr rr)rrrQrrr'r2r1rZgru_cellrrrrrrrrr r#s"    zGRUCell.forward)TNN)N) rrrrrrLrFrrrrrrr`r rs=r)r)r)#rr"rJrhtypingrrrrrQrmodulerZ parameterr Z utils.rnnr r5r r__all__rrZ _rnn_implsrr!r$rrrrrrrrrrrr s>      6T=+bY