Models Sequence-to-Sequence Part 1
3r33300. 3r3-31. Good day everyone!
3r33300. And we have again opened a new stream for the revised course 3r336. "Data scientist"
: another Excellent teacher , slightly modified based on the program updates. Well, as usual interesting 3r310. open lessons
and selections of interesting materials. Today we will begin the analysis of seq2seq models from Tensor Flow.
3r33300. As discussed in 3r320. tutorial RNN
(we recommend to read it before reading this article), recurrent neural networks can be taught to simulate a language. And an interesting question arises: is it possible to train the network on certain data to generate a meaningful answer? For example, can we teach a neural network to translate from English to French? It turns out that we can.
3r33300. This guide will show you how to create and train such an end-to-end system. Copy main repository Tensor Flow and TensorFlow model repository with GitHub . Then, you can start by running the translation program:
cd models /tutorials /rnn /translate
python translate.py --data_dir[your_data_directory]
3r33300. She will upload the data to translate English to French from 3r348. WMT’15 3-333272. , prepare them for training and train. This will require about 20 GB of hard disk space and quite a lot of time for downloading and preparation, so you can start the process now and continue reading this tutorial.
3r33300. The manual will refer to the following files:
3r33300. 3r3-300. 3r33300. 3r3365. File
3r33300. 3r3365. What is in it? 3r366. 3r33300. 3r3108. 3r33300. 3r3-300. 3r33300. 3r3105. tensorflow /tensorflow /python /ops /seq2seq.py
3r33300. 3r3105. Library for creating sequence-to-sequence models
3r33300. 3r3108. 3r33300. 3r3-300. 3r33300. 3r3105. models /tutorials /rnn /translate /seq2seq_model.py
3r33300. 3r3105. Sequence-to-sequence of the neural translation model
3r33300. 3r3108. 3r33300. 3r3-300. 3r33300. 3r3105. models /tutorials /rnn /translate /data_utils.py
3r33300. 3r3105. Auxiliary functions for preparing translation data 3r3106. 3r33300. 3r3108. 3r33300. 3r3-300. 3r33300. 3r3105. models /tutorials /rnn /translate /translate.py
3r33300. 3r3105. A binary that trains and runs the translation model
3r33300. 3r3108. 3r33300. 3r31-10.
3r33300. The basics of sequence-to-sequence
3r33300. The basic sequence-to-sequence model, as presented in Cho et al., 2014 ( Pdf ), Consists of two recurrent neural networks (RNN): an encoder (encoder) that processes the input data, and a decoder (decoder) that generates output data. The basic architecture is shown below:
3r33300. Each box in the picture above represents a cell in the RNN, usually a GRU cell - a managed recurrent block, or an LSTM cell - a long short-term memory (read 3r3189. Tutorial RNN 3r37272. For more details). Encoders and decoders can have common weights or, more often, use different sets of parameters. Multi-layered cells have been used successfully in sequence-to-sequence models, for example, for translation Sutskever et al., 2014 ( Pdf 3r3323272.).
3r33300. In the base model described above, each input must be encoded in a state of a fixed-size state, since this is the only thing that is transmitted to the decoder. To give the decoder more direct access to the input data, in 3r3142. Bahdanau et al., 2014 ( Pdf 3r3323272.) The attention mechanism was presented. We will not go into the details of the mechanism of attention (for this you can get acquainted with the work of the link); suffice it to say that it allows the decoder to look into the input data at each decoding step. A multi-layered sequence-to-sequence network with LSTM cells and the attentional mechanism in the decoder is as follows:
3r33300. TensorFlow library seq2seq
3r33300. As you can see above, there are different sequence-to-sequence models. All of them can use different RNN cells, but all of them accept encoder input data and decoder input data. This is the basis of the TensorFlow seq2seq library interface (tensorflow /tensorflow /python /ops /seq2seq.py). This basic, RNN, codec, sequence-to-sequence model works as follows.
outputs, states = basic_rnn_seq2seq (encoder_inputs, decoder_inputs, cell)
3r33300. In the call above,
encoder_inputs is a list of tensors representing the encoder input data, corresponding to the letters A, B, C from the image above. Similarly,
decoder_inputs - tensors representing decoder input data. GO, W, X, Y, Z from the first picture.
cell - instance class
tf.contrib.rnn.RNNCell 3r3-33284. which determines which cell will be used in the model. You can use existing cells, for example, , for examples.
LSTMCell or you can write your own. In addition,
tf.contrib.rnn provides shells for creating layered cells, adding exceptions to cell input and output data, or other transformations. Read 3r3189. RNN Tutorial
basic_rnn_seq2seq returns two arguments:
states . They both represent a list of tensors of the same length as 3r3r-3283. decoder_inputs .
outputs corresponds to the output of the decoder at each time step, in the first picture it is W, X, Y, Z, EOS. Returned
states represents the internal state of the decoder at each time step.
3r33300. In many applications using the model's sequence-to-sequence, the decoder output at time t is transmitted back to input to the decoder at time t + 1. When testing, during the decoding sequence, this is how a new one is constructed. On the other hand, during training, it is customary to transmit to the decoder correct input data at each time step, even if the decoder was previously mistaken. Functions in 3r38383. seq2seq.py support both modes with argument
feed_previous . For example, analyze the following usage of the nested RNN model.
outputs, states = embedding_rnn_seq2seq (
encoder_inputs, decoder_inputs, cell,
embedding_size, output_projection = None,
3r33300. In the model
embedding_rnn_seq2seqAll input data (as 3r3-3833. encoder_inputs 3r-3-33284., and 3-333283. decoder_inputs 3r-3-3284.) are integer tensors that represent discrete values. They will be embedded in a solid representation (for details on the attachment, refer to 3r3-33235. Guide to Vector Representations 3r-33272.), But to create these attachments, you need to specify the maximum number of discrete characters: 3r-33283. num_encoder_symbols on the side of the encoder and
num_decoder_symbolson the decoder side.
3r33300. In the call above, we ask
feed_previousFalse value. This means that the decoder will use the tensors 3r-3283. decoder_inputs in the form in which they are provided. If we ask
feed_previoustrue value, the decoder will use only the first element
decoder_inputs. All other tensors from the list will be ignored, and the previous value of the decoder output will be used instead. This is used to decode translations in our translation model, but it can also be used during training, to improve the resilience of the model to its errors. Approximately like Bengio et al., 2015 ( Pdf ).
3r33300. Another important argument used above is
output_projection. Without clarification, the conclusions of the nested model will be the form tensors of the number of training samples on 3r-3383. num_decoder_symbols because they represent the logs of each generated character. When training models with large output dictionaries, for example, with a large
num_decoder_symbols, storing these large tensors becomes impractical. Instead, it is better to return smaller tensors, which will subsequently be projected onto a large tensor using 3r-3383. output_projection . This allows us to use our seq2seq models with softmax sampled losses, as described in 3r-3269. Jean et. al., 201? ( Pdf ).
3r33300. In addition to
seq2seq.pyThere are some more sequence-to-sequence models. Pay attention to them. All of them have a similar interface, so we will not go into their details. For our translation model below, we use
3r33300. Continuation will follow.
3r33300. 3r33300. 3r33300.
3r33300. 3r33300. 3r33300. 3r33300.
It may be interesting
Hey what a brilliant post I have come across and believe me I have been searching out for this similar kind of post for past a week and hardly came across this. Thank you very much and will look for more postings from you. [Url = https: //mtsoul.net] 먹튀 검증 [/ url]