3r33300. 3r3-31. Good day everyone!
 3r33300.
 3r33300. And we have again opened a new stream for the revised course 3r336. "Data scientist"
: another Excellent teacher , slightly modified based on the program updates. Well, as usual interesting 3r310. open lessons
and selections of interesting materials. Today we will begin the analysis of seq2seq models from Tensor Flow.
 3r33300.
 3r33300. Go.
 3r33300.
 3r33300. As discussed in 3r320. tutorial RNN
(we recommend to read it before reading this article), recurrent neural networks can be taught to simulate a language. And an interesting question arises: is it possible to train the network on certain data to generate a meaningful answer? For example, can we teach a neural network to translate from English to French? It turns out that we can.
 3r33300.
 3r33300. This guide will show you how to create and train such an end-to-end system. Copy main repository Tensor Flow and TensorFlow model repository with GitHub . Then, you can start by running the translation program:
 3r33300.
 3r33300.
cd models /tutorials /rnn /translate
python translate.py --data_dir[your_data_directory]
 3r33300.
 3r33300.
 3r33300. She will upload the data to translate English to French from 3r348. WMT’15 3-333272. , prepare them for training and train. This will require about 20 GB of hard disk space and quite a lot of time for downloading and preparation, so you can start the process now and continue reading this tutorial.
 3r33300.
 3r33300. The manual will refer to the following files:
 3r33300.
 3r33300.
 3r33300. 3r3-300.  3r33300. 3r3365. File
 3r33300. 3r3365. What is in it? 3r366.  3r33300. 3r3108.  3r33300. 3r3-300.  3r33300. 3r3105. tensorflow /tensorflow /python /ops /seq2seq.py
 3r33300. 3r3105. Library for creating sequence-to-sequence models
 3r33300. 3r3108.  3r33300. 3r3-300.  3r33300. 3r3105. models /tutorials /rnn /translate /seq2seq_model.py
 3r33300. 3r3105. Sequence-to-sequence of the neural translation model
 3r33300. 3r3108.  3r33300. 3r3-300.  3r33300. 3r3105. models /tutorials /rnn /translate /data_utils.py
 3r33300. 3r3105. Auxiliary functions for preparing translation data 3r3106.  3r33300. 3r3108.  3r33300. 3r3-300.  3r33300. 3r3105. models /tutorials /rnn /translate /translate.py
 3r33300. 3r3105. A binary that trains and runs the translation model
 3r33300. 3r3108.  3r33300. 3r31-10.
 3r33300. The basics of sequence-to-sequence
 3r33300.
 3r33300. The basic sequence-to-sequence model, as presented in Cho et al., 2014 ( Pdf ), Consists of two recurrent neural networks (RNN): an encoder (encoder) that processes the input data, and a decoder (decoder) that generates output data. The basic architecture is shown below:
 3r33300.
 3r33300.  3r33300.
 3r33300. Each box in the picture above represents a cell in the RNN, usually a GRU cell - a managed recurrent block, or an LSTM cell - a long short-term memory (read 3r3189. Tutorial RNN 3r37272. For more details). Encoders and decoders can have common weights or, more often, use different sets of parameters. Multi-layered cells have been used successfully in sequence-to-sequence models, for example, for translation Sutskever et al., 2014 ( Pdf 3r3323272.).
 3r33300.
 3r33300. In the base model described above, each input must be encoded in a state of a fixed-size state, since this is the only thing that is transmitted to the decoder. To give the decoder more direct access to the input data, in 3r3142. Bahdanau et al., 2014 ( Pdf 3r3323272.) The attention mechanism was presented. We will not go into the details of the mechanism of attention (for this you can get acquainted with the work of the link); suffice it to say that it allows the decoder to look into the input data at each decoding step. A multi-layered sequence-to-sequence network with LSTM cells and the attentional mechanism in the decoder is as follows:
 3r33300.
 3r33300. 3r33150.
 3r33300.
 3r33300. TensorFlow library seq2seq
 3r33300.
 3r33300. As you can see above, there are different sequence-to-sequence models. All of them can use different RNN cells, but all of them accept encoder input data and decoder input data. This is the basis of the TensorFlow seq2seq library interface (tensorflow /tensorflow /python /ops /seq2seq.py). This basic, RNN, codec, sequence-to-sequence model works as follows.
 3r33300.
 3r33300.
outputs, states = basic_rnn_seq2seq (encoder_inputs, decoder_inputs, cell)
 3r33300. In the call above, encoder_inputs is a list of tensors representing the encoder input data, corresponding to the letters A, B, C from the image above. Similarly, decoder_inputs - tensors representing decoder input data. GO, W, X, Y, Z from the first picture.
 3r33300.
 3r33300. Argument cell - instance class tf.contrib.rnn.RNNCell 3r3-33284. which determines which cell will be used in the model. You can use existing cells, for example, GRUCell or LSTMCell or you can write your own. In addition, tf.contrib.rnn provides shells for creating layered cells, adding exceptions to cell input and output data, or other transformations. Read 3r3189. RNN Tutorial , for examples.
 3r33300.
 3r33300. Call basic_rnn_seq2seq returns two arguments: outputs and states . They both represent a list of tensors of the same length as 3r3r-3283. decoder_inputs . outputs corresponds to the output of the decoder at each time step, in the first picture it is W, X, Y, Z, EOS. Returned states represents the internal state of the decoder at each time step.
 3r33300.
 3r33300. In many applications using the model's sequence-to-sequence, the decoder output at time t is transmitted back to input to the decoder at time t + 1. When testing, during the decoding sequence, this is how a new one is constructed. On the other hand, during training, it is customary to transmit to the decoder correct input data at each time step, even if the decoder was previously mistaken. Functions in 3r38383. seq2seq.py support both modes with argument feed_previous . For example, analyze the following usage of the nested RNN model.
 3r33300.
 3r33300.

    outputs, states = embedding_rnn_seq2seq (
encoder_inputs, decoder_inputs, cell,
num_encoder_symbols, nr_decoder_symbols,
embedding_size, output_projection = None,

 3r33300. In the model embedding_rnn_seq2seq All input data (as 3r3-3833. encoder_inputs 3r-3-33284., and 3-333283. decoder_inputs 3r-3-3284.) are integer tensors that represent discrete values. They will be embedded in a solid representation (for details on the attachment, refer to 3r3-33235. Guide to Vector Representations 3r-33272.), But to create these attachments, you need to specify the maximum number of discrete characters: 3r-33283. num_encoder_symbols on the side of the encoder and num_decoder_symbols on the decoder side.
 3r33300.
 3r33300. In the call above, we ask feed_previous False value. This means that the decoder will use the tensors 3r-3283. decoder_inputs in the form in which they are provided. If we ask feed_previous true value, the decoder will use only the first element decoder_inputs . All other tensors from the list will be ignored, and the previous value of the decoder output will be used instead. This is used to decode translations in our translation model, but it can also be used during training, to improve the resilience of the model to its errors. Approximately like Bengio et al., 2015 ( Pdf ).
 3r33300.
 3r33300. Another important argument used above is output_projection . Without clarification, the conclusions of the nested model will be the form tensors of the number of training samples on 3r-3383. num_decoder_symbols because they represent the logs of each generated character. When training models with large output dictionaries, for example, with a large num_decoder_symbols , storing these large tensors becomes impractical. Instead, it is better to return smaller tensors, which will subsequently be projected onto a large tensor using 3r-3383. output_projection . This allows us to use our seq2seq models with softmax sampled losses, as described in 3r-3269. Jean et. al., 201? ( Pdf ).
 3r33300.
 3r33300. In addition to basic_rnn_seq2seq and embedding_rnn_seq2seq at seq2seq.py There are some more sequence-to-sequence models. Pay attention to them. All of them have a similar interface, so we will not go into their details. For our translation model below, we use embedding_attention_seq2seq .
 3r33300.
 3r33300. Continuation will follow.
3r33300. 3r33300. 3r33300.
3r33300.
3r33300. 3r33300. 3r33300. 3r33300.