Skip to content

adding tutorial for using the MXNet RNN API #36

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
karishmamalkan wants to merge 1 commit into
base: master
Choose a base branch
from
Open

adding tutorial for using the MXNet RNN API #36

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
302 changes: 302 additions & 0 deletions python/rnn_api_tutorial/RNN API tutorial.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,302 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"# Using MXNet RNN API"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"## Introduction\n",
"\n",
"You can use this tutorial by:\n",
"- pulling this git repository\n",
"- run \"jupyter notebook\" command in the folder where this tutorial is\n",
"- execute the cells\n",
" "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You will accomplish the following:\n",
"\n",
"- Build a language model using MXNet\n",
"- Learn to use BucketSentenceIter to iterate through and bucket variable length sentences\n",
"- Learn to use BucketingModule for optimization\n",
"\n",
"\n",
"## Prerequisites\n",
"\n",
"To complete this tutorial, you need:\n",
"\n",
"- MXNet\n",
"- Familiarity with basic MXNet functionalities (iterators, optimizers)"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"sys.path.insert(0, \"../../python\")\n",
"import numpy as np\n",
"import mxnet as mx\n",
"\n",
"from bucket_io import BucketSentenceIter, default_build_vocab\n",
"\n",
"def Perplexity(label, pred):\n",
" # collapse the time, batch dimension\n",
" label = label.reshape((-1,))\n",
" pred = pred.reshape((-1, pred.shape[-1]))\n",
"\n",
" loss = 0.\n",
" for i in range(pred.shape[0]):\n",
" loss += -np.log(max(1e-10, pred[i][int(label[i])]))\n",
" return np.exp(loss / label.size)\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Preparing the data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here, we make use of the BucketSentenceIter to iterate through our sentences. This module is used to group together sentences of the same length into buckets. The sentences which fall in between bucket lengths are padded with 0's to make their length match the length of the next smallest bucket."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"N = 4\n",
"batch_size = 32 * N\n",
"buckets = [10, 20, 30, 40, 50, 60]"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Summary of dataset ==================\n",
"bucket of len 10 : 19479 samples\n",
"bucket of len 20 : 19336 samples\n",
"bucket of len 30 : 12208 samples\n",
"bucket of len 40 : 3962 samples\n",
"bucket of len 50 : 845 samples\n",
"bucket of len 60 : 160 samples\n",
"Summary of dataset ==================\n",
"bucket of len 10 : 19479 samples\n",
"bucket of len 20 : 19336 samples\n",
"bucket of len 30 : 12208 samples\n",
"bucket of len 40 : 3962 samples\n",
"bucket of len 50 : 845 samples\n",
"bucket of len 60 : 160 samples\n"
]
}
],
"source": [
"vocab = default_build_vocab(\"./data/ptb.train.txt\")\n",
"data_train = BucketSentenceIter(\"./data/ptb.train.txt\", vocab,\n",
" buckets, batch_size, [])\n",
"data_val = BucketSentenceIter(\"./data/ptb.eval.txt\", vocab,\n",
" buckets, batch_size, [])\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Defining the model symbol"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"While working with RNNs, we cannot work with a static symbol and optimize it. This is because the symbol which we are working with changes based on the length of the sequence we are working with. In order to overcome this problem, instead of defining a fixed symbol, we write a function to generate the symbol, which takes the sequence length as its input and uses it to generate the appropriate symbol. We then pass this function in place of the symbol to the optimization module. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For this particular model, our model looks as follows:\n",
" 1. The BucketSentenceIter iterator passes input words in the following shape (batch_size, seq_length), where each value in the input corresponds to an individual word index in the vocabulary.\n",
" 2. This data is fed into the variable called \"data\"\n",
" 3. We extract the embeddings for each of the individual words by making use of mx.sym.Embedding to get an output of the shape (batch_size, seq_length, embedding_size)\n",
" 4. To feed the data into the RNN, we need to slice it at the word level, so that each slice consists of the ith word for each sentence of the batch.\n",
" 5. Next we define the structure of the individual RNN cell. We stack multiple lstm layers in our implementation\n",
" 6. After this the RNN is rolled out to the length of the sequence. Since we are doing language modelling, an output will be created at each time-step of the sequence.\n",
" 7. The outputs of each time-step are concatenated, followed by reshaped by stacking each output over the other.\n",
" 8. Each of the outputs is multiplied by a fully connected layer, to produce an output the size of the vocabulary.\n",
" 9. We then softmax over the vocabulary for each output time-step to get the distribution over the vocabulary for the next word.\n",
" "
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import logging\n",
"\n",
"head = '%(asctime)-15s %(message)s'\n",
"logging.basicConfig(level=logging.DEBUG, format=head)\n",
"\n",
"\n",
" # buckets = [32]\n",
"num_hidden = 200\n",
"embedding_size = 200\n",
"num_lstm_layer = 2\n",
"\n",
"num_epoch = 1\n",
"learning_rate = 0.01\n",
"momentum = 0.0\n",
" # contexts = [mx.context.gpu(i) for i in range(N)]\n",
"contexts = mx.cpu()\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"def sym_gen(seq_len):\n",
" data = mx.sym.Variable('data')\n",
" label = mx.sym.Variable('softmax_label')\n",
" embed = mx.sym.Embedding(data=data, input_dim=len(vocab), output_dim=embedding_size, name='embed')\n",
" wordvec = mx.sym.SliceChannel(data=embed, num_outputs=seq_len, squeeze_axis=1)\n",
"\n",
" stack = mx.rnn.SequentialRNNCell()\n",
" for i in range(num_lstm_layer):\n",
" stack.add(mx.rnn.LSTMCell(num_hidden=num_hidden, prefix='lstm_l%d_' % i))\n",
" outputs, states = mx.rnn.rnn_unroll(stack, seq_len, inputs=wordvec)\n",
"\n",
" outputs = [mx.sym.expand_dims(x, axis=1) for x in outputs]\n",
" pred = mx.sym.Concat(*outputs, dim=1)\n",
" pred = mx.sym.Reshape(pred, shape=(-1, num_hidden))\n",
" pred = mx.sym.FullyConnected(data=pred, num_hidden=len(vocab), name='pred')\n",
"\n",
" label = mx.sym.Reshape(label, shape=(-1,))\n",
" pred = mx.sym.SoftmaxOutput(data=pred, label=label, name='softmax')\n",
"\n",
" return pred, ('data',), ('softmax_label',)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Model Optimization"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Optimizing the model for the current task is similar to creating MXNet models in general, except for one difference. Since we are working with RNNs, where the length of the input could vary across training examples, the symbol which we are trying to optimize varies based on the sequence length. To take care of this, we make use of the bucketing module for optimization, which takes care of this by optimizing the appropriate symbol based on the sequence length while sharing the weights accross different optimizations, so all of them are modifying the same parameters. "
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Fitting...\n",
"Done.\n"
]
}
],
"source": [
"model = mx.mod.BucketingModule(\n",
" sym_gen=sym_gen,\n",
" default_bucket_key=data_train.default_bucket_key,\n",
" context=contexts)\n",
"\n",
"print \"Fitting...\"\n",
"\n",
"model.fit(\n",
" train_data=data_train,\n",
" eval_data=data_val,\n",
" eval_metric=mx.metric.np(Perplexity),\n",
" optimizer_params={'learning_rate': learning_rate,\n",
" 'momentum': momentum,\n",
" 'wd': 0.00001},\n",
" initializer=mx.init.Xavier(factor_type=\"in\", magnitude=2.34),\n",
" num_epoch=num_epoch,\n",
" batch_end_callback=mx.callback.Speedometer(batch_size, 200))\n",
"print \"Done.\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 2",
"language": "python",
"name": "python2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.12"
}
},
"nbformat": 4,
"nbformat_minor": 1
}
Loading