Have you got a data set which you are sure contains some really interesting temporal relationships? Do you suspect that if only you could exploit this knowledge you could improve your target predictor / classifier? If yes, then you probably already know about LSTM as a potential solution to your problem.
LSTM (Long Short Term Memory) has a storied history. It dates back to 1997 so is not particularly new, yet it has only really come to prominence in the last two years (2013 - 2015) as:
LSTM has usurped previous state of the art approaches in applications such as handwriting recognition, phoneme recognition and more.
LSTM has replicated these successes in real-world applications (by contrast, many research ideas do not map across to industry use well, if at all).
I'm interested in LSTM for four reasons:
As noted above, it represents current state of the art for a wide range of problem domains / applications. I have a hunch that there is something intrinsic to LSTM which can port to other problem domains and add value by improving classification / prediction accuracy.
Ecommerce data has a strong temporal component - we denote web site users by some unique identifier (e.g. a cookie) and then we log their observable actions as a series of events on their timeline. Cause and effect is important, and we need something like LSTM to remember the important events and discard the less relevant examples. I work with Ecommerce data a lot, and not so much generating new sonnets or inception images :).
LSTM is essentially deep learning where the depth is created by unrolling the network through time.
LSTM is probably one of the main precursors to RAM (Reasoning, Attention, Memory) networks, and understanding / using LSTM can serve as a good base to apply RAM to target datasets. RAM is growing quickly..
Motivation
Most of the data sets used in deep learning revolve around image and text corpuses. These sets and domains are very interesting, but they are too far away from ecommerce data to allow any useful conclusions or correlations to be drawn. So the goal for this series of posts is as follows:
Build up a good understanding of the Torch7-based RNN framework by creating the simplest body of code which builds and trains an LSTM recurrent neural network.
Use these simple building blocks to construct and train a model which consumes and classifies an Ecommerce data set with strong temporal connections embedded in it.
Evaluate the model performance and apply well-known techniques to improve that initial performance.
RNN Framework selection criteria
What matters when picking / scoring an ML framework to use? Some criteria simply relate to ease of use and tooling support, while others speak directly to the ML ability of the framework. For me, the criteria are:
Good Language support (debugger, editor)
Speed of development
Active community
Good performance (training speed, data set scalability, accuracy / precision)
Near transparent migration from CPU to GPU training
Reproducible results
Ability to modify the framework / template code easily
The Torch + RNN layer we delve into here is by no means perfect, but it scores highly enough on these criteria to make it my current choice. Realistically, it pays to have good familiarity with multiple frameworks. TensorFlow (TF) for example may morph into some kind of generic back-end like gcc or clang where other ML frameworks output TF models for training. So there's no reason for me to move to TF until it demonstrates something superior - speed, ability to harness a cluster of nodes. And when it does, then I'll use it as well as Torch / RNN.
Other candidates
There are multiple frameworks available that "do" LSTM, for example:
Theano
Caffe
Keras
Brainstorm
TensorFlow
I don't mind having to learn Lua or Python whereas a lot of researchers prefer the Python frameworks. I find the compile time overhead of Theano to be quite onerous so I prefer the interpreted nature of Torch7. Brainstorm is pretty new still and so is TensorFlow. I've played with Keras and it looks good, although I don't know that it lets me control enough directly.
Torch and the RNN library
I don't propose to cover Torch itself in this post - there is simply too much to cover. Suffice it to say that the core Torch distro does not support LSTM or recurrent neural networks (RNNs) natively. It is possible of course to construct RNNs / LSTM using the Torch nngraph (specifically the gModule). Three specific implementations that I m aware of that take this approach are:
Wojciech Zaremba's LSTM code for the Penn TreeBank corpus.
Andrej Karpathy's LSTM (char-rnn) code, built on Zaremba's code.
Brendan Shillingford's tutorials for Nando de Freitas' ML course at Oxford.
I find this code hard to understand / change so I was looking for a better abstraction of LSTM cells with the various forget gates "already baked in". That is the primary thing that brought me to @nicholas-leonard'sRNN addition to Torch.
The RNN library
Now, the interesting thing about the RNN library (and this in no small part prompted this sequence of posts) is that you don't need to use all of the RNN library to take advantage of LSTM. Therefore we will write the simplest code we can here and only add in the parts we need. One specific things we won't use just yet are:
The dp package (Experiment, DataSource etc.). It's powerful, but confusing..
Structure of this tutorial
This tutorial builds as follows:
Post 1 (this post) - setting out the groundwork and building the base code for future posts.
Post 2 - Moving to use a real-world data set and to train on the GPU.
Post 3 - Performance tuning -
how to improve the performance of the model by applying tried and tested optimisations / tricks of the trade when (a) engineering features / input layers for neural networks and (b) tuning the model itself.
The core code
Ok, so our minimum requirements for the core code at this point are as follows:
Create the requisite data structures so we can train an LSTM on a synthetic data set using RNN.
Create an LSTM using the RNN library that is an adjunct to the main Torch repo.
Train the LSTM.
Save the trained LSTM and associated data structures to file so we know we can re-use trained models for prediction.
Load in the saved model and data sets to validate all of our work so far.
Without further ado, here is the code to do exactly that! There's not really much point in building it up slowly / line by line - hopefully the inline comments will make it clear exactly what each non-obvious line is doing. Obvious lines are self-documenting :)
(Thanks to @nicholas-leonard for help in debugging this code as I wrote it)
require 'rnn'
The code is also available here with better syntax highlighting.
Running and testing the code
On our first run, the code minimises the loss / error in about 200 epochs and this takes about 6 minutes. With a working torch installation, you can test this as follows:
time th example_part1.lualoss
After you run the code you will see *.t7 files in the directory:
These files represent the trained model and inputs and targets - if we re-run the network again but load in these files, then the network should carry on training from where it left off:
And it does! Note that we have simply over-trained our LSTM on the random problem - but it's hard at this point to get it to do well on a similarly generated random validation or test set. That will have to wait until part two, but we now have the plumbing code in place to perform our cross-validation in part two.
Part two..
In the next part of this series of posts, we will:
Replace the play / synthetic data set with a real data set (this will expose a lot of "in the trenches" coding that is needed to make LSTM work), including having a separate validation and test set.
Add code so we can target dedicated hardware (efficiently) to speed up training - using CUDA and OpenCL. We can expect a 5x - 15x speedup running on GPUs, depending on how well we can keep the GPU fed with data without context-switching between it and the CPU.
