Visualize Training Results With TensorFlow summary and TensorBoard

We have a simple neural network written in TensorFlow and we want to figure out some way to visualize the training.

# visualize.py
#
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data

mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)

x = tf.placeholder(tf.float32, shape=[None, 784])

W = tf.get_variable("weights1", shape=[784, 10],
                    initializer=tf.glorot_uniform_initializer())

b = tf.get_variable("bias1", shape=[10],
                    initializer=tf.constant_initializer(0.1))

y = tf.nn.relu(tf.matmul(x, W) + b)

y_ = tf.placeholder(tf.float32, [None, 10])

cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits=y, labels=y_)
train_step = tf.train.GradientDescentOptimizer(0.001).minimize(cross_entropy)

correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

sess = tf.InteractiveSession()
tf.global_variables_initializer().run()

for step in range(50):
    print(f"training step: {step}")
    batch_xs, batch_ys = mnist.train.next_batch(100)
    sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})
    if step % 10 == 0:
        print("model accuracy: ")
        print(sess.run(accuracy, feed_dict={x: mnist.test.images,
                                            y_: mnist.test.labels}))

print("final model accuracy: ")
print(sess.run(accuracy, feed_dict={x: mnist.test.images,
                                    y_: mnist.test.labels}))

So maybe we have a really long running network. We want to understand what’s going on.

for step in range(100):

We just want to get a better understanding of how training is progressing.

# Command Line
$ python visualize.py

So we have a network we’re training.

As you can see, it takes a reasonable amount of time to train.

It’s still pretty fast.

But you can imagine if we had bigger images or a deeper network, it would take a lot longer.

for step in range(1000):

# Command Line
$ python visualize.py

As you can see, we have to wait here.

It’s not instant anymore and we want to get some understanding of what’s going on.

So say we want to track accuracy over time.

TensorFlow has a really handy, built-in way to do this.

What we’re going to do is we’re going to add in a summary and call it accuracy because that’s what it is.

tf.summary.scalar('accuracy', accuracy)

And we’re going to name each of our runs so that we can have some understanding of what’s going on.

This just gives us a nice date format.

time_string = datetime.datetime.now().isoformat()

# Python Interpreter
import datetime
datetime.datetime.now().isoformat()

It gives you the day and the second timestamp.

We then can name each of our runs – however you want, just some string.

experiment_name = f"one_hidden_layer_1000_steps_{time_string}"

So one_hidden_layer_1000_steps_{time_string}, with the format string here which is a nice feature of Python 3.6.

Then we call the merge_all function.

merged = tf.summary.merge_all()

Finally, we want to make sure that we separate our test data from our training data.

So we’re going to define two copies of the FileWriter object, one for training and one for testing.

train_writer = tf.summary.FileWriter(f'./train/{experiment_name}', sess.graph)
test_writer  = tf.summary.FileWriter(f'./est/{experiment_name}', sess.graph)

They’re exactly the same.

It’s just that we’re going to call them at different times to preserve the data.

Now, we’re going to run through our data set and every 10 steps, we’re going to record the accuracy of the model against the test set.

So we’re going to get the summary and the accuracy.

summary, acc = sess.run([merged, accuracy], feed_dict={x: mnist.test.images,
                                                       y_: mnist.test.labels}))

It’s equal to the result of running our merged variable which is just all of the summary steps we’ve defined, which in this case is just accuracy, and then accuracy.

Because we’re using the test data set, we’re going to call it our test writer and this is going to write out our summary.

test_writer.add_summary(summary, step)

We’ll also print out what’s going on to the command line.

print(f'Step {step}; Model accuracy: {acc}')

If it’s not a tenth step

elif

then this is where we’re going to run our training step now.

We’re going to do the exact same thing as above.

else:
    batch_xs, batch_ys = mnist.train.next_batch(100)
    sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})
    summary = sess.run(merged, feed_dict={x: batch_xs.images,
                                          y_: batch_ys})

We calculate summary and accuracy.

Here, all we care about is summary.

We’re going to use our batch variables as you’d expect.

We’re going to call our train writers.

train_writer.add_summary(summary, step)

We’ll work the training data.

Then we’ll run this and we’ll see the results.

# Command Line
$ python visualize.py

# FIX: Remove a parentheses
summary, acc = sess.run([merged, accuracy], feed_dict={x: mnist.test.images,
                                                       y_: mnist.test.labels})

Then we’ll run this and see the results.

# Command Line
$ python visualize.py

# FIX: import datetime library at top of the file
import datetime

Then we’ll run this and we’ll see the results.

# Command Line
$ python visualize.py

# FIX: Remove .images from "batch_xs.images"
summary = sess.run(merged, feed_dict={x: batch_xs,
                                      y_: batch_ys})

Then we’ll run this and we’ll see the results.

# Command Line
$ python visualize.py

To actually look at the visualizations, we need to use TensorBoard.

# Command Line
$ tensorboard --logdir ./

# FIX: change /est/ to /test
test_writer  = tf.summary.FileWriter(f'./test/{experiment_name}', sess.graph)

# FIX: change directory name from est to test
# Command Line
$ ls
$ mv est test
$ ls

Okay, now that we’ve finished running our network, or while our network is running, we can use the TensorBoard tool that TensorFlow provides to look at the results.

# Command Line
$ tensorboard --logdir ./

So we run this command from the command line and then we can go to local host on port 6006 to see the results.

Chrome Web Browser

So as you see here, for each of the experiments that we’ve run, we have the results recorded.

This can be really handy.

The thing that comes to mind is we can compare the training results versus the test results.

You see that the training results are a lot noisier.

It also lets us change things.

For instance, we could try to see what happens if we had multiple layers.

Full Source Code For Lesson

# visualize.py
#
# to run
# $ python visualize.py
#
# to use TensorBoard
# $ tensorboard --logdir ./
#
import tensorflow as tf
import datetime
from tensorflow.examples.tutorials.mnist import input_data

mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)

x = tf.placeholder(tf.float32, shape=[None, 784])

W = tf.get_variable("weights1", shape=[784, 10],
                    initializer=tf.glorot_uniform_initializer())

b = tf.get_variable("bias1", shape=[10],
                    initializer=tf.constant_initializer(0.1))

y = tf.nn.relu(tf.matmul(x, W) + b)

y_ = tf.placeholder(tf.float32, [None, 10])

cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits=y, labels=y_)
train_step = tf.train.GradientDescentOptimizer(0.001).minimize(cross_entropy)

correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

sess = tf.InteractiveSession()

tf.summary.scalar('accuracy', accuracy)

time_string = datetime.datetime.now().isoformat()
experiment_name = f"one_hidden_layer_1000_steps_{time_string}"

merged = tf.summary.merge_all()
train_writer = tf.summary.FileWriter(f'./train/{experiment_name}', sess.graph)
test_writer  = tf.summary.FileWriter(f'./test/{experiment_name}', sess.graph)

tf.global_variables_initializer().run()

for step in range(1000):
    print(f"training step: {step}")
    if step % 10 == 0:
        summary, acc = sess.run([merged, accuracy], feed_dict={x: mnist.test.images,
                                                               y_: mnist.test.labels})
        test_writer.add_summary(summary, step)
        print(f'Step {step}; Model accuracy: {acc}')
    else:
        batch_xs, batch_ys = mnist.train.next_batch(100)
        sess.run(train_step, feed_dict={x: batch_xs, y_:batch_ys})
        summary = sess.run(merged, feed_dict={x: batch_xs,
                                              y_: batch_ys})
        train_writer.add_summary(summary, step)