Tell PyTorch To Do An In Place Operation

This video will show you how to tell PyTorch to do an in-place operation by using an underscore after an operation’s name.

First, we import PyTorch.

import torch

Then we print the PyTorch version we are using.

print(torch.__version__)

We are using PyTorch 0.3.1.post2.

Let’s now create a PyTorch tensor for our example.

pt_tensor_ex = torch.Tensor([2, 3, 4])

So we use torch.Tensor.

It’s going to be a tensor of size 3 – the number 2, the number 3, the number 4 – and we assign it to the variable pt_tensor_ex.

Let’s print the pt_tensor_ex Python variable to see what we have.

print(pt_tensor_ex)

We see that it’s a torch.FloatTensor of size 3 and the numbers 2, 3, 4.

Let’s now create a second tensor, torch.Tensor.

pt_tensor_to_add = torch.Tensor([8, 7, 6])

The elements will be 8, 7, 6, and we’re going to assign it to pt_tensor_to_add Python variable, and we’re going to use this to add to our first tensor.

Let’s print pt_tensor_to_add to see what it is:

print(pt_tensor_to_add)

and we see that it’s a torch.FloatTensor of size 3 – 8, 7, 6.

So if we were doing a normal PyTorch tensor addition:

pt_normal_sum_ex = pt_tensor_ex.add(pt_tensor_to_add)

we pass in the first tensor, then we say dot add, then we pass in the second tensor, and we assign the result to the Python variable pt_normal_sum_ex.

Let’s print the pt_normal_sum_ex Python variable to see what we have.

print(pt_normal_sum_ex)

We see 10, 10, 10.

So 2 plus 8 is 10, 3 plus 7 is 10, 4 plus 6 is 10.

If we look at the initial pt_tensor_ex to see if it’s changed:

print(pt_tensor_ex)

we see that it hasn’t changed.

It’s still 2, it’s still 3, and it’s still 4.

So when we use the PyTorch add operation, it returned a new tensor and then do in-place operation.

In this video, we want to tell PyTorch to do an in-place operation.

So what we’re going to do is we’re going to do an in-place addition using an underscore.

So rather than using the PyTorch dot add operation, we’re going to use the PyTorch dot add operation with this underscore.

So what we’re saying here is that we’re going to pass in our pt_tensor_ex, we want to do an in-place addition with our second tensor, then we’re going to assign that to the Python variable pt_in_place_sum_ex.

pt_in_place_sum_ex = pt_tensor_ex.add_(pt_tensor_to_add)

We check the variable assignment, so we’re going to print the pt_in_place_sum_ex to see what it is:

print(pt_in_place_sum_ex)

and we see that it’s 10, 10, 10, which is what we would expect.

It’s a torch.FloatTensor of size 3.

However, what we really want to see is we want to check what happened to our initial pt_tensor.

Because we said pt_tensor_ex.add with an underscore, so we were telling PyTorch to do an in-place operation, we should expect that this has changed from being 2, 3, 4 to 10, 10, 10.

print(pt_tensor_ex)

So it did the addition in place, so it changed the value of the pt_tensor_ex.

Perfect - We were able to tell PyTorch to do an in-place operation by using an underscore after an operation’s name.