1. Getting Started with Pre-trained TSN Models on UCF101file_download

Download Python source code: demo_tsn_ucf101.py

file_download

Download Jupyter notebook: demo_tsn_ucf101.ipynb

UCF101 is an action recognition dataset of realistic action videos, collected from YouTube. With 13,320 short trimmed videos from 101 action categories, it is one of the most widely used dataset in the research community for benchmarking state-of-the-art video action recognition models.

TSN (Temporal Segment Network) is a widely adopted video classification method. It is proposed to incorporate temporal information from an entire video. The idea is straightforward: we can evenly divide the video into several segments, process each segment individually, obtain segmental consensus from each segment, and perform final prediction. TSN is more like a general algorithm, rather than a specific network architecture. It can work with both 2D and 3D neural networks.

In this tutorial, we will demonstrate how to load a pre-trained TSN model from gluoncv-model-zoo and classify video frames from the Internet or your local disk into one of the 101 action classes.

Step by Step

We will show two exmaples here. For simplicity, we first try out a pre-trained UCF101 model on a single video frame. This is actually an image action recognition problem.

First, please follow the installation guide to install MXNet and GluonCV if you haven’t done so yet.

import matplotlib.pyplot as plt
import numpy as np
import mxnet as mx
from mxnet import gluon, nd, image
from mxnet.gluon.data.vision import transforms
from gluoncv.data.transforms import video
from gluoncv import utils
from gluoncv.model_zoo import get_model

Then, we download and show the example image:

url = 'https://github.com/bryanyzhu/tiny-ucf101/raw/master/ThrowDiscus.png'
im_fname = utils.download(url)

img = image.imread(im_fname)

plt.imshow(img.asnumpy())
plt.show()

Out:

Downloading ThrowDiscus.png from https://github.com/bryanyzhu/tiny-ucf101/raw/master/ThrowDiscus.png...

  0%|          | 0/147 [00:00<?, ?KB/s]
148KB [00:00, 42543.83KB/s]

In case you don’t recognize it, the image is a man throwing discus. :)

Now we define transformations for the image.

transform_fn = transforms.Compose([
    video.VideoCenterCrop(size=224),
    video.VideoToTensor(),
    video.VideoNormalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
])

This transformation function does three things: center crop the image to 224x224 in size, transpose it to num_channels*height*width, and normalize with mean and standard deviation calculated across all ImageNet images.

What does the transformed image look like?

img_list = transform_fn([img.asnumpy()])
plt.imshow(np.transpose(img_list[0], (1,2,0)))
plt.show()

Can’t recognize anything? Don’t panic! Neither do I. The transformation makes it more “model-friendly”, instead of “human-friendly”.

Next, we load a pre-trained VGG16 model. The VGG16 model is trained using TSN with three segments.

net = get_model('vgg16_ucf101', nclass=101, pretrained=True)

Out:

Downloading /root/.mxnet/models/vgg16-e660d456.zip from https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/models/vgg16-e660d456.zip...

  0%|          | 0/500159 [00:00<?, ?KB/s]
  0%|          | 102/500159 [00:00<10:08, 822.39KB/s]
  0%|          | 508/500159 [00:00<03:42, 2242.36KB/s]
  0%|          | 2189/500159 [00:00<01:07, 7370.77KB/s]
  2%|1         | 7737/500159 [00:00<00:20, 23881.16KB/s]
  3%|2         | 14941/500159 [00:00<00:12, 39717.30KB/s]
  5%|4         | 23243/500159 [00:00<00:08, 53559.31KB/s]
  6%|6         | 30519/500159 [00:00<00:07, 59574.23KB/s]
  8%|7         | 38771/500159 [00:00<00:06, 66346.25KB/s]
  9%|9         | 47002/500159 [00:00<00:06, 71219.66KB/s]
 11%|#         | 54249/500159 [00:01<00:07, 60878.94KB/s]
 12%|#2        | 62376/500159 [00:01<00:06, 66369.23KB/s]
 14%|#3        | 70015/500159 [00:01<00:06, 69153.14KB/s]
 16%|#5        | 78414/500159 [00:01<00:05, 73369.37KB/s]
 17%|#7        | 86106/500159 [00:01<00:05, 74393.38KB/s]
 19%|#8        | 94127/500159 [00:01<00:05, 75654.78KB/s]
 20%|##        | 102288/500159 [00:01<00:05, 77398.70KB/s]
 22%|##2       | 110236/500159 [00:01<00:05, 77659.49KB/s]
 24%|##3       | 118677/500159 [00:01<00:04, 79652.40KB/s]
 25%|##5       | 126683/500159 [00:02<00:04, 78653.01KB/s]
 27%|##6       | 134841/500159 [00:02<00:04, 79473.76KB/s]
 29%|##8       | 142888/500159 [00:02<00:04, 79761.82KB/s]
 30%|###       | 150881/500159 [00:02<00:04, 79466.58KB/s]
 32%|###1      | 159188/500159 [00:02<00:04, 80536.33KB/s]
 33%|###3      | 167251/500159 [00:02<00:04, 79463.19KB/s]
 35%|###5      | 175502/500159 [00:02<00:04, 80363.36KB/s]
 37%|###6      | 183546/500159 [00:02<00:03, 79353.28KB/s]
 38%|###8      | 191503/500159 [00:02<00:03, 79415.10KB/s]
 40%|###9      | 199733/500159 [00:02<00:03, 80266.61KB/s]
 42%|####1     | 207765/500159 [00:03<00:03, 79916.81KB/s]
 43%|####3     | 216026/500159 [00:03<00:03, 80586.48KB/s]
 45%|####4     | 224088/500159 [00:03<00:03, 79786.19KB/s]
 46%|####6     | 232091/500159 [00:03<00:03, 79857.17KB/s]
 48%|####8     | 240126/500159 [00:03<00:03, 80002.08KB/s]
 50%|####9     | 248128/500159 [00:03<00:03, 79393.54KB/s]
 51%|#####1    | 256414/500159 [00:03<00:03, 80421.37KB/s]
 53%|#####2    | 264459/500159 [00:03<00:02, 79576.44KB/s]
 54%|#####4    | 272420/500159 [00:03<00:02, 79223.13KB/s]
 56%|#####6    | 280875/500159 [00:03<00:02, 80799.21KB/s]
 58%|#####7    | 288959/500159 [00:04<00:02, 79796.98KB/s]
 59%|#####9    | 297093/500159 [00:04<00:02, 80241.96KB/s]
 61%|######1   | 305121/500159 [00:04<00:02, 79417.63KB/s]
 63%|######2   | 313067/500159 [00:04<00:02, 79363.78KB/s]
 64%|######4   | 321006/500159 [00:04<00:02, 73621.50KB/s]
 66%|######5   | 328449/500159 [00:04<00:02, 71644.60KB/s]
 67%|######7   | 337042/500159 [00:04<00:02, 75659.26KB/s]
 69%|######8   | 344676/500159 [00:04<00:02, 75509.68KB/s]
 71%|#######   | 353133/500159 [00:04<00:01, 78129.81KB/s]
 72%|#######2  | 361012/500159 [00:05<00:01, 78320.10KB/s]
 74%|#######3  | 368875/500159 [00:05<00:01, 78131.49KB/s]
 75%|#######5  | 377456/500159 [00:05<00:01, 80398.89KB/s]
 77%|#######7  | 385514/500159 [00:05<00:01, 79246.65KB/s]
 79%|#######8  | 393522/500159 [00:05<00:01, 79229.79KB/s]
 80%|########  | 401952/500159 [00:05<00:01, 80726.96KB/s]
 82%|########1 | 410035/500159 [00:05<00:01, 79028.49KB/s]
 84%|########3 | 418403/500159 [00:05<00:01, 80284.26KB/s]
 85%|########5 | 426444/500159 [00:05<00:00, 79842.60KB/s]
 87%|########6 | 434437/500159 [00:05<00:00, 79308.67KB/s]
 89%|########8 | 442749/500159 [00:06<00:00, 80432.77KB/s]
 90%|######### | 450799/500159 [00:06<00:00, 79286.61KB/s]
 92%|#########1| 458835/500159 [00:06<00:00, 79297.79KB/s]
 93%|#########3| 467224/500159 [00:06<00:00, 80652.63KB/s]
 95%|#########5| 475295/500159 [00:06<00:00, 79412.27KB/s]
 97%|#########6| 483796/500159 [00:06<00:00, 81057.99KB/s]
 98%|#########8| 491910/500159 [00:06<00:00, 79352.41KB/s]
100%|#########9| 499857/500159 [00:06<00:00, 78741.45KB/s]
100%|##########| 500159/500159 [00:06<00:00, 73971.77KB/s]
Downloading /root/.mxnet/models/vgg16_ucf101-d6dc1bba.zip from https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/models/vgg16_ucf101-d6dc1bba.zip...

  0%|          | 0/486863 [00:00<?, ?KB/s]
  0%|          | 101/486863 [00:00<09:25, 861.43KB/s]
  0%|          | 515/486863 [00:00<03:22, 2398.61KB/s]
  0%|          | 2187/486863 [00:00<01:02, 7790.81KB/s]
  2%|1         | 8017/486863 [00:00<00:18, 25783.45KB/s]
  3%|3         | 14838/486863 [00:00<00:11, 40091.01KB/s]
  4%|4         | 20502/486863 [00:00<00:10, 45470.18KB/s]
  6%|5         | 27947/486863 [00:00<00:08, 52272.19KB/s]
  8%|7         | 36694/486863 [00:00<00:07, 62911.05KB/s]
  9%|9         | 44678/486863 [00:00<00:06, 68014.72KB/s]
 11%|#         | 52303/486863 [00:01<00:06, 70489.27KB/s]
 12%|#2        | 60337/486863 [00:01<00:05, 73450.34KB/s]
 14%|#4        | 68779/486863 [00:01<00:05, 76744.97KB/s]
 16%|#5        | 76496/486863 [00:01<00:05, 74867.84KB/s]
 17%|#7        | 84022/486863 [00:01<00:05, 74726.04KB/s]
 19%|#8        | 92041/486863 [00:01<00:05, 76338.98KB/s]
 21%|##        | 100279/486863 [00:01<00:04, 78131.49KB/s]
 22%|##2       | 108750/486863 [00:01<00:04, 80082.47KB/s]
 24%|##4       | 117447/486863 [00:01<00:04, 82129.02KB/s]
 26%|##5       | 125725/486863 [00:01<00:04, 82320.97KB/s]
 28%|##7       | 134062/486863 [00:02<00:04, 82575.26KB/s]
 29%|##9       | 142945/486863 [00:02<00:04, 84438.21KB/s]
 31%|###1      | 151394/486863 [00:02<00:04, 83855.77KB/s]
 33%|###2      | 159784/486863 [00:02<00:03, 83619.15KB/s]
 35%|###4      | 168549/486863 [00:02<00:03, 84817.07KB/s]
 36%|###6      | 177034/486863 [00:02<00:03, 84332.44KB/s]
 38%|###8      | 185470/486863 [00:02<00:03, 83978.42KB/s]
 40%|###9      | 193976/486863 [00:02<00:03, 84298.08KB/s]
 42%|####1     | 202604/486863 [00:02<00:03, 84888.09KB/s]
 43%|####3     | 211095/486863 [00:02<00:03, 84255.18KB/s]
 45%|####5     | 219581/486863 [00:03<00:03, 84412.42KB/s]
 47%|####6     | 228131/486863 [00:03<00:03, 84735.36KB/s]
 49%|####8     | 236606/486863 [00:03<00:02, 84384.64KB/s]
 50%|#####     | 245046/486863 [00:03<00:02, 83365.53KB/s]
 52%|#####2    | 253403/486863 [00:03<00:02, 83424.59KB/s]
 54%|#####3    | 261783/486863 [00:03<00:02, 83535.59KB/s]
 56%|#####5    | 270380/486863 [00:03<00:02, 84079.17KB/s]
 57%|#####7    | 278873/486863 [00:03<00:02, 84331.63KB/s]
 59%|#####9    | 287308/486863 [00:03<00:02, 84046.85KB/s]
 61%|######    | 295781/486863 [00:03<00:02, 84248.99KB/s]
 63%|######2   | 304439/486863 [00:04<00:02, 84943.53KB/s]
 64%|######4   | 312935/486863 [00:04<00:02, 83879.66KB/s]
 66%|######5   | 321327/486863 [00:04<00:01, 83619.83KB/s]
 68%|######7   | 330319/486863 [00:04<00:01, 85476.39KB/s]
 70%|######9   | 338870/486863 [00:04<00:01, 84535.53KB/s]
 71%|#######1  | 347328/486863 [00:04<00:01, 84126.56KB/s]
 73%|#######3  | 355967/486863 [00:04<00:01, 84795.94KB/s]
 75%|#######4  | 364450/486863 [00:04<00:01, 84663.47KB/s]
 77%|#######6  | 372919/486863 [00:04<00:01, 84196.48KB/s]
 78%|#######8  | 381532/486863 [00:05<00:01, 84767.77KB/s]
 80%|########  | 390011/486863 [00:05<00:01, 84564.13KB/s]
 82%|########1 | 398469/486863 [00:05<00:01, 83788.37KB/s]
 84%|########3 | 406850/486863 [00:05<00:00, 82494.52KB/s]
 85%|########5 | 415199/486863 [00:05<00:00, 82784.89KB/s]
 87%|########6 | 423482/486863 [00:05<00:00, 82451.18KB/s]
 89%|########8 | 432047/486863 [00:05<00:00, 82731.13KB/s]
 91%|######### | 440845/486863 [00:05<00:00, 84282.41KB/s]
 92%|#########2| 449338/486863 [00:05<00:00, 84472.04KB/s]
 94%|#########4| 457788/486863 [00:05<00:00, 83029.48KB/s]
 96%|#########5| 466693/486863 [00:06<00:00, 84802.73KB/s]
 98%|#########7| 475181/486863 [00:06<00:00, 82984.17KB/s]
 99%|#########9| 483492/486863 [00:06<00:00, 82702.06KB/s]
486864KB [00:06, 77608.12KB/s]

Note that if you want to use InceptionV3 series model, please resize the image to have both dimensions larger than 299 (e.g., 340x450) and change input size from 224 to 299 in the transform function. Finally, we prepare the image and feed it to the model.

pred = net(nd.array(img_list[0]).expand_dims(axis=0))

classes = net.classes
topK = 5
ind = nd.topk(pred, k=topK)[0].astype('int')
print('The input video frame is classified to be')
for i in range(topK):
    print('\t[%s], with probability %.3f.'%
          (classes[ind[i].asscalar()], nd.softmax(pred)[0][ind[i]].asscalar()))

Out:

The input video frame is classified to be
        [ThrowDiscus], with probability 0.998.
        [HorseRace], with probability 0.001.
        [VolleyballSpiking], with probability 0.001.
        [Hammering], with probability 0.000.
        [TennisSwing], with probability 0.000.

We can see that our pre-trained model predicts this video frame to be throw discus action with high confidence.

The next example is how to perform video action recognition, e.g., use the same pre-trained model on an entire video.

First, we download the video and sample the video frames at a speed of 1 frame per second.

from gluoncv.utils import try_import_cv2
cv2 = try_import_cv2()

url = 'https://github.com/bryanyzhu/tiny-ucf101/raw/master/v_Basketball_g01_c01.avi'
video_fname = utils.download(url)

cap = cv2.VideoCapture(video_fname)
cnt = 0
video_frames = []
while(cap.isOpened()):
    ret, frame = cap.read()
    cnt += 1
    if ret and cnt % 25 == 0:
        video_frames.append(frame)
    if not ret: break

cap.release()
print('We evenly extract %d frames from the video %s.' % (len(video_frames), video_fname))

Out:

Downloading v_Basketball_g01_c01.avi from https://github.com/bryanyzhu/tiny-ucf101/raw/master/v_Basketball_g01_c01.avi...

  0%|          | 0/281 [00:00<?, ?KB/s]
282KB [00:00, 59973.32KB/s]
We evenly extract 0 frames from the video v_Basketball_g01_c01.avi.

Now we transform each video frame and feed them into the model. In the end, we average the predictions from multiple video frames to get a reasonable prediction.

if video_frames:
    video_frames_transformed = transform_fn(video_frames)
    final_pred = 0
    for _, frame_img in enumerate(video_frames_transformed):
        pred = net(nd.array(frame_img).expand_dims(axis=0))
        final_pred += pred
    final_pred /= len(video_frames)

    classes = net.classes
    topK = 5
    ind = nd.topk(final_pred, k=topK)[0].astype('int')
    print('The input video is classified to be')
    for i in range(topK):
        print('\t[%s], with probability %.3f.'%
              (classes[ind[i].asscalar()], nd.softmax(final_pred)[0][ind[i]].asscalar()))

We can see that our pre-trained model predicts this video to be basketball action with high confidence. Note that, there are many ways to sample video frames and obtain a final video-level prediction.

Next Step

If you would like to dive deeper into training TSN models on UCF101, feel free to read the next tutorial on UCF101.