Action recognition in images and video
Recognises what people are doing, first in still photos and then in video clips, with a residual network, transfer learning and optical flow. TensorFlow.
- When
- Spring 2022
- Where
- MSc coursework, Computer Vision course at Utrecht University, in a group of two
- Built with
- Python, TensorFlow, Keras, Transfer learning, Optical flow
- Made as
- Team project
In pictures


About the project
The problem. Action recognition: look at a person in a photo or a video and say what they are doing. The assignment went from still photos to video in three steps, in TensorFlow with Keras.
Step 1: still photos. The data is Stanford-40, a set of photos of forty everyday actions. We trained a network modelled on ResNet-20 from scratch on it. ResNet is a residual network: each block of layers adds its input back to its output, a shortcut that lets a deep network keep training well. The picture shows its accuracy and loss during training.
Step 2: video, with transfer learning. TV-HI is a set of short TV clips of four interactions: handshake, high five, hug and kiss. Instead of starting again from zero, we reused the Stanford-40 network; this is called transfer learning, because what the network learned about people and poses carries over. First its layers were frozen (their weights kept fixed) and only a new head, the last layers that give the four classes, was trained on top. Then everything was fine-tuned: the whole network trained further at a tenth of the original learning rate, so what it learned before is adjusted rather than overwritten.
Step 3: motion. A second network looks at motion instead of appearance. Its input is optical flow: for every pixel, how far and in which direction it moved from one frame to the next, computed here with the Farnebäck method. The network uses 3D convolutions, filters that slide over time as well as over the image, on sixteen frames of optical flow.
Limitations. We designed a two-stream model that merges the appearance network and the motion network, but did not get it running in time.
My part. I worked on the two-stream model that combines the image network with the optical flow network, and on the data generator that feeds both of them at once. We did not get the combined model to train before the deadline. I also kept the repository and assembled the final notebooks.




