Assignment 2: Video Morphing

Due: Wednesday Apr 24, 2019

The goal of this assignment is to automatically create morphing video transitions between faces, similar to these.

Raw footage

Record several sequences of similar head motion across different people. The head should take up most of the frame, and the background should be a uniform color.

Setting up

This is a programming assignment, which requires a few packages to be installed. While each computer is different, below we provide installation resources you might find useful. You are not required to follow these — install in a way that makes sense for your setup. If you are unable to sort out installation issues on your own, please contact us on Piazza.

You will need the following:

• Python3
  Mac: can install via brew by typing brew install python3 in a console window. Note that the system comes with an older version of python, so make sure you are using the correct one.
  Windows: Available via Anaconda
• OpenCV with python binding
  Mac: can be installed via brew (or pip). Also, there are many online tutorials such as this one.
  Windows: pip install opencv-python.
• Numpy and Scipy
  Mac: see OpenCV tutorial.
  Windows: Available via Anaconda or pip.
• Dlib
  Mac: brew install dlib or by following these instructions
  Windows: Instructions or via Anaconda
• Jupyter
  Instructions

Using the Jupyter notebook

Clone the assignment repository.

The bulk of this assignment (including further instructions) is inside a Jupyter notebook. If you haven’t used one before, this is a useful tool that allows markdown, code and results to coexist in an interactive environment. To start the notebook, navigate to the assignment folder and type jupyter notebook assignment2.ipynb.

Note that the Jupyter notebook references code which is implemented in other files. You will need to edit these using your favorite text editor or IDE (e.g. PyCharm is free for students). You can run each individual file independently of the Jupiter notebook, which is useful for development and debugging. Once your external code is working, don’t forget to make sure it also works in the notebook.
Important: changes in external files are not automatically loaded by the notebook, so you will need to click Kernel –> Restart & Run All in your notebook to use updated files.

Tips and tricks

Numpy arrays require a bit of getting used to, but it is well worth your time. Performing array manipulation “the numpy way” will often lead not only to shorter and more readable code, but also to faster code because the built-in array manipulation functions are highly optimized (e.g. using AVX). In many cases, instead of using for loops, you can find a one-liner numpy function that performs the exact same operation. Specifically for this assignment, you can solve it all (except the very last section) without adding any for loops.

Take the time to understand the concept of array slicing, both basic and advanced. You will never look back!

Some useful function to know are zeros, full, shape, arange, concatenate, hstack, newaxis, astype. This list is by no means comprehensive.

Deliverables

Your final submission should include:

• Raw video footage.
• Video morphing results.
• A working implementation. Running the Jupyter notebook should generate all results with no errors.

Grading

• Coding tasks:
  • Image warping: 10%
  • Face landmark detection: 15%
  • Vector field extrapolation: 15%
  • Face morphing (image to image): 20%
  • Face morphing (video to video): 20%
• Overall code readability, elegance and correctness: 10%
• Quality of raw video footage and video morphing results: 10%

Submission Details

To submit your assignment, prepare a ZIP file containing your deliverables with the filename: A2-FirstnameLastname.ZIP

Upload this ZIP file to Canvas.