Regressing Robust and Discriminative 3D Morphable Models with a very Deep Neural Network

• This paper is published in CVPR 2017
• Paper link
• Github repository

Install code

<font color=red>** Follow the instructions on the github repo readme page. ↓↓↓↓↓ **</font>

<font color=green>//Green letters and comments are my own words, others are all from README.MD</font>

Data requirements

Before running the code, please, make sure to have all the required data in the following specific folder:

• Download our CNN and move the CNN model (3 files: 3dmm_cnn_resnet_101.caffemodel,deploy_network.prototxt,mean.binaryproto) into the CNN folder
• Download the Basel Face Model and move 01_MorphableModel.mat into the 3DMM_model folder
• Acquire 3DDFA Expression Model, run its code to generate Model_Expression.mat and move this file the 3DMM_model folder

<font color=green>//After running the code, I only got Model_Shape.mat, but I found a Model_Expression.mat in the folder where it should be generated. It seems that this file comes with the original download. So I upload that one to the 3DMM_model folder. (Anyway, this file seems unnecessary to the demo code.)</font>

• Go into 3DMM_model folder. Run the script python trimBaselFace.py. This should output 2 files BaselFaceModel_mod.mat and BaselFaceModel_mod.h5.
• Download dlib face prediction model and move the .dat file into the dlib_model folder.

Installation

<font color=green>If you don't have apt-get install, use yum install instead.</font>

• Install cmake:

$ apt-get install cmake
$ #yum install cmake

• Install opencv (2.4.6 or higher is recommended):

(http://docs.opencv.org/doc/tutorials/introduction/linux_install/linux_install.html)

• Install libboost (1.5 or higher is recommended):

$ apt-get install libboost-all-dev
$ #yum install boost-devel

• Install OpenGL, freeglut, and glew

$ sudo apt-get install freeglut3-dev
$ #yum install freeglut-devel

$ sudo apt-get install libglew-dev
$ #yum install glew-devel   #not sure, may need to build from source

• Install libhdf5-dev library

$ sudo apt-get install libhdf5-dev
$ #yum install hdf5-devel

• Install Dlib C++ library

$ (http://dlib.net/)

<font color=green>
To get libdlib.so, do not follow the instructions on the offical website for Dlib. Use the following commands, which are originally from stackoverflow.
</font>

$ cd dlib
$ mkdir build
$ cd build
$ cmake ..
$ make
$ sudo make install

<font color=green>
libdlib.so will appear in build foler.
</font>

• Update Dlib directory paths (DLIB_INCLUDE_DIR and DLIB_LIB_DIR) in CMakeLists.txt

<font color=green>
DLIB_INCLUDE_DIR = /path/to/dlib-19.4
DLIB_LIB_DIR = /path/to/dlib-19.4/dlib/build
</font>

• Make build directory (temporary). Make & install to bin folder

$ mkdir build
$ cd build
$ cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=../bin ..
$ make
$ make install

This code should generate TestVisualization in bin folder

Usage

3DMM fitting on a set of input images

• Go into demoCode folder. The demo script can be used from the command line with the following syntax:

$ Usage: python testBatchModel.py <inputList> <outputDir> <needCrop> <useLM>

where the parameters are the following:

• <inputList> is a text file containing the paths to each of the input images, one in each line.
• <outputDir> is the path to the output directory, where ply files are stored.
• <needCrop> tells the demo if the images need cropping (1) or not (0). Default 1. If your input image size is equal (square) and has a CASIA-like [2] bounding box, you can set <needCrop> as 0. Otherwise, you have to set it as 1.
• <useLM> is an option to refine the bounding box using detected landmarks (1) or not (0). Default 1.

Example for <inputList>:
<pre>
data/1.jpg
data/2.jpg
....
</pre>

Other errors I have encountered

• import caffe error: add caffe path to PYTHONPATH

$ export PYTHONPATH=/path/to/caffe/python:$PYTHONPATH

• "xxx" is not a member of cv: add corresponding header files

#include "opencv2/calib3d/calib3d.hpp"
#include "opencv2/imgproc/imgproc.hpp"

//my markdown doesn't support cpp :(

• import dlib error: Go to the base folder of the dlib repository and run python setup.py install or pip install dlib. I don't know which one works.

• libcudart.so.8.0: cannot open shared object file: No such file or directory: I failed on this. _(:3」∠)_

<font color=green>After words: I finally managed to run the demo with two servers. I generated TestVisualization on a RedHat server. But I can't run the demo code. I copied TestVisualization to a Ubuntu server, with some other preparations, it works, miraculously! </font>