FaceDet3D: Facial Expressions with 3D Geometric Detail Prediction

ShahRukh Athar, Albert Pumarola, Francesc Moreno-Noguer, Dimitris Samaras



Paper
  
Code (On the way!)



Predicting and Rendering Expression conditioned Facial Details: In the first row, the first column is the input image, the second is the output rendering with facial details, the third is the output rendering without facial details, the fourth is the shaded geometry with details and the fifth is the shaded geometry without details. In the second row, a subset of predicted details, both on the detailed shaded geometry and the rendering, are marked with green rectangles and zoomed-in. Beside the zoomed-in details are plotted the zoom-ins of the same regions of the face without the details (marked with red rectangles) of both the non-detailed shading and the non-detailed rendering.


TL;DR From single images, FaceDet3D predicts facial geometric details that are consistent with any given target expression and photo-realistically renders them using Neural Rendering.





Abstract

Facial Expressions induce a variety of high-level details on the 3D face geometry. For example, a smile causes the wrinkling of cheeks or the formation of dimples, while being angry often causes wrinkling of the forehead. Morphable Models (3DMMs) of the human face fail to capture such fine details in their PCA-based representations and consequently cannot generate such details when used to edit expressions. In this work, we introduce FaceDet3D, a first-of-its-kind method that generates - from a single image - geometric facial details that are consistent with any desired target expression. The facial details are represented as a vertex displacement map and used then by a Neural Renderer to photo-realistically render novel images of any single image in any desired expression and view.

Some Results

FaceDet3D renders the predicted details consistently across different identites and expressions.






Acknowledgements

This work is supported in part by a Google Daydream Research award, by the Spanish government with projects HuMoUR TIN2017-90086-R and María de Maeztu Seal of Excellence MDM-2016-0656, by NSF-IUCRC on CVDI, Medpod Inc, by NICHD 1R21 HD93912-01A1, the Partner University Fund, and the SUNY2020 Infrastructure Transportation Security Center.

Citation

``` @article{Athar2020, title={FaceDet3D: Facial Expressions with 3D Geometric Detail Prediction}, author={Athar, ShahRukh and Pumarola, Albert and Moreno-Noguer, Francesc and Samaras, Dimitris}, journal={arXiv preprint arXiv:2012.07999}, year={2020} } ```