Lifelike Humans: Detailed Reconstruction of Expressive Human Faces

Gemma Rotger Moll

Lumbreras Ruiz, Felipe Antonio Agudo

The development of human-like digital characters (CGI) is a difficult task, since humans are used to recognizing each other and find CGIs to be unhumanized. To meet the standards of video game and digital film productions, it is necessary to model and animate these characters as close to human beings as possible. However, it is a difficult and expensive task, since it requires many artists and specialists working on a single character. Therefore, to meet these requirements, we find the automatic creation of detailed faces using low-cost setups an interesting option to study. In this work, we develop new techniques to achieve detailed faces by combining different aspects that stand out when developing realistic characters, such as skin details, hair and facial expressions and microexpressions. We examine each of the aforementioned areas with the aim of recovering them automatically without user interaction or training data. We study the problems looking for their robustness, but also for the simplicity of the setup, preferring solutions based on a single image with uncontrolled illumination and methods that can be easily computed with a standard laptop. A detailed face with wrinkles and skin details is vital for developing a realistic character. In this work, we introduce our method to automatically describe facial wrinkles from the image and transfer them to the recovered base face. We then move on to facial hair recovery by solving a parameterization problem with a new facial hair model. Finally, we develop a mapping function that allows to transfer expressions and microexpressions between different facial meshes, which provides realistic animations to our detailed face. We cover all the above points paying attention to key aspects such as (i) how to describe facial wrinkles in a simple way, (ii) how to recover 3D from 2D detections, (iii) how to recover and model facial hair from 2D to 3D, (iv) how to transfer expressions between models with skin and facial hair details, (v) how to perform all the described actions without training data or user interaction. In this work, we present our proposals to solve these aspects with an efficient and simple setup. We validate our work with several sets of both synthetic and real data, obtaining remarkable results even in such difficult cases as occlusions by glasses, dense beards, even working with different facial topologies such as one-eyed cyclops.

The immediate application of the results of the thesis is foreseen for sectors such as: Audiovisual production Audiovisual production is surely the most obvious beneficiary. Since not only can they create their own characters ready to be animated in less than a week and without any special equipment, but they can work directly with any character in the database without having to wait. Health: pain detection Research on pain expressions in ICU patients is a topic of interest and in the past was worked on in collaboration between the Computer Vision Center and a hospital in Catalonia. Using our innovative model, we can create a large number of faces to which we can extrapolate different pain expressions and label the automatic response that the detection algorithms should give, so that we can generate a meaningful and properly labeled data set. Security: facial recognition and biometrics In security, we can provide a large number of faces with labeled facial metrics, taken directly on the 3D model. So we can provide a set of both training and validation for facial recognition and biometrics algorithms for the security and video security sector. Driving aids: fatigue detection Driving aids are increasingly common to avoid accidents, and the Computer Vision Center has also carried out research on this topic. In a similar way to the study of pain, we can generate different expressions of fatigue, from mild to moderate, and transfer them automatically to a large number of faces. Cosmetic / aesthetic industry It would be interesting to provide clients of aesthetic or cosmetic centers with a personal avatar before undergoing treatments, so that a very detailed personal simulation of different treatments such as facial element modifications, facelifts, blemish removal, etc. could be generated. Also offer a personal avatar to those people who want to see how a certain cosmetic product will look on them. Avoid racial and gender bias It is true that facial analysis algorithms evolve at a frenetic speed, but many times they fail to detect and represent people of races other than Caucasian, acquiring an implicit bias that makes them in a way "racist". This is due to the fact that in most cases the data has not been properly selected and not enough examples of other ethnicities, especially minority ones, have been included. The goal of our facial database is to include abundant examples of people of all genders (including non-binary), gender-transitioning people, of all ages and ethnicities, always reinforcing those that have a lower tendency to be correctly interpreted.

Human-like Digital Characters (CGI); Unhumanized Recognition; Video Game Production; Digital Film Production; Realistic Characters; Automatic Creation; Detailed Faces; Low-Cost Setups; Skin Details; Hair; Facial Expressions; Microexpressions; Automatic Recovery; No User Interaction; No Training Data; Robustness; Setup Simplicity; Single Image; Uncontrolled Illumination; Standard Laptop Computation; Facial Wrinkles; Image Description; Wrinkle Transfer; Recovered Base Face; Facial Hair Recovery; Parameterization Problem; Facial Hair Model; Mapping Function; Expression Transfer; Microexpression Transfer; Different Facial Meshes; Realistic Animations; 3D Recovery from 2D Detections; Facial Hair Modeling (2D to 3D); Expression Transfer with Details; Efficient Setup; Synthetic Data; Real Data; Occlusions by Glasses; Dense Beards; Different Facial