On October 28th, the launching ceremony of the “Digital Mei Lanfang” Master Reproduction Project was held at the Mei Lanfang Grand Theatre in Beijing. The first high-precision Peking Opera digital human, the Mei Lanfang’ Twin, Digital Human, developed by the Beijing Institute of Technology (BIT), made a stunning appearance.

The “Mei Lanfang’ Twin, Digital Human” project was initiated by the Central Academy of Drama (CAD) and BIT, with technical support from Tencent, and funded by the Outstanding Young Scientists in Beijing Universities program. The project was supported by the Institute of Automation of the Chinese Academy of Sciences, the Central Academy of Fine Arts (CAFA), Beijing Foreign Studies University and the family members and disciples of Mei Lanfang, and experts such as academician Zhao Qinping. With the support of the “Virtual Digital Human” technology of BIT, Mei Lanfang, the master of Peking Opera, was reproduced at the age of 26, creating a “twin digital human of Mei Lanfang”, which is close to the real person in appearance, physical appearance, voice and performance. In the future, the model will be upgraded in real-time and applied in many scenes, striving to create the first set of “Peking Opera Digital Human” that can be interacted with in real-time in China. The launch of this project marks new cooperation and exchange achievement of the “Yanhe University Talent Training Alliance” led by BIT.

Give the thumbs-up to BIT for virtual digital human technology!

Let’s see the technology behind the master reproduction!

Research Background

Virtual Digital Human Model

A virtual digital human is a human existing in the digital world, using information technology and artificial intelligence technology to digitize and visualize the human body and limb movements in all aspects, and finally achieve the accurate construction of real-life people in the digital world simulation.

Thanks to the breakthrough of deep learning algorithms, virtual digital human technology has developed rapidly in recent years, with virtual idol anchors, virtual employees and other virtual characters. The digital human industry driven by intelligence is gradually getting better. Virtual digital human technology is the core support technology for digital performance, and has a pivotal role in the integration of culture and technology.

BIT’s Advantages on Virtual Digital Human Research

01Highly realistic expression light field acquisition system

Demonstration of the operation of the acquisition system

Adopting the light field acquisition technology, the 3D data of the real actor with millimeter-level precision is collected and the physical attribute skin texture is constructed based on computational optics, thus realizing a high degree of recovery reconstruction of the real actor. Highly realistic digital human assets based on a large amount of data are presented to users in real-time through advanced real-time rendering technology.

02AI-driven

Lip high-precision data acquisition case

For high fidelity real-time digital humans, based on self-developed lip high-precision data acquisition, a new deep neural network algorithm is used to achieve accurate control from speech to lip shape. Combined with multimodal interaction intent understanding, a multi-round language dialogue platform based on a knowledge graph is built, thus realizing natural human-machine dialogue with high accuracy in a specific closed domain.

03Real-time interactivity

Merchandise guide digital human

The understanding of surrounding scenes and real users is realized through cameras, and real-time accurate understanding of user intent through multimodal data such as pressure sensors, ultrasonic sensors, and muscle electrical sensors. People can use VR, AR, desktop, holographic and other ways to achieve real-time interaction with digital people. By interacting with people, users can really feel that digital people are alive in this world.

Master Mei Lanfang will return in this way

In order to accurately restore Mei Lanfang, the team also upgraded the existing technology.

01Replacing real person with sculpture

Since it was impossible to collect a real-life 3D data source of Mei Lanfang, after collecting a large number of photos of Mr. Mei as a reference for the construction of digital assets, the team invited teachers and students from the CAFA to complete a physical portrait of Mei Lanfang based on historical photos. Through repeated cooperation of the technical team and the sculpture team, the team finally completed the production of a digital person of a real-life statue with the goal of 3D digitization.

02Make the statue real

The team scanned the statue with a high-precision laser scanner so as to obtain the basic expression of Mr. Mei. After getting the basic scanned model, the details of the model were processed through the digital sculpting process. Because the sculpture art itself is not figurative enough, the team also needed to make additional refinements in the model bone image, skin details, hair details, eye light and other details.

03The real persona is constantly being improved

Static model comparison chart

The team continuously polished the static model to reach consistency with historical photos, then constructed high realistic costumes based on real three-dimensional tailoring and made realistic costume material simulations with the Republic of China’s surviving cloth samples as a model, finally successfully creating Mei Lanfang’s twin digital people.

PS: This technical update was supported by the VR base in Honggutan, Nanchang

BITers behind the virtual digital human

The research group of Weng Dongdong from the School of Optics and Photonics of BIT, affiliated with Beijing Mixed Reality and New Display Engineering Technology Research Center led by Professor Wang Yongtian, and relying on the high precision discipline of digital performance and simulation technology led by Professor Ding Gongyi, has carried out a lot of cutting-edge research on the integration of culture and technology. The team is currently engaged in the research of virtual reality, augmented reality and human-computer interaction. The scientific research achievements that the team participated in, Key Technology and Application of Interactive Display, won the first prize of the technical invention of the Ministry of Education in 2014 and the second prize of the national technical invention in 2017. In addition, Research and Application of Head-mounted Display Technology won the first prize of Innovation Technology Award of the Chinese Optical Engineering Society in 2017.

The team organized and wrote several national and industry standards in this field, and won the 2018 Outstanding Contribution Award of China Virtual Reality and Visualization Industry Technology Innovation Strategic Alliance.

The key technologies and systems developed by the project team, such as interactive display key technology and application, virtual-real fusion display technology, human-computer interaction technology in the new display, astronaut psychological relief system, have been applied in many units such as astronaut training center, software institute of Chinese Academy of Sciences, and secondary school of Renmin University. These technologies have been successfully applied in important industries such as heritage conservation, education and science, culture and entertainment.

Dialogue with the times with the power of science and technology, and write development chapters with the ink of innovation

Generations of BIT people have pioneered and innovated, forged ahead to climb to the top, and continuously contributed to the wisdom and power of BIT for a strong science and technology nation.