The problem of forming natural lighting conditions for virtual objects and interaction between real and virtual objects is not yet solved. The main goal of this study is to investigate the possibility of eliminating the causes of the mixed reality visual perception conflicts by replacing real-world objects with their virtual counterparts. The main idea is to transfer all of the real-world objects into a virtual environment. This solution transforms the mixed reality system into a virtual reality system and ensures the correct physical and light interaction between objects of different worlds. This will open up new possibilities for mixed reality systems, e.g., illuminating a dark room with a virtual light source, or seeing the reflection of virtual objects in a real mirror. This paper presents an algorithm that uses machine learning tools and neural network technologies along with RGB-D sensors and a Z-buffer to obtain a real-world point cloud. This approach allows not only to select scene items but also to determine their position and size. The PointNet network architecture trained on the ScanNet dataset was used to annotate and segment scene data. The 'Total3D understanding' network was used to build a triangular grid. Finally, a real-world room reconstruction system was implemented using RGB images and point clouds as input parameters. An example of the reconstruction of a simple room interior and reconstruction quality assessment is presented.

neural networks, mixed reality systems, 3D scanning, environment reconstruction

Sorokin Maxim Igorevich,  ,  orcid.org/0000-0001-9093-1690,  ITMO University

Zhdanov Dmitri Dmitrievich,  ,  orcid.org/0000-0001-7346-8155,  ITMO University

Zhdanov Andrei Dmitrievich,  ,  orcid.org/0000-0002-2569-1982,  ITMO University