We propose a system that use the most popular deep learning technology to analyze panorama video and image to extract the 3D layout of the panorama image to create a virtual scene. This 3D layout structure allows the system to add 3D model to the original panorama video, making it more editable and capable of synthesize virtual objects to the real world of panorama videos.
Our system uses the 3D layout to create low-poly mesh with almost the same quality of the real scenery.
This allow users to browse the 3D result on website with low latency.
We propose a novel network architecture that contains two encoder-decoder branches to analyze the input panorama in two different projections.
Our neural network is an important step towards building an end-to-end architecture, which directly outputs a probability map of the 2D floor plan. This output requires significantly less post-processing to obtain the final 3D room layout than the output of the current state of the art.
We compare our method to Layoutnet through a series of qualitative experiments on our Realtor360 data set. The comparison is listed the supplementary attachment.
Research shows that XR is an important technical issue in a variety of industry not only just in entertainment industry. In terms of current market conditions, instead of using high-poly mesh or point clouds, using low-poly mesh with high-quality texture that stitches the texture from different view is the potential technology. Using deep learning technology can automatically produce these images. Our technology uses the images taken from smart-phone instead of expensive scanners. The system can instantly create the mesh and texture. This will be the competitive advantage of our technology.

線上展網址：
https://tievirtual.twtm.com.tw/iframe/1a986f5b-2d8d-45e6-83b1-9de0437843bc?group=23bfb1fa-dd5b-4836-81a1-4a1809b1bae5&lang=en