Early on, we discovered that 360-degree video is worse than real VR, so there are articles that stand out and say that these are "pseudo-VR."
The reason they give is that the vast majority of 360-degree video is not shot in 3D, let alone 6-DOF movement.
It is also seen that this point, including Facebook, Sony, Intel in the VR leading companies are trying 6 degrees of freedom VR video. However, they usually need to use several high-end cameras in combination to produce huge amounts of data, and they must be edited in the cloud due to the large amount of computation.
However, Adobe may be able to change this. They have just demonstrated a new technology that can turn flat 360-degree video into 6-DOF video. In this way, "pseudo VR" can also be changed to "true VR".
Adobe's research was published by the company's chief investigator this week at the National Association of Broadcasters Show. Originally reported by foreign media Variety.
The so-called 360-degree video to 6-DOF video, from the perspective of user experience, means that users can only turn their heads around in 360-degree video to see the surroundings. The 6-DOF allows users not only to turn their heads but also to Moving the head up, down, left, and right is like seeing things in the real world.
So how did Adobe do it?
According to the company’s researchers, they used a new warping algorithm that can simulate new viewpoints in fixed-view video while maintaining a frame rate of 120 fps.
In simple terms, ordinary 360-degree video has only one viewpoint, which is the position of the camera. To change 6 degrees of freedom, you have to create a new viewpoint around this position so that you can move forward and backward, and the surrounding image is not move.
In fact, this new algorithm can be used not only to create a 6-DOF experience, but also to use video stabilization to make some motion blur images caused by camera shake more smooth.
However, there is a condition for achieving these functions, that is, the camera must be able to move, after all, shooting at only one point is not accurate depth.
In the research paper published by Adobe, the researcher introduced the basic flow of this function:
First, they used a structure-from-motion (SfM) algorithm to calculate camera motion and create a basic three-dimensional reconstruction architecture.
After inferring the 3D geometry values ​​using this method, they map each frame of the video onto a six-sided sphere and then run a standard computer vision tracking algorithm for each surface.
For distortions that are inevitably generated when mapping the screen to a hexahedron, they use a field of view angle (FOV) greater than 45 degrees to create a picture overlap area.
Although very powerful, this technology is not a panacea. In addition to requiring the camera to move, the quality of the 3D reconstruction also depends on the distance between the position of the synthesized viewpoint and the position of the initial viewpoint. That is to say, although a 6-DOF experience can be produced, if you move too far too far, the effect will be a bigger discount.
In addition, some natural phenomena can also cause problems. For example, there is a large area of ​​texture in the picture, occlusion and changes in lighting. They may cause serious noise in the picture of the reconstruction point, and bring in 3D effects. The "holes" effect.
In fixed-point pictures, distortion of some non-stationary objects can also be seen, because the algorithm tries to fuse the synthesized frame with the original frame.
As the saying goes, it's better to shoot the material better after the last period. So many companies directly use a light field camera to shoot 6-DOF video, such as the HypeVR Lidar 6K/60 FPS camera, as well as the giant Immerge camera Lytro previously reported. Using these cameras can obviously produce higher quality 3D effects.
However, then again, these cameras are all large and expensive customized products that ordinary people cannot afford. Therefore, I believe that Adobe's technology is not perfect, but also gives ordinary users the hope of making 6-DOF video.