User Perspective Rendering approaches

The app that was required to be constructed was highly responsive and susceptible to even the small movements of the head. The movements of the augmented object was not proper and was highly irregular, hence, I made the movements regular so that it is perceptible and more concrete.

User Perspective Rendering

After discussion with the mentor, we opted for mirror based calibration approach so as to obtain the precise relative calibration. Adding to this, dynamic frustum approach and matrix transformation approach seemed to be most promising. The paper for the calibration method that were referred were as follows :

• Camera pose estimation using images of planar mirror reflections This paper provides a way to estimate the pose of the camera using image of planar mirror reflection (IPMRs). The paper presents and idea of recovering poses by solving a system of linear equations. The position of mirror planes are also recovered.

• Two Cameras and a Screen: How to Calibrate Mobile Devices? This paper provides an approach to calibrate a set of two non-overlapping cameras rigidly connected to a reference pattern. The paper aims at estimating the relative pose of screen and front camera using the previously mentioned paper. While, the back camera’s relative pose is estimated using structure from motion approach from virtual front camera and back camera’s scene. The estimations are further refined using bundle adjustment.

However, there were some doubts regarding the paper and so I have mailed the author regarding the virtual front camera pose being available which doesn’t seems to be true.

I also thought that instead, the essential relative pose between screen and rear camera would involve merely translation, which can be computed using manual offests, as mentioned in the paper - Towards User Perspective Augmented Reality for Public Displays . I tested it with some images, the approach seemed to work, however, to concretely ensure it to be working, a tripod stand is required.

There was another issue that relates to modifying the camera’s extrinsic parameter in Android as Pupr = PeMs->rMr where Pe is the projection matrix of eye, Ms->r is the relative pose between screen and rear camera while Mr is the rear camera’s extrinsic parameter. Pupr is the user perspective rendering matrix and Pdpr = Mr where Pdpr is the device perspective rendering. However, after intense surfing on the web, I couldn’t find any concrete material that will alter the camera’s extrinsic parameter for the scene, as this would require changing the Android.hardware.camera API.

Thus alternatively, I derived the following way that can possibly render the required user perspective rendering :

This approach needs to be discussed and validated.

Novel View Synthesis for UPR

As continuation to the NVS idea for UPR, I thought to explore it more, I read the paper : View Synthesis by Appearance Flow This paper provides the NVS approach using auto-encoders and seem to be a good approach to move forward with.