Skip to main content

Advertisement

Springer Nature Link
Log in

Stereoscopic image stitching with rectangular boundaries

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

This paper proposes a novel algorithm for stereoscopic image stitching, which aims to produce stereoscopic panoramas with rectangular boundaries. As a result, it provides wider field of view and better viewing experience for users. To achieve this, we formulate stereoscopic image stitching and boundary rectangling in a global optimization framework that simultaneously handles feature alignment, disparity consistency and boundary regularity. Given two (or more) stereoscopic images with overlapping content, each containing two views (for left and right eyes), we represent each view using a mesh and our algorithm contains three main steps: We first perform a global optimization to stitch all the left views and right views simultaneously, which ensures feature alignment and disparity consistency. Then, with the optimized vertices in each view, we extract the irregular boundary in the stereoscopic panorama, by performing polygon Boolean operations in left and right views, and construct the rectangular boundary constraints. Finally, through a global energy optimization, we warp left and right views according to feature alignment, disparity consistency and rectangular boundary constraints. To show the effectiveness of our method, we further extend our method to disparity adjustment and stereoscopic stitching with large horizon. Experimental results show that our method can produce visually pleasing stereoscopic panoramas without noticeable distortion or visual fatigue, thus resulting in satisfactory 3D viewing experience.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Explore related subjects

Discover the latest articles and news from researchers in related subjects, suggested using machine learning.

Notes

  1. We take the result in Fig. 5e directly from the [24] paper, as the code is not publicly available; thus, the zoom-in view is not very clear.

References

  1. Brown, M., Lowe, D.G.: Automatic panoramic image stitching using invariant features. Int. J. Comput. Vis. 74(1), 59–73 (2007)

    Article  Google Scholar 

  2. Chang, C., Liang, C., Chuang, Y.: Content-aware display adaptation and interactive editing for stereoscopic images. IEEE Trans. Multimed. 13(4), 589–601 (2011)

    Article  Google Scholar 

  3. Chang, C., Sato, Y., Chuang, Y.: Shape-preserving half-projective warps for image stitching. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3254–3261 (2014)

  4. Chen, Y., Chuang, Y.: Natural image stitching with the global similarity prior. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 186–201 (2016)

  5. Du, S., Hu, S., Martin, R.R.: Changing perspective in stereoscopic images. IEEE Trans. Vis. Comput. Graph. 19(8), 1288–1297 (2013)

    Article  Google Scholar 

  6. Du, S., Masiá, B., Hu, S., Gutierrez, D.: A metric of visual comfort for stereoscopic motion. ACM Trans. Graph. 32(6), 222:1–222:9 (2013)

    Article  Google Scholar 

  7. Gao, J., Kim, S.J., Brown, M.S.: Constructing image panoramas using dual-homography warping. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 49–56 (2011)

  8. von Gioi, R.G., Jakubowicz, J., Morel, J., Randall, G.: LSD: a fast line segment detector with a false detection control. IEEE Trans. Pattern Anal. Mach. Intell. 32(4), 722–732 (2010)

    Article  Google Scholar 

  9. Guo, H., Liu, S., He, T., Zhu, S., Zeng, B., Gabbouj, M.: Joint video stitching and stabilization from moving cameras. IEEE Trans. Image Process. 25(11), 5491–5503 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  10. He, K., Chang, H., Sun, J.: Rectangling panoramic images via warping. ACM Trans. Graph. 32(4), 79:1–79:10 (2013)

    MATH  Google Scholar 

  11. Lang, M., Hornung, A., Wang, O., Poulakos, S., Smolic, A., Gross, M.H.: Nonlinear disparity mapping for stereoscopic 3D. ACM Trans. Graph. 29(4), 75:1–75:10 (2010)

    Article  Google Scholar 

  12. Lee, S., Kim, Y., Lee, J., Kim, K., Lee, K., Noh, J.: Depth manipulation using disparity histogram analysis for stereoscopic 3D. Vis. Comput. 30(4), 455–465 (2014)

    Article  Google Scholar 

  13. Lin, C.C., Pankanti, S.U., Ramamurthy, K.N., Aravkin, A.Y.: Adaptive as-natural-as-possible image stitching. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1155–1163 (2015)

  14. Lin, K., Jiang, N., Cheong, L., Do, M.N., Lu, J.: SEAGULL: seam-guided local alignment for parallax-tolerant image stitching. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 370–385 (2016)

  15. Lin, K., Liu, S., Cheong, L., Zeng, B.: Seamless video stitching from hand-held camera inputs. Comput. Graph. Forum 35(2), 479–487 (2016)

    Article  Google Scholar 

  16. Liu, S., Yuan, L., Tan, P., Sun, J.: Bundled camera paths for video stabilization. ACM Trans. Graph. 32(4), 78:1–78:10 (2013)

    Google Scholar 

  17. Liu, Y., Sun, L., Yang, S.: A retargeting method for stereoscopic 3D video. Comput. Vis. Med. 1(2), 119–127 (2015)

    Article  Google Scholar 

  18. Martínez, F., Rueda, A.J., Feito, F.R.: A new algorithm for computing Boolean operations on polygons. Comput. Geosci. 35(6), 1177–1185 (2009)

    Article  Google Scholar 

  19. Mu, T., Sun, J., Martin, R.R., Hu, S.: A response time model for abrupt changes in binocular disparity. Vis. Comput. 31(5), 675–687 (2015)

    Article  Google Scholar 

  20. Mu, T., Wang, J., Du, S., Hu, S.: Stereoscopic image completion and depth recovery. Vis. Comput. 30(6–8), 833–843 (2014)

    Article  Google Scholar 

  21. Szeliski, R.: Image alignment and stitching: a tutorial. Found. Trends Comput. Graph. Vis. 2(1), 1–104 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  22. Tang, Y., Tong, R., Tang, M., Zhang, Y.: Depth incorporating with color improves salient object detection. Vis. Comput. 32(1), 111–121 (2016)

    Article  Google Scholar 

  23. Tong, R., Zhang, Y., Cheng, K.: Stereopasting: interactive composition in stereoscopic images. IEEE Trans. Vis. Comput. Graph. 19(8), 1375–1385 (2013)

    Article  Google Scholar 

  24. Wang, H., Zhou, Y., Wang, X., Fang, L.: A natural shape-preserving stereoscopic image stitching. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1812–1816 (2018)

  25. Wang, M., Zhang, X., Liang, J., Zhang, S., Martin, R.R.: Comfort-driven disparity adjustment for stereoscopic video. Comput. Vis. Med. 2(1), 3–17 (2016)

    Article  Google Scholar 

  26. Yan, W., Hou, C., Lei, J., Fang, Y., Gu, Z., Ling, N.: Stereoscopic image stitching based on a hybrid warping model. IEEE Trans. Circuits Syst. Video Technol. 27(9), 1934–1946 (2017)

    Article  Google Scholar 

  27. Yan, W., Hou, C., Wang, B., Wang, L.: Content-aware disparity adjustment for different stereo displays. Multimed. Tools Appl. 76(8), 10465–10479 (2017)

    Article  Google Scholar 

  28. Zaragoza, J., Chin, T., Tran, Q., Brown, M.S., Suter, D.: As-projective-as-possible image stitching with moving DLT. IEEE Trans. Pattern Anal. Mach. Intell. 36(7), 1285–1298 (2014)

    Article  Google Scholar 

  29. Zhang, F., Liu, F.: Parallax-tolerant image stitching. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3262–3269 (2014)

  30. Zhang, F., Liu, F.: Casual stereoscopic panorama stitching. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2002–2010 (2015)

  31. Zhang, Y., Lai, Y., Zhang, F.: Content-preserving image stitching with regular boundary constraints. ArXiv e-prints (2018)

  32. Zhu, Z., Lu, J., Wang, M., Zhang, S., Martin, R.R., Liu, H., Hu, S.: A comparative study of algorithms for realtime panoramic video blending. IEEE Trans. Image Process. 27(6), 2952–2965 (2018)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (Grant No. 61602402), Zhejiang Province Public Welfare Technology Application Research (Grant No. LGG19F020001), and the Royal Society (Grant No. \(\hbox {IES}{\setminus }\hbox {R}1{\setminus }180126\)).

Author information

Authors and Affiliations

  1. Institute of Zhejiang Radio and TV Technology, Communication University of Zhejiang, Hangzhou, 310018, China

    Yun Zhang

  2. School of Computer Science and Informatics, Cardiff University, Cardiff, CF24 3AA, UK

    Yu-Kun Lai

  3. School of Engineering and Computer Science, Victoria University of Wellington, Wellington, 6012, New Zealand

    Fang-Lue Zhang

Authors
  1. Yun Zhang
    View author publications

    Search author on:PubMed Google Scholar

  2. Yu-Kun Lai
    View author publications

    Search author on:PubMed Google Scholar

  3. Fang-Lue Zhang
    View author publications

    Search author on:PubMed Google Scholar

Corresponding author

Correspondence to Yun Zhang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, Y., Lai, YK. & Zhang, FL. Stereoscopic image stitching with rectangular boundaries. Vis Comput 35, 823–835 (2019). https://doi.org/10.1007/s00371-019-01694-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-019-01694-7

Keywords