FlowNet 2.0: Evolution of optical flow estimationwith deep networks. The FlowNet demonstrated that optical flow estimation can be cast as a learning problem. However, the state of the art with regard to the quality of the flow has still been defined by traditional methods. Particularly on small displacements and real-world data, FlowNet cannot compete with variational methods. In this paper, we advance the concept of end-to-end learning of optical flow and make it work really well. The large improvements in quality and speed are caused by three major contributions: first, we focus on the training data and show that the schedule of presenting data during training is very important. Second, we develop a stacked architecture that includes warping of the second image with intermediate optical flow. Third, we elaborate on small displacements by introducing a sub-network specializing on small motions. FlowNet 2.0 is only marginally slower than the original FlowNet but decreases the estimation error by more than 50%. It performs on par with state-of-the-art methods, while running at interactive frame rates. Moreover, we present faster variants that allow optical flow computation at up to 140fps with accuracy matching the original FlowNet.

References in zbMATH (referenced in 20 articles , 1 standard article )

Showing results 1 to 20 of 20.
Sorted by year (citations)

  1. Ayed, Ibrahim; de Bézenac, Emmanuel; Pajot, Arthur; Gallinari, Patrick: Modelling spatiotemporal dynamics from Earth observation data with neural differential equations (2022)
  2. Michaël Fonder, Damien Ernst, Marc Van Droogenbroeck: M4Depth: Monocular depth estimation for autonomous vehicles in unseen environments (2022) arXiv
  3. Duran, Joan; Navarro, Julia; Buades, Antoni: Variational densification and refinement of registration maps (2021)
  4. Hernandez, Monica: Combining the band-limited parameterization and semi-Lagrangian Runge-Kutta integration for efficient PDE-constrained LDDMM (2021)
  5. Ma, Ding; Zhou, Yong; Yao, Rui; Zhao, Jiaqi; Liu, Bing; Gua, Banji: Shape robust siamese network tracking based on weakly supervised learning (2021)
  6. Ma, Jiayi; Jiang, Xingyu; Fan, Aoxiang; Jiang, Junjun; Yan, Junchi: Image matching from handcrafted to deep features: a survey (2021)
  7. Schmoderer, Timothée; Aviles-Rivero, Angelica I.; Corona, Veronica; Debroux, Noémie; Schönlieb, Carola-Bibiane: Learning optical flow for fast MRI reconstruction (2021)
  8. Chapel, Marie-Neige; Bouwmans, Thierry: Moving objects detection with a moving camera: a comprehensive review (2020)
  9. Fukami, Kai; Fukagata, Koji; Taira, Kunihiko: Machine-learning-based spatio-temporal super resolution reconstruction of turbulent flows (2020)
  10. Hackel, Timo; Usvyatsov, Mikhail; Galliani, Silvano; Wegner, Jan D.; Schindler, Konrad: Inference, learning and attention mechanisms that exploit and preserve sparsity in CNNs (2020)
  11. Innamorati, Carlo; Ritschel, Tobias; Weyrich, Tim; Mitra, Niloy J.: Learning on the edge: investigating boundary filters in CNNs (2020)
  12. Liu, Yan; Lv, Bingxue; Wang, Yuheng; Huang, Wei: An end-to-end stereo matching algorithm based on improved convolutional neural network (2020)
  13. Ranjan, Anurag; Hoffmann, David T.; Tzionas, Dimitrios; Tang, Siyu; Romero, Javier; Black, Michael J.: Learning multi-human optical flow (2020)
  14. Song, Taeyong; Kim, Youngjung; Oh, Changjae; Jang, Hyunsung; Ha, Namkoo; Sohn, Kwanghoon: Simultaneous deep stereo matching and dehazing with feature attention (2020)
  15. de Bézenac, Emmanuel; Pajot, Arthur; Gallinari, Patrick: Deep learning for physical processes: incorporating prior scientific knowledge (2019)
  16. Edgar Riba, Dmytro Mishkin, Daniel Ponsa, Ethan Rublee, Gary Bradski: Kornia: an Open Source Differentiable Computer Vision Library for PyTorch (2019) arXiv
  17. Pengpeng Liu, Irwin King, Michael R.Lyu, Jia Xu: DDFlow: Learning Optical Flow with Unlabeled Data Distillation (2019) arXiv
  18. Wang, Song; Wang, Zengfu: Optical flow estimation with occlusion detection (2019)
  19. Kuzmin, A. I.: Learning the regularization operator for the optical flow problem (2018)
  20. E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, T. Brox: FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks (2016) arXiv