Advanced Search
Article Contents
Article Contents

On a spatial-temporal decomposition of optical flow

  • * Corresponding author: Aniello Raffaele Patrone

    * Corresponding author: Aniello Raffaele Patrone 
The first author is supported by WWTF.
Abstract Full Text(HTML) Figure(8) / Table(3) Related Papers Cited by
  • In this paper we present a decomposition algorithm for computation of the spatial-temporal optical flow of a dynamic image sequence. We consider several applications, such as the extraction of temporal motion features and motion detection in dynamic sequences under varying illumination conditions, such as they appear for instance in psychological flickering experiments. For the numerical implementation we are solving an integro-differential equation by a fixed point iteration. For comparison purposes we use a standard time dependent optical flow algorithm, which in contrast to our method, constitutes in solving a spatial-temporal differential equation.

    Mathematics Subject Classification: Primary: 65F22, 35A15, 68U10.


    \begin{equation} \\ \end{equation}
  • 加载中
  • Figure 1.  $f(x,t)=x(1-x)(1-t)$ from (7). Level lines of $f$ are parametrized by $(\Psi(x,t),t)$

    Figure 2.  $g(t)=\exp \left\{-\frac{1}{\beta}(1-t)^\beta\right\}$

    Figure 3.  Color Wheel

    Figure 4.  ${\vec u^{\left( 2 \right)}}$ at different frequencies of rotations: $2$, $4$ and $8 \times$ faster than the original motion frequency. $\alpha^{(1)}=1$, $\alpha^{(2)}=\frac{1}{4}$. The intensity of ${\vec u^{\left( 2 \right)}}$ increases when the frequency of rotation is increased

    Figure 6.  ${\vec u^{\left( 1 \right)}}$: Movement of a Ferris wheel and people walking in the foreground (top left). ${\vec u^{\left( 2 \right)}}$ consists of blinking lights and the reflections of the wheel (top right). The third image (bottom) is a reference frame

    Figure 5.  The dynamic sequence consists of the smooth (translation like) motion of a cube and an oscillating background. The oscillation has a periodicity of four frames and takes place along the diagonal direction from the bottom left to the top right, moving at a rate of 5% of the frame size in each frame. The proposed model decomposes the motion, obtaining the global movement of the cube in ${\vec u^{\left( 1 \right)}}$ (left) and the background movement exclusively in ${\vec u^{\left( 2 \right)}}$ (right).

    Figure 8.  The two frames of the flickering sequence containing information (top), the difference between these two frames (down left), and the ${\vec u^{\left( 2 \right)}}$ flow field resulting from the proposed approach (down right). As predicted in Section 3 and Appendix A the ${\vec u^{\left( 1 \right)}}$ component is negligible, instead ${\vec u^{\left( 2 \right)}}$ detects the change of intensity across the blank sheet.

    Figure 7.  Result with Horn-Schunck

    Table 1.  Continuous notation

    $\vec x = (x_1,x_2)$vector in two-dimensional Euclidean space
    $\partial_k = \frac{\partial}{\partial x_k}$differentiation with respect to spatial variable $x_k$
    $\partial_t = \frac{\partial}{\partial t}$differentiation with respect to time
    $\nabla = (\partial_1, \partial_2)^T$gradient in space
    $\nabla_3 = (\partial_1, \partial_2, \partial_t)^T$gradient in space and time
    $\nabla \cdot = \partial_1 + \partial_2$divergence in space
    $\nabla_3 \cdot = \partial_1 + \partial_2 + \partial_t$divergence in space and time
    $\vec{n}$outward pointing normal vector to $\Omega$
    $f$input sequence
    $f(\cdot,t)$movie frame
    ${\vec u^{\left( i \right)}}$optical flow module, $i=1,2$
    $\vec u = {\vec u^{\left( 1 \right)}} + {\vec u^{\left( 2 \right)}}$optical flow
    $u_j^{\left( i \right)}$$j$-th optical flow component of the $i$-th module
    $\widehat{u}(\cdot,t) = \int_0^t u(\cdot,\tau)\,{\rm{d}} \tau$primitive of $u$
    $\widehat{\widehat{u}}(\cdot,t) = -\int_t^1 \widehat{u}(\cdot,\tau)\,{\rm{d}} \tau$2nd primitive of $u$ -note that $\partial_t \widehat{\widehat{u}}(\cdot,t)=\widehat{u}(\cdot,t)$
     | Show Table
    DownLoad: CSV

    Table 2.  Discrete Notation

    $f= f(r,s,t) \in \mathbb{R}^{M \times N \times T}$input sequence
    ${\vec u^{\left( i \right)}} = {\vec u^{\left( i \right)}}(r,s,t;k) \in \mathbb{R}^{M \times N \times T \times K \times 2}$discrete optical flow approximating the
    continuous flow ${\vec u^{\left( i \right)}}$ at $(\frac{r}{M-1},\frac{s}{N-1},\frac{t}{T-1})$
    $\partial_k^h$finite difference approximation in direction $x_k$
    $\partial_t^h$finite difference approximation in direction $t$
    $\Delta_x=\frac{1}{M-1}$, $\Delta_y=\frac{1}{N-1}$ and $\Delta_t=\frac{1}{T-1}$Discretization
    $ \hat u_j^{\left( 2 \right)}(r,s,t;k) = \Delta_t \sum_{\tau=1}^t u_j^{\left( 2 \right)}(r,s,\tau;k)$, $j=1,2$finite difference approximation of $\widehat{u}(\cdot,t)$
    $ \hat {\hat {u}}_j^{\left( 2 \right)}(r,s,t;k)= - \Delta_t \sum_{\tau = t}^T \hat u_j^{\left( 2 \right)}(r,s,\tau;k)$finite difference approximation of $\widehat{\widehat{u}}(\cdot,t)$
     | Show Table
    DownLoad: CSV

    Table 3.  Comparison of squared residuals over space and time $\mathcal{E}$ between Weickert-Schnörr and the proposed method

    Weickert-SchnörrProposed model
    Hamburg Taxi1374.91021
     | Show Table
    DownLoad: CSV
  • [1] J. Abhau, Z. Belhachmi and O. Scherzer, On a decomposition model for optical flow, in Energy Minimization Methods in Computer Vision and Pattern Recognition, Lecture Notes in Computer Science, 5681, Springer-Verlag, Berlin, Heidelberg, 2009,126-139. doi: 10.1007/978-3-642-03641-5_10.
    [2] R. AndreevO. Scherzer and W. Zulehner, Simultaneous optical flow and source estimation: Space-time discretization and preconditioning, Applied Numerical Mathematics, 96 (2015), 72-81.  doi: 10.1016/j.apnum.2015.04.007.
    [3] G. Aubert and J. F. Aujol, Modeling very oscillating signals. Application to image processing, Appl. Math. Optim., 51 (2005), 163-182.  doi: 10.1007/s00245-004-0812-z.
    [4] J. F. Aujol and A. Chambolle, Dual norms and image decomposition models, Int. J. Comput. Vision, 63 (2005), 85-104. 
    [5] J. F. Aujol and S. Kang, Color image decomposition and restoration, J. Vis. Commun. Image Represent, 17 (2006), 916-928.  doi: 10.1016/j.jvcir.2005.02.001.
    [6] J. F. AujolG. AubertL. Blanc-Féraud and A. Chambolle, Image decomposition into a bounded variation component and an oscillating component, J. Math. Imaging Vision, 22 (2005), 71-88.  doi: 10.1007/s10851-005-4783-8.
    [7] J. F. AujolG. GilboaT. Chan and S. Osher, Structure-texture image decomposition-modeling, algorithms, and parameter selection, Int. J. Comput. Vision, 67 (2006), 111-136. 
    [8] S. BakerD. ScharsteinJ. P. LewisS. RothM. J. Black and R. Szeliski, A database and evaluation methodology for optical flow, Int. J. Comput. Vision, 92 (2011), 1-31.  doi: 10.1007/s11263-010-0390-2.
    [9] M. BauerM. Bruveris and W. P. Michor, Overview of the geometries of shape spaces and diffeomorphism groups, J. Math. Imaging Vision, 50 (2014), 60-97.  doi: 10.1007/s10851-013-0490-z.
    [10] B. Berkels, A. Effland and M. Rumpf, Time discrete geodesic paths in the space of images, SIAM J. Imaging Sci., 8(2015),1457-1488, arXiv: 1503.02001. doi: 10.1137/140970719.
    [11] A. BorziK. Ito and K. Kunisch, Optimal control formulation for determining optical flow, SIAM J. Sci. Comput., 24 (2002), 818-847.  doi: 10.1137/S1064827501386481.
    [12] A. Bruhn, Variational Optic Flow Computation: Accurate Modeling and Efficient Numerics, PhD thesis 2006, Saarland University, Germany.
    [13] V. DuvalJ. F. Aujol and L. Vese, Mathematical modeling of textures: Application to color image decomposition with a projected gradient algorithm, J. Math. Imaging Vision, 37 (2010), 232-248.  doi: 10.1007/s10851-010-0203-9.
    [14] M. Hanke and O. Scherzer, Inverse problems light: Numerical differentiation, Amer. Math. Monthly, 108 (2001), 512-521.  doi: 10.2307/2695705.
    [15] B. K. P. Horn and B. G. Schunck, Determining optical flow, Artificial Intelligence, 17 (1981), 185-203. 
    [16] A. Jain and L. Younes, A kernel class allowing for fast computations in shape spaces induced by diffeomorphisms, J. Comput. and Applied Math., 245 (2013), 162-181.  doi: 10.1016/j.cam.2012.10.019.
    [17] C. KirisitsL. Lang and O. Scherzer, Decomposition of optical flow on the sphere, GEM-Int. J. on Geomathematics, 5 (2014), 117-141.  doi: 10.1007/s13137-013-0055-8.
    [18] T. Kohlberger, E. Memin and C. Schnörr, Variational dense motion estimation using the helmholtz decomposition, in Scale Space Methods in Computer Vision (eds. L. D. Griffin and M. Lillholm), Lecture Notes in Computer Science, 2695, Springer, Berlin, 2003,432-448. doi: 10.1007/3-540-44935-3_30.
    [19] J. Lee and J. C. Wright, A versatile parallel block-tridiagonal solver for spectral codes, MIT Plasma Science & Fusion Center, 2010.
    [20] B. McCaneK. NovinsD. Crannitch and B. Galvin, On benchmarking optical flow, Comput. Vision Image Understanding, 84 (2001), 126-143. 
    [21] Y. Meyer, Oscillating Patterns in Image Processing and Nonlinear Evolution Equations, University Lecture Series, vol 22, American Mathematical Society, Providence, RI, 2001. doi: 10.1090/ulect/022.
    [22] M. I. Miller and L. Younes, Group actions, homeomorphisms, and matching: A general framework, Int. J. Comput. Vision, 41 (2001), 61-84. 
    [23] J. K. O'Regan, Change blindness, E Cognitive Science, 2007.
    [24] O. Scherzer, M. Grasmair, H. Grossauer, M. Haltmeier and F. Lenzen, Variational Methods in Imaging, Applied Mathematical Sciences, vol 167, Springer, New York, 2009. doi: 10.1007/978-0-387-69277-7.
    [25] L. Vese and S. Osher, Modeling textures with total variation minimization and oscillating patterns in image processing, J. Sci. Comput., 19 (2003), 553-572.  doi: 10.1023/A:1025384832106.
    [26] L. Vese and S. Osher, Image denoising and decomposition with total variation minimization and oscillatory functions, J. Math. Imaging Vision, 20 (2004), 7-18.  doi: 10.1023/B:JMIV.0000011316.54027.6a.
    [27] C. M. WangK. C. Fan and C. T. Wang, Estimating optical flow by integrating multi-frame information, Journal of Information Science and Engineering, 24 (2008), 1719-1731. 
    [28] J. WeickertA. BruhnT. Brox and N. Papenberg, A survey on variational optic flow methods for small displacements, Mathematical Models for Registration and Applications to Medical Imaging, Springer Berlin Heidelberg, 10 (2006), 103-136.  doi: 10.1007/978-3-540-34767-5_5.
    [29] J. Weickert and C. Schnörr, A theoretical framework for convex regularizers in PDE-based computation of image motion, Int. J. Comput. Vision, 45 (2001), 245-264. 
    [30] J. Weickert and C. Schnörr, Variational optic flow computation with a spatio-temporal smoothness constraint, J. Math. Imaging Vision, 14 (2001), 245-255. 
    [31] J. YuanC. Schörr and G. Steidl, Simultaneous higher-order optical flow estimation and decomposition, SIAM J. Sci. and Stat. Comput., 29 (2007), 2283-2304 (electronic).  doi: 10.1137/060660709.
    [32] J. Yuan, G. Steidl and C. Schnörr, Convex Hodge decomposition of image flows, in Pattern Recognition, Lecture Notes in Comput. Sci., 5096, Springer, Berlin, 2008,416-425. doi: 10.1007/978-3-540-69321-5_42.
    [33] J. YuanC. Schnörr and G. Steidl, Convex Hodge decomposition and regularization of image flows, J. Math. Imaging Vision, 33 (2009), 169-177.  doi: 10.1007/s10851-008-0122-1.
  • 加载中




Article Metrics

HTML views(156) PDF downloads(225) Cited by(0)

Access History

Other Articles By Authors



    DownLoad:  Full-Size Img  PowerPoint