American Institute of Mathematical Sciences

Advanced
October  2021, 15(5): 843-864. doi: 10.3934/ipi.2021016

Inbetweening auto-animation via Fokker-Planck dynamics and thresholding

Yuan Gao 1, , Guangzhen Jin 2,3,, and  Jian-Guo Liu 4,

1. 

Department of Mathematics, Duke University, Durham, NC 27708, USA
2. 

College of Marine Ecology and Environment, Shanghai Ocean University, China
3. 

Southern marine science and engineering Guangdong laboratory, Zhuhai, China
4. 

Department of Mathematics and Department of Physics, Duke University, Durham, NC 27708, USA

* Corresponding author: Guangzhen Jin

Received  May 2020 Revised  October 2020 Published  October 2021 Early access  February 2021

Fund Project: J.-G. Liu was supported in part by the National Science Foundation (NSF) under award DMS-1812573. G. Jin was supported in part by the the Natural Science Foundation of Guangdong Province under award 2019A1515011487, the China Postdoctoral Science Foundation under award 2020M681269 and the Fundamental Research Funds for the Central Universities under award 20184200031610059

We propose an equilibrium-driven deformation algorithm (EDDA) to simulate the inbetweening transformations starting from an initial image to an equilibrium image, which covers images varying from a greyscale type to a colorful type on planes or manifolds. The algorithm is based on the Fokker-Planck dynamics on manifold, which automatically incorporates the manifold structure suggested by dataset and satisfies positivity, unconditional stability, mass conservation law and exponentially convergence. The thresholding scheme is adapted for the sharp interface dynamics and is used to achieve the finite time convergence. Using EDDA, three challenging examples, (I) facial aging process, (II) coronavirus disease 2019 (COVID-19) pneumonia invading/fading process, and (III) continental evolution process are computed efficiently.

Keywords: Morphing, shape evolution, ergodicity, upwind scheme, Voronoi tessellation.
Mathematics Subject Classification: Primary: 68U10, 94A08; Secondary: 37A30, 65M08.
Citation: Yuan Gao, Guangzhen Jin, Jian-Guo Liu. Inbetweening auto-animation via Fokker-Planck dynamics and thresholding. Inverse Problems & Imaging, 2021, 15 (5) : 843-864. doi: 10.3934/ipi.2021016
References:
[1]

M. Cong, M. Bao, Jane L. E., K. S. Bhat and R. Fedkiw, Fully automatic generation of anatomical face simulation models, In Proceedings of the 14th ACM SIGGRAPH/Eurographics Symposium on Computer Animation, (2015), 175–183. doi: 10.1145/2786784.2786786.  Google Scholar

[2]

Q. Du, M. D. Gunzburger and L. Ju, Voronoi-based finite volume methods, optimal voronoi meshes, and pdes on the sphere, Computer Methods in Applied Mechanics and Engineering, 192 (2003), 3933-3957.  doi: 10.1016/S0045-7825(03)00394-3.  Google Scholar

[3]

R. Eymard, T. Gallouët and R. Herbin, Finite volume methods, Handbook of numerical analysis, Vol. VII, Handb. Numer. Anal., VII, North-Holland, Amsterdam, 2000, 713–1020. doi: 10.1086/phos.67.4.188705.  Google Scholar

[4]

Y. Gao, T. Li, X. Li and J.-G. Liu, Transition path theory for langevin dynamics on manifold: optimal control and data-driven solver, arXiv: 2010.09988, 2020. Google Scholar

[5]

Y. Gao, J.-G. Liu and N. Wu, Data-driven efficient solvers and predictions of conformational transitions for langevin dynamics on manifold in high dimensions, arXiv: 2005.12787, 2020. Google Scholar

[6]

Z. Leng, R. Zhu, W. Hou, Y. Feng, Y. Yang, Q. Han, G. Shan, F. Meng, D. Du, S. Wang, J. Fan, W. Wang, L. Deng, H. Shi, H.Li, Z. Hu, F. Zhang, J. Gao, H. Liu, X. Li, Y. Zhao, K. Yin, X. He, Z. Gao, Y. Wang, B. Yang, R. Jin, I. Stambler, L. W. Lim, H. Su, A. Moskalev, A. Cano, S. Chakrabarti, K.-. Min, G. Ellison-Hughes, C. Caruso, K. Jin and R. C. Zhao, Transplantation of ACE2-mesenchymal stem cells improves the outcome of patients with COVID-19 pneumonia, Aging and disease, 11 (2020), 216. Google Scholar

[7]

The Sydney Morning Herald, July 4, 2014, 'Can your face reveal how long you'll live?', http://www.dailylife.com.au/health-and-fitness/dl-fitness/can-your-face-reveal-how-long-youll-live-20140708-3bjsk.html. Google Scholar

[8]

R. J. Renka, Algorithm 772: Stripack: Delaunay triangulation and voronoi diagram on the surface of a sphere, ACM Transactions on Mathematical Software (TOMS), 23 (1997), 416-434.  doi: 10.1145/275323.275329.  Google Scholar

[9]

M. Zollhöfer, J. Thies, P. Garrido, D. Bradley, T. Beeler, P. Pérez, M. Stamminger, M. Nießner and C. Theobalt, State of the art on monocular 3d face reconstruction, tracking, and applications, In Computer Graphics Forum, volume 37. Wiley Online Library, 2018, 523–550. Google Scholar

show all references

References:
[1]

M. Cong, M. Bao, Jane L. E., K. S. Bhat and R. Fedkiw, Fully automatic generation of anatomical face simulation models, In Proceedings of the 14th ACM SIGGRAPH/Eurographics Symposium on Computer Animation, (2015), 175–183. doi: 10.1145/2786784.2786786.  Google Scholar

[2]

Q. Du, M. D. Gunzburger and L. Ju, Voronoi-based finite volume methods, optimal voronoi meshes, and pdes on the sphere, Computer Methods in Applied Mechanics and Engineering, 192 (2003), 3933-3957.  doi: 10.1016/S0045-7825(03)00394-3.  Google Scholar

[3]

R. Eymard, T. Gallouët and R. Herbin, Finite volume methods, Handbook of numerical analysis, Vol. VII, Handb. Numer. Anal., VII, North-Holland, Amsterdam, 2000, 713–1020. doi: 10.1086/phos.67.4.188705.  Google Scholar

[4]

Y. Gao, T. Li, X. Li and J.-G. Liu, Transition path theory for langevin dynamics on manifold: optimal control and data-driven solver, arXiv: 2010.09988, 2020. Google Scholar

[5]

Y. Gao, J.-G. Liu and N. Wu, Data-driven efficient solvers and predictions of conformational transitions for langevin dynamics on manifold in high dimensions, arXiv: 2005.12787, 2020. Google Scholar

[6]

Z. Leng, R. Zhu, W. Hou, Y. Feng, Y. Yang, Q. Han, G. Shan, F. Meng, D. Du, S. Wang, J. Fan, W. Wang, L. Deng, H. Shi, H.Li, Z. Hu, F. Zhang, J. Gao, H. Liu, X. Li, Y. Zhao, K. Yin, X. He, Z. Gao, Y. Wang, B. Yang, R. Jin, I. Stambler, L. W. Lim, H. Su, A. Moskalev, A. Cano, S. Chakrabarti, K.-. Min, G. Ellison-Hughes, C. Caruso, K. Jin and R. C. Zhao, Transplantation of ACE2-mesenchymal stem cells improves the outcome of patients with COVID-19 pneumonia, Aging and disease, 11 (2020), 216. Google Scholar

[7]

The Sydney Morning Herald, July 4, 2014, 'Can your face reveal how long you'll live?', http://www.dailylife.com.au/health-and-fitness/dl-fitness/can-your-face-reveal-how-long-youll-live-20140708-3bjsk.html. Google Scholar

[8]

R. J. Renka, Algorithm 772: Stripack: Delaunay triangulation and voronoi diagram on the surface of a sphere, ACM Transactions on Mathematical Software (TOMS), 23 (1997), 416-434.  doi: 10.1145/275323.275329.  Google Scholar

[9]

M. Zollhöfer, J. Thies, P. Garrido, D. Bradley, T. Beeler, P. Pérez, M. Stamminger, M. Nießner and C. Theobalt, State of the art on monocular 3d face reconstruction, tracking, and applications, In Computer Graphics Forum, volume 37. Wiley Online Library, 2018, 523–550. Google Scholar

Figure 1.  The semilog plot of temporal evolution of relative root mean square errors for the RGB facial aging transformation with parameters $ \Delta t = 0.01, T = 100 $ and $ \Delta x = \Delta y = 10^{-4} $. The red, green and blue lines represent the relative errors of the corresponding color modes
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 2.  Facial aging transformation from initial to equilibrium with parameters $ \Delta t = 0.01, T = 100 $ and $ \Delta x = \Delta y = 10^{-4} $. The updated results after time step $ 40,100 $, $ 200 $, 400, $ 1000, 2000 $, $ 4000 $, $ 10000 $ are shown and compared to the initial and equilibrium images
Figure Options

Download full-size image

Download as PowerPoint slide

6]. The left column of figures illustrates the evading of pneumonia from January 23th to February 2nd and the right column illustrates the fading away of pneumonia from February 2nd to February 15th after treatment. Red circles indicate the significant COVID-19 pneumonia invading areas and blue circles indicate the significant pneumonia fading away areas">Figure 3.  Chest CT images of the critically severe COVID-19 patient [6]. The left column of figures illustrates the evading of pneumonia from January 23th to February 2nd and the right column illustrates the fading away of pneumonia from February 2nd to February 15th after treatment. Red circles indicate the significant COVID-19 pneumonia invading areas and blue circles indicate the significant pneumonia fading away areas
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 4.  The semilog plot of temporal evolution of the relative root mean square errors for COVID-19 pneumonia invading and fading away on CT scan images with the parameters $ \Delta t = 0.01, T = 100 $ and $ \Delta x = \Delta y = 10^{-4} $. (up) The error evolution for the pneumonia invading process simulation. (down) The error evolution for the pneumonia fading away process simulation
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 5.  The simulated COVID-19 pneumonia invading and fading away process on CT scan images with the parameters $ \Delta t = 0.01, T = 100 $ and $ \Delta x = \Delta y = 10^{-4} $. Results after the step 20, 50, 100, 200, 500, 1000, 2000, 5000 are illustrated and compared to the initial and equilibrium scan images. The white part inside the lungs shown on images indicates the evidence of pneumonia. (up) The pneumonia invades into the patient's lungs caused by COVID-19. (down) The pneumonia fades away from the lungs after a stem cell treatment is applied to the patient. Red circles indicate the significant COVID-19 pneumonia invading areas and blue circles indicate the significant pneumonia fading away areas
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 6.  The unit sphere and the Voronoi cells on it. There are totally 3000 cells on the sphere. The black dots indicate locations of the point clouds on the sphere. The polygons with black edges are the Voronoi polygons generated with CVT algorithm
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 7.  The semilog plot of temporal evolution of relative root mean square errors for the continental evolution process with thresholding for shape dynamics with $ \Delta t = 0.05 $ and the total number of the thresholding adjustments is $ N_t = 50 $. The linear iterations before the $ (k+1) $th thresholding adjustment is $ 2k $. The red line indicates the error of simulations with only the linear Fokker-Planck algorithm while the blue line is the error of the linear algorithm combined with the thresholding scheme. The black dotted lines indicate the time steps when the thresholding adjustments are applied
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 8.  The evolutions of continental movements on the unit sphere with the parameter $ \Delta t = 0.05 $ and the total number of the thresholding adjustments is $ N_t = 50 $. The continental evolution on the sphere after the 2th, 5th, 10th, 30th thresholding adjustment are illustrated and compared with the initial and equilibrium states. The black dots and polygons in each subplot illustrate the point clouds and the Voronoi cells, respectively. The orange and blue patch indicate the land and ocean, respectively. 'TH' is short for 'thresholding step'. The formation of the Antarctic is revealed at the bottom (southern part) of the globe (black arrow in TH 5). Note that the globes are shown in the same view angle so the Antarctic continental is out of view in the last two subplots
Figure Options

Download full-size image

Download as PowerPoint slide

Table 1.  Comparison of the time steps needed to reach the torenlence between linear method and threshold method
Tolerance $ 10^{-3} $ $ 10^{-4} $ $ 10^{-5} $ $ 10^{-6} $
Steps (Linear) 1737 4026 6304 8581
Steps (Threshold) 960 960 960 960
Table Options

Download as excel

Tolerance $ 10^{-3} $ $ 10^{-4} $ $ 10^{-5} $ $ 10^{-6} $
Steps (Linear) 1737 4026 6304 8581
Steps (Threshold) 960 960 960 960
[1]

Filippo Dell'Oro, Olivier Goubet, Youcef Mammeri, Vittorino Pata. A semidiscrete scheme for evolution equations with memory. Discrete & Continuous Dynamical Systems, 2019, 39 (10) : 5637-5658. doi: 10.3934/dcds.2019247
[2]

Yuan-Nan Young, Doron Levy. Registration-Based Morphing of Active Contours for Segmentation of CT Scans. Mathematical Biosciences & Engineering, 2005, 2 (1) : 79-96. doi: 10.3934/mbe.2005.2.79
[3]

Charles Pugh, Michael Shub, Alexander Starkov. Unique ergodicity, stable ergodicity, and the Mautner phenomenon for diffeomorphisms. Discrete & Continuous Dynamical Systems, 2006, 14 (4) : 845-855. doi: 10.3934/dcds.2006.14.845
[4]

Jon Chaika, Rodrigo Treviño. Logarithmic laws and unique ergodicity. Journal of Modern Dynamics, 2017, 11: 563-588. doi: 10.3934/jmd.2017022
[5]

Gabriel Rivière. Remarks on quantum ergodicity. Journal of Modern Dynamics, 2013, 7 (1) : 119-133. doi: 10.3934/jmd.2013.7.119
[6]

Amy Allwright, Abdon Atangana. Augmented upwind numerical schemes for a fractional advection-dispersion equation in fractured groundwater systems. Discrete & Continuous Dynamical Systems - S, 2020, 13 (3) : 443-466. doi: 10.3934/dcdss.2020025
[7]

Andy Hammerlindl, Jana Rodriguez Hertz, Raúl Ures. Ergodicity and partial hyperbolicity on Seifert manifolds. Journal of Modern Dynamics, 2020, 0: 331-348. doi: 10.3934/jmd.2020012
[8]

Karl Grill, Christian Tutschka. Ergodicity of two particles with attractive interaction. Discrete & Continuous Dynamical Systems, 2015, 35 (10) : 4831-4838. doi: 10.3934/dcds.2015.35.4831
[9]

Henk Bruin, Gregory Clack. Inducing and unique ergodicity of double rotations. Discrete & Continuous Dynamical Systems, 2012, 32 (12) : 4133-4147. doi: 10.3934/dcds.2012.32.4133
[10]

Dubi Kelmer. Quantum ergodicity for products of hyperbolic planes. Journal of Modern Dynamics, 2008, 2 (2) : 287-313. doi: 10.3934/jmd.2008.2.287
[11]

Federico Rodriguez Hertz, María Alejandra Rodriguez Hertz, Raúl Ures. Partial hyperbolicity and ergodicity in dimension three. Journal of Modern Dynamics, 2008, 2 (2) : 187-208. doi: 10.3934/jmd.2008.2.187
[12]

Amin Boumenir. Determining the shape of a solid of revolution. Mathematical Control & Related Fields, 2019, 9 (3) : 509-515. doi: 10.3934/mcrf.2019023
[13]

Jan Philipp Schröder. Ergodicity and topological entropy of geodesic flows on surfaces. Journal of Modern Dynamics, 2015, 9: 147-167. doi: 10.3934/jmd.2015.9.147
[14]

Marianne Akian, Stéphane Gaubert, Antoine Hochart. Ergodicity conditions for zero-sum games. Discrete & Continuous Dynamical Systems, 2015, 35 (9) : 3901-3931. doi: 10.3934/dcds.2015.35.3901
[15]

David Ralston, Serge Troubetzkoy. Ergodicity of certain cocycles over certain interval exchanges. Discrete & Continuous Dynamical Systems, 2013, 33 (6) : 2523-2529. doi: 10.3934/dcds.2013.33.2523
[16]

F. Rodriguez Hertz, M. A. Rodriguez Hertz, A. Tahzibi and R. Ures. A criterion for ergodicity for non-uniformly hyperbolic diffeomorphisms. Electronic Research Announcements, 2007, 14: 74-81. doi: 10.3934/era.2007.14.74
[17]

Yunhua Zhou. The local $C^1$-density of stable ergodicity. Discrete & Continuous Dynamical Systems, 2013, 33 (7) : 2621-2629. doi: 10.3934/dcds.2013.33.2621
[18]

Mark Pollicott. Ergodicity of stable manifolds for nilpotent extensions of Anosov flows. Discrete & Continuous Dynamical Systems, 2002, 8 (3) : 599-604. doi: 10.3934/dcds.2002.8.599
[19]

Jon Aaronson, Michael Bromberg, Nishant Chandgotia. Rational ergodicity of step function skew products. Journal of Modern Dynamics, 2018, 13: 1-42. doi: 10.3934/jmd.2018012
[20]

Peter Hinow, Ami Radunskaya. Ergodicity and loss of capacity for a random family of concave maps. Discrete & Continuous Dynamical Systems - B, 2016, 21 (7) : 2193-2210. doi: 10.3934/dcdsb.2016043

2020 Impact Factor: 1.639

Tools

Metrics

  • PDF downloads (155)
  • HTML views (334)
  • Cited by (0)

Other articles
by authors

[Back to Top]

Copyright © 2021 American Institute of Mathematical Sciences | Privacy Policy