When overviewing time-series data in video editing software and simulation software, one-dimensional data can be represented with line graphs or bar charts, but difficulties arise when the target data spreads in the spatial direction. In thumbnail displays of video viewing software, there is a problem that either they occupy a large display area, or when displayed sparsely, continuity on the time axis is lost. As a method aimed at solving this, there is the Video slicing method that defines scan lines in the target video and displays scanned line segment images arranged chronologically, but this method has the issue of losing spatial continuity.
In this research, we developed a method called TimeSpaceSlice that extends the Video slicing method and aims to partially maintain spatiotemporal continuity by moving scan lines within a specified range according to the passage of time. When we applied the proposed method to several videos and experimented, we were able to present temporal changes at the shortest possible time intervals while preserving some spatial information of the observation target.
TimeSpaceSlice Method
The figure above shows a display example using the TimeSpaceSlice method. The left image shows the target video with a white rectangle indicating the scan line movement range. The right image shows the generated timeline display.
In the TimeSpaceSlice method, we first define a region in the target video and move scan lines within that range. Specifically, we use scan lines with height equal to the rectangular region’s height and width of 1 pixel, moving them back and forth within the specified range. By moving the scan line 1 pixel per frame and arranging the pixels on the line chronologically from left to right, we generate a timeline display that partially maintains spatiotemporal continuity.
The example uses candle combustion video. Since there was no wind and no flame movement for a while after the video started, the flame shape is preserved and displayed on the timeline. Later, when wind blows and the flame begins to move, the flame shape on the timeline becomes disturbed, visualizing the presence of movement. Additionally, from the flame height on the scan line, we can estimate the wind strength and its temporal changes.
Application Examples
The above figure shows a TimeSpaceSlice display of a Chinese cooking scene. The timeline on the right progresses from bottom-left to top-right. An effective example of the proposed method is footage from fixed cameras. In this Chinese cooking scene, we can clearly observe on the timeline when seasonings are added or when the pan stops for washing, and we can roughly grasp the state of the pan at those times.
The above figure shows the same video displayed using the video slicing method. Due to lack of spatial continuity, it is difficult to understand what is shown. Compared to this video slicing display, our proposed method maintains better spatial continuity when the pan is stationary, making it easier to understand the video content.
We applied this method to develop a gaze movement data visualization technique called “GazeTiling”.
Acknowledgments
The Chinese cooking video used as an example in this research was used with the permission of the copyright holder as follows:
『【営業中】中華一筋厨房営業中を俯瞰で覗く動画』 https://youtu.be/7cWrCBRnAa4?si=MXzCFOKqYTU9cp_Q
We thank 中華一筋 for kindly granting permission to use the video.