Space Time Velocimetry for Measuring Surface Velocity Vector Distribution from UAV

. Due to the remarkable development of unmanned aerial vehicle (UAV) in recent years, its application in river engineering increases widely mainly for the measurement of ground topography such as by the technique Structure from Motion (SfM) using a series of high-resolution static images. However, although UAV usually installed a high density video camera, the use of the movie is limited just for watching and observing the geometrical feature of the ground. In the light of such a present status, the authors have developed an aerial space-time image velocimetry (STIV) technique to measure streamwise river surface velocity distributions. However, as STIV is insensitive to the change of flow direction, the aerial space-time volume velocimetry (STVV) technique, which is an extension of STIV, was developed in this research. STVV examines the change of volumetric texture within a space-time volume (STV) instead of examining the change of image intensity on a line segment as in STIV. The performance of STVV was investigated during the measurement of snowmelt flood of the Shinano River by comparing it with those obtained by the other techniques such as STIV, LSPIV and ADCP. It was made clear the aerial STVV has a great advantage over the existing image-based techniques.


Introduction
In recent years, thanks to the development of unmanned aerial vehicle (UAV), usually called drone, its application to the river engineering has boadened significantly. One of the application is for the measurement of ground topography by the Structure from Motion (SfM) technique [1][2][3]. In another case, UAV is used for measuring gravel size distributions by a sofisticated image analysis technique [4,5]. As for the aerial river flow measurement techniques, Fujita and Hino [6] and Fujita and Kunita [7] analyzed flood video images captured from a manned helicopter by using the aerial Large Scale Particle Image Velocimetry (LSPIV) technique. The use of UAV has become available with a development of efficient feature detection techniques for stabilizing video images taken from the air at different positions [8,9]. Regarding the flow measurement technique, Fujita et al. [8] and Notoya et al. [9] used the space-time image velocimetry (STIV) [10,11] by using hundreds of consetutive images stabilized. STIV is a technique to measure streamwise velocity distributions more efficiently and accurately when compared with LSPIV by generating a space-time image (STI) for a search line set in the direction of main flow. However, STIV requires predetermined streamwise direction for setting a search line, which is sometimes difficult to perform when the flow direction changes from place to place as in a meandering flow. To overcome the shortcoming of STIV, the space time volume image velocimetry (STVV) was developed in this research, allowing the measurement of surface velocity vector distributions more stably than LSPIV. In the following sections, the algorithm of STVV and its application to the actual flow field will be presented, including the comparison with an acoustic Dopper current profiler (ADCP).

SPACE TIME VOLUME VELOCIMETRY (STVV)
In STVV, the time evolution of image intensity distribution within a rectangular area set on a river surface is expressed as a space time volume (STV) having three axes with two image coordinate directions and the time direction. This is an extension of space time image (STI) in the STIV technique. To extract the flow direction and flow velocity from STV, a threedimensional autocorrelation function (3D-acf) of the image intensity distribution wihin an STV is first calculated. Then the velocity vector is extracted after applying the spherical logarithmic transformation of the 3D-acf.

Generation of STV
In the application of STVV to surface flow measurements, a rectangular area, termed the search area, similar to a template window of PIV is set at a measurement point within a water surface. The size of the search area has to be large enough to include the time change of surface texture. Therefore, unlike PIV measurements in laboratory flume using a large amount of small tracer particles, the size of one side can not be made very small. The length of one side of the area varies from several meters to more than ten meters depending on the actual situation. A space time volume is generated by stacking consecutive image intensity distributions in a vertical direction as shown in Figure 1. When the direction of u is taken in the streamwise direction, the surface u-t corresponds to the space-time image in STIV. In STVV, the direction of the search area is arbitrary and the texture orientations appeared on the surfaces u-t and v-t indicate the velocity component in the u-axis and the v-axis, respectively. The surface u-v includes the information of the velocity vector direction.

Calculation of autocorrelation function
In order to clarify the general direction of the volumetric texture orientation contained in STV, the three-dimensional autocorrelation function (3D-acf) is calculated by the following equation, Here,  is the shift amount vector (u,v,t and f(x) is the 3D-image intensity distribution within an STV. In STVV, 3D-acf is calculated from the inverse Fourier transform of the power spectrum density by using the Wiener Khinchin's theorem as follows, with the angular frequency. The distribution of the autocorrelation function is normalized such that R(0) =1. An example of the autocorrelation function is shown in Figure 2, with the isosurface of 0.65. It can be seen that an inclined ellipsoid centered at R(0,0,0) is extracted from the STV indicating the local flow direction. The major axis corresponds to the direction of the texture gradient with time. The gradients in the planes t-u and t-v indicate the velocities in the u and v directions, respectively and the gradient in the u-v plane indicates the velocity vector within the search area.
(a) 3D view (b)t-u plane (c)t-v plane (d)u-v plane

Logarithmic spherical transformation of 3D-acf
It may be possible to calculate the velocity vector from the gradient of the ellipsoid indicated in Figure 2, but for improving the measurement accuracy the distribution of the 3D-acf expressed in the Cartesian coordinate is transformed into a logarithmic spherical (LS) coordinates. The transformation is defined by Here, src stands for the three dimensional autocorrelation function in the Cartesian coordinates and dst is the transformed distribution in the LS coordinates. M is an intensity parameter for emphasizing the data close to the origin because the important distribution is concentrated near the origin. The default value of M is one. An example of the transformed distribution is provided in Figure 3. The coordinates showing the peak value in the  plane provide the detected direction of the flow.

Averaging of 3D-acf in  direction
In order to find the peak coordinates of 3D-acf more efficiently, the 3D distribution is averaged in the direction of the axis using the following equation, with nt the number of elements in the axis. An example of the directional average distribution of 3D-acf is shown in Figure 4, indicating two sharp peaks in the distribution. As one of the peaks differs from the other by a phase of , the two peaks corresponds to the axis of the ellipsoid.

Fig.4. Distribution of ).
Once the peak of the directional average distribution is obtained as (peakpeak), the velocity vector can be calculated from the following equations.
The coefficient k is a constant value obtained by the following relation, where F [frame/s] stands for the frame rate of the video and S [m/pixel] is the physical size of one pixel. In STV, since each frame is stacked in the time direction as an image, the unit of the frame is pixel; therefore the unit of k becomes meter per second.

APPLICATION OF STVV TO AIRBORNE SURFACE IMAGE
The new image analysis technique STVV developed so far was applied to the aerial video images of the snowmelt flood of the Shinano River captured from an UAV. In the image capturing, the target area of about 900m in length was covered by changing the posision of UAV place by place.

Outline of the site and UAV measurements
The measurement site was near the Asahi Bridge of the Shinano River in Ojiya City, Niigata Prefecture, Japan. The measurements were conducted downstream and upstream of the bridge from April 26 to 28 in 2017. During the measurement, snowmelt flood occurred and it made the flow faster than the usual normal flow condition. The Shinano River is the longest river in Japan with a main channel length of 367 km. The Asahi Bridge location, 34.5km from the river mouth shown in Figure 5, is the official stream gauging location of the Shinano River. The water surface width in the observation area was about 140m and the maximum depth was 9.37 m in the downstream part of the bridge. The average wind speed was about 2.0 m/s, and its influence on the surface flow velocity was considered minimal.
For shooting the airborne video image, an UAV (Inspire 1 manufactured by DJI) capable to capture 4K videos for about fifteen minutes in one flight was used. As seen from Figure 5, the flow bends upstream of the bridge and flows almost straight after the bend. The measurement by UAV was conducted by subdividing the target area into several overlapping zones and by shooting the flow image of each zone. The video image was taken repeatedly while hovering for about 1 minute above one zone.

Image stabilization and Correction of shooting angle
The video image taken from the air is subject to background shift effect even when the mechanical stabilizing system is activating; i.e. the first image and the last image captured after about one minute does not overlap completely and this causes a serious impact on the application of STVV because it takes all frame images into account as a form of STV. Therefore, all of the images used in STVV were stabilized before the analysis by using the sophisticated image stabilizing technique [8,9].
In the present methodology, airborne images have to be obtained normal to the water surface plane so that overlapping of the images at different locations is conducted simply and efficiently. However, the camera angle was found to be tilted at 4 degrees after the shooting, which caused a change in the field of view of about 20 meters from the normal image because the image was taken from an altitude of 150 m from the water surface. Therefore, an angle correction was performed before the analysis. In adjusting the angle of tilt, information on the camera's depression angle, focal length, sensor size, and camera height was used.

Comparison with LSPIV
To examine the performance of STVV, the velocity distribution measured at a bend upstream of the Asahi Bridge was compared with the measurement by LSPIV. The template of LSPIV and the search area of STVV are set to have the same area, 100 by 100 pixel and 450 frame images were used for the analysis. The results in Figure 6 show almost similar distributions between the two techniques except for the vectors close to the inner and outer banks. In order to compare the measurement accuracy, a search line was set in the direction of the each vector and the velocity in the direction was measured by STIV. Some example of STI is indicated in Figure 6. The vertical scale is 15 seconds and the horizontal scale is 15.5m in this case. At No.8, STVV and LSPIV yielded almost the same results because STI displays clear texture.
On the other hand, at No.13, STI in the direction of STVV generates much clearer texture than the one in the direction detected by LSPIV, indicating STVV can provide more reliable data. In order to confirm this feature, distribution of relative velocity errors of STVV and LSPIV against STIV is plotted in Figure 7. Generally, STVV provides relatively smaller errors than LSPIV

Comparison with ADCP
In the present field measurement, a remote-controlled boat installed an ADCP was used to measure the upstream section by navigating the boat in a zigzag manner. Since the UAV  measurement was conducted during the ADCP measurement as well, the ADCP data closest to the water surface, about 63cm below the water surface, can be compared with STVV and LSPIV. Before the analysis, an image process to enhance the surface texture was applied by controlling the image mapping table. The measurement results were compared for the same trajectory of the ADCP boat as shown in Figure 8. It can be noted that STVV provides smoothly varying data than the other methods, except at the position with a red circle where an ADCP boat was navigating. LSPIV yielded comparable data in a deeper region near the left bank but unreliable data was obtained in a relatively shallower region in which surface texture were vague. The difference between STVV and ADCP is due to the difference between measurement depth and measurement time; i.e. the former measures surface velocity for about 15 seconds while the latter measures subsurface flow almost instantaneously.

Surface velocity map by STVV
Since the performance of STVV was verified in the above, the target area was expanded 200 m downstream and 700 m upstream of the Asahi Bridge. The upstream and downstream areas were covered by four and three zones respectively as shown in Figure 9. In this figure, each vector indicates only the direction and its magnitude is expressed by color chart. In this case, each flight took about ten minutes while hovering for one minute above each zone. Although the data in the shallower area are missing, velocity vector fields for each zone were successfully overlapped smoothly. It can be noted that the flow bends smoothly at the deflection corner without changing its velocity. On the other hand, in the downstream of the bridge, the flow is accelerated significantly after passing through the bridge piers with the maximum velocity more than 4 m/s and the accelerated flow region extended farther downstream of the bridge. It should be noted that a surface flow field was successfully measured only by tracking the surface features without any seeding to visualize the flow.

Conclusions
A novel image analysis technique for measuring river surface flow, STVV, was developed. STVV is an extension of STIV that uses all frame images to analyse averaged space-time velocity vectors at one time, which is different from LSPIV that requires averaging process after conducting instantaneous measurement. It was shown that STVV is capable to extract velocity data even where LSPIV yields erroneous value. With the use of UAV, it becomes possible to map a much longer river reach by repeating image shooting by hovering. The measurement accuracy of STVV has to be investigated by varying the size of the search area and flow conditions such as flows with a large shear or unsteady flows.
Part of this research was subsidized by scientific research grants and river improvement fund. The measurement result is one of the results of the joint observation of the flow observation technology advancement study subcommittee and the civil engineering laboratory (ICHARM) of the Japan Society of Water Engineering Committee. I'd like to express my appreciation here.