Online supervised attention-based recurrent depth estimation from monocular video

Supervised depth estimation

Depth estimation from a single image is classified as an ill-posed problem, as long as one input image can be projected to multiple plausible depths (across sensors and methods of acquisition). Recently, many depth estimation methods, which are based on the deep learning, have achieved great results.

In Xie, Girshick & Farhadi (2016), shortcut connections were used in network in order to fuse low-level and high-level features. In Eigen, Puhrsch & Fergus (2014), authors used a multi-scale neural network with two components in order to generate coarse estimations globally and refine the corresponding results locally. Another view on the depth estimation problem was introduced in Cao, Wu & Shen (2018), where authors formulated a classification problem instead of a regression problem.

Further improvements required new ideas for loss functions. In Laina et al. (2016), authors employed a reverse Huber loss to estimate depth distributions, while in Yin et al. (2019) authors implemented the loss term that enforces geometric constraints. Same time, perceptual loss, which was introduced in Johnson, Alahi & Fei-Fei (2016), was successfully used in the dense depth estimation task (Wang et al., 2018a; Makarov et al., 2017). To further boost performance, some works have integrated continuous CRFs in their DL framework (Liu et al., 2015).

One of the ways to enhance performance of the methods, which predict depth from RGB images, is to use additional information from other sources. One such example of a source is sparse depth, which could be extracted from SLAM systems (Mal & Karaman, 2018). A LiDAR sensor can also serve as a sparse depth input (Liao et al., 2017). Model based on the semi-dense depth interpolation was presented in Makarov, Aliev & Gerasimova (2017), where authors proposed an end-to-end learnable residual convolutional neural network architecture, that achieved fast interpolation of semi-dense depth maps. The suggested approach was later improved for fast depth estimation from sparse (Makarov, Korinevskaya & Aliev, 2018a; Makarov, Korinevskaya & Aliev, 2018b) and low resolution (Korinevskaya & Makarov, 2018) depth values.

Recently, excellent performance was achieved in Fu et al. (2018), where authors proposed a SID policy and ordinal regression loss. Although, the results are great, this method can not be used in the mobile robotics platforms, due to complex network architecture, thus, it can not be applied in real-time for video processing.

In general, the problem of encoder–decoder architectures or GANs for the supervised depth estimation lies either in current limitations for error improvement or slow performance making approaches inapplicable for real-world scenarios on constraint resources.

Self-supervised depth estimation

Acquiring the ground truth depth is quite a challenging task. That is why the alternative methods rise, where image reconstruction is used as a supervisory signal. Mainly, there are two types of methods: one use stereo pairs for the training, while another use monocular sequences. Thus, this type of models are trained via minimizing the image reconstruction error, where depth for certain image is projected in nearby views.

For the methods, which use stereo pairs as input, the pixel disparities between the synchronized stereo pair are predicted during training. Authors from (Xie, Girshick & Farhadi, 2016) introduced a model with discretized depth in context of the novel view synthesis problem. In Godard, Mac Aodha & Brostow (2017), authors enhanced performance by using left–right view consistency as the supervisory signal, while in Garg et al. (2016) performance was improved by predicting continuous disparity values.

Using monocular sequences as input leads to slightly different methods, as long as camera pose estimation between frames plays a crucial role in the whole training pipeline. Estimation of the camera pose is quite challenging, due to the object motion, but it only requires during training stage. Great progress has been made in this field, starting from Zhou et al. (2017), where separate pose network was trained along depth estimation network, to Godard, Mac Aodha & Brostow (2017), where a novel multi-scale sample method and an auto-masking loss were introduced.

Although the sequential video frames are used in view-synthesis loss, the spatiotemporal data at longer range is still missing.

Video depth estimation

One of the earliest work in this field was Karsch, Liu & Kang (2014), in which authors improved depth estimation by using local motion cues and secured temporal depth consistency via optical flow. However, this method is offline, which is not suitable for our online setting. In Ranftl et al. (2016), consecutive frame information was used for optical flow segmentation and depth estimation via geometry reconstruction. In Xu et al. (2017), depth estimation based on multi-scale convolutional neural networks with continuous Conditional Random Fields (CRFs) refinement was proposed. The authors used either cascade of multiple CRFs, or unified graphical model. Later, several works were presented, which focused on unsupervised video depth estimation methods (Mahjourian, Wicke & Angelova, 2018; Wang et al., 2018b).

Recently, convLSTM was proposed for the weather forecasting task (Xingjian et al., 2015) and convGRU was introduced for solving both human action recognition and video captioning tasks (Ballas et al., 2015). Recently, convLSTM was successfully used in the real-time video depth estimation task (Zhang et al., 2019), in which authors boosted performance with temporal consistency loss and generative adversarial learning scheme. In Vaishakh et al. (2020), authors have also exploited convLSTM in the real-time depth estimation task, however, they focused mostly on self-supervised setting and training strategy, which involved pre-training of the initial hidden states.

Thus, according to our knowledge, still there were no works, which integrated the convGRU block in real-time depth estimation pipeline for videos and, hence, there were no comparison between convGRU and convLSTM in this online setting. As we know, GRU is yet another gated architecture. In Chung et al. (2014), authors showed, that GRU has similar performance in comparison with LSTM. Also, reduced number of gates leads to fewer parameters in model, which reduces complexity of the whole pipeline and that is a crucial point for online mobile robot applications. That is why we compare both recurrent blocks: convGRU and convLSTM as a part of the real-time depth estimation task.

Integration of the recurrent block in pipeline already leads to depth estimation accuracy improvement. Moreover, this pipeline can be further enhanced by explicitly exploiting previous frames information, particularly by using attention mechanism as we show below.

Supervised depth estimation

We formulate the depth estimation task as a regression problem: The task is to learn function f:X→^Y, where X is an input RGB image and $\hat{Y}$ is a predicted depth, where $\hat{Y}$ minimizes loss function $L\left(\hat{Y},Y\right)$, where Y correspond to ground truth depth. We follow the common architecture for the depth estimation network and represent it as encoder–decoder (following notations of Vaishakh et al. (2020)). Thus, it takes the next form: (1)${\displaystyle Z=f_{enc}\left(X\right),\hat{Y}=f_{dec}\left(Z\right),}$ where Z correspond to encoder bottleneck features.

Another subject to consider is the choice of a loss function. The common loss functions are: L₁, L₂ and Reversed Huber loss (denoted as berHu), which was introduced in Owen (2007). According to Makarov, Aliev & Gerasimova (2017), L₂ loss function tends to give oversmoothed results and often has poor perceptual quality. Same time, Mal & Karaman (2018) showed that using L₁ loss leads to better results compared with berHu loss. Following mentioned above considerations, we decided to use L₁ loss in our work. Following (Mal & Karaman, 2018), we apply a binary mask M_i,j of dimensions m × n, where M_i,j = 1 for valid values of the ground truth depth map $\hat{Y}$. Same as in Godard, Mac Aodha & Brostow (2017), we add an edge-aware smoothing loss term, which can be introduced as: (2)${\displaystyle L_{smooth}=\left|{\partial }_x{\hat{Y}}^{\ast }\right|e^{\left|\right|-{\partial }_{x}X\left|\right|}+\left|{\partial }_y{\hat{Y}}^{\ast }\right|e^{\left|\right|-{\partial }_{y}X\left|\right|},}$ where ${\hat{Y}}^{\ast }=\hat{Y}∕\widehat{\stackrel{̄}{Y}}$ is mean normalized inverse depth from Wang et al. (2018b) to prevent shrinking of the estimated depth.

In the next section we describe two recurrent blocks: convGRU and convLSTM, which will be integrated between encoder and decoder modules.

Recurrent blocks: convLSTM and convGRU

Recently, LSTMs achieved great results in various sequence-to-sequence tasks, for example in speech recognition (Graves, rahman Mohamed & Hinton, 2013) and machine translation (Parmar & Devi, 2018). Long and short term temporal dependencies can be captured by utilizing the memory cell mechanism. Still, standard LSTM uses one dimensional input, that is why we can not directly apply it to image sequences. Xingjian et al. (2015) overcame this problem by introducing the convLSTM mechanism, which allowed to handle two-dimensional feature maps. In this case i_t, f_t, o_t gates, cell outputs C_t, and hidden states H_t are 3D tensors, which last two dimensions are spatial dimensions.

The original structure of the convLSTM cell did not work properly in our experiments, that is why we used the cell structure, described in Zhang et al. (2019). The key equations are shown below in ??, where ∗ denotes convolutional operator and W_xk, W_hk, and b_k, k ∈ {f, i, o, c}, denotes the kernel and bias terms for the corresponding convolutional layers: (3)${\displaystyle f_t=\sigma \left(W_{xf}\ast X_t+W_{hf}\ast H_{t-1}+b_f\right),i_t=\sigma \left(W_{xi}\ast X_t+W_{hi}\ast H_{t-1}+b_i\right),o_t=\sigma \left(W_{xo}\ast X_t+W_{ho}\ast H_{t-1}+b_o\right),\widetilde{C_t}=tanh\left(W_{xc}\ast X_t+W_{hc}\ast H_{t-1}+b_c\right),C_t=f_t\circ C_{t-1}{i}_t\circ \widetilde{C_t},H_t=o_t\circ tanh\left(C_t\right)}$

GRU follows the same gated principal as LSTM, but with a little simpler architecture. It has reduced number of gates thus fewer parameters (Chung et al., 2014). The ConvGRU was first introduced in Ballas et al. (2015) for the video captioning task. It also performed well in the video segmentation task, as shown in Siam et al. (2017). Equation (4) describe mathematical model of the ConvGRU, where * is a convolutional operator, z_t, r_t correspond to gates, H_t correspond to a hidden state, W_xk, W_hk, and b_k, k ∈ {z, r, h} correspond to kernel and bias terms, respectively: (4)${\displaystyle z_t=\sigma \left(W_{xz}\ast X_t+W_{hz}\ast H_{t-1}+b_z\right),r_t=\sigma \left(W_{xr}\ast X_t+W_{hr}\ast H_{t-1}+b_r\right),\widetilde{h_t}=tanh\left(W_x\ast X_t+W_h\ast \left(r_t\circ H_{t-1}\right)+b_h\right),H_t=\left(1-z_t\right)\left(\circ H_{t-1}+z\circ \widetilde{h_t}\right)}$

Figure 2 shows architectures of both convLSTM (A) and convGRU blocks (B).

In the next section, we will declare the recurrent depth estimation pipeline and demonstrate the role of recurrent blocks described above in it.

Supervised recurrent depth prediction

In ‘Supervised depth estimation’ we described a baseline encoder–decoder architecture, which estimates depth for a single frame separately from other frames, which leads to loss of the spatiotemporal information across sequence. In this section we formulate the recurrent depth estimation problem and formalize the convGRU and the convLSTM role in recurrent pipeline.

The problem is formulated as follows: at timestamp t, given the encoder representation Z_t for image X_t and given previous hidden states of the recurrent block (H_t−1, H_t−2, .., H_t−n), we need to estimate depth map $\widehat{Y_t}$: (5)${\displaystyle \widehat{Y_t}=\underset{\widetilde{Y_t}}{argmin}P\left(\widetilde{Y_t}|Z_t,H_{t-1},H_{t-2},..,H_{t-n}\right),}$ where n equals to the number of frames from the beginning of the sequence.

So, after adding the recurrent block, which can be either a convGRU or a convLSTM, the basic supervised depth estimation pipeline looks like this: at time t, we pass image X_t through the encoder Z_t = f_enc(X_t), then we pass Z_t through convLSTM (6)${\displaystyle H_t,C_t=f_{convLSTM}\left(Z_t,H_{t-1},C_{t-1}\right),}$ or through convGRU (7)${\displaystyle H_t=f_{convGRU}\left(Z_t,H_{t-1}\right),}$ where H_t and C_t correspond to hidden and cell states, respectively. Finally, in order to get a depth estimation ${\hat{Y}}_t$, a hidden representation H_t is passed through decoder ${\hat{Y}}_t=f_{dec}\left(H_t\right)$. Figure 3 demonstrates the basic recurrent depth estimation pipeline with convLSTM as recurrent block.

Thus, the integration of recurrent block in our pipeline helps us to capture and exploit the spatiotemporal information across frame sequence. In our work, we get boost in the performance, in comparison with the non-recurrent approach. We explain it by ability of the recurrent block to capture motions of visual components via transitional kernels, whereas we consider a hidden state as a hidden representation of visual structure according to Xingjian et al. (2015).

Integration of the recurrent block in the non-recurrent based architecture, already gives us improvements in the depth prediction accuracy. However, there are still ways to improve current architecture, by exploiting previous frames information for current frame depth prediction. In the next section, we propose yet simple, but quite efficient architecture modifications.

Exploiting previous frames information

Recently, there were few works, which focused on exploiting the temporal information via concatenation of multiple frames at input, such as Sun et al. (2015) and Diba et al. (2019). The problem of these approaches lies in inability to scale well on long sequences. Same time, the explicit injection of previous frame information can benefit in predicting the depth for current frame was suggested in Kaneko & Yamamoto (2017). Thus, in this section, we propose three modifications to the recurrent depth estimation pipeline, which utilize previous frames information.

One of straightforward methods is the explicit injection of just one previous frame, which can be described as follows: (8)${\displaystyle \widehat{Y_t}=f_{dec}\left(g\left(H_t,H_{t-1}\right)\right),}$ where function g refers to either concatenation or fusion. Although, the explicit injection can be useful, it can not give significant boost in the performance, due to a little difference between adjacent frames. Hence, we propose a third attention-based modification, which aggregates information from previous k frames hidden representations.

First, let us denote f_RecBlock1, f_RecBlock2 as Layer 1 and Layer 2 of the recurrent block respectively, where recurrent block can be either a convGRU or a convLSTM. Let us refer to H1_t, H1_t−1, …, H1_t−k and H2_t, H2_t−1, …, H2_t−k as hidden states at timestamps t, t − 1, …, t − k for the Layer 1 and the Layer 2 respectively. The idea is to form the context vector, which preserve information from k previous frames. We propose an attention mechanism, which is based on relevance between two hidden states.

Let us define an alignment score between hidden state s and v as: (9)${\displaystyle score\left(H_s,H_v\right)=H_s^T{H}_v,}$ which is a scalar product between two vectors. The alignment weights for the previous k frames are obtained by the following formula: (10)${\displaystyle {\alpha }_t\left(s\right)=\frac{score\left(H{1}_t,H{1}_{t-s}\right)}{\sum _{\grave{s}=1}^{k}score\left(H{1}_t,H{1}_{t-\grave{s}}\right)},}$ where s ∈ 1, 2.., k.

To get the context vector for the current timestamp t, we do weighted average on k previous hidden states from Layer 2: (11)${\displaystyle u_t=\sum _{s=1}^k{\alpha }_t\left(s\right)H{2}_{t-s}}$ Thus, we obtain attention scores from Layer 1 hidden states and construct context vector from Layer 2 hidden states. For the final step, we concatenate the context vector and the hidden state for timestamp t and pass it to decoder: (12)${\displaystyle \widehat{Y_t}=f_{dec}\left(H{2}_t\oplus u_t\right),}$

Figure 4 contains illustrations for all three modifications. While first two modifications (A) are rather simple and can not give significant enhance in depth prediction accuracy, due to exploiting information only from the previous frame, the third one (B) is more complex and provide an accuracy improvement, by utilizing attention-based mechanism. In section ‘Experiment with model modifications’ we analyze the benefits of the proposed modifications in comparison with baseline non-recurrent and recurrent pipelines.

It is important to note few things:

• Before constructing the context vector, all hidden representations are reshaped from 3-dimensional to 1 dimensional vector;

• Due to high dimensionality of hidden states, it is impossible to implement the classical softmax attention mechanism, as long as it contains exponent step.

Dataset

In our work, we report results on KITTI Dataset (Geiger et al., 2013). This dataset contains 61 outdoor video scenes captured by cameras and depth sensors, fixed on a driving car. The initial resolution of the videos is 375x1242. Training on frame sequences is quite challenging task, considering the GPU RAM issues. Hence, in order to decrease memory costs, in all our experiments we resize images to a resolution of 192 × 640. Works Vaishakh et al. (2020) and Godard et al. (2019) use the same resolution, which leads to fair comparison with their results. We use Eigen split, defined in Eigen & Fergus (2015), which separates 32 monocular sequences for training and 29 sequences for testing. 697 specific samples from test sequences are used as a standard evaluation test set (Vaishakh et al., 2020; Godard et al., 2019) for the non-recurrent based experiments. Considering the recurrent-based experiments, we split 32 sequences into 27 for training and 5 for validation. We divide sequences into small sub-sequences of 10 frames long. This sub-sequences are used as training samples. During test phase we evaluate on complete video sequences. We cap the maximum predictions to 80 m.

Training Details

For non-recurrent based experiments we use the batch size of 24; while for recurrent based experiments we use the batch size of 6, due to GPU RAM limitations, as long as we use sequences as training samples. We resize all images to a resolution of 192 × 640, due to GPU RAM limitations. We choose the Adam optimizer and the learning rate of 10⁻⁴. We apply exponential decay for learning rate with the decay rate 0.96. For the smooth loss term, we use weight 0.001. To achieve competitive results, we follow Vaishakh et al. (2020), Godard et al. (2019) and Luo et al. (2018) and use pretrained ImageNet (Krizhevsky, Sutskever & Hinton, 2017) weights for the encoder. Additionally, we apply following augmentations:

• color and depths are both horizontally flipped with a 50% chance;

• brightness, contrast, saturation and hue jitter with respective ranges of ±0.2, ±0.2, ±0.2 and ±0.1.

For the attention modification, we look at previous 3 frames. We train both recurrent-based and non-recurrent based architectures for 20 epochs.

Baseline architecture

For the baseline approach, we follow the U-net work (Ronneberger, Fischer & Brox, 2015) and use the encoder–decoder architecture with skip connections, where ResNet-18 (He et al., 2016) performs the role of an encoder. As for the decoder, we apply upconvolutional blocks same as in DispNet work by Mayer et al. (2016). At the final step, we get the depth prediction via inverse transformation of disparity output: $\hat{Y}=1∕\left(a\sigma +b\right)$, where a and b are selected to limit depth $\hat{Y}$ from 0.1 to 100 m.

The basic approach is expanded to recurrent, by adding a recurrent module between the encoder and decoder, either convGRU presented in Ballas et al. (2015) or convLSTM preented in Xingjian et al. (2015), such that encoder output is passed to this module. Details are provided in ‘Supervised recurrent depth prediction’.

Training strategy

There are several important aspects to consider when we design the training strategy for the recurrent depth estimation pipeline.

The first one is about the impact of the initial hidden state of a recurrent block. As we know, in a vanilla LSTM and in a vanilla GRU, the latter are initialized by zeros. This is a common practice, taking into account the tasks in which those blocks are applied (for example, Machine Translation or Time Series Analysis). It works fine in these type of tasks for several reasons: length of sequences is relatively long compared to the size of hidden state and the impact of initial hidden state is trivial. However, when we deal with monocular videos, we have some restrictions on memory resources, which leads to shortening the length of sequences at training stage. Thus, the impact of the initial hidden state becomes crucial. Following work (Vaishakh et al., 2020), we divide our training process in two stages: at the first stage, we consider the initial hidden state as learnable parameter, then, at the second stage, the trained initial hidden state is used at the start of every frame sequence.

The second aspect concerns the impact of the pretrained encoder and the pretrained decoder on the training process of the recurrent pipeline. As we know, the common way is to train architecture in end-to-end manner, however, we can let the recurrent block to adapt to sequence faster, by preloading weights, which were obtained as a result of training in the supervised baseline approach. Thus, we perform a finetuning of architecture with additional recurrent block on monocular video sequences.

System configuration

All models were implemented with Pytorch Paszke et al., 2017. All experiments were carried out with the RTX6000 GPU, Intel(R) Xeon(R) CPU and 32 Gb RAM. The operating system is Ubuntu 18.04. All process was performed using free and open-source distribution Anaconda with python version 3.7.4.

Experiment with model modifications

The experiments are carried out in the following order: At first, we train a non-recurrent architecture (Table 1 Baseline row). Then we add a recurrent block between encoder and decoder modules and train in end-to-end manner. We test both convGRU and convLSTM (Table 1 convGRU and convLSTM rows). After that, we preload encoder and decoder weights, received from the first experiment, and train in same manner (Table 1 recurrent Block + weights rows). Then we test all the architecture modifications: concatenation/fusion of previous frame hidden state with current frame hidden state and modification based on attention mechanism. Following (Vaishakh et al., 2020), we experiment with activation function: Tanh activation function is replaced with ELU activation function (Table 1 last row). We conduct this change only with convGRU architecture, because activation function change in convLSTM cell leads to worse results.

The results of these experiments are presented in Table 1. As we can see, the recurrent-based architectures, which use preloaded weights from the non-recurrent based model, outperforms the baseline supervised model (4.181 against 4.397 RMSE). Moreover, we see an improvement by 8% in comparison with the model, trained from scratch, both for convLSTM, convGRU, which confirms the correct use of preloaded weights for encoder, decoder blocks. As we can see, convLSTM slightly outperforms convGRU (by 0.007 RMSE), however the difference is not that great. By using the attention modification, the results are improved even further. First two modifications, which utilize previous frame information by doing concatenation/fusion with current frame hidden state, only make results worse. This happens due to little difference between adjacent frames.

It is important to note, that the attention modification works better with convLSTM (4.168 RMSE). At last, the change of Tanh activation function to ELU activation function in convGRU cell, leads to even better results (4.104 RMSE). We see an improvement, comparing with Tanh activation function, because, in this case we match the scale with the output of encoder. Summing up, the attention-based recurrent architecture showed an improvement in comparison with baseline non-recurrent architecture by 0.293 RMSE, which demonstrates the effectiveness of proposed method.

Comparison with state-of-the-art models

Table 2 contains results from previous state-of-the-art works and result of our best recurrent-based architecture. By comparing results of our best supervised-based approach with other supervised approaches, we see a significant improvement. Fu et al. (2018) method still outperforms our method, however, their method is not suitable for real-time depth estimation task (autonomous driving), due to complex network architecture; also it provides poorer results in terms of visual perception compared to our approach. The results of speed tests on Nvidia GTX-1060 with MAX-Q Design GPU are shown in Table 3.

Our method, based on convGRU and attention mechanism, shows better results, than methods, based on convLSTM in Zhang et al. (2019) and Vaishakh et al. (2020), which proves efficiency of attention-based modification together with online performance.