A semi-automatic motion-constrained Graph Cut algorithm for Pedestrian Detection in thermal surveillance videos

Introduction

Video surveillance technology is rapidly proliferating across public and private spaces. Traditionally, surveillance systems could only be found on buildings owned by the Government and large organisations. Currently, they can be found in a variety of settings such as shops, stadia, airports, schools and private residences. Two main factors are responsible for the ubiquity of video surveillance systems (VSS). The first is increased ease of acquisition and installation of VSS. This is due to the advancements in technology from analogue to digital systems and the significant drop in the cost of acquisition. The second factor is the increasing need for security globally. There is a high demand for persistent surveillance systems which can monitor round the clock all year round. As most VSS use visible-light cameras, the presence or absence of light hinders their ability to monitor persistently. Thermal cameras are viable substitutes because they function in poor lighting and at night. These cameras contain sensors which measure and create images from the thermal infrared energy emitted from objects in the scene (Negied, Hemayed & Fayek, 2015).

The amount of infrared detected determines how bright or how dark an object will appear in the final image. Emissivity is the ratio of infrared energy radiated from an object to that radiated from a perfect emitter under the same conditions. Given that 1 is the emissivity of a perfect emitter, also called a blackbody, pedestrians have a value of 0.98 (Fluke, 2020). Thermal imaging finds extensive application in pedestrian detection and tracking because pedestrians have high emissivity which creates a good enough contrast between them and the background. The challenge to detecting pedestrians in thermal images arises from the fact that, while pedestrians can emit infrared energy almost perfectly, only a fraction of the emissions are detected by the thermal camera. The amount of infrared energy reaching the thermal camera sensors depends on the prevailing weather conditions, the reflectivity of other objects in the scene and even the thermal camera itself. Thus, thermal images have lower resolution and lack the number of details present in visible-light images and the applications of thermal imaging are not as varied as those of visible imaging.

The motivation of this article is to propose a new method to detect pedestrians in thermal imaging acquired under different conditions. State-of-the-art algorithms for visible images usually do not perform with similar accuracy on thermal images and generally do not perform well across different datasets. This is because Image Analysis is slightly different when performed on visible and thermal images. Some of the characteristics of thermal images introduce additional challenges and/or nullify some steps in algorithms used for visible light images. For instance, there are immediate changes in appearance as illumination changes in visible images while appearance changes much slowly because detected radiation increases or decreases gradually in thermal images. Also, objects in thermal images do not cast shadows. Therefore, applying algorithms such as background subtraction to thermal images will not urgently need steps for scene update and shadow removal as will be the case for visible images. Furthermore, objects in visible images are commonly differentiated by their colour and displayed in the RGB (Red-Green-Blue) colour space while thermal information is commonly mapped to grayscale. It is important to remember that while RGB can be converted to grayscale, they still do not present the same information as thermal infrared images even if both images capture the same scene.

Furthermore, many of the methods put forward for pedestrian detection in thermal images require several steps grouped broadly into two: candidate generation and validation. Candidate generation involves extracting likely regions containing pedestrians. Candidate validation involves examining the extracted regions and discriminating between pedestrian and non-pedestrian. Errors tend to accumulate from each of these steps. Thus, different from other methods put forward, the proposed method is a single-model algorithm for pedestrian detection that eliminates the need for separate modules of candidate generation and validation. It integrates the appearance properties of the image with motion patterns such that all the fine-tuning and adjustment happens during energy formulation.

The contribution of this article is a novel Graph Cut energy function, referred to as motion-constrained energy (MCE), which repurposes binary segmentation for pedestrian detection in infrared images. Inspired by the semi-automatic framework of Boykov & Jolly (2001) that integrates the image region and boundary information into a single energy function, the proposed energy function incorporates an additional term to penalise pixels based on motion characteristics to accurately detect pedestrians in thermal images. The formulation in Boykov & Jolly (2001) presents an energy function E incorporating a region D(h) and boundary term S(h) shown as follows

(1) $$E(h)=\lambda \cdot D(h)+J(h)$$where

$$\begin{array}{cc}D(h)\hfill & =\sum _{y\in \mathcal{Y}}{D}_y(h_y)\hfill \\ J(h)\hfill & =\sum _{\{y,z\}\in \mathcal{N}}{J}_{y,z}(h_y,h_z)\cdot ϵ(h_y,h_z)\hfill \end{array}$$and

$$ϵ(a,b)=\{\begin{array}{cc}1,\hfill & \mathrm{i}\mathrm{f}a\ne b\hfill \\ 0,\hfill & \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\hfill \end{array}$$where $\mathcal{N}$ are unordered pairs of neighbouring pixels from a standard neighbourhood system e.g., 4-, 8- or 26- neighbourhood system and $\lambda$ is used to balance the contribution of the region and boundary term to the final segmentation result. D(h) measures how well pixels fit into the object or background models. J(h) is also called the smoothness term and it measures the similarity of intensity values between neighbouring pixels.

There are two areas where this formulation falls short in thermal images. Firstly, the low resolution and noisy nature of IR images mean that more importance will be given to the region term in many instances using this formulation. This means that a robust model for each class will have to be determined. As mentioned earlier, most models and approximated distributions in the literature do not generalise well across datasets, therefore, it is important to add another element to reduce over-dependence on the region term. Secondly, this formulation produces solutions where regions with similar intensity values as the pedestrians are included in the solution irrespective of their location.

The proposed energy function (MCE) incorporates motion constraints and is defined in Eq. (2) as

(2) $$E(h)=D(h)+J(h)+M(h)$$where

$$\begin{array}{cc}D(h)\hfill & =\sum _{y\in \mathbf{Y}}{D}_y(h_y)\cdot \delta (h_y)\hfill \\ J(h)\hfill & =\sum _{\{y,z\}\in \mathbf{N}}{J}_{y,z}(h_y,h_z)\cdot ϵ(h_y,h_z)\hfill \\ M(h)\hfill & =\sum _{y\in \mathbf{Y}}{M}_y(h_y)\cdot \beta (h_y)\hfill \end{array}$$and

$$\delta (h_y)=\{\begin{array}{cc}1,\hfill & \mathrm{i}\mathrm{f}y\in \mathcal{T}\wedge y\notin \mathcal{D}comb\hfill \\ 0,\hfill & \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\hfill \end{array}$$

$$ϵ(h_y,h_z)=\{\begin{array}{cc}1,\hfill & \mathrm{i}\mathrm{f}y\in \mathcal{T}\wedge z\in \mathcal{T}\wedge h_y\ne h_z\hfill \\ 0,\hfill & \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\hfill \end{array}$$

$$\beta (h_y)=\{\begin{array}{cc}1,\hfill & \mathrm{i}\mathrm{f}y\in \mathcal{D}comb\hfill \\ 0,\hfill & \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\hfill \end{array}$$where M(h) is the motion term, $\mathcal{T}$ is the set of pixels containing one or more motion pixels and $\mathcal{D}comb$ is the set of pixels with the highest energies from four directional difference images.

The impact of the proposed energy is expressed in Fig. 1. The result of using the energy of Boykov & Jolly (2001) produces topologically unconstrained solutions shown in Fig. 1B. This means that all pixels with the same properties as the object of interest will be included in the final result. However, MCE constrains the solution to only the object of interest as shown in Fig. 1C.

The rest of the article is organised as follows. Section 2 presents the related works. Section 3 presents the proposed framework. Section 4 provides the experimental results. Section 5 presents the conclusion and future work.

Related works

The task of detecting pedestrians is necessary for understanding and recognising human activity and behaviour in video surveillance footage. In thermal infrared images, this task is carried out in two major steps. The first step is to detect all regions likely to contain pedestrians. This is called Candidate Generation. The second step is to discriminate from among the extracted regions those belonging to the pedestrians. This is called Candidate Validation.

Many methods put forward for candidate generation in thermal infrared images depend on the contrast between the pedestrian and background. Thresholding methods have, therefore, found extensive use in this domain and are into two categories: parametric and non-parametric. Parametric thresholding algorithms obtain a threshold by parameter estimation while non-parametric thresholding algorithms optimise an objective function. Examples of Parametric methods can be found in the works of Soundrapandiyan & Mouli (2015), Wu et al. (2017), Manda et al. (2020) and Manda & Kim (2020). Soundrapandiyan & Mouli (2015) proposed a method comprising background subtraction, high-boost filtering to highlight the pedestrian pixels and suppress the background before applying local adaptive thresholding. Wu et al. (2017) produced formulas to compute two threshold limits from the image histogram. One limit eliminates cold regions while the other eliminates extremely bright regions. To obtain the binary image, the pixels with intensity values between the low and high threshold limit are set to the maximum pixel intensity of 255 while all other values are set to 0. Manda et al. (2020) made use of the raised cosine distribution function to determine a threshold value which separates the pedestrian from the background. Manda & Kim (2020) assumed the distribution of the image follows that of the transient response of the first-order linear circuit to determine a threshold value for pedestrian detection. Non-parametric approaches can be found in the works of Li et al. (2011) and Wu, Hou & Chen (2016). Li et al. (2011) found that previous non-parametric methods proposed which perform well on visible images do not perform satisfactorily on thermal images because the object and background distributions are similar and proposed a new criterion for thresholding infrared images where the distribution for both classes have similar standard deviations. Wu, Hou & Chen (2016) proposed a new threshold criterion for infrared images by assuming normal distributions for both the object and background histograms and comparing the hyper-entropies of both distributions. Thresholding methods produce excellent results when the approximated distribution of the image fits the dataset under consideration. However, this means that they can easily become too dataset–dependent. Also, in situations where the contrast is not pronounced, the pedestrians are not of uniform appearance, or polarity reversal occurs due to change of weather and the presence of artefacts such as halos, detection based on appearance alone suffers setbacks.

To reduce dependence on the contrast for pedestrian detection, candidate generation has been carried out by detecting moving regions. Background Subtraction and Optical flow-based methods are commonly used for detecting moving regions, but Background Subtraction is less computationally expensive (Choudhury et al., 2018). Generally, Background Subtraction is carried out by creating a model of the image background and comparing that model with each video frame. A similarity function is employed to determine which pixels are likely to belong to the object of interest. Background Subtraction by Frame differencing detects moving regions and is commonly used in tracking algorithms (Gawande, Hajari & Golhar, 2020). The presence of motion can be obtained from the absolute difference between consecutive image pairs. Jeon et al. (2015) created a background model using pixel difference image and combined edge information with the result of background subtraction to detect the pedestrians. Jeyabharathi & Dejey (2018) made use of frame differencing to extract likely pedestrian regions and reflectional symmetrical patterns to provide geometrical information for accurate background modelling. Motion is one feature that can cut across a wide range of infrared images.

Candidate validation have be performed using unsupervised and supervised approaches. Unsupervised methods make use of known or calculated physical properties of the pedestrians to discriminate between pedestrian and non-pedestrian. Younsi, Diaf & Siarry (2020) proposed a global similarity function that uses the sum of sub-similarity functions to discriminate between human moving objects and non-human moving objects. The drawback of unsupervised methods is that they also tend to be data–dependent. Supervised methods depend on feature extraction and training. Although recent efforts are moving towards the use of Convolutional Neural Networks (CNN) where feature representation is an inherent part of the training framework, feature representation is still a challenge because thermal images have low resolution and fewer details compared with visible images. Recent efforts such as those of Dai et al. (2019), Park et al. (2019), Chen & Shin (2020), Gao, Zhang & Li (2020), Huda et al. (2020), Krišto, Ivasic-Kos & Pobar (2020), Tumas, Nowosielski & Serackis (2020) and Haider, Shaukat & Mir (2021) rely on features for a (pedestrian/non-pedestrian) classifier. Park et al. (2019) proposed a CNN-based classifier with three input channels for fine-grained pedestrian detection. The input channels take in the original image, a Difference image from the previous frame and a background subtraction mask. In their results, they noted that training and testing needed to be carried out on similar datasets for best performance. Chen & Shin (2020) developed an attention-guided autoencoder network that includes a skip-connection block which combines features from the encoder-decoder modules to increase contextual information for robust and distinguishable features in infrared images with low SNR and resolution. YOLOv3 was used by Krišto, Ivasic-Kos & Pobar (2020) and Tumas, Nowosielski & Serackis (2020) for pedestrian detection under different weather conditions. Gao, Zhang & Li (2020) redesigned the visual geometry group (VGG-19) CNN to extract more features from infrared images for better detection results. The rationale for using these methods is that they perform well on visible images and achieve state-of-the-art results. However, their performance is lower on infrared images for two reasons. First, the models developed by Huda et al. (2020) for testing infrared images were trained on visible images. Second, different thermal cameras output different levels of detail. Therefore, even for models trained on infrared images such as done by Krišto, Ivasic-Kos & Pobar (2020) and Park et al. (2019), the performance of the trained model depends on how similar the test dataset is to the training dataset.

To the best of our knowledge, semi-automatic methods requiring human inputs have not yet found extensive application in the thermal domain. This work is inspired by the methods put forward by Boykov & Jolly (2001) and Viola, Jones & Snow (2003). Graph Cut is a powerful optimization method that guarantees an exact solution for binary labelling problems. Graph Cut’s effectiveness is shown in the framework of Boykov & Jolly (2001) which seamlessly combines edge and appearance information into its energy formulation to produce topologically unrestrained solutions Boykov & Funka-Lea (2006). This means that all pixels with the same properties are given the same label regardless of their location. Viola, Jones & Snow (2003) proposed a method which eliminates the need for separate modules for pedestrian detection and put forward a detector that integrates appearance and motion patterns such that all the fine-tuning and adjustment happens during training. Both methods are similar in that they seamlessly combine different attributes to accomplish one goal that would otherwise have required several steps. Also, both methods were tested and achieved state-of-art on visible images. However, the framework of Boykov & Jolly (2001) is semi-automatic while that of Viola, Jones & Snow (2003) is supervised.

Proposed method

This work considers a Graph-Cut based method for pedestrian detection which combines intensity (region and boundary) information with motion characteristics. The task of pedestrian detection is formulated as a binary labelling problem where the goal is to partition the image into two classes. Formally, the labelling problem is a function that maps observed data to labels. For our purposes, the observed data is the image and the labels are the classes. Let labels $\mathcal{Z}$ assigned to a pixel be given as $\mathcal{Z}=($‘ped’, ‘bkg’) where ‘ped’ refers to the ROI and ‘bkg’ refers to the rest of the scene. The labelling of $\mathcal{X}$ over $\mathcal{Z}$ is a function $h:\mathcal{X}\to \mathcal{Z}$. $h_x$ specifies the label assignments to x in $\mathcal{X}$ and is taken from $\mathcal{Z}$. To solve the binary labelling problem, Graph Cut performs efficient searches for the optimal labels among the possible set of labels. A graph is constructed over the image and a cut on the graph corresponds to the binary partitioning of the image. An energy function is used to represent the information in the image and the global minimum of the energy corresponds to the optimal partitioning. The overview of the proposed method is presented in Fig. 2.

Graph construction

The first step is to construct a graph $\mathcal{G}$ over an image. $\mathcal{G}=⟨\mathcal{V},\mathcal{E}⟩$ where $\mathcal{V}$ are the nodes of the graph and $\mathcal{E}$ are the edges. $\mathcal{V}$ correspond to the pixels of the image and include two additional nodes, source s and sink t, called terminals. The edges which connect the pixels to each other are referred to as N-links while the edges which the pixels to the two terminals are referred to as T-links. A neighbourhood system N determines the placement of edges between the nodes. A non-negative weight, discussed in “Weight assignment”, is assigned to each edge. An illustration of a graph constructed over an image is shown in Fig. 3

Weight assignment

The non-negative weights for each edge edge $e\in \mathcal{E}$ of the graph $\mathcal{G}$ are calculated from the region, boundary and motion terms of Eq. (2).

The region term, $D_y(h_y)$ reflects the extent to which each pixel fits into the image intensity model of “object” and “background”. These weights, $D_y($“object”) and $D_y($“background”), are computed as negative log-likelihoods as follows.

(3) $$\begin{array}{cc}{D}_y(‘‘\mathrm{o}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}")\hfill & =-\ln \mathrm{P}\mathrm{r}(Y_y|‘‘\mathbf{o}\mathbf{b}\mathbf{j}\mathbf{e}\mathbf{c}\mathbf{t}")\hfill \\ D_y(‘‘\mathrm{b}\mathrm{a}\mathrm{c}\mathrm{k}\mathrm{g}\mathrm{r}\mathrm{o}\mathrm{u}\mathrm{n}\mathrm{d}")\hfill & =-\ln \mathrm{P}\mathrm{r}(Y_y|‘‘\mathbf{b}\mathbf{a}\mathbf{c}\mathbf{k}\mathbf{g}\mathbf{r}\mathbf{o}\mathbf{u}\mathbf{n}\mathbf{d}")\hfill \end{array}$$

The intensity model for $D_y(h_y)$ is built using pixels, called seeds, which definitely belong to the “object” and “background”. These seeds are chosen interactively by the user.

The boundary term $J_{y,z}(h_y,h_z)$ assigns penalties to discontinuities between neighbouring pixels y and z. Therefore, the edge weights between pixels with dissimilar pixel intensity values will be higher and vice versa. These weights are calculated as follows

(4) $$S_{y,z}=\exp \left({\displaystyle \frac{{(y-z)}^2}{2{\sigma }^2}}\right)$$

In the above equation, $\sigma$ has been calculated as the variance of the video frame under consideration.

The motion term $M_y(h_y)$ computes the cost of labelling a pixel as “object” or “background” as determined by the motion constraint $\mathcal{D}comb$. $\mathcal{D}comb$ provides an estimate of the location of each pedestrian in the image obtained by thresholding and combining four images obtained by frame differencing. The direction of motion can be obtained from the absolute differences $\mathcal{D}{L}_f$, $\mathcal{D}{R}_f$, $\mathcal{D}{U}_f$ and $\mathcal{D}{D}_f$ between consecutive image pairs $I_f$ and shifted versions of $I_{f+1}$ to the left, to the right, up and down respectively. The difference image computations are given as follows

(5) $$\begin{array}{cc}{\mathcal{D}}_f\hfill & =|I_f-I_{f+1}|\hfill \\ \mathcal{D}{L}_f\hfill & =|I_f-I_{f+1}\leftarrow |\hfill \\ \mathcal{D}{R}_f\hfill & =|I_f-I_{f+1}\to |\hfill \\ \mathcal{D}{U}_f\hfill & =|I_f-I_{f+1}\uparrow |\hfill \\ \mathcal{D}{D}_f\hfill & =|I_f-I_{f+1}\downarrow |\hfill \end{array}$$

Figure 4 shows how the shifted difference images provide information about the direction of motion. In our experiments, we found that the energy of the image was highest when the image was shifted in the direction of motion and the least when shifted in the opposite direction. Also, because the surveillance footage is taken from different angles and there are usually several pedestrians going in different directions, we found that the energy for each subject is higher in at least two directions, that is, either in the ↑ or ↓ direction and either in $\leftarrow$ or $\to$ direction.

$\mathcal{D}comb$ is, therefore, created by combining the pixels with the highest energies from each directional difference image and is defined as

(6) $$\mathcal{D}comb=\{(\mathcal{D}{L}_f>\mathrm{T}\mathrm{h})\cup (\mathcal{D}{R}_f>\mathrm{T}\mathrm{h})\cup (\mathcal{D}{U}_f>\mathrm{T}\mathrm{h})\cup (\mathcal{D}{D}_f>\mathrm{T}\mathrm{h})\}$$where Th is used to extract the highest energies from each directional difference image.

$M_y(h_y)$ is, thus, defined as follows

(7) $$M_y(‘‘\mathrm{o}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}")=\{\begin{array}{c}\eta ,\mathrm{i}\mathrm{f}y\in \mathcal{D}comb\hfill \\ 0,\mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\hfill \end{array}$$

(8) $$M_y(‘‘\mathrm{b}\mathrm{a}\mathrm{c}\mathrm{k}\mathrm{g}\mathrm{r}\mathrm{o}\mathrm{u}\mathrm{n}\mathrm{d}")=\{\begin{array}{c}\eta ,\mathrm{i}\mathrm{f}y\notin \mathcal{D}comb\hfill \\ 0,\mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\hfill \end{array}$$where $\eta$ is an arbitrarily large number to ensure that the object or background label is assigned to a pixel if the stated condition for each class of assignment is satisfied.

Table 1 provides the weights for the graph edges. As discussed in “Graph construction”, the elements of $\mathcal{V}$ for graph $\mathcal{G}$ are the image pixels. Each node, corresponding to pixel y, is connected to the source s and sink t terminals using edges {y, s} and {y, t} called T-links. Also, each node is connected to other nodes in its neighbourhood. A four-neighbourhood system, for example, would mean that a pixel was connected to its four neighbours above, below, to the left and the right of it. The edges which connect a node to its neighbours {y, z} are called N-links. A higher weight on the T-link connecting a node to either s or t implies a higher likelihood of a pixel to be labelled as “object” or “background” respectively. Likewise, a higher weight on the N-link between vertices implies a greater dissimilarity between pixels. It should be noted that $D_y(h_y)$ and $M_y(h_y)$ are unary terms acting on each pixel to compute the weight on the T-links of the graph while $J_{y,z}(h_y,h_z)$ is a binary term acting on pixel pairs y and z in a specified neighbourhood N to compute the weight on the N-links.

Table 1:

Edge weights of the graph constructed from the image.

Edge	Weight	For
{y, z}	$J_{y,z}$	$\{y,z\}\in \mathbf{N},y\in \mathcal{T}\wedge z\in \mathcal{T}$
{y, s}	$D_y(‘‘\mathrm{p}\mathrm{e}\mathrm{d}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{r}\mathrm{i}\mathrm{a}\mathrm{n}")$	$y\in \mathbf{Y},y\in \mathcal{T}$
	$M_y(‘‘\mathrm{p}\mathrm{e}\mathrm{d}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{r}\mathrm{i}\mathrm{a}\mathrm{n}")$	$y\in \mathbf{Y},y\in \mathcal{D}comb$
	0	$y\in \mathbf{Y},y\notin \mathcal{D}comb\wedge y\notin \mathcal{T}$
{y, t}	$D_y(‘‘\mathrm{b}\mathrm{a}\mathrm{c}\mathrm{k}\mathrm{g}\mathrm{r}\mathrm{o}\mathrm{u}\mathrm{n}\mathrm{d}")$	$y\in \mathbf{Y},y\notin \mathcal{T}$
	$M_y(‘‘\mathrm{b}\mathrm{a}\mathrm{c}\mathrm{k}\mathrm{g}\mathrm{r}\mathrm{o}\mathrm{u}\mathrm{n}\mathrm{d}")$	$y\in \mathbf{Y},y\notin \mathcal{D}comb$
	0	$y\in \mathbf{Y},y\notin \mathcal{D}comb\wedge y\notin \mathcal{T}$

DOI: 10.7717/peerj-cs.1064/table-1

To obtain $\mathcal{T}$, the image is divided into non-overlapping equal-sized detection windows such that only windows which have one or more pixels from $\mathcal{D}comb$ are considered by D(h) and S(h).

Energy minimization

Following the graph construction and weight assignment, the energy is minimized using the Boykov-Kolmogorov minimum cut/maximum flow algorithm (Boykov & Kolmogorov, 2004). The aim of this algorithm is to find the cut $\mathcal{C}$ that partitions a two-terminal graph into two disjoint sets S and T such that s is in S and t is in T. The optimization problem, to find the minimum among all possible cuts, is solved by finding the maximum flow moving from the source s to the sink t. The cost of the cut $\mathcal{C}=\{S,T\}$ is the sum of the weights on the edges (y, z) where y ∈ S and z ∈ T. The final labelling on the original image is produced by the minimum cut separating the two terminals shown in Fig. 5.

Experimental results

Comparison of MCE with energy function of Boykov & Jolly (2001)

To show the improvements of the proposed method by the inclusion of an additional term in the energy function, the detections results of the proposed method (MCE) are compared with those produced by the energy of Boykov & Jolly (2001) (GC). The results of the comparison are presented in Table 2 and Fig. 6. The visual comparison of GC and MCE is shown in Figs. 7–10.

Table 2:

Quantitative results for GC and MCE comparison.

	Method	LITIV	LTIR	Terravic	OSU
Recall	GC	0.9859	0.9562	0.9842	0.9933
	MCE	0.9992	0.9771	0.9902	0.9995
Precision	GC	0.9785	0.9504	0.9780	0.9801
	MCE	0.9969	0.9805	0.9889	0.9907

DOI: 10.7717/peerj-cs.1064/table-2

The performance of both GC and MCE is lowest on the LTIR database. This could be because LTIR has the most varied scenes of all the datasets. The images were either too bright or too dark and there were cases of slight camera motion and reversed polarity. Conversely, it shows the greatest improvement in performance when MCE is used.

LITIV database has the most uniform appearance but is the most varied in perspective; images were captured from different angles from the side view to the top view. Most of the images were very dark and the contrast was poor except in images taken from the top view. Significant improvement in performance is also observed when MCE is used.

The Terravic database had the best contrast, but the pedestrians were not always moving and, compared to the other databases, it took a long time for the pedestrians to move significantly. The impact of this slow or lack of movement is in deciding the interval between consecutive frames. Ideally, the next immediate frame should be used but this might depend on the footage.

The OSU database is the oldest and most extensively used because it was created specifically for evaluating pedestrian detection algorithms. The database contains details about the weather condition and comprehensive ground truth. The images were taken over different days and under different weather conditions but from the same scene. As mentioned in the introduction, temporal changes in appearance do not occur in thermal images unless there is a drastic change in weather conditions, and these changes occur much slowly as detected radiation increases or decreases gradually. Table 3 shows the weather conditions for each video sequence in the database, the total number of pedestrians in the database and the true positive (TP) and false positive (FP) detection results using the proposed algorithm. It can be concluded that the proposed method is quite robust to changes in weather.

Table 3:

Table showing the weather conditions, true positive (TP) and false positive (FP) detection results for the OSU thermal database.

Video	Atmospheric phenomenon	Time of day	Temp (°C)	Total pedestrians	TP	FP
1	Light rain	Afternoon	13	91	90	0
2	Partly cloudy	Morning	5	100	98	2
3	Partly cloudy	Morning	21	101	98	3
4	Fair	Morning	9	109	109	0
5	Partly cloudy	Morning	25	101	99	1
6	Mostly cloudy	Morning	21	97	95	0
7	Light rain	Afternoon	36	94	92	1
8	Light rain	Afternoon	30	99	95	0
9	Haze	Afternoon	18	95	95	1
10	Haze	Afternoon	23	97	90	1
				984	961	9

DOI: 10.7717/peerj-cs.1064/table-3

Conclusion

In this article, a motion-constrained Graph Cut framework for pedestrian detection in thermal infrared videos has been presented which integrates appearance information with motion characteristics in a single model. The proposed method has been compared with the framework of Boykov & Jolly (2001) to show the advantages of including an additional constraint and the performance of the detection framework. In addition, the method has been tested on four publicly available datasets and with different methods in the literature which make use of the same datasets to showcase the robustness of the framework. As the process of selecting seeds significantly increases the execution time, future work will involve optimising the algorithm to require as little human input as possible.

Information on images used in the figures

Supplemental Information