Deep learning—a first meta-survey of selected reviews across scientific disciplines, their commonalities, challenges and research impact

General computer vision

A review about selected deep learning methods that have been used in the general area of computer vision is presented by Voulodimos et al. (2018). They introduce convolutional neural networks, deep Boltzmann machines, deep belief networks and stacked denoising autoencoders, alongside with their history and application tasks they have been used for. They conclude their review with an outlook on how future deep learning-based methods can be designed for tasks in computer vision and the challenges that arise in doing so.

Another general review on deep learning in computer vision comes from Guo et al. (2016). They also start their review with a comprehensive overview of different deep learning architectures, like CNNs, covering neural network layer types, like convolutional layers and pooling layers, but also training strategies, like dropout, DropConnect and pre-training/fine-tuning. For the deep learning-based architectures, they discuss characteristics, advantages and disadvantages. They conclude their review with current trends and challenges, and provide future directions for a theoretical understanding, human-level vision, training with limited data, time complexity and more powerful models in deep learning.

Object detection

Object detection is one of the most basic, but as well challenging problems in computer vision. Object detection deals with the localization of objects from predefined categories, like cats, dogs, etc., in natural images. Liu et al. (2020) outline more than 300 research contributions in their survey about object detection. In doing so, they cover many general aspects in the field of object detection, which includes, for example, detection frameworks, but also object feature representation and object proposal generation. On top, they address context modelling, training strategies, and, eventually, evaluation metrics for object detection.

A second review paper in the area of computer vision and object detection is from Zhao et al. (2019), which first provides a short introduction on convolutional neural networks, deep learning, and their history. They focus on typical generic object detection architectures and briefly survey various particular tasks. These cover salient object detection, but also pedestrian and face detection. In addition, an experimental analysis is provided, which allows the comparison of different approaches and hence, draw constructive conclusions amongst them.

Finally, Jiao et al. (2019) review existing approaches of general detection models and additionally introduce a common benchmark dataset for them. The authors also outline a comprehensive and systematic overview of numerous approaches for object detection, which include one-stage and multi-stage object detectors. Furthermore, they list and analyse established, as well as new, applications of object detection and its most representative branches.

Image segmentation

Image segmentation is usually the first step in different computer vision applications, like scene understanding, video surveillance, and robotic perception. Further applications can be medical image analysis, augmented reality and image compression, to name a few. Garcia-Garcia et al. (2018) provide a survey of deep learning approaches for semantic segmentation that can be translated and applied to numerous areas. In doing so, datasets, but also challenges, are outlined to guide researchers in the decision which method is most suitable for their needs and aims. Subsequently, the existing approaches and methods, like CNNs, are surveyed in the contribution. Additionally, they review common loss functions and error metrics and provide quantitative results for the introduced methods, but also the datasets that have been used for an evaluation.

Minaee et al. (2020) present a comprehensive review that covers a wide spectrum of contributions in the area of semantic and instance-level segmentation. This includes fully convolutional pixel-labelling networks and encoder-decoder architectures. Moreover, recurrent networks, multi-scale and pyramid-based methods. Further, visual attention and generative models in an adversarial setting. They studied the strengths and challenges of the proposed deep learning models, but also their similarity. Finally, the authors investigated the most commonly applied datasets and present performance results for them.

Face recognition

Face recognition is a significant biometric method for identity authentication that has been applied to numerous application areas. This includes public security, daily life, military, but also finance. Masi et al. (2018) introduce the main benefits of face recognition with deep learning, also called deep face recognition. They focus on identification and verification by learning representations of the face. The review gives a structured overview of works from the past years, covering the principals and state-of-the-art in face recognition methods.

Li & Deng (2020) provide a survey on facial expression recognition with deep learning, including datasets and algorithms. They present datasets that are available and have been commonly applied in previous works. Further, they outline commonly recognised data selection and evaluation concepts that have been used for these datasets. Next, they outline the general pipeline and workflow for a deep facial expression recognition approach, covering the corresponding background knowledge, and finally propose, for each stage, a feasible implementation.

Wang & Deng (2018) also provide a survey of the latest trends on deep facial expression detection, including the design of algorithms, but also possible applications, protocols and databases. They outline various network architectures and loss functions that have been introduced in the general field of the deep facial expression. They categorized the face processing approaches into two different classes: “one-to-many augmentation” and “many-to-one normalization”. They also give a summarized overview of common databases and compare them concerning model training and model evaluation. Finally, they explored further scenarios for deep facial expression, like cross-factor, heterogeneous, industrial and multiple-media.

Action and motion recognition

Understanding human actions and motions in visual data, like surveillance videos, is closely connected to research fields like object recognition, semantic segmentation, human dynamics and domain adaptation. Herath, Harandi & Porikli (2017) review notable steps that have been taken towards recognizing human actions. Therefore, they start with a discussion of first approaches that applied handcrafted representations. Subsequently, they review deep learning-based approaches suggested in this field.

Wang et al. (2018b) give an outline of latest trends and improvements of motion recognition in RGB-D images. They categorized the surveyed approaches into four groups. The groups are based on the particular modality used for recognition, and can be RGB-, skeleton, depth-, or RGB+D-based. Finally, they discuss the advantages and limitations, with a focus on approaches that encode spatial-temporal-structural information, which is inherent in video sequences.

Biometric recognition

Biometric recognition, or biometrics, studies the identification of people utilizing their unique phenotypical characteristics, like fingerprints or the iris, for applications ranging from cell phone authentication to airport security systems. Sundararajan & Woodard (2018) review one hundred distinct methods that study recognizing individuals with deep learning applying different biometric modalities. They conclude that the majority of research in biometrics based on deep learning has been conducted around face recognition and speaker recognition so far.

Minaee et al. (2019) conduct a survey of more than 120 works on biometric recognition, including face, fingerprint, iris, palm print, ear, voice, signature, and gait recognition. For each biometric recognition task, they present the available datasets used in the literature and their characteristics and outline the performance on popular public benchmarks.

Image super-resolution

Image super-resolution is a basic task in image processing that has seen a rise in popularity with the advent of deep learning. Image super-resolution methods and algorithms are used to improve the resolution of (low-resolution) images and videos. Wang, Chen & Hoi (2020) give a comprehensive overview on latest trends and advances in the field of image super-resolution focusing on deep learning methods. They divide the existing papers on image super-resolution methods into three main categories, namely supervised image super-resolution, unsupervised image super-resolution and finally, domain-specific image super-resolution. Moreover, they cover other topics in the field of image super-resolution, like public accessible benchmark data collections and metrics for a performance evaluation.

Image captioning

Image captioning refers to the generation of a description for an image. Therefore, their primary objects, but also their attributes and their relationships to each other within the image need to be recognized. Furthermore, image captioning must produce sentences that are syntactically and semantically correct. Hossain et al. (2019) introduce a broad survey of works for image captioning based on deep learning. They analyse their main strengths, performances, but also their limitations. In addition, they explore the datasets and the evaluation metrics that have been used for automatic image captioning with deep learning.

Data augmentation

Data augmentation can be used for the expansion of (limited) datasets to obtain larger training and evaluation sets. Shorten & Khoshgoftaar (2019) review image augmentation algorithms that cover geometric transformations, but also colour space augmentations and feature space augmentation. Further, techniques like kernel filters, mixing images, random erasing, neural style transfer and meta-learning. They also cover generative adversarial network-based augmentation methods. In addition, they explore and study further characteristics in the area of data augmentation, like test-time augmentation, final size of the dataset, the impact of the resolution, but also curriculum learning. Finally, they give an overview of available approaches for meta-level decisions for implementing data augmentation.

Generative adversarial networks

Generative adversarial networks or GANs belong to the field of generative models in machine learning. GANs have experienced an in-depth exploration during the last few years with the most significant impact in the field of computer vision. Wang, She & Ward (2019) survey three real-world problems that have been approached with GANs: the generation of high-quality images, diversity of image generation, and stable training. They give a detailed overview of the current state-of-the-art in generative adversarial networks. Furthermore, they structure their review using a specific taxonomy, which they have adopted based on variations in generative adversarial network-based architectures and loss functions.

Going deeper: common architectures, methods, evaluations, pros, cons, challenges and future directions in computer vision

Table 3 gives more details about the presented methods, pros, cons, evaluations and challenges and future directions of the surveys in the category computer vision. The two surveys about general computer vision from Voulodimos et al. (2018) and Guo et al. (2016) introduce mainly CNNs, Boltzmann and autoencoder architectures. The authors list several pros and cons, regarding which networks can learn features automatically, generalize well, are computationally demanding during training, can be trained in real-time and do not perform well on small training sets. A challenge is, that the underlying theory of the models is not well understood, which leads to the problem of selecting an optimal or effective architecture or algorithm for a given task and that there is no clear understanding of what kind of architectures should perform better than other ones. Further challenges and future directions are the training with limited data, reducing the time complexity, the development of more powerful models and a better understanding in evolving and optimizing CNN architectures.

The three surveys about object detection with deep learning (Liu et al., 2020; Zhao et al., 2019; Jiao et al., 2019), all outline the main object detectors in deep learning, like R-CNN, Fast R-CNN, Faster R-CNN, Mask R-CNN, etc. in a temporal sequence. In doing so, showing the improvement from one detector to the other over time, which are for example a more arcuate object detection, improved detection speed and quality, variable image sizes, memory consumption, small-size object detection and localization, and end-to-end training possibilities. Future common challenges are even better and more efficient detection frameworks (e.g. the research field of generic object detection is still far from complete), unsupervised and weakly supervised learning, network optimization and combination, video object detection, 3D object detection and multi-domain object detection.

The two surveys about image segmentation (Garcia-Garcia et al., 2018; Minaee et al., 2020) review numerous deep learning architectures and variants, like CNN, AlexNet, VGG-16, GoogLeNet, ResNet, ReNet and further custom architectures. Garcia-Garcia et al. (2018) list the pros and cons for accuracy, efficiency, training, instance sequences, multi-modal and 3D. Minaee et al. (2020) consider the quantitative accuracy, speed (inference time), storage requirements (memory footprint) and further present a metrics evaluation for pixel accuracy/mean pixel accuracy (MPA), intersection over union (IoU)/Jaccard Index and Dice coefficient. Both surveys see future challenges in more memory efficient models, real-time segmentations and diverse datasets, like 3D datasets, sequence datasets, and in general more challenging datasets. Both also point out point cloud segmentation. Further challenges are for example execution time, accuracy, reproducibility and weakly-supervised and unsupervised learning.

The three surveys in face recognition (Masi et al., 2018; Li & Deng, 2020; Wang & Deng, 2018) have all a strong focus on datasets and loss functions. A reason for that is that there exist many large public facial databases, which make comparable evaluations among the methods feasible. In this regard, Li & Deng (2020) focus on databases and evaluations specifically for Facial Expression Recognition (FER) in their survey. However, the surveys see the common challenges in boarded databases, handling biases in large datasets, like ethnicity, gender, age or other factors, having more non-saturated benchmark datasets and ubiquitous face recognition across applications and scenes. Further open challenges are privacy and security issues, pursuing extreme accuracy and efficiency, understanding deep face recognition and fusion issues. Finally, Masi et al. (2018) state reducing intra-class variance and increasing the margin between classes while training as two main challenges, and that video processing and clustering are currently the next frontiers for face recognition.

Table 3 presents two surveys in Action/motion recognition (Herath, Harandi & Porikli, 2017; Wang et al., 2018b), the latter one specific on RGB-D-based human motion recognition with deep learning. Both surveys present numerous deep learning-based methods, like VGG, Decaf, RNN, LSTM, LRCN, autoencoder, adversarial models and others, and present accuracy evaluations on numerous public available benchmark datasets. Common challenges of the surveys point out the need for practical (carefully engineered) systems that can handle more realistic activities and (outdoor) real-life scenarios. Further challenges are the development of more generic architectures that can handle also multiple modalities, have a better performance and can better train video data. Moreover, the surveys point out a deeper understanding in action recognition, better results on smaller training data, but also better results on large complex datasets, solutions for online motion recognition and prediction, and zero/one-shot learning.

In the sub-category biometric recognition, we present two surveys (Sundararajan & Woodard, 2018; Minaee et al., 2019). Both survey publications are dominated by CNN architectures and variants. Sundararajan & Woodard (2018) divide their survey by biometric modalities, namely physiological biometrics (face, fingerprint, palmprint, and iris) and behavioral biometrics (voice, signature, gait, and keystroke), and present evaluation results on public and private datasets. Minaee et al. (2019) arrange their survey by the following biometric recognitions: fingerprint, iris, palmprint, ear, voice, signature and gait. However, Sundararajan & Woodard (2018) conclude, that deep learning in biometrics has so far been very little explored beyond face and speaker recognition. Among others, Sundararajan & Woodard (2018) point out that future challenges are computing resources and training speed-up, also in regards to large-scale identification. Furthermore, there is a need for models that are more robust, the modeling of biometric aging and the fusion of multiple modalities. Minaee et al. (2019) see future directions, for example, in more challenging datasets, interpretable deep learning models, few shot and self-supervised learning, real-time and memory efficient models, and the dealing with security and privacy issues.

Wang, Chen & Hoi (2020) present in their survey specific deep learning architecture and methods, like SRCNN, DRCN, FSRCNN, ESPCN, LapSRN, DRRN, SRResNe, that have been developed for the field of super-resolution. For evaluation, they show the performance (PSNR) of various models like FSRCNN, LapSR, SRCNN, CARN-M, FALSR-B, FALSR-C, BTSRN, CARN, on four benchmark datasets. They see one future challenge in more efficient architectures and better learning strategies in regards to loss functions and normalization, but also unsupervised super-resolution. Further, they state that the evaluation metrics need more exploration, specifically they want to see more accurate metrics and the development of blind IQA methods in the field of super-resolution. Moreover, they request for combining low- and high-level information, but also combining local and global information. Final challenges concern real-world scenarios dealing with various degradation and domain-specific applications.

A survey about image captioning with deep learning is presented by Hossain et al. (2019). In doing so, they survey image encoder methods, like AlexNet, VGGNet, GoogLeNet, ResNet, Inception-V3, and language models, like LBL, LSTM, SC-NLM, RNN, DTR, MELM, Language CNN, in the field of image captioning. They also list the used evaluation metrics applied in the survey publications for most commonly and public benchmark datasets in that field, like +BLEU, PPLX, R@K, mrank, METEOR, CIDEr, PPLX, AP, IoU, PPL, Human Evaluation, E-NGAN, E-GAN, SPICE, SPIDEr. The authors see future challenges in detecting prominent objects and attributes, and their relationships, and generating accurate and multiple captions. They also point out that open-domain datasets are needed. Future directions are the generation of high-quality captions, adding external knowledge and the generation of attractive image captions. Finally, they state that supervised learning needs a large amount of labeled training data, so the focus will be on unsupervised learning and reinforcement learning in image captioning.

Shorten & Khoshgoftaar (2019) present a survey on data augmentation for deep learning. Thereby, dividing deep Learning-based methods in adversarial training, neural style transfer and GAN data augmentation. They introduce several deep learning architectures in the field of data augmentation, like CNN, LeNet-5, AlexNet, GAN, NAS, DCGAN, CycleGAN, Progressively-Growing GAN, WGAN and variants. Evaluation of the data augmentation methods are outlined for a wide range of tasks and datasets with the task specific metrics and scores, like (validation) accuracy, visual Turing test, ARS, C10, C10+, C100, C100+, SVHN, rank, class, etc. Future works are seen in establishing a taxonomy of augmentation techniques. Further, improving the quality of GAN samples, combing meta-learning and data augmentation, exploring the relationships between data augmentation and classifier architectures. Moreover, extending the data augmentation principles to other data types, identify the impact on video data and a translation to text, bioinformatics, and tabular records. Finally, the authors call for a practical integration of data augmentation into deep learning software tools and common data augmentation APIs.

In their survey, Wang, She & Ward (2019) give a comprehensive overview of generative adversarial networks, like FCGAN, SGAN, BiGAN, CGAN, InfoGAN, AC-GAN, LAPGAN, DCGAN, BEGAN, PROGAN, SAGAN, BigGAN, rGANs, YLG, AutoGAN, MSG-GAN, Loss-Variant GANs, WGAN, WGAN-GP, LSGAN, f-GAN, UGAN, LS-GAN, MRGAN, Geometric GAN, RGAN, SN-GAN, RealnessGAN, Sphere GAN, SS-GAN, variants. They also provide a compact overview of the pros and cons of different GAN-based methods, outlining which methods can, for example, handle the vanishing gradient problem, generate low quality images, have not been tested or investigated enough, and which are easy or hard to implement. In addition, they present several evaluation tables and images, about time performance (DCGAN, BEGAN, PROGAN, RFACE), accuracy (DCGAN, BEGAN, PROGAN), scoring (DCGAN, BEGAN, PROGAN) and further performances (FCGAN, BEGAN, PROGAN, LSGAN, DCGAN, WGAN-GP, SN-GAN, Geometric GAN, RGAN, AC-GAN, BigGAN, RealnessGAN, MSG-GAN, SS-GAN, YLG, Sphere GAN). The authors conclude, that GAN-based research is still limited in other non-computer vision areas and that GANs are hard to apply to the natural language application field. However, GANs could be used in NLP for generating comments to live streaming. They also state that GANs could have a significant impact on neuroscience by tackling privacy issues. On the other hand, they outline society and safety concerns when tampered videos are generated with GANs. Hence, a future challenge is to develop detectors for AI-generated images. Future directions are solving the lack of efficient evaluation metrics in some areas, GPU memory problems for large batched images and video application are in general still under-researched.

To conclude this section, most of the surveys reviewed in this category have also a similar structure: starting with a general introduction of machine/deep learning techniques followed by a discussion of the reviewed publications categorized using different strategies, among which task- or application-based categorization is the most widely used. The works usually conclude by stating challenges and future directions of deep learning in their fields from a high-level point of view. The future challenges are very broad depending on the field and application. However, despite the successes and advances of deep learning in the field of computer vision, there is still research needed to make them more reliable for real world applications.

Natural language processing

In general, natural language processing is a theory-inspired variety of computational methods and algorithms for the automatic studying of the human language that can be used, for example, for voice commands in all kind of applications. Young et al. (2018) survey several important deep learning-based approaches that have been utilized for various tasks of natural language processing. They provide a walk-through of their evolution during the last years and overview, compare and contrast the numerous models. Finally, they provide an in-depth overview of the past, the present and future role of deep learning in natural language processing.

Language generation and conversation

A task of natural language processing is the generation (NLG) of text or speech from a non-linguistic input. This can be the generation of new texts from (often human-written) existing ones from one language to another language by machine translation, or a summarization and fusion of texts with the goal to make them more concise. Gatt & Krahmer (2018) provide an overview of the published research in common NLG tasks and the corresponding neural architectures. They explore common research areas between NLG and general artificial intelligence and outline the particular challenges in evaluation in the field.

Santhanam & Shaikh (2019) provide an overview of classical methods, statistical methods, and methods that utilize deep neural networks and review publications on open domain dialogue systems. Thereby, they recognize three further research directions for the development of more effective dialogue systems. These are incorporating conversation context and world knowledge, but also including larger contexts. Further, they highlight the problem of generic or dull responses and mention ways to improve the quality of NLG systems, for example by incorporating personae or personality information. Gao, Galley & Li (2018) present a tutorial that surveys neural approaches to conversational artificial intelligence and arrange conversational systems into three different categories. These are agents for question-answering, task-oriented dialogues and social bots. For each of these categories, they introduce the state-of-the-art overview of neural methods, but also make connections between these methods and classical methods. Finally, they outline the current progress in this field and present the remaining challenges by applying certain models and systems in case studies.

Chen et al. (2017) propose a survey overview of the current progress in the area of dialogue systems from different angles and explore future research areas and topics. Established dialogue systems are grouped by the authors into two model categories: Task-oriented and non-task-oriented. Then, they outline how deep learning-based approaches can support these with specific algorithms. Finally, they highlight several further research areas, which can support and advance the field of dialogue-systems.

Named entity recognition

Named entity recognition deals with the identification of named entities (for example real-world objects, like persons or locations) and furthermore, their classification into specific categories. It functions as foundation for natural language approaches, like the summarization of text, answering questions or machine translation. Li et al. (2020) provide a large panoramic of established deep learning-based approaches for named entity recognition. They introduce named entity recognition resources, including tagged named entity recognition corpora, and further off-the-shelf named entity recognition applications. Available works are systematically arranged, depending on a taxonomy along three axes, namely the distributed representations for the input, the context encoder, and the tag decoder. Finally, they review the main approaches for deep learning-based techniques that have recently been applied and outline the particular challenges for named entity recognition systems in their contribution.

Yadav & Bethard (2019) present a broad overview of deep neural network approaches for the field of named entity recognition. They compare them with existing methods for named entity recognition that use feature engineering and further supervised or semi-supervised learning methods. Finally, they outline the advantages that have been gained by neural networks and depict how including specific works on feature-based named entity recognition systems can yield further improvements.

Sentiment analysis

Sentiment analysis is the (automatic) recognition and analysis of people’s opinions, sentiments, emotions and appraisals. This can be used in data mining applications for the exploration of this subjective information source and, therefore, opinions of specific entities, like products, events or services, but also topics, individuals or organizations. In their contribution, Zhang, Wang & Liu (2018) start with an introduction of deep learning, then they provide an extensive review of its recent sentiment analysis applications. They divide the area of recent sentiment analysis in different sub-categories, like a sentiment classification on a document-level, sentence-level and aspect-level, and a further opinion expression extraction.

Do et al. (2019) wrote a comprehensive and context-based overview of deep learning methods that have been used in aspect-based sentiment analysis. For their review contribution, they categorised and summarised 40 approaches by their main deep learning architecture and their specific classification tasks. The review works consisting of general, but also adaptions of common convolutional neural networks, long-short term memory methods, and gated recurrent units.

Text summarization

Text summarization targets the summarization of (long) documents into shorter ones while, at the same time, keeping the key meaning and information of the original text documents. Shi et al. (2018) give a broad technical literature review on various seq2seq (sequence-to-sequence learning) methods for the summarization of text from the angle of training strategies, network structures and summary generation methods. In addition to the review, they implemented an open-source library, called the Neural Abstractive Text Summarizer (NATS) toolkit, which can be used for abstractive text summarization. Further, they conducted a set of experiments on the common CNN/Daily Mail dataset to evaluate the capabilities and results of diverse neural network components. They conclude by benchmarking two implemented NATS methods on the Newsroom and Bytecup datasets.

Answer selection

The aim of (automatic) answer selection is identifying correct (and incorrect) answers. For example, for a given question and a number of possible candidate answers, answer selection identifies which of the candidates answered the question correctly (and, concurrently, which of the candidates do not). In their survey, Lai, Bui & Li (2018) outline an extensive, systematic analysis of numerous deep learning-based approaches for answer selection along two main dimensions: (1) neural network architectures, such as attentive architecture, siamese architecture and compare-aggregate architecture, and (2) learning strategies, such as listwise, pairwise, and pointwise. Moreover, they examined the most common datasets for answer selection and their evaluation metrics, and present various possible research directions for the future in this field.

Word embedding

Word embedding, or short embedding, covers feature learning and modelling strategies for representing words or phrases as numbers or vectors. In their work, Zhang et al. (2016) review the state-of-the-art of neural information retrieval research, with a focus on the usage of queries and document representations that have been learned, like neural embeddings. In doing so, they outline the achievements in the field of neural information retrieval, but also point out limitations for a broader usage, and conclude by proposing possible and favourable future research directions.

The survey contribution of Almeida & Xexéo (2019) depicts and delineates the main recent strategies in the field of word embedding. The authors introduce two main categories for word embeddings and the corresponding publications: prediction-based models and count-based models.

Financial forecasting

Financial forecasting tries to (automatically) predict financial market trends, like stock market predictions. Financial forecasting can, for example, be based on financial statements and reports, but also news articles and press releases, with the goal to keep a competitive business advantage. Xing, Cambria & Welsch (2018) present in their work the scope of natural language-based financial forecasting (NLFF) research by arranging and organizing the methods and approaches from the reviewed works. Their review publication targets on providing a greater knowledge of the advancements and NLFF hotspots.

Going deeper: architectures, evaluations, pros, cons, challenges and future directions in language processing

Table 5 provides more details about the presented methods, pros, cons, evaluations and challenges and future directions of the surveys in the category language processing.

General language processing is covered by Young et al. (2018), where at first word and character embeddings are described, highlighting the challenge of out of vocabulary (OOV) words, which are partly addressed by character embeddings. Still, for embeddings it has been found to be limited in terms of perceptual understanding, which could motivate grounded learning. For CNNs the limitation of capturing long distance dependencies has been noted, which also motivated the development of dynamic convolutional neural networks and Recurrent Neural Networks. The survey also mentions the role of the pooling layer, where in the case of max pooling an information loss can be observed, calling for modified techniques, such as dynamic multi-pooling CNNs. The fixed size representation of traditional Sequence-to-Sequence models can be overcome by means of the Attention Mechanism. Memory-augmented networks capture a similar intuition to model the complex dependencies and relationships in text. Finally, the survey provides a good overview of the performance of a range of approaches on various tasks, including POS tagging, Semantic Role Labelling, and Sentiment Classification. In general, recurrent network architectures, like LSTMs, are ranked among the highest across the tasks.

For machine translation the generic encoder-decoder architecture has been mentioned to be well suited (Gatt & Krahmer, 2018). In particular, Sequence-to-Sequence models are considered to provide good performance on this task. Another important point raised is in relation to suitability in practical settings, as the availability of data is often limited. Hence, the application of data-driven methods, especially deep learning, is limited in relation to rule-based approaches. The task of open domain dialogue systems is highlighted by Santhanam & Shaikh (2019). Here Sequence-to-Sequence models are a popular choice. Making use of the attention mechanism enhanced the performance. As key challenges they identified the encoding of contextual information and the incorporation of personality into the text generation. They mention the risk of generating dull and generic responses. As future directions they foresee taking insights from cognition into account and ways to encode emotional content. In Chen et al. (2017) a difference is made between task-oriented and non-task-oriented dialogue systems. For task-oriented systems, often external knowledge bases are queried. Such procedure has two main downsides. First, there is no information about the uncertainty, and second, such approaches are not differential. As a response, techniques like key-value memory networks might address these issues. It has been noted that trivial responses is one of the key challenges, which can be tackled via improved objective functions. Another noteworthy approach being mentioned are hybrid methods, combining the strength of multiple approaches. Challenges for future research include shortening the warm-up phase required to adapt to novel domains. A deeper understanding of learning should also facilitate improved dialogue systems. Finally, protecting the privacy of users should be a relevant aspect of future systems.

For Named Entity Recognition, Li et al. (2020) point out the importance of the input representation. When using language model embeddings (e.g., Transformer), the performance of NER systems improved. Also, leveraging external knowledge improves performance. Yadav & Bethard (2019) highlights that feature-inferring systems outperform systems that rely on feature engineering. Using a combination of word embedding and character embedding performs better than each of them individually. What exact architecture to choose depends on the data and domain. If large datasets are available, learning RNNs from scratch is an viable option. For cases where data is scarce, transfer learning is a promising approach, with fine tuning on the target domain is often an effective way. Crucially, the manual annotations are required to be of high accuracy and consistency. Taking insights from traditional approaches and lifting them into contemporary DL-based approaches may improve the current state of the art. Fine-grained NER will be of more importance in the future, as well as jointly solving NER and Entity Linking as one combined task.

For sentiment analysis, Zhang, Wang & Liu (2018) highlights the downsides of following a Bag-of-Words representation, including their inability to represent sequence information and the lack of semantic encoding. In Do et al. (2019) the usage and importance of word embeddings for sentiment detection is being covered. They found that CNNs require more training data in contrast to recursive neural networks. Furthermore, CNNs lack the ability to capture broader contextual information. Recursive Neural Networks, at the other hand, are well suited to capture the complex relationships found in text, which are required to capture opinions, emotions and sentiment. Challenges include changes in the domain and languages, where under-resourced languages with different structure are less well researched. As for future trends, the surveys identified multimodal data, including acoustic and visual data, and a stronger focus on resource-poor languages.

For the task of text summarization in Shi et al. (2018) most of the presented approaches are also known to be used for other tasks, for example sentiment detection. A prominent exception is the development of Pointer-Generator Networks, where the input is directly copied to the output, which is particularity interesting for the challenge of Out-of-Vocabulary words. This challenge is relevant for all abstractive text summarization approaches, which also suffer from repetitions and insufficient handling of salient information. In terms of computation, recurrent neural networks cannot easily be parallelized. Here, CNNs have the advantage of a fixed size input and a short path between the input and output. As challenges and future work for the task of text summarization the survey mentions large transformers, reinforcement learning schemes, the generation of summaries via language generation, the adoption of more diverse datasets and improvements in evaluation, for example via improved evaluation measures.

The task of answer selection is covered by Lai, Bui & Li (2018). First, three learning approaches are compared. It is found that the listwise approach outperformed the pointwise, as well as the pairwise approach. As explanation it is mentioned that the listwise approach considered the task as a list of candidates, instead of considering all candidates in isolation. Next, multiple architectures are presented. Here the Compare-aggregate architecture has been mentioned to provide improved performance over the Siamese and Attentive architectures. This can be explained by the ability of this architecture to capture interactions between the input sentences. As future research, synergies between closely related tasks have been mentioned, for example open domain question answering and community question answering. Furthermore, it is expected that answer selection will find its way into real-world approaches, for example applications related to truth discovery.

Word embedding is studied as a standalone task in Almeida & Xexéo (2019) and in the context of information retrieval in Zhang et al. (2016). Both surveys shed light on a number of approaches that precede word embeddings, which share the motivation of exploiting the distributional semantics. A difference is being made between prediction-based and count-based methods, with word2vec being the most prominent example for the first and GloVe the most popular for the second. It has been mentioned that GloVe is able to outperform many other count-based approaches, as well as prediction-based methods. In the context of information retrieval, a noteworthy approach is the computation of local embeddings, based on the query. While there is a performance improvement associated with such approach, the computational complexity is high and the embeddings need to be computed at query time. As future work, word embeddings are expected to get more rich and integrate more contextual information, for example the word order.

In the domain of financial forecasting Xing, Cambria & Welsch (2018) the temporal aspect is being stressed. Hence, approaches including recurrent neural networks have been applied in this domain. Interestingly, the usage of self-organizing networks has been researched, also for the task of event embeddings. Self-organizing fuzzy neural network have been found to be faster than similar approaches like adaptive neuro-fuzzy inference system (ANFIS). Similar to other surveys one of the challenges are domain-specific datasets and resources, including knowledge bases and ontologies. The need for improved evaluation measures is another common topic for future research. Furthermore, the ability to interpret the model is an added benefit in the financial forecasting domain and will attract more research in the future.

To conclude the majority of surveys in this section highlight the importance and contribution of word embeddings to wide variety of tasks. Currently, there does not appear to exist a consensus, if CNN-based or LSTM-based approaches are better suited for the majority of NLP problems. Dedicated architectures, such as attention-based, encoder-decoders, and especially the transformer architecture gained a lot of popularity recently. Another common emphasis of many surveys is the topic of evaluation measures. Currently used measures were considered to only partly reflect the true performance in relation to the target tasks, with novel measures being considered one of the future research topics. Similarly, there is an emphasis on need of more datasets, especially for dedicated domains and resource-poor languages.

Health informatics

Health or healthcare informatics is the field that applies informatics to the medical domain with the aim to improve health care. Ravì et al. (2016) provide a broad overview about the application of deep learning for health informatics and cover the areas medical informatics, sensing, bioinformatics, imaging and public health, also listing the application, input data and underlying base method. They start their review with a comprehensive background about deep learning architectures like deep neural networks, deep autoencoder and deep belief networks. They conclude their work by discussing the limitations and challenges of deep learning in healthcare.

Medical image analysis

Medical image analysis is the task of automatically or semi-automatically extracting information from (patient-specific) medical images. For example, this could be an automatic determination of the tumour volume from a patient’s magnetic resonance imaging scan with the aim to choose a therapy strategy. The publication of Litjens et al. (2017) gives an overview of main deep learning techniques in regard to medical image analysis. It presents over 300 works in that area and reviews the application of deep learning in topics like organ or disease detection, image classification, segmentation, registration, and further tasks. Furthermore, compact outlines of studies are presented by the application areas: abdominal, breast, cardiac, digital pathology, musculoskeletal, neuro, pulmonary and retinal. Finally, they summarize recent works at that time and discuss remaining challenges and areas for future research work.

Shen, Wu & Suk (2017) present the basics of deep learning-based approaches and analyse the reported results in fields like medical image registration, tissue segmentation, anatomical and cell structure detection, computer-aided disease diagnosis and prognosis. They conclude their work by raising remaining research questions and proposing future research directions for additional improvements in the field of medical image analysis.

Xing et al. (2017) give an overview about microscopy image analysis. They start with a brief introduction of common deep neural networks and provide a compact outline of recent deep learning successes in numerous applications, including detection, classification and segmentation in the area of microscopy image analysis. They present the background of (fully) convolutional neural networks, deep belief networks, recurrent neural networks and stacked autoencoders, and connect their basic principles and modelling to certain application in different microscopy images. They conclude by discussing remaining research challenges and outline possible directions for future research in deep learning-based microscopy image analysis.

The review of Haskins, Kruger & Yan (2020) present the progress of deep learning-based approaches for the field of medical image registration by outlining research challenges, but also significant advancement in the last years. They divide their article in three main categories, namely unsupervised transformation estimation, supervised transformation estimation and deep iterative registration. Each main category is then divided again in sub-categories. They conclude with surveying highlights of future research directions in this field.

Medical imaging

Medical imaging deals with the acquisition of human and animal images from cellular to body scale. Common examples include computed tomography (CT) and magnetic resonance imaging (MRI), which allow to acquire images at vascular and organ level for diagnostic and therapeutic reasons (Pepe et al., 2019; Gsaxner et al., 2019a; Gsaxner et al., 2018). In this context, Lundervold & Lundervold (2019) provide an analysis of deep learning-based methods in medical imaging with a focus on MRI acquisitions. The goal of their review is threefold: First, they provide a compact background introduction of deep learning and its main contributions. Then, they outline how deep learning-based methods have been used for the whole MRI workflow from image acquisition to diagnosis, and give a starting point for researchers in the area of deep learning-based medical imaging. Finally, they point to open-source code repositories, educational resources, and further related data sources that are of interest for medical imaging.

Health record analysis

Health record analysis explores the digital information stored in electronic health records. Originally intended to archive patient information and performing administrative tasks in healthcare, such as billing, researchers starred to utilize these records also for numerous other applications in clinical informatics. Shickel et al. (2017) survey the applications of deep learning for the analysis of health-record data. They report several deep learning-based methods and techniques that have been used for different clinical applications, such as information extraction, outcome prediction, representation learning, but also de-identification and phenotyping. In their review, they identify a number of limitations for the current research regarding the data heterogeneity, model interpretability, but also missing universal benchmarks. They conclude their review by outlining the field and proposing directions for upcoming deep health-record analysis research.

Cancer detection and diagnosis

As previously mentioned, medical image acquisitions, like MRI or CT, can be used for the diagnosis of pathologies, such as tumours, and internal injuries like bone fractures. However, the manual processing of these images can be cumbersome and time-consuming, even for experts (Hahn et al., 2020). This led to the investigation of automatic approaches. Hu et al. (2018) review deep learning-based applications for cancer detection and diagnosis. In their survey, they start with a background on deep learning and common architectures applied to the detection and diagnosis of cancer. They focus on four common architectures in deep learning, such as (fully) convolutional neural networks, deep belief networks, but also auto-encoders. Additionally, they present a review on studies that exploit deep learning for cancer detection and diagnosis, grouped by cancer type. Finally, they provide a summary and personal comments to the reviewed works and suggest future research directions.

Bioinformatics

Bioinformatics is an interdisciplinary field developing approaches and software tools for the understanding of biological data with a strong focus on large and complex data sets. Lan et al. (2018) concentrate on the review of research contributions using deep learning and data mining approaches for the analysis of domain-specific knowledge in bioinformatics. Their review article provides a summary of several data mining methods that have been utilized for pre-processing, classification and clustering along with different optimized neural network architectures and deep learning methods. Furthermore, they present the advantages and disadvantages of such methods in practical applications, discuss, and compare them in terms of their industrial usage.

Radiotherapy

Radiotherapy can be used, for example, to treat cancer patients. However, planning and delivering radiotherapy treatment is a complicated procedure, which artificial intelligence tries to automate and therefore facilitate. In their review, Meyer et al. (2018) introduce the basics of deep learning and its position in the overall context of machine learning. They introduce popular neural architectures with a particular emphasis on classic convolutional neural networks. Subsequently, they give an overview of contributions on deep learning-based methods that can be utilized for radiotherapy. Thereby, they classify them into seven different categories in regards to the overall patient workflow.

Pharmacogenomics

Pharmacogenomics (pharmaco- + genomics) is an interdisciplinary field between pharmacology and genomics that studies the role of the genome in drug response. In their review, Kalinin et al. (2018) introduce recent works and future applications for deep learning in the area of pharmacogenomics. This includes the exploration of new regulatory variants situated in noncoding genomic regions and their role not only in pharmacoepigenomics, but also in patient stratification from clinical records. They aim at the application of deep learning for the prediction of patient-specific drug responses to optimize the process of drug selection and dosing. This process is automated by applying data-driven deep learning algorithms on large and complex data collections, which can provide different sets of information, ranging from the micro- to the macroscopic level, such as from molecular to epidemiological and from clinical to demographic domains.

Radiology

Radiology is the medical field that deals with the extraction of useful information from images, like CT or MRI patient acquisitions, for diagnosis and treatment of humans and animals. Mazurowski et al. (2019) give an overview of the common fields of radiology and present options and chances for deep learning-based approaches there. They also present fundamental deep learning concepts, such as convolutional neural networks before presenting research contributions focused on deep learning and its application to radiology. The reviewed works are grouped by application task. They conclude their work discussing opportunities and challenges for the inclusion of deep learning-based approaches into the clinical practice.

Going deeper: architectures, evaluations, pros, cons, challenges and future directions in medical informatics

Table 7 gives more details about the presented methods, pros, cons, evaluations and challenges and future directions of the surveys in the category medical informatics. Ravì et al. (2016) presented a review on using deep learning in health informatics, where the authors specifically focused on applications in translational bioinformatics (e.g., genomics, pharmacogenomics, epigenomics), medical imaging (CT, X-Ray, MRI, fMRI, positron emission tomography (PET)), pervasive sensing (signal abnormality detection (e.g., ECG, EEG), assistive device for physically impaired patient and energy and activity monitoring for obesity prevention), medical informatics (aggregated medical data such as Electronic health records (EHR)) and public health (e.g., pandemic disease control and surveillance, social public behavior modelling). In each of the categories covered, the authors reviewed the representative publications for each specific task such as using deep learning for RNA binding protein prediction, myocardium identification, calorie measurement using CNN and EEG anomaly detection based on a deep belief network (DBF). Where applicable, the limitations of the reviewed publications and methods for the respective tasks are briefly analyzed. The survey is concluded by presenting the limitations and challenges of using deep learning in healthcare and future perspectives. For example, iterative deep learning that enabled the collaboration between human experts and the machine would benefit the health informatics field, which would at the same time overcome the shortcomings of deep learning models such as lack of reliability and interpretability.

Four surveys were included in the ‘Medical image analysis’ category, where two of them (Litjens et al., 2017; Shen, Wu & Suk, 2017) specifically used ‘deep learning in medical image analysis’ as the titles and the survey from Xing et al. (2017) focused on the analysis of microscopy images, and Haskins, Kruger & Yan (2020) focused on medical image registration. In Litjens et al. (2017), the reviewed publications were grouped/categorized by two means: First, by different medical tasks such as classification (e.g., image, lesion), detection (e.g., organ, landmark, lesion), segmentation (e.g., organ, lesion), registration and other medical tasks (image retrieval, generation, enhancement, medical reports). Second, by anatomical areas such as brain, eye, chest, breast, cardiac, etc. Different medical tasks and anatomical areas constitute the subcategories of the survey. For each subcategory, the authors selected and reviewed the corresponding publications, providing information, if available, about the deep learning architecture used, image modalities and the limitations of the respective studies. The authors concluded the survey by summarizing the challenges and outlook of deep learning in medical image analysis and the desirable properties deep learning models should have in medical tasks. Shen, Wu & Suk (2017) used a similar task-based grouping strategy to that of Litjens et al. (2017), where the authors reviewed deep learning methods that were applied to structure detection (e.g., organ, body part, cell), image segmentation, computer-aided detection (CADe) and computer-aided diagnosis (CADx). Xing et al. (2017) also grouped the reviewed publications by applications in microscopy images, such as detection (e.g., mitosis detection in histology images, cell and nuclei detection), segmentation (e.g., nucleus, cell, neural membrane segmentation) and classification (e.g., human epithelial-2 cell images classification). Due to that microscopy images are distinct from other medical image modalities such as CT and MRI, tailored pre/post-processing methods and sometime network architectures were involved. Haskins, Kruger & Yan (2020) divided the reviewed publications based on the registration techniques (iterative registration, supervised and unsupervised transformation estimation) used. High expense of obtaining the training data and the labels have been identified as the most prevalent challenges of using deep learning for medical image analysis.

The survey from Lundervold & Lundervold (2019) specifically focused on the application of deep learning in magnetic resonance imaging (MRI), from the acquisition and reconstruction of raw MRI data to the registration, restoration, synthesis, retrieval and segmentation of MRI images. Limited data availability, difficulty of interpreting deep learning models and integrating deep learning models into existent clinical workflows has been identified by the authors as the main challenges of deep learning in MR image analysis.

Shickel et al. (2017) reviewed the recent development of electronic heath record (EHR) analysis using deep learning, which is a relatively new and less touched area compared to image-based deep learning applications. Due to the distinct characteristics of EHR, LSTM, RNNs and their variants (mostly NLP techniques) are among the most widely used deep learning architectures in this field, compared to medical image analysis where CNNs were prevalently adopted. The survey starts by reviewing techniques regarding the representation learning of EHR, which is fundamental for all deep learning-based EHR applications. Then, the survey goes into application-specific publications such as outcome prediction and the deidentification of clinical data. The authors concluded the survey by pointing out aspects specific to EHR data, such as data heterogeneity and irregular measures worthy of future investigation. Challenges in this field are, for example, the EHR data even present greater heterogeneity compared to other well-structured and standardized medical data such as CT and MRI, making the learning of their representation using deep learning more challenging.

The survey by Hu et al. (2018) is focused on the detection and diagnosis of cancers based on medical images. The reviewed publications in this survey are grouped according to the areas where the cancer occurs, such as breast cancer, lung cancer, skin cancer, prostate cancer, brain cancer, colonial cancer and other types of cancers. For each type of cancer, the deep learning-based methods used, including the architectures like FCN and AE were discussed. The authors pointed out that, even though large amount of imaging data of cancers has been accumulated over the years, it is still a challenge to make these valuable datasets publicly available for the development of deep learning models.

Lan et al. (2018) reviewed data mining and deep learning techniques that can be used in bioinformatics. Unlike previous surveys, which are built upon publications about specific medical applications, the survey from Lan et al. (2018) focused more on the introduction of the data mining and deep learning methodologies, such as traditional machine learning algorithms (e.g., nearest neighbor, decision tree, clustering) and various deep learning structures and compositions (CNN, SAE, DBN, RNN). Another large part of the overall content in the survey is about a comparison between data mining and deep learning. Class imbalance is common in many medical datasets, which is pointed out to be one of the main challenges.

Meyer et al. (2018) focused on a specific application—radiotherapy used for cancer treatment. They have identified the common steps in the radiology workflow including patient consultation, image acquisition, target structure segmentation, treatment planning and delivery and follow-up. The deep learning methods that have been applied in these steps were reviewed and discussed. These include, for example, that some researchers have used CNNs to reduce the artifacts on CT images acquired from limited angles and some researchers used a CNN to reconstruct MRI images at 7T fidelity level given an image taken under 3T. Building coherent, large and balanced medical datasets is identified by the authors as the main challenges, which should be paid more attention to in the future.

Kalinin et al. (2018) restrict their survey to pharmacogenomics. The publications in the survey were grouped by area of applications such as drug discovery, patient stratification, toxicology. The authors stated that when the dataset used for training deep learning models is small, deep learning is susceptible to overfitting. Lack of interpretability of the deep learning models is identified as another limitation of deep learning based methods in pharmacogenomics. The authors envisaged that future deep learning in pharmacogenomics will revolutionize from ‘prediction’ to ‘prescription’.

Mazurowski et al. (2019) gave an overview of deep learning in radiology, where the imaging modality is restricted to MRI. Like the survey from Lundervold & Lundervold (2019), Mazurowski et al. (2019) grouped the reviewed papers in terms of their applications including classification, segmentation, detection, registration, reconstruction, etc. For classification of radiological data, the most adopted classifier is composed of several convolutional layers, followed by a fully connected layer. For segmentation fully convolutional neural network (FCNN) is a popular choice. The survey concludes by stating challenges and outlooks of deep learning in radiology. For example, the authors pointed out that more studies are needed in the future to seamlessly integrate deep learning models in existent radiology workflows, so that the current radiology practice can be improved. Another challenge is that medical image datasets are usually much smaller than natural image datasets and training on small datasets could lead to overtraining. Class imbalance (healthy cases far outnumber pathological cases) in medical image datasets is another factor that could lead to suboptimal training of deep learning models.

To conclude, most of the surveys reviewed in this category followed a similar structure: a general introduction of machine/deep learning techniques followed by the discussion of the publications categorized using different strategies, among which task- or application-based categorization is the most widely used. The surveys usually conclude by stating challenges and future perspective of deep learning in the medical field from a high level. It should be noted that, the topics in subcategories ‘health informatics’, ‘medical image analysis’, ‘medical imaging’ and ‘radiology’, are highly overlapping and too broad, such that the respective surveys are unable to provide ample low-level details regarding the deep learning models. Some surveys such as the work of Shickel et al. (2017), focused on a very specific and narrow topic (Electronic Health-record analysis), so that they can provide some low-level insights of deep learning usage for the task.

Big data

Big data is the field that analyses data that is too comprehensive, or too complex, to handle using classic data-processing tools. The aim of big data is to systematically extract information from large and complex data sets. Application areas include e-commerce, industrial control, and precision medicine. Zhang et al. (2018) review in their contribution the works on emerging deep learning models for feature learning with big data. They review deep learning-based methods and models that have been used with large data collections, heterogeneous data, but also real-time and low-quality data.

Mohammadi et al. (2018) give an overview on deep learning-based approaches that have been used to support the learning and analytics in the domain of internet of things (IoT). They begin with the characteristics of IoT data, but also the treatments of IoT data, namely the analytics and streaming of big data in the domain of IoT. Next, they review the promising aspects of deep learning-based methods for getting certain results in analytics regarding these data types’ applications. In addition, they outline the potential of upcoming deep learning-based methods for data analytics in the IoT domain. Besides a comprehensive background on different deep learning methods, they review further research efforts, which affected the IoT domain by applying deep learning. Furthermore, they state how smart IoT devices have incorporated deep learning and review methods for fog computing and cloud computing in aiding IoT approaches.

Emmert-Streib et al. (2020) start their contribution with a background analysis in deep learning-based methods, like convolutional neural networks, deep feed-forward neural networks, but also deep belief networks, long short-term memory networks and autoencoders, because, according to the authors, these are currently the most commonly used architectures. Additionally, they introduce related concepts such as restricted Boltzmann machines and resilient backpropagation and discuss the differences when dealing with big data vs. small data and specific data types. They state that the adaptiveness of these (network) architectures enables a “Lego-like” generation of countless new neural networks.

Reinforcement learning

Reinforcement learning is an algorithmic learning strategy where the algorithm tries to maximize an agent’s performance via rewards, based on observations within the task environment. Reinforcement learning-based approaches have proven to be successful in numerous fields, like the robotics domain. The article of Mousavi, Schukat & Howley (2016) surveys the recent advances in supervised and unsupervised deep reinforcement learning, putting an emphasis on most commonly applied deep architectures, like convolutional neural networks, recurrent neural networks, but also autoencoders that have effectively been incorporated within reinforcement learning-based frameworks. They structure their review in three main categories, these are: supervised reinforcement learning, unsupervised reinforcement learning, and deep reinforcement learning in environments that allow to observe parts of the process as Markov decisions.

Li (2017) gives also an overview of deep reinforcement learning strategies, discussing six main elements, six significant mechanisms, but also twelve related applications. After presenting the fundaments of machine and deep learning, he also introduces the main elements of reinforcement learning, such as value function, policy, reward, and exploration strategies. Afterwards, the mechanisms, including attention and memory, transfer learning, unsupervised learning, multiagent reinforcement learning, but also hierarchical reinforcement learning and learning to learn, are presented. Finally, numerous possible applications are outlined, such as games (like AlphaGo), natural language processing, covering dialogue systems, text generation, machine translation, but also computer vision, robotics, finance, business management, education, healthcare, Industry 4.0, intelligent transportation systems, smart grids and further computer systems.

Arulkumaran et al. (2017) start their review by giving a general overview of the reinforcement learning field and continuing afterwards to the central areas of value-based methods, but also policy-based methods. The review covers the main approaches and methods in the field of deep reinforcement learning, such as deep q-networks, asynchronous advantage actor critic and trust region policy optimization. Additionally, they outline the main benefits of deep neural networks, in particular on visual understanding, with reinforcement learning.

Mobile and wireless networking

Mobile and wireless networking, or short networking, has rapidly evolved during the last years, thanks to the spread of mobile devices and mobile applications (Bevilacqua et al., 2015; Labini et al., 2019). In addition, the recently released 5G technology is expected to massively increase the mobile traffic volumes. Zhang, Patras & Haddadi (2019) present a comprehensive survey in deep learning-based research in mobile and wireless networking. They start with an introduction about fundamentals on deep learning-based methods that could lead to networking applications and review certain approaches and platforms with the potential to promote the progression of mobile systems using deep learning. Next, they give an overview on deep learning-based research in mobile and wireless networking, by categorizing it in several domains.

Mobile multimedia

Mobile multimedia refers to various applications that can be accessed or created by portable devices, like smartphones (Karner et al., 2020). This includes mobile applications such as audio and video players, games and e-healthcare. Ota et al. (2017) introduce the basics of deep learning for multimedia, thereby focusing on the core parts of deep learning in regard to mobile environments, namely low-complexity deep learning methods, software tools and frameworks for mobile and other resource-constrained environments, but also specific hardware available in mobile devices, which can be used to facilitate the computationally intensive training and inference of deep networks. In addition, they present numerous deep learning-based, mobile applications to show possible real-life scenarios for such a technology.

Multimodal learning

Multimodal learning uses data of different modalities in a learning strategy. An example for data from different modalities are the acquisitions from positron emission tomography-computed tomography (PET-CT) scanners, where the tissue data from the CT and metabolically active regions from the PET are acquired from a patient (Gsaxner et al., 2019b). In their survey, Ramachandram & Taylor (2017) first classify the architectures for deep multimodal learning. Afterwards, they introduce certain methods to combine multimodal representations that have been learned with these deep learning-based architectures. In particular, they outline two main research fields for potential upcoming works, namely regularization methods and strategies that learn and optimize structures in the domain of multimodal fusion.

Remote sensing

Remote sensing covers technologies for the remote analysis of objects or scenes. Examples are satellite-based imaging, aerial imaging, crowdsourcing (such as tweets or phone imagery), but also advanced driver-assistance systems and unmanned aerial vehicles. Ball, Anderson & Chan (2017) provide a compact analysis of recent deep learning-based research in the domain of remote sensing. They introduce the theories and tools, but also challenges within the remote sensing field. Thereafter, they present remaining research questions and opportunities, like modelling physical phenomena with human-understandable solutions, inadequate data sets and big data. Furthermore, they focus on specific non-traditional data sources that are heterogeneous, deep learning-based architectures and algorithms to learn spatial, spectral and temporal data, but also transfer learning. Further, they provide a fundamental insight into deep learning-based systems, and outline obstacles for training, but also optimizing deep learning-based methods.

Graphs

Graphs are representations of objects and of their relationships with other objects. Common examples include social networks, traffic networks, e-commerce networks, but also biological networks. Zhang, Cui & Zhu (2020) provide a survey on various types of deep learning-based approaches on graphs. They split the existing approaches into five different categories in regards to the underlying model architectures, but also training strategies, namely graph convolutional networks, graph recurrent neural networks, graph reinforcement learning, graph autoencoders and graph adversarial methods. They propose a systematic outline of these techniques, mostly by following the historical appearance and review the structures and differences. They conclude their review by outlining the applications in this area.

Anomaly detection

Anomaly detection is a strategy used to detect unexpected events or items in data sets. It can be used in areas like signal processing, statistics, finance, manufacturing, econometrics, networking, but also data mining. Kwon et al. (2019) propose an outline of deep learning-based methods, covering deep neural networks, restricted Bolzmann machine-based deep belief networks and recurrent neural networks. They also cover machine learning techniques that are related to network anomaly detection. Furthermore, they present the latest work from the literature that used deep learning-based techniques with an emphasis on network anomaly detection. Finally, they outline their own results with deep learning-based methods for the analysis of network traffic.

Recommender systems

Recommender systems are utilized to predict the preferences of a user, for example, to provide web users with personalized information about products and services, like movies, insurances, or restaurants. Zhang et al. (2019) propose a survey of latest deep learning-based research for recommender systems. They formulate and introduce a deep learning-based taxonomy for recommendation models, together with an outlining of recent contributions from the literature. They conclude their contribution by outlining current trends and propose new perspectives in regard to the development in the field.

Agriculture

Agriculture is the scientific process that deals with the cultivation of plants and livestock to produce products like food, feed and fibres. Kamilaris & Prenafeta-Boldú (2018) review 40 research works in deep learning, which have been used for numerous agricultural, but also food production problems. For each problem, they explored the specific agricultural challenge, the frameworks and models that have been used, but also the data source, data nature and data pre-processing methods. They also report and present the overall performance results for the selected metrics. In addition, they survey the comparison of deep learning-based approaches with other existing common methods, with a focus on the differences in the performances for classification and regression.

Multiple areas

At last, this sub-section presents deep learning review contribution that span over multiple disciplines and applications with no main focus on a specific area. Pouyanfar et al. (2018) present a more general review in the field of deep learning-based algorithms and techniques, but also their applications. Their survey proposes an in-depth analysis of historical, but also novel methods in visual processing, audio processing, and text processing, but also the analysis of social networks, and further the processing of natural language. Next, they provide a comprehensive review on improvements in deep learning-based approaches and survey deep learning challenges, like unsupervised learning and online learning, but also black-box models, and show how these remaining demands may be addressed in the upcoming works.

Dargan et al. (2019) focus in their contribution on common deep learning concepts, including fundamental and more sophisticated architectures, characteristics, techniques, limitations and motivational aspects. They introduce several main differences amongst classical machine learning, deep learning and approaches in conventional learning, but also main challenges that still need to be solved. They chronologically analyse and conduct an extensive review of significant deep learning-based applications, thereby including numerous fields, techniques, methods and architectures that have been applied, and discuss the works of their application for usage in a real world scenario.

In their review of deep learning-based scientific discovery, Raghu & Schmidt (2020) provide an analysis of several deep learning-based models that have been applied to topics like sequential, visual, but also graph structured data. Further, they present related tasks and varying training approaches, together with methods that enable the usage of deep learning-based methods for sparse data, and how to get a better understanding of complex models. They also give different outlines of overall design processes, hints for implementations, and point to tutorials, open-sourced pipelines in deep learning, research summaries, but also pretrained models that have been implemented within the community, with the aim to speed up the application of deep learning-based approaches across multiple scientific fields and domains.

Going deeper: architectures, evaluations, pros, cons, challenges and future directions in additional works

Table 9 provides more details about the presented methods, pros, cons, evaluations and challenges and future directions of the surveys in the category additional works. The three surveys about big data from Zhang et al. (2018), Mohammadi et al. (2018) and Emmert-Streib et al. (2020) introduce common architectures, like DBN, CNN, RNN, AutoEncoders, whereby Mohammadi et al. (2018) and Emmert-Streib et al. (2020) introduced further models and variants, like LSTM. Zhang et al. (2018) and Mohammadi et al. (2018) have also a focus on real-time data. Zhang et al. (2018) and Mohammadi et al. (2018) make a complexity analysis, which includes for both works the forward passes. Emmert-Streib et al. (2020) do not perform such an analysis, but extract the pros and cons when going over the different deep learning models, for example that CNN can easily process high-dimensional input and is good at extracting local information. On the downside CNN can overfit, be hard to train and has the vanishing gradient problem. RNNs can be used for problems with sequential data, but have problems with long-term dependencies. ResNets can be used to tackle the degradation problem. A property of AutoEncoder is that the input and output layer have the same size and they can be used for dimensionality reduction. On the downside AutoEncoder often need an adequate amount of data. LSTM, for example, can handle long-term dependencies. Zhang et al. (2018) see future challenges in the development of large-scale deep learning models for huge amount of data and multi-modal deep learning models. They state that the gain in computational performance is lagging far behind the growth rate of big data. Further challenges are the development of new learning frameworks and computing infrastructures, compressing the large-scale deep learning models and the focus on more than two modalities, exploring effective fusion models, reduce the computational complexity, optimize network structures and reliable deep learning models for low-quality. Mohammadi et al. (2018) see the main challenges in: (1) Service discovery; (2) Model and task distribution; (3) Design factors for fog environments; (4) Energy management of mobile edge devices; and (5) mobile edge computing environments. Emmert-Streib et al. (2020) see future works in finding the right application for a deep learning model, but also understand the decisions of deep learning models (black boxes), especially for the healthcare domain. Further challenges are small data sizes, the transfer of knowledge between models and exploring more advanced models, like reinforcement learning, graph CNN and VAE.

Mousavi, Schukat & Howley (2016), Li (2017) and Arulkumaran et al. (2017) present all surveys about deep reinforcement learning. They also have in common that all three surveys cover various types of input data like, images, videos, text and language, for a range of applications, like natural language processing or robotics. The contributions go over classic concepts in reinforcement learning, like Q-learning, policy search and actor critic, and how these have been handled in deep reinforcement learning, thereby introducing deep learning methods, like CNN, AutoEncoder, RBM, RNN, LSTM, and their combinations with classic reinforcement learning concepts, resulting in deep reinforcement learning methods and concepts, like DQN, DFQ, DRQN, or DRQN. All surveys also introduce and discuss the performance of deep reinforcement learning methods in playing Atari games, which seems to be one of the main benchmarks for evaluating (deep) reinforcement learning methods. Beside pros and cons from classic deep learning methods, Mousavi, Schukat & Howley (2016) identified several pros and cons in deep reinforcement learning, like DQN can learn an optimal policy and can have a good stability, but needs sufficient data. They also state that model-free reinforcement learning algorithms, like Q-learning, can have stability issues and DRQN can have better performance for some tasks. They also extract several pros for combined methods, for example MDRNN+LSTM can play a variety of games, MDRNN+LSTM tackles, the vanishing gradient problem, CNN+RNN+RL can predict action-conditional frames, and LSTM+DQN can handle noisy and incomplete data. Li (2017) identified several pros and cons in deep reinforcement learning, e.g. that DQN stabilized the learning, achieved outstanding results, is an end-to-end RL approach and performs well on many different tasks. However, Li (2017) also states that PGQ can outperform DQN. D-DQN on the other hand can tackle the problem of over-estimation in DQN and have better policies on Atari games than DQN. The author further states, that some specific extensions, like dueling network architectures can converge faster, but also that DDPG can avoid the optimization of action at every time step, or that VIN are fully differentiable CNN. A mature con is that deep reinforcement learning is a black box compared to many classic reinforcement learning techniques. Arulkumaran et al. (2017) also state that DRL are black box optimization methods. However, they can handle high-dimensional problems and have powerful function approximation properties. On the downside, the policy search can be difficult, they can suffer from severe local minima and DNN require in general many samples. They find that DQNs are comparable to professional video games tester, can compactly represent both high-dimensional observations and the Q-function, and address the instability problem. Further duelling DQNs are able to easier-to-learn relative values, Finally, ALVINN cannot adapt to new situations and can be unable to recover from specific situations they ran into. Mousavi, Schukat & Howley (2016) see future challenges in learning optimal control policies in problems with raw visual input, investigating deep architectures for end to end leaning and deep state representation and transfer learning. They also see challenges in real application such as robotics and end to end learning in real world applications. Future challenges of Li (2017) are a better understanding of how deep learning works and the investigation of comments/criticisms, e.g., from cognitive science, also considering perspectives of government, academia and industry. Li (2017) states that there are only a few products so far and there is a need for products and market validation. Arulkumaran et al. (2017) see future challenges in improving the data efficiency of neural networks and that they are not yet ready for complex real-world problems. Further open issues are a deeper integration with other traditional AI approaches for a better sample complexity, generalization, and interpretability. Finally, they hope for a better theoretical understanding of neural networks within deep reinforcement learning. Note, that the survey of Li (2017) seems to be unfinished.

Zhang, Patras & Haddadi (2019) give a comprehensive survey in the domain of deep learning for mobile and wireless networking. In doing so, they introduce common deep learning architectures used in this field, like MLP, RBM, AutoEncoder, CNN, RNN, LSTM, GAN, DRL, but also tailored deep learning methods for mobile networks. They provide a compact overview of the pros and cons of these architectures, e.g. if they are straightforward to build, can generate virtual samples, if they are ideal for high-dimensional environment modeling or which architectures have modest performance, slow convergence, are difficult to train or expensive to pretrain with big data, to name a few. In this regards, they also state the learning scenarios: supervised, unsupervised, reinforcement, and suitable problems, like modeling data with simple correlation or extracting robust representations. They conclude their survey by listing five main and important remaining research issues that should be addressed in the future in mobile and wireless networking: (1) Serving deep learning with massive high-quality data, for example high-quality and large-scale labeled datasets still lack for mobile network applications; (2) Deep learning for spatio-temporal mobile data mining, by means of spatio-temporal distribution of mobile traffic and application popularity are difficult to understand; (3) Deep learning for geometric mobile data mining, because of inherent complexities of representations in mobile and wireless networking, traditional machine learning tools can struggle to interpret geometric data and make reliable inferences; (4) Deep unsupervised learning in mobile networks, because data labeling is costly and requires domain-specific knowledge, hence unsupervised learning becomes essential in extracting insights from unlabeled data; (5) Deep reinforcement learning for mobile network control by making no strong assumptions (e.g. about the objective functions, like function convexity, or data distribution, like Gaussian or Poisson distributed) about the target system and employ function approximation.

Ota et al. (2017) went through the basics of deep learning for multimedia, focusing on the main components of deep learning for mobile environments, namely the low-complexity deep learning algorithms, the software frameworks that are optimized for mobile and resource constrained environments, and the specific hardware for mobile devices supporting the computationally expensive processes of training deep networks and inference. They also highlight several mobile deep learning applications and give an overview of real-life usage in this field. In doing so, the authors introduce several deep learning-based architectures, like CNN, DBN, RCN, NMT, GNMT and variants. Because of the hardware constraints of mobile devices, the authors focus on different hardware acceleration solutions for DNNs (GPU, FPGA, ASIC, VPU, Distributed Computing, High Performance Computing) in the mobile domain and provide the pros and cons for performance, energy consumption and cost. Future challenges are the resource-constrained mobile platforms and the development of techniques for optimizing such architectures by “compressing” them and further developing new hardware that can optimally cater to the needs of deep learning-based architectures. Further challenges are the energy constraints that exist in the mobile world and coupling deep network-based architectures and the mobile computing capabilities in an optimal way. They foresee that more and more mobile applications can run deep learning-based engines, as mobile devices become more powerful, resulting in a new world of possibilities where we have seen only the tip of the iceberg so far.

Ramachandram & Taylor (2017) survey deep learning in the area of multimodal learning. Therefore, they introduce various deep learning-based architectures, like Sparse RBM, DBN, DBM, FCNN, CNN, RNN, AutoEncoder, MLP, and LSTM. In their contribution, they have a strong focus on multimodal fusion, namely, intermediate and late fusion, being implemented with deep architectures. They state that incorporating multiple modalities into the learning problem almost always results in much better performance. Further, they state that deep learning methods facilitate a flexible intermediate-fusion approach, but also that they make it simpler to fuse modality-wise representations and learn a joint representation. Finally, they allow multimodal fusion at various depths in the architecture. As downside, the authors see the involvement of a great deal of manual design for multimodal learning with deep learning-based architectures. In addition, that architecture learning can be extremely compute-intensive in multimodal learning. Future challenges are a full exploration of fusion architectures and truly generic learning methods with minimal or no human intervention. Moreover, taking advantage of advances in hardware acceleration and distributed deep learning. Specifically for the medical domain, they see future challenges in (1) the relatively few multimodal medical data set collections that are available and in addition (2) the relatively small medical data set sizes, involving only between ten and 50 subjects, and on top (3) a high class imbalances (normal vs abnormal cases). Hence, they encourage researches in the medical domain to make data sets publicly available in the future.

Ball, Anderson & Chan (2017) provide a comprehensive survey of state-of-the-art remote sensing deep learning research. Thereby, they introduced the most commonly used deep learning architectures in remote sensing, like AutoEncoder, DBN, RNN, CNN and variants of them. They also list some popular deep learning tools, like Caffe and Keras, and the subject areas of deep learning works in remote sensing, like human detection, animal detection or vehicle detection/recognition. For evaluation, the authors focus on the overall accuracy results in percent for hyperspectral images of common open-source data sets (Indian Pines, Kennedy Space Center, Pavia City Center, Pavia University, Salinas, Washington DC Mall). Supplemental material from the authors, in form of more detailed tables, can be found online. According to the authors, the main unsolved challenges and opportunities in remote sensing are: (1) inadequate data sets; (2) human-understandable solutions for modeling physical phenomena; (3) big data; (4) nontraditional heterogeneous data sources; (5) deep learning-based architectures and learning algorithms for spectral, spatial, and temporal data; (6) transfer learning; (7) an improved theoretical understanding of deep learning-based systems, (8) high barriers to entry; and (9) training and optimizing the deep learning approaches.

Zhang, Cui & Zhu (2020) provide a deep learning survey on graphs, more specific on deep learning for graph data with its unique characteristics. In doing so, they divide the existing methods into five categories based on their deep learning-based architectures and training strategies, namely graph recurrent neural networks, graph convolutional networks, graph autoencoders, graph reinforcement learning and graph adversarial methods. For evaluation, the authors focus on the time complexity, divided into tables for Graph RNN, GCN, GAE, Graph Reinforcement Learning, and Graph Adversarial Methods. The author identified the following main future challenges for deep learning on graphs: (1) new models for unstudied graph structures, because the existing methods are not suitable for all graph structures due to the extremely diverse structures of graph data; (2) compositionality of existing models, by means of incorporating interdisciplinary knowledge in a more general way, in contrast to a case-by-case basis remains an open problem; (3) most methods focus on static graphs rather than dynamic graphs, however, many real graphs are dynamic in nature, because their nodes, edges, and features can change over time; (4) interpretability and robustness is also a future challenge, because interpreting deep learning-based results on graphs is critical in decision-making problems, for example in the medical domain or disease-related tasks.

Kwon et al. (2019) present a survey about anomaly detection with deep learning-based methods. In doing so, they give a general overview about deep learning architectures, like RBM, DBM, DBN, DNN, AutoEncoder, and RNN, but also briefly discuss the classification techniques supervised learning, unsupervised learning and semi-supervised learning. For evaluation, they focus on their own FCN model applying it to the public NSL-KDD dataset. They present the accuracy, precision, recall and F1-score for their experiments. They conclude that the FCN model has an improved detection accuracy when compared to conventional machine learning techniques, like SVM, random forest, and Adaboosting. The authors see future challenges in studying other deep learning techniques, like GANs, for anomaly detection. They foresee that missing data can be trained through prediction with the reinforced learning algorithms in the GAN model. Further, they foresee that GANs can work with machine learning to generate multi-modal outputs, which may lead to an improved performance for network anomaly detection with greater accuracy.

Zhang et al. (2019) give a survey on deep learning for recommender systems. Thereby, dividing deep learning-based recommendation models into the following two main categories: (1) Recommendation with Neural Building Blocks (MLP, AutoEncoder, CNN, RNN, RBM, Neural Autoregressive Distribution Estimation, Attentional Models, Adversarial Network, Deep Reinforcement Learning); (2) Recommendation with Deep Hybrid Models (RNN+CNN, AutoEncoder+CNN, RNN+AutoEncoder, etc.). By going over these models, they discuss the pros and cons addressing specific recommender task-related problems. For example, MLP can approximate any measurable function to any desired degree of accuracy and has been widely used in many areas, but might not be as expressive as AutoEncoders, CNNs, or RNNs. The authors identified that almost all AutoEncoder variants can be applied to the recommendation tasks and CNNs are powerful in processing unstructured multimedia data. On the downside, AutoEncoder can fail to deal with non-integer ratings and can lead to worse prediction accuracy and CNN can sometimes have only slightly performance improvements over other methods. RNN are suitable for sequential data processing, but trivial methods can sometimes achieve the same or even better accuracy results. RBMs can be easy to design but can also be limited to binary values and are not tractable in contrast to NADE, which however, have problems with sparse data. Attentional models can process long and noisy inputs and outperform CNNs and RNNs. DRL can have advantages in personalized recommendation and in the performance of real-world applications. Finally, GANs can have advantages in personalized citation recommendation tasks and sometimes significantly improve performance. They conclude that existing works have already established a solid foundation for deep recommender systems, but identified the following open issues: (1) Joint representation learning from user and item content information; (2) Explainable recommendation with deep learning; (3) Going deeper (more than three to four layers) for recommendation; (4) Machine reasoning for recommendation; (5) Cross domain recommendation with deep neural networks; (6) Deep multi-task learning for recommendation; (7) Scalability of deep neural networks for recommendation; (8) and finally, they state that the recommender field needs a better, more unified and harder evaluation of the methods.

Kamilaris & Prenafeta-Boldú (2018) survey deep learning methods in the domain of agriculture. Thereby, giving a summary of the most commonly used methods and architectures, like CNN, AlexNet, VGG16, ResNet, DRNN, LSTM, DBN, AutoEncoder and further variants of these. For a comparison and evaluation of the surveyed works, they focus on metrics like Classification Accuracy, Precision, Recall, F1 score, LifeCLEF metric, Quality Measure, Mean Square Error, Root Mean Square Error, Mean Relative Error, Ratio of total fruits counted, L2 error, Intersection over Union and combined scores. Future challenges they see in the need of large datasets and data augmentation techniques with label-preserving transformations. Other challenges are the low variation among the different classes, the need for experts in order to annotate input images and that experts or volunteers are susceptible to errors during data labeling. They state also that current methods cannot generalize beyond the “boundaries of the dataset’s expressiveness” have difficulties in detecting heavily occluded and distant objects and that there are not many publicly available datasets for researchers in the domain of agriculture. Further, they state that some agricultural-related problems are under-researched, such as seeds identification. Future directions they see in the usage of aerial imagery (i.e. by means of drones), higher performance classification or prediction, like a growth estimation of plants, trees, etc., combining hand-crafted features with automatic features, and demonstrating the ability of the models to generalize to various real-world situations. Finally, they encourage researchers to make their datasets publicly available and that some works can already be used commercially in the near future, e.g. for automatic robots that collect crops, and overall conclude and encourage that deep learning can be used towards smarter, more sustainable farming and more secure food production.

The surveys from Pouyanfar et al. (2018), Dargan et al. (2019) and Raghu & Schmidt (2020) span over multiple areas and various applications in the context of deep learning and provide a broad overview. In doing so, they all survey over numerous domains and fields, like visual, audio, and text processing, natural language processing, and applications, like sentiment analysis, social network analysis, parking systems, smart city, stock market analysis, pose estimation, answer selection, etc., and how they been tackled with deep learning. All works cover the main deep learning architecture and techniques and some variants and combinations, including DNN, RvNN, RNN, CNN, DBN, DBM, GAN, VAE. By going over the different deep learning techniques and methods, Dargan et al. (2019) make a direct comparison to methods from classic machine learning. Raghu & Schmidt (2020) introduce an overall general workflow to design and work with deep learning, starting with the data, over the learning to a validation and analysis of the results, and provide an implementation guideline pointing out to even further resources, like tutorials. They also provide specific sections for transfer learning, domain adaptation, multitask learning and weak supervision (distant supervision). Furthermore, they go over several learning and training strategies, like self-supervised learning, semi-supervised learning, self-training (bootstrapping), co-training, also in combination, like self-supervision with semi-supervised learning, for different data types and domains, like images and NLP, and discuss data augmentation for image data and sequence data. Because of the broadness of the topics, the authors also provide more high level pros and cons in the area of deep learning. Examples are, that they all agree that deep learning had major breakthroughs in different applications and provides solution to many real-world application, and state that it is the most effective, supervised, time and cost efficient machine learning approach. Further pros in deep learning are that it can solve highly computational tasks, does not rely on prior data and knowledge, requires little engineering by hand, and that there is a lot of open sourced code from the community. All criticize that deep learning is a black box approach, which is difficult to understand (decision understanding and weak statistical interpretability), and that deep learning-based approaches need massive data and computational hardware resources to work well. Moreover, it is hard to know where to start with deep learning, by means of how to determine the optimal method for a scientific problem, because of the breadth and diversity of different techniques, and furthermore which combination of methods may be the most promising. It was also stated that deep learning is not always the best technique for a problem. All surveys discuss deep reinforcement learning and Dargan et al. (2019) state that deep reinforcement kind of learning will be the future direction. Other challenges are the large amount of data, on one hand handling these efficiently, on the other hand developing unsupervised methods to avoid the need of manual human labels. Raghu & Schmidt (2020) do not provide a section about future challenges, but state the difficulty of determining how the neural network model makes a specific prediction. Hence, all surveys agree that a future challenge lies in enlighten the black box behavior of deep learning.

To conclude this section, most of the surveys reviewed in this category have also a similar structure: a general introduction of machine/deep learning techniques followed by a discussion of the reviewed publications categorized using different strategies, among which task- or application-based categorization is the most widely used. The surveys usually conclude by stating challenges and future perspectives of deep learning in their fields from a high level. The main challenge amongst the majority of the surveys is seen in deep learning being a black box approach, which makes it hard to understand how results and decisions have been achieved. A further common challenge is seen in the lack of (annotated) data, to train deep learning approaches more reliable.