Neuro-Symbolic Learning: Principles and Applications in Ophthalmology

Neural networks have been rapidly expanding in recent years, with novel strategies and applications. However, challenges such as interpretability, explainability, robustness, safety, trust, and sensibility remain unsolved in neural network technologies, despite the fact that they will unavoidably be addressed for critical applications. Attempts have been made to overcome the challenges in neural network computing by representing and embedding domain knowledge in terms of symbolic representations. Thus, the neuro-symbolic learning (NeSyL) notion emerged, which incorporates aspects of symbolic representation and bringing common sense into neural networks (NeSyL). In domains where interpretability, reasoning, and explainability are crucial, such as video and image captioning, question-answering and reasoning, health informatics, and genomics, NeSyL has shown promising outcomes. This review presents a comprehensive survey on the state-of-the-art NeSyL approaches, their principles, advances in machine and deep learning algorithms, applications such as opthalmology, and most importantly, future perspectives of this emerging field.


I. INTRODUCTION
N EURO-SYMBOLIC learning (NeSyL) is a relatively new field that combines aspects of both symbolic learning and artificial intelligence (AI). NeSyL hybridizes good oldfashioned AI into the advanced machine learning (ML) and deep networks towards an exciting interpretable AI. NeSyLs have extended AI capabilities and outperformed state-of-theart deep learning models with higher accuracy in different domains, specifically in medical imaging and video reasoning. Researchers in this field are interested in understanding how the brain learns and represents knowledge, and how this knowledge can be used to improve AI based systems. In this review, we first provide an overview of the current state of the field. Furthermore, we discuss some of the key challenges that need to be addressed in order to advance the field. Finally, we suggest possible future directions to broaden the employment of NeSyL to distinct critical fields.
In clinical practices, AI has possessed increasing significance in the fields of ophthalmic diseases [9], [10], [11] (Figure 1), including screening, diagnosis, lesion segmentation, treatment, and prognosis. AI has a broad spectrum of applications in clinics and ophthalmology exemplifies in academic research and entrepreneurship. Thus, we use ophthalmology as example to illustrate the application of NeSyL in clinics. The application of AI in ophthalmology increased exponentially in the last decade ( Figure 2). AI in ophthalmology encompasses a wide range of ocular diseases such as cataracts, glaucoma, myopia, and diabetes retinopathy. The workflow of AI guided analysis of these diseases is composed of image classification, segmentation, and object detection and prediction. Although various AI algorithms have been adopted in the field of ophthalmology; however, deep learning (DL) is the dominant one.
We divide AI applications in ophthalmology into three tasks considering current progress. Then we discuss NeSyL approach to address the challenges in AI applications such as classification, segmentation, and detection into futuristic applications. The AI applications in opthalmology include, 1) the image classification and detection of ophthalmic disease for the examination of pathological severity; 2) image segmentation and partition of the retinal image into different parts according to their features and properties; and 3) disease prediction such as eyes related complications (Figure 3).

A. Classification
Several DL approaches have been proposed to classify healthy and pathological cases based on image features in an end-to-end learning manner. The study [12] used a two-stage approach to classify the retrieved disks as healthy or glaucoma patients. In the first step, the optic disc was localized and extracted from retinal fundus images using a convolutional neural network. In the second step, a deep convolutional network was applied to achieve the area under the curve (AUC) of 0.874, in which a 2.7% improvement over the previous best results was observed [13]. Zhang et al. [14] proposed a residual attention based multi-model ensemble method as an object identification network to extract the eyeball from the b-ultrasound image of the eye via ensembling three neural  [1], RF (2005) [2], and CNNs (2006) [3] have been widely used for ophthalmic disease classification, diagnosis, and prediction. With the rapid development of deep learning, network structures such as generative adversarial networks (2017) [4], ResNet (2017) [5], UNet (2018) [6], graph convolutional networks (2020) [7], and transformer (2021) [8] have been introduced to achieve more accurate results. Besides, the public availability of several ophthalmic datasets has significantly advanced the field of ophthalmic AI. networks for cataract classification. The AUC was 0.975 and outperformed the counterpart approaches. However, certain studies embedded additional information in terms of disease severity to categorize images. Wan et al. [15], uses a Convolutional Neural Networks (CNNs) model with transfer learning to perform diabetic retinopathy (DR) classification on the Kaggle, with a classification accuracy of 95.68% and an AUC of 0.97. Peng et al. [16] proposed the first patient-based scoring system, simulating the agerelated macular degeneration (AMD) classification process by merging the De.pdfeeNet model with the age-related eye disease study (AREDS) simplified severity scale (0-5 points) of bilateral color fundus images.

B. Segmentation
The variation in retinal image patterns demands sophisticated DL segmentation models. For instance, Yan et al. [17] proposed a U-Net like autoencoder for vessel segmentation which, combines segment and pixel level losses. Such a framework with two branches learns distinguishable features for vessel segmentation compared to monolithic loss based model. Akram et al. [18] proposed generative adversarial network for retinal blood vessel segmentation with satisfactory results in terms of accuracy, F1-score and AUC. However, the method may perform poorly on thin vessel segmentation. For optic disc (OD) segmentation, Fu et al. [13] developed M-Net to solve the OD and optic cup (OC) segmentation jointly in a one-stage multi-labels framework. The M-Net achieved significant results given multiscale input and side-output layers as compared to single-scale networks. Al-Bander et. [11] used multiscale sequential CNNs for simultaneous detection of fovea and OD on both Messidor [19] and Kaggle datasets [20]. The proposed method is less sensitive to preprocessing and does not need of vessel segmentation or border localization in order to detect the OD and foveal centers. A region proposal and cascaded network were proposed by Huang et al. [21] to realize robust and effective OD detection and fovea localization.

C. Prediction
The ML application has also been extended to the estimation of ocular disease. Among others, refractive error is a leading cause of visual impairment [22]. Traditional ML algorithms,  such as Random Forest (RF) and Support Vector Machine (SVM) were used to predict the onset or development of myopia through numerical data like spherical equivalence (SE), and age [23], [24], while deep neural network (DNN) was employed to predict the progress of refractive error using retinal fundus images [25]. Gordon et al. [26] applied Conventional Cox proportional hazards regression modeling to assess the risk of glaucoma progression with untreated ocular hypertension and evaluated the influence of long-term intra-ocular pressure (IOP). Jiang et al. [27] utilized a deep temporal sequence, which extracts features using CNN and explores temporal information via long short term memory (LSTM) to predict the progression of posterior capsular opacification. Apart from the progression, predicting the outcome of ophthalmic therapy is another regular application of ML. Wolf-Dieter Vogl et al. [28] applied a longitudinal mixed effects model to predict the outcome of anti-VEGF therapy for patients with exudative macular diseases.

D. AI Limitations and NeSyL
Despite the wide applications of AI in ocular image processing, there are limitations in terms of available datasets and ML models which hinder its implementation in clinical trials. Generally, annotated data obtained from hospitals are limited, specifically in segmentation or detection tasks, where the ground truths require annotations by experts. Sometimes the data is imbalanced with different quality and distributions. Some of the datasets are homogeneous in ethnic groups and lacking generalizability in real world applications. Medical data sets reflect true distributions in real clinical settings, but a minute imbalance may cause poor results toward the minority labeled class by a DL model. DL is the mainstream approach to process ocular images due to its capability of extracting features. However, the black box representation in terms of interpretability, adaptation, and reasoning-ability may lead to poor results [29]. Regarding the aforementioned limitations, this survey discusses the basic principle of NeSyL, and proposes its potential applications in ophthalmic diseases, diagnosis and prediction, and circumventing the limitation of DNNs.

II. PRINCIPLE OF NEURO-SYMBOLIC AI
A. Symbolic AI versus Connectionist AI For many years, there have been two main schools of thought in AI fields, namely symbolic AI (sAI) and connectionist AI (cAI). Hubert and Stuart Dreyfus [33] have defined the correlation between sAI and cAI by phrasing as: making a mind versus modeling the brain. According to human cognitive ability, the creation of the mind (sAI) usually starts at a higher cognitive level through symbolic manipulation and language processing. On the other hand, the creation of the brain (cAI) starts at the lowest cognitive level by developing neurons and their interconnections. Moreover, Kahneman [34] proposed a concept for a binary architecture (called System-1 and System-2) to explain the results of processing speed differences in humans on an assortment of mental tasks. Similarly, the cAI could be classified as System-1, while sAI is considered to belong to System-2. Figure 4 shows the main differences between sAI and cAI.  1) Symbolic AI: The sAI has a list of aliases, including classic AI, rule-based AI, and good old-fashioned AI (GOFA) [35]. From the mid-1950s until the middle 1990s, much of the AI research focused on sAI, which depends on embedding human knowledge and behavioral rules into computer codes [36].
The sAI assumes symbols are the basic units of human intelligence. Similarly, the human cognitive processes are a series of explicit inferences about symbolic representations. From a cognitive point of view, sAI corresponds to analytical operations, which are mostly related to high-level, decomposable, conscious, and reasoning tasks. In analytical operations, there exists a conscious thought process and conscious decisions using explicit high-level information. The knowledge involved in symbolic and propositional knowledge is concerned with language. The sAI and analytical operations are found to share many features such as sequential decomposability and propositional knowledge. The application of sAI processes strings of characters, representing real-world entities or concepts through symbols. All the symbols can be arranged hierarchically or via lists and networks, which dictate the algorithm association among symbols [37].
The sAI system often refers to logical inference, deductive reasoning, and some search algorithms that aim to solve the constraints of a specified model [38]. An expert system belongs to this system, which is based on human-created knowledge and generally contains if/then pairing statements instructing algorithms behaviors. An inference engine model refers to the knowledge-base and selects the rules corresponding to a given symbol or symbol sets. The sAI has been widely used in tasks that have clear-cut rules and goals [39], [40]. However, the algorithmic complexity of discrete reasoning is nondeterministic polynomial time (NP) hard in such algorithms requiring substantial hand-tuning, which made them difficult to address complex problems. The sAI faces difficulty in modeling uncertain and ambiguous knowledge.
The attempt of solving real-world tasks and super-massive spaces is a challenging problem.
2) Connectionist AI: The cAI receives its name from the typical network topology employed in this category. It is the dominant paradigm in the AI field because of the technological successes of connectionism in the presence of large data sets. The cAI models the processes based on how the human brain works via its interconnected neurons, which usually applies a perceptron to represent a single neuron. From a cognitive point of view, it corresponds to synthetic operations, and they are mostly concerned with low-level, unconscious, non-decomposable, and perceptual tasks. Vision and speech recognitions are the typical tasks that involve the knowledge limitations to express verbally and in a non-propositional way. However, the effective use of semantic knowledge can improve CNN performance.
Typically, cAI systems learn associations from large-scale data with no or little prior knowledge. On the contrary, sAI can be built via hand-coded programs by humans.The cAI is composed of all kinds of neural networks, such as CNNs, DNNs, and graph neural network (GNN). The cAI offers good performance in pattern recognition and generalization. However, it often cannot explain how it arrived at a solution, which is highly needed for safety-critical fields such as selfdriving cars and medicine. It also requires a wealth of data to ensure high training and testing accuracies, which is unrealistic for some tasks.
3) Symbolic and Connectionist AI: Many scientists reach an agreement that information at different levels of abstraction differs in structure and makeup. They assume that lower levels of abstraction are subsymbolic and higher levels are symbolic. Hugo Latapie [41] proposed that cognition is not dualistic in this way and that knowledge at any level of abstraction involves neural-symbolic information, in other words, both symbolic and subsymbolic information are contained in data at both levels. This implies the realization of neural-symbolic models as natural mimic machine for information processing, which could cumulate connectionist (cAI) and symbolic (sAI) capabilities.
Neural-symbolic AI or NeSyL could extract features from data using connectionist approaches and then manipulate these features via symbolic approaches. Neural-symbolic AI has proven to converge quickly with only the 10 th part of the training data. Scientists believe that neural-symbolic AI will empower AI with the ability to learn and reason while performing a wide variety of tasks without extensive training. Symbolic component regularizes the neural learning, while neural components help model scaling and guide discrete decisions.

B. Integration Methods: Hybrid Versus Unified
Various methods are used to integrate neural and symbolic AI. Generally, different tasks and researches are suitable for using different integration methods. NeSyL is more difficult to be implemented in parallel computation than traditional neural models due to complex control flow and low-operationalintensity operations. The neural learning of computation dominates the symbolic reasoning when they are clearly separable.
In 1995, Yannic et al. made an effort to lay the cognitive foundation for neurosymbolic integration, and described different strategies employed up to that time [42]. The study [43] identified two main methods of constructing neural symbolic integration models: hybrid and unified.
1) Hybrid Methods: Hybrid methods combine independent submodules, including several neural and symbolic submodules. Each submodule has its own role and performs different functions. Neural submodules are mainly responsible for perceptual tasks, while symbolic submodules are responsible for inference analysis. Translational and functional hybrids are the two hybrid approaches summarized by Hilario et al. [44], in which the functional hybrids have been further classified into chain processing, sub-processing, meta-processing, and coprocessing. Garcez et al. presented their study of hybrid approaches in [45], [46], [47], as early efforts for neurosymbolic integration. In recent years, a number of studies [48], [49], [50], [51] on neuro-symbolic have utilized hybrid approaches. The submodels in hybrid systems can be constructed separately, in which the hybrid methods can directly benefit from advances in neural networks and symbolics. However, hybrid systems may not reflect biological reality [52]. Gary Marcus claims that Hybrids are often effective [53].
2) Unified Methods: Unified methods utilize both connectionism and symbolic ability by adding another paradigm to neural or symbolic approaches. The unified methods are further divided into two cases. The first case integrates symbols into the neural network and the second constructs neural architectures for symbolic processing. A unified approach can also be viewed as an end-to-end approach with exception of a submodel. However, a unified model can be used as submodel in a hybrid system. As Garcez et al. proposed logic belonging to the first case can be thought of as the language for compiling neural networks [54]. The unified architecture of the brain should be equipped with biological plausibility and may lead to discoveries. In addition, unified systems have the advantages of computational efficiency and biological fidelity in massively parallel processing. However, unified approaches are less scalable [52].

C. NeSyL Components
In this section, we describe the components of a neurosymbolic system, which include attention mechanisms, knowledge graphs, and representation methods. They can collaborate to achieve better results.
1) Attention Mechanism: Bahdanau [55] introduced an attention mechanism in machine translation to solve the fixedlength encoding problem successfully and retroactively. The attention mechanism was computed separately for the alignment score, weight, and context vector. The alignment model was designed as a feed-forward neural network, which was trained with other components. The decoder structure was designed to pay extra attention to a certain part of the data set. Thus, the alignment mechanism increases the accuracy of prediction facilitated by the attention mechanism to focus only on the relevant contents.
It is important to remove irrelevant terms from the symbolic reasoning process where the excessive number of log-ical combinations makes computation challenging. Symbolic distillation and relaxation of symbolic programs into neural networks convert combinatorial search over symbolic code to scalable gradient-based techniques. Distillation emerges as an alternative solution for learning symbolic and neurosymbolic programs. Latapie [41] applied an attentional mechanism to neuro-symbolic reasoning to filter huge combinations of data and realize parallel calculations of different combinations.
2) Knowledge Graphs: The graph is the associated link of all elements or vertices. The knowledge base and the reasoning machine jointly form a knowledge system. The graph can serve as a knowledge base to store knowledge and perform reasoning [56]. Graphs link these entities by mimicking the connections of neurons in the brain. Neurosymbolic systems also have this capability, benefiting from knowledge graphs as a bionic technology. Graphs enable AI to reason and learn based on common sense, prior knowledge, and statistics. For a graph-based AI, new knowledge will be integrated into the larger graph via logics. Such models can learn new knowledge continuously even after completing training which saves computations in large systems [57]. In addition, the combination of the attention mechanism and the graph is designed to compute independent relationships among neighboring nodes' weights of a graph [58].
3) Representation Methods: Knowledge representation is considered to be one of the core concepts of AI, which combines objects, logical representation, and semantic networks [59]. Logic represents a rule without ambiguity such as propositional logic, production rules, frames, and predictive logic. Programming languages are designed based on logical representations which allow the logical deduction. Ambiguity in natural language may lead to unclear representations and it is difficult to correspond to logical representations [60]. The knowledge can also be represented via semantic network to classify objects and represent interrelationships through wires to bring ease to network expansion. The gap between semantic networks and knowledge graphs need to be filled for a better knowledge representation [57].
On the other hand, knowledge representation has two routes, localist and distributed representation [54]. The localist representation has advantages of interpretation and high computational efficiency on simple models. However, it has obvious disadvantages of loss relevant features to be encoded as oneheat codes. Distributed representation is more conducive to generate dense data, which is favorable to improve the computational efficiency of complex features. Most importantly, it preserves the correlation between features [61]. Distributed representation is applied to compute similarity and supplement the knowledge base distributed representation [61]. Distributed representation has been widely used in natural languages and entities [62], [63]. The structured embedding (SE) [54], single layer model (SLM) [64], semantic matching energy (SME) [65], [66], latent factor model (LFM) [67], [68], and neural tensor network (NTN) [64] have been developed to implement the knowledge representation. Compared with knowledge graph, knowledge representation can save computational cost and improve accuracy [61]. The knowledge representation potential will be further unleashed after combining with neural networks.
4) Decision Rules: The decision rules are an indispensable component in the incorporation of knowledge representation. Decision rules are the If-The statements with constraints that have similarities to natural language and is suitable for interpretation. To avoid overlap and rule missing, decision lists and decision sets are proposed to solve the combination problem [69]. The rules in the decision set are given different weights based on properties such as accuracy. Algorithmic processing of the data can produce new rules. The OneR algorithm [70] uses the entire dataset to learn and obtain multiple rules according to the produced error. In addition, suitable intervals are used to divide the continuous values to extract discrete features. However, for a dataset containing multiple rules, OneR is not applicable. Cohen [65] proposed RIPPER as an improvement on sequential covering, which applied rule pruning for optimization [69]. Bayesian Rule Lists are another way of rule learning. The first step of this approach applies pre-mining of frequent patterns using frequent pattern (FP) growth or Apriori [71]. There are other algorithms used to generate decision rules [72]. Decision rules have been used together with deep learning [73]. Decision rules are considered suitable for interpretation and fast prediction but may lack regression problems [69].
NySyL Components are closely related to the integration methods of Neuro-symbolic AI ( Figure 5). Components such as attention mechanisms and knowledge graphs can be used in each submodule in hybrid integration methods. In the unified methods, such components can be added to another paradigm to form single neural or symbolic arithmetic.

D. Task Adaptive NeSyL Structure
Various categorizations have emerged trying to classify neuro-symbolic structures. In this section, we present the most widely accepted scenarios followed by some leading examples in the respective field. The first approach to neuro-symbolic taxonomy is presented in [74] where 5 different structures are illustrated in Tabl II.
Another approach to taxonomy has common characteristics with the illustrated one (Table II) and is largely accepted by the literature [54], [76], [77], [78] (Figure 6). Pointer networks or Graph Neural Networks (GNNs) may be deployed to solve graph problems with attention mechanisms ( Figure 6). We provide an overview of the 6 structures in Table III.

E. Proposed Large Frameworks
This section presents frameworks that have attracted attention as building components in NeSyL. Among these frameworks, DeepProbLog is a probabilistic logic programming language that uses neural predicates to accomplish deep learning [79]. DeepProbLog stems from ProbLog, while maintaining all its functionalities [80]. Moreover, the pure neural network, logical and probabilistic methods are presented to allow varieties of framework combinations [81].
The Neuro-symbolic concept learner is a model that learns visual concepts by parsing sentences without explicit supervision [48]. The induction approach extracts novel concepts  The input is first processed by neural components and then fed to either logic or functional symbolic codes. The Houdini is an example of such a concept, which consists of a symbolic program synthesizer that searches over a library of functions and a neural module to train these functions [75]. Neural-aftersymbolic pipeline The input first passes via a set of synthesized program components followed by a neural network. Neural module networks Several neural modules appear as components to provide high-level functionality (network layout predictor, modular meta-learning). Algebraic compositions Neural and symbolic programs run independently and then results are combined with algebraic operators such as control regularized reinforcement learning (CORE-RL) and imitation-projected programmatic reinforcement learning (PROPEL). Neurally accelerated symbolic programs Machine learning is used to guide the execution of a program without affecting its functionality and improved its efficiency (safe learned controllers). from noisy perceptual experience as by mimicking childrens' ability to label objects without being explicitly taught. In such concept, first an object-based scene representation is built together with translated sentences in parallel into executable symbolic programs. These two modules are integrated by a neurosymbolic reasoning module. The structured representation following a transfer learning might be applicable to different tasks and the replacement of attributes with placeholders allows new attributes to be learnt [82].
Neuro symbolic forward reasoner (NSFR) is a framework for reasoning tasks using first-order logic and converts raw input into object-centric representations. For differential forward-chaining, weighted rules follow to generat probabilis- All symbols are converted to vector embeddings followed by the processing of neural models and produces symbol as the standard deep learning procedure. Symbolic [Neuro] Refers to symbolic solvers, such as the Monte Carlo tree search, that use neural models as subroutines, e.g., neural state estimator. Neuro; Symbolic Neural and symbolic systems are assigned different tasks in the parallel pipeline and communicate the extracted information to improve the individual or collective systems performance (e.g. neuro-symbolic concept learner, deepProbLog). Neuro:Symbolic → Neuro The symbolic knowledge is compiled into a neural network model. Various forms of symbolic knowledge is embedded in the neural network by assigning its initial weights. NeuroSymbolic A symbolic logic rules transformation regularize the network's loss function. Such rich transformation and logic embeddings are present in Logic Tensor Networks (LTNs) and Tensor Product Representations.

Neuro [Symbolic]
The neural network model performs combinatorial symbolic reasoning. This is achieved either by learning the relations between the symbols or by paying attention to selected symbols at a certain point (e.g. GNNs). tic logical atoms [83]. The whole process is divided into three components: I) the object-centric perception module, ii) the facts' converter, and iii) the differentiable reasoning module. In this way, a new set of atoms can be deduced from given facts.
Commonsense transformers (COMET) is a generative model based on commonsense knowledge that learns to generate commonsense descriptions in natural language resulting in the construction of a knowledge base(KB) [84]. COMET extracts KB structure, relations, and representations among nodes and edges to seed the knowledge to a graph.
Bidirectional encoder representations from transformers (BERT) is a language representation model, which is designed to pre-train deep bidirectional (both left and right context) representations from unlabeled text, and then fine tunes with labeled data [85]. This makes BERT appropriate for various tasks. During training and fine-tuning the same architecture is used except for the last layer. Another important feature is the questions answering representation into token sequence which makes it appropriate for visual question answering (VQA) applications.
Logic tensor networks (LTNs) is a statistical relational learning (SRL) framework that applies real logic constants and interprets function symbols as real-valued functions and fuzzy logic relation [86]. Therefore, the soft and hard constraints on real-valued data can be presented in first-order logic to improve learning constraints [87].
Graph neural networks (GNNs) generally learn an embed-ding that maps nodes and edges into a continuous vector space with limitations on the number of vertices in the input graph [77]. This approach has been enriched with attention mechanisms. For instance, a multi-hop attention graph neural network (MAGNA) first computes the attention scores on all edges and then the attention diffusion module assigns attention scores between pairs of nodes that are not directly connected [58].

III. REVISITING CNN THROUGH NESYL A. Ocular Applications of Neural Network
The most widely used image recognition methods such as CNN [88], [89], GCN [90] can be used to identify, diagnose and infer from ocular images. For diagnosing glaucoma, a double tier deep convolutional neural network was used with convincing accuracy rate of 92.64% [88]. Nusinovici et al. [89] used a deep learning approach to predict biological age based on retinal photographs. Upadhyay et al [91] used coherent convolutional neural networks to detect retinal disease on optical coherence tomography images. Neural network-based methods have the advantages of availability, robustness, and accuracy.
Neural Network and Deep Learning are very popular these days for their powerful magic in solving varieties of problems; however, uninterpretability and explainability like challenges still need to be addressed. Although by complex network architecture and computational approaches, DL models can produce a viable solution to many challenging problems; however, the latent space under the jillion numbers in the latent space is unexplainable to humans. It is inevitable to exploit the principles behind the results produced in the legal domain. Good old fashioned AI (GOFAI), also known as symbolic AI, is an interpretable and generalized concept equipped with reasoning capability. However, the GOFAI needs to manually establish the knowledge dataset in the representation form of symbols. Thus, it is very natural to bring these two concepts together into NeSyL.

B. Potential Superiority of NeSyL over Classical Deep Learning
NeSyL can be applied to many fields. Compared with the classical DNN, NeSyL has the advantages of lower cost, better generalization and robustness, interpretation and explainablity, efficient training, and mimicking human cognitive ability.
Several efforts [92] have already been made to demonstrate that the loss function matters for the generalization abilities of DNN. Some researchers have used the symbolic reasoning knowledge to optimize the learning of the DNN and derive a better loss function [93]. Therefore, applying a NeSyL before the DNN to choose the loss function may help improve the performance of the entire system. A GNN may not compute important graph properties without additional information [94]. Common methods like optimizing the structure or adding additional message such as labeling can be a huge cost in DL. CNN model can be pruned by specializing the polynomial to train the regression more effectively [95], which is similar to synthesis in the NeSyL. In comparison, NeSyL has more advantages for its low cost in labeling. Researches have pointed out that both the depth and the width of a network matter in the deep learning, which makes it harder to balance the time and resource consumption [96]. Compared with CNN, the NeSyL needs few resources and can achieve satisfying performance in many situations.
The study [91] proposed that DNNs are vulnerable to adversarial perturbations which can be solved by considering domain knowledge such as road signs in autonomous vehicles. NeSyL can produce a robust system by applying the knowledge enhancement strategy. A system generalization is also a well-worth paying attention sector; however, both generalization and domain adaptation is hard to jointly improve [97]. Henceforth, deep learning has its limits in domain adaptation and representational learning, where the NeSyL can perform much better. There is also a problem of overfitting in deep learning networks which weaken the robustness to adversarial attacks [98]. It is hard to explain the observed overfitting. NeSyL maybe helpful in the researching fields for its advantages such as domain adaptation, interpretations, and auto reasoning.
Moreover, large scale pretraining networks have been leveraged to do the neural symbolic regression and satisfying performance in re-discovering physical equations [99]. Equivariant network, a promising fusion of NeSyL and image processing is a capable model to improve the performance and efficiency of the symmetrical neural network in computer vision and medical imaging which can be extended to wide applications [100]. An overlap can be observed for the top 10 research fields of CNN and the NeSyL, such as Computer Science, Engineering and Mathematical Computational Biology ( Figure 7). However, efforts have been made on CNN while the NeSyL received relatively less attention. Considering the advantages over traditional networks, more attention is needed for in-depth research in NeSyL.

C. Image Processing via NeSyL
The CNN models contributed tremendously in image processing during the last decade specifically in medical imaging and ocular diseases diagnosis. On the other hand, NeSyL integrates expert knowledge and adds explainability to ensure transparent decision system to humans. A transparent system in clinical domain becomes a requirement; thus, the research potential and progress in the field of NeSyL is replacing CNN in ocular images. An explainable and interpretable NeSyL method was utilized for diabetic retinopathy classification. They used a U-Net segmentation network to extract humanreadable symbolic representation proceeded to a fully connected network for decision making purposes.
The NeSyL model obtained comparable performance to state-of-the-art methods on Indian diabetic retinopathy image dataset (IDRiD) featured with achieving interpretability and explainability. To combine the representative power of deep learning and the interpretable capabilities of symbols, Chowdhury et al. [101] demonstrated an emergent language based classification framework that involved a sender, generator, and a receiver for medical image classification. The sender extracts feature representations and feeds into a symbol generator to generate symbols. The receiver accepts generated symbols to perform the classification. Han et al. [102] provided a novel method for generating spinal medical report automatically. They embedded a symbolic graph based reasoning module into an adversarial network for semantic segmentation of spinal structures. Based on these outputs from neural learning, the framework conducted causal effect analysis to detect abnormalities through symbolic-logical reasoning and generated spinal medical reports. The pool of concepts influences the semantic information contained in the system and logical reasoning.
To tackle this problem, Diaz-Rodriguez et al. [103] leveraged symbolic representations and presented the explainable NeSyL (X-NeSyL) methodology to fuse deep learning representations with expert domain knowledge which offers faced images classification and architectural element annotations. Alirezaie et al. [104] designed a semantic referee which extracts features from errors produced by a deep convolutional network as additional image channels to assist miss classification correction. Although, neural symbolic methods perform impressively in visual question answering on synthetic images, they suffer from the long-tail distribution of visual concepts and unequal reasoning st.pdf which are two challenges in real images. Li et al. [105] proposed a paradigm, calibrating concepts and operations (CCO), which contained an executor to capture the underlying data characteristics for handling distribution imbalance and an operation weight predictor to highlight important operations and suppress redundant operations.
Requiring large scale, high quality, and labeled datasets are the hurdles of deep learning applications. To train an end-to-end neural-to-symbolic model, Agarwal et al. [106] constructed a neural-symbolic-neural architecture. The model composed of three st.pdf providing a new fashioned way for image construction. Manigrasso et al. [107] utilized logic tensor networks (LTNs) and a neural-symbolic technique to impose logical constraints on the training of the convolutional layers, which made the framework no longer purely datadriven.
The practical framework of neural symbols is summarized as shown in Figure 8. The symbolic learning module embedded in the neural network can help .loklinterpretability and expalinability. Placing neural network structure before symbolic learning can make the system more robust in learning.

Encoder
Decoder

D. The Possibility of Ocular Applications Using Neurosymbolic AI
Some efforts [108], [109], [48], [110], [111], [112] have been made to use NeSyL to assist deep learning representation to process VQA , bringing more transparency and explainability to the reasoning process.The neuro-symbolic visual question answering (NS-VQA) [108] based model composed of three parts, scene parser, question parser, and program executor, to disentangle the reasoning of visual perception and textual comprehension. The core part designs symbolic programs to reason both visual and textual representations obtained through deep learning and question-answering in ocular clinical scenarios. The knowledge reasoning with implicit and symbolic rePresentations (KRISP) was proposed by [109] to adopt two branches of reasoning, one is to integrate explicit symbolic knowledge into the knowledge graph for graph-based inference, and the other is to complete the supervised implicit knowledge. The look, read, think, answer (LRTA) [110] treats the VQA task as a complete answer-generating question, proposing an end-to-end trainable modular framework that is not just a superficial guess to answer the question, but highly rationalizes throughout the process.
In addition, NeSyL plays a similar role in other reasoning tasks [113], [114], [115], [100], [116]. The probabilistic abduction and execution (PrAE) Learner [113] uses a NeSyL concept to process Raven's progressive matrices (RPM) like spatiotemporal inference tasks and to design probabilistic abductions to generate answers like humans. A generative neuro symbolic (GNS) model was proposed by [114] to utilize compositional and causal knowledge of NeSyL to generate task-general representations of handwritten character concepts. Through the above research, it can be found that combined with interpretable NeSyL, deep representation learning can provide more transparent processes of reasoning while achieving satisfactory performance. Specifically, deep learning modules tend to be responsible for generating certain tasks followed by symbolic learning to complete the entire reasoning process and make it interpretable to human.
The neuro-symbolic concept learning (NS-CL) [48] built on object-based scene representation and translation into an executable symbolic program can learn visual concepts and semantic information without explicit supervision. Neural networks build a knowledge base instead of manually extracting features, and the amount of data is only 10% of that of neural networks. This outcome makes the neural notation accessible on images including all types of ocular images.

IV. INNOVATIVE IDEAS OR FUTURE DIRECTIONS
NeSyL integrates contemporary neural network-based models with features of symbolic learning and reasoning. The research community is intrigued about how to make interpretable neural network models that transcend the black-box representation by incorporating properties like interpretability and explainability. This section offers novel concepts that may be applied to many fields in real-world situations. These concepts are both generic and specialized. This part is divided into two subsections: NeSyL-based generalized in clinical settings including medical imaging and the diagnosis of ocular diseases.

A. Domain Specific NeSyL for Medical Imaging
Given the challenges in neural computing in terms of transparency, interpretability, explainability, safety and trust, perturbation attack, and general robustness, bring commonsense to machine learning which is unavoidable for a reliable neural learning. In biomedical science, health, therapeutic practices, and reasoning in machine learning are becoming increasingly relevant.
1) NeSyL for Multi-class and Fine-grained Classification of Ocular Diseases: NeSyL applications in clinical practices is advantageous considering the interpretability and explainability. It can be broadened to various medical imaging including fundus images, which can be utilized to diagnose up to 50 disease where each disease may have multiple subtypes. Among these disease, retina image classification is critical considering the subtle differences between sub-categories and large intra-class differences [117]. The signal-to-noise ratio of fine-grained images is very small where the information containing sufficient discrimination often only exists in very small local areas. Effective utilization of local area information has become the key to the success of fine-grained image classification algorithms. Moreover, the medical imaging, especially in the field of ocular diseases, often demands the involvement of the expertise for an expensive task of objects labeling [118].
The fine-grained image classification extracts objects with clear object shapes, outlines, appearance, and requirements of large scale reliable dataset. However, for objects with unclear contours and no obvious components, such as extracted lesions with different grades and minor differences, neural network algorithms may not achieve satisfactory performance or interpretability. On the other hand, NeSyL not only extract features via Connectionist AI methods, but also manipulate features using symbolic AI methods. The introduction of attention mechanisms address more in depth processing of extracted features, but still lack symbolic reasoning toward interpretablity and explainability models. Therefore, NeSyL has shown better performance, while requiring a small amount of training data, and the capability of learning and reasoning in performing distinct tasks. Thus, NeSyL has good application prospects in medical images specifically in the field of ocular disease classification.
2) Knowledge as Symbolics Regarding Clinical Practices: Knowledge-driven platforms for health-care decision-making are useful tools for giving actionable and intelligible information to patients and clinicians. Building knowledge into learning algorithm improves data efficiency and learning rates. Hierarchical knowledge or learning group sequence of related low-level actions into subgoals, where tasks are abstracted into subgoals for efficient learning. A capsule network can be employed to capture hierarchical relations and feature selection. To bring facilitation and reasoning about a patient's health, it is inevitable to integrate and utilize the domain knowledge to enable personalized health care system [119]. Symbolic information representation into domain knowledge can be focused to areas such as patient personal knowledge. The Patient personalized knowledge representation as a graph (PKG) encapsulates relevant knowledge about a patient. For instance, regarding a diabetic retinopathy patient, the patients' health, food, diet, lifestyle, and history are needed to model the health care decision system. The PKG about a person health's can be captured from various sources [120]. The PKG in case of patient health can capture knowledge about patient history including allergies, eating habits, and other related diseases.
Given the PKG as domain knowledge about a patient's health history through symbolics, a neural network could be trained to reason about the health informatics such as meal recommendation, allergic restriction, and carbohydrate prescription (Figure 9). The PKG knowledge can be translated into a graph or hypergraph prior to integration (symbolics) and inference with the neural network while training. The trained NeSyL network may not only predict patients diagnosing but also prescribe health care. It can be challenging to collect personal health information, linking or embedding such information to neural network, and most importantly associating against the clinical statistics of the same patient to map the full health history. Using machine learning in isolation may not reflect the efficient utilization of the domain knowledge. Thus, NeSyL can better inference, reason, and provide a health decision system akin to human clinicians. Fig. 9: Personal knowledge as symbolic representation into an encoding-decoding structure of a neural network. The reconstructed, generated, and segmented image can be reasoned for decision making purpose.
3) NeSyL Parallelism with Synthetic Biology: Synthetical biology transforms the technique of functional biomaterial production, leading to the development of new classes of inexpensive, and rapidly deployable diagnostics. AI has been incorporated both in NeSyL and Synthetic biology (synbio). In genetics, AI has been employed to annotate protein function, heterologous genes optimization, estimation of plasmid expression, genotype-phenotype understanding and association. Furthermore, synthetic biology aims to control complex biological system by leveraging engineered principle and rules. The engineered rules standardize the genetic parts to achieve a projected outcome. Similarly, NeSyL utilizes, presents, integrates, and enables inferencing of symbolic knowledge in terms of domain knowledge with neural architecture. A veriety of AI approaches have been employed to Synbio such as multiagent system, expert system, heuristic search, and constrained based reasoning [121], [122], [123]. An attempt has been made to incorporate biological or domain knowledge into graph neural network [124].
Following the similarities of Synbio and NeSyL, a joint model can be built to reason at the genetic information level by utilizing biological knowledge and neural computing. This idea will not only engineer the genetic information but also has the potential to apply inferencing such as protein-protein interaction, DNA and RNA sequencing, identifying proteins functional sites, and the construction of new biological designs.
Furthermore, there are a number of challenges that could be faced while modeling by incorporating symbolics into neural computing, for instance, dataset (lack of dataset to merge AI into synthetic biology, omics dataset), no robust algorithms (to capture system dynamics in high dimensional space), metrics (to account underlying biological system), and so on. NeSyL is the concept of imitating the human inferencing system while synbio deals with modifying genetics, thus their mutual collaborations would be an interesting future research direction. AI can help bioengineering, while in reverse, bioengineering principles and findings can help AI. Thus, by deploying NeSyL in synthetic biology, a gene can be modified at a gnomic level and inference for our desire outcome (cell self-repair and self-replication). A generalized model has been depicted in Figure 10, in which the miRNA synthesis and neural network (NN) perform in parallel as siamese networks. The miRNA editing utilizes biological knowledge which may compose a sequence of stages. Similarly, a neural network that is inspired by biological learning can tune network parameters for similar fed domain knowledge. Both the parallel systems can be used for reasoning and inferencing toward an optimized NeSyL approach. For instance, NN and Micro-RNA synthesis can mutually benefit from each other simultaneously and finally contribute to biosynthesis in genetic editing, reasoning, and inferencing neural networks by forming a NeSyL platform.

4) Diseases Diagnosis, Prognosis, and Treatment:
Explainable machine learning approach (ExplainDR) has been used to diagnose ocular disease (i.e., Diabetic Retinopathty) [125]. A generalized proposed architecture for diagnosing ocular diseases is shown in Figure 11. The study used static vector for symbolic information given 4 eye conditions (severity level) and statistically decides segmented lesion sizes (small, medium, and large). The accuracy increased from 47% to 64% given 4 to 12 dimensions of symbolic features.
The increases have been observed in accuracy corresponding to the growing number of symbolic features. Therefore, additional feature dimensions inclusion may make the framework significant; nevertheless, this may complicate the symbolic reasoning. In medical image diagnosis, prognosis, and treatment, the domain (expert) knowledge has much importance to be infused with the neural network at the different possible levels of integrations. The domain knowledge can be incorporated into the latent space and generating segmented regions simultaneously with symbolic information with reasoning capacity [126], [127].
In medical imaging practices, the acquisition of detailed information is indispensable for clinics in order to diagnose and treat diseases. Biomarkers or lesions could require detection, segmentation, classification, and spatial information extraction to correctly retrieve the information of interest. This may demand the employment of multi-tasking. The multi-tasking in this case would need to utilize the domain knowledge and symbolic reasoning to predict accurate results. Similarly, the segmented lesions may have variations in contrast and intensity that represent symptoms and their correlation is necessary for prognosis and treatment. The embedding of domain knowledge in terms of symbols needs special attention in neural computing, for instance, glaucoma estimation not only lies in cup-to-disk-ratio (CDR) but factors, such as age, race, and family history can also be utilized to carry out symbolic reasoning. 5) Tele-Medicine: Telemedicine, especially in the field of ophthalmology, can improve the sensitivity of eye disease screening in communities or primary care settings and provide care to patients in resource-depleted areas [128]. Over the past few decades, there have been numerous teleophthalmology pilot projects around the world, many advanced vision algorithms can be introduced into this area given ophthalmology data as images. For instance, supervised and unsupervised ML has achieved new breakthroughs in automated glaucoma screening, proving the feasibility of telemedicine detection and management.
The safety of remote triage in emergency ophthalmology remains to be explained and improved. An early study showed that 1% of 500 patients undergoing teletriage in the emergency department experienced delays in treatment due to misdiagnosis [128]. A more accurate description on potential harms should be made before teletriage is widely available and establish mechanisms to mitigate the disadvantages of remote censorship. The advantage of NeSyL is not only to maximize the efficiency of the network but also to use symbolic reasoning to improve the interpretability of pathology and make the model prediction more accurate, which has far-reaching significance for ophthalmic remote triage research. Figure 12 shows a tele-ophthalmic medical diagnosis system that is based on NeSyL, and the trained model can achieve more accurate diagnosis with the advantages of neural network and symbolism (NeSyL). The pathological data of patients is obtained online, and the trained model makes a detailed diagnosis conclusion, then the clinician supervises its accuracy, which greatly reduces the miss-diagnosis rate.

1) Addressing ML and DL Challenges in NeSyL:
This section covers and leads experiments using NeSyL to address issues in recent machine and deep learning approaches.
Addressing black-box challenge through NeSyL: The ability to comprehend a machine learning model's prediction Fig. 11: The ExplainDR utilizes a feature vector as a symbolic representation and achieves explainable and interpretable DR classification. The end-to-end architecture receives retinal images into a U-Net based segmentation network to generate four ouput images for each eye's condition. Then a quantization technique is designed to extract symbolic representation followed by a fully connected network to predict decision.

Hospital Database
Neural-symbolic...  builds user trust and aids understanding of the underlying processes. In terms of interpretability and reasoning, one of the major challenges faced by ML and DL-based modeling is the lack of a white-box idea. Model interpretability is important in a variety of application domains, including natural sciences, where understanding and prediction are important for decisionmaking [129], [130], [131]. In this regard, some studies have been carried out to convert the black-box challenging aspect to a white-box characteristic in the form of visible mathematical equations that human subjects can readily understand and evaluate [29].

Diagnosis
Optimization: Varieties of approaches have been proposed to optimize a network parameters and hyper parameters [132]. In discrete spaces where comprehensive enumeration is impossible, combinatorial optimization of symbolic learning seeks to identify optimal configurations. Nonetheless, despite a worst case exponential time complexity, there is a wide range of accurate combinatorial optimization algorithms that are guaranteed to discover an optimal solution [133]. Branchand-bound, which is a classic precise approach for solving mixed-integer linear systems, was reformulated as a Markov decision process [134]. In this context, the study proposed and tested a novel approach for tackling the branching problem that involves expressing the state of the branch-and-bound process as a bipartite graph, which eliminates the need for feature engineering by leveraging the variable constraint structure of MLP problems naturally, and allows for the encoding of branching policies as a graph. A Knowledge distillation model might be employed to reduce the NeSyL model's complexity so that it can be efficiently fine-tuned.
The study in [135] proposed a bayesian optimization method for combinatorial search spaces. Most of the baysian optimization methods are focusing on continuous rather than combinatorial search spaces [136]. The innovated frameworks transform discrete constraints into smooth functions that are differentiable and optimized using gradient descent. To efficiently tackle the exponentially increasing complexity of combinatorial search space, the study [135] represented the search space as the combinatorial graph by combining subgraphs given to all combinatorial variables using the graph cartesian product. Moreover, the combinatorial graph reflects a natural metric on categorical choices (Hamming distance) when all combinatorial variables are categorical. Graph neural networks can be used to solve combinatorial optimization problems with a relaxation strategy to the hamiltonian problem to generate a differentiable loss function with which they train the graph neural network.
Conditioning on Neural Network: Conditioning in the neural network aims to derive the process or signal among many forms toward the intended direction. The conditioning on neural networks have been carried out in varieties of studies, such as text-to-speech [137], style generation [138], attention mechanism [139], literature generation, and text-toimage synthesis [140]. Humans innate ability for reasoning in terms of conditioning, such as textual description based visualization, is attempted in neural networking [140]. A hypergraph based conditioning can be applied at different levels, such as sentence-level and word-level conditioning for text-to-image synthesis, to bring flexibility in generation. The embedding of hypergraph as conditioning facilitates both the continuous and discrete image generation beyond the training dimensions. Furthermore, meta internal learning is deployed for image synthesis and generation (GAN) as condition on the network statistics, such as one-hot vector, vector signa-ture, conditional signal, style as condition, and conditioning hypernetwork (rather than a primary network) [141]. Similarly, hypernetwork based variational autoencoder (VAE) has been conditionalized for self driving vehicle to predict the stochastic behavior of the multi-dimensional agent in novel environment featured with context driven and generalized scenarios [142].
It is a genuine source to reason the neural network learning by imposing distinct conditions toward the intended goal. For example, the text-to-image synthesis which is crucial to learn from unstructured descriptions and encoding different statistical properties of language inputs. The future work can be aimed to apply controllable conditional (both discrete and continuous) based generation given hyper network parameters [140]. Furthermore, improvement can be brought by incorporating adversarial hypernetwork and meta learning such as few-shot learning [141]. Implicit learning can be one of the future directions, in which the overall solution is based on sub-solutions, for reasoning in machine learning models [143] Discussion on Black-box to White-box Conversion, Optimization, and Conditioning: Regarding interpretability and explainability toward white-box learning, we proposed a generalized approach as depicted in Figure 13. The proposed model is based on NeSyL which address the hindrance of machine learning in terms of black-box representation. Overall, the model's structure has been divided into two layers (which can be extended as per requirement) including deeper layer (neural computation) and upper layer (symbolic learning). Each submodule in the deeper level receives input (can be symbolics) and learns parameters in neural network architecture and then produces output to a symbolic learning (SL) unit. The proposed approach deals the black-box to white-box interpretation by incorporating optimization and conditioning (BWOC). The key idea is to divide the whole end-to-end ML platform into submodules infused with symbolics at different levels. The architecture of BWOC carries out NeSyL at low level to overcome the abstraction and enable understanding neural network to achieve white-box architecture. The BWOC still has the abstraction up to some extent to address the phenomena like adversarial attack. Thus, BOWC provides the opportunity to include and exclude modules given their performances while learning in a particular domain.
The BOWC optimizes the parameters both at local and global levels ( Figure 13). The local loss (LL) enables the optimization at particular module when realized to be higher than fellow modules. Similarly, the model also cumulatively computes the loss function and optimizes in an end-to-end manner. The global optimization can be carried out by selecting a module or number of modules that have high a contribution to performance. The selection of the optimized module can be performed using genetic programming (GP). Meijer G-function can be used for selection purposes given symbolic expressions such as polynomial, arithmetic, algebraic, closedform, and analytic expressions, as well as some special function including hypergeometric and bessel functions. One of the key aspects of the model is the path selection (PS), in which a global optimized path to be selected while training the model.
The third key aspect of the proposed model is the customization which can be in terms of embedding information and applying attention mechanism (AM). For the natural language processing, the embedding can be carried out using word, sentence, and structural levels. The AM can be employed both at global and local levels. For global level attention, particular module with low performance can be reparameterized given the learned knowledge by the optimized module. The PS can facilitate global attention mechanism while choosing a robust path. The local attention mechanism (LA) penalizes and prioritizes particular neuron(s) in a submodule level (see red colored neurons in M 1 and M 3 of Figure 13). In this way, the BOWC module following NeSyL concept exploits black-box learning, enable optimization at different levels, and infusion of different conditions toward a robust model.
2) Symbolic Reasoning: NeSyL is mainly focused on the typical machine learning problems with employing reasoning at the symbolic levels. There is no systematic way for reasoning which need more attention to reach a maturity level. It is necessary to propose innovations in reasoning by achieving human-level reasoning. A number of reasoning methods have been proposed, such as object-centric [51] (reasoning about object and its attribute), logical reasoning [144] (combining expressive NN with symbolic reasoning), neuro-symbolic forward reasoning-NSFR [83] (differentiable forward-chaining using first-order logic reasoning), common sense reasoning [145], and visual question answering [146].
The object centric suffers from the complexity of performing low-level visual perception while reasoning on upper or higher level, which has been addressed by employing logical reasoning (symbolic representation) with deep learning [144]. A dynamic concept learner approach has been deployed to not only capture features about objects but also to track trajectories of each object movement [51]. Similarly, adaptive neurosymbolic network adapts to its environment by building symbols based on perceptual sequences.
For building intelligent systems, intuitive learning (intuitive physics) has been utilized for machine cognition to reason with the physical world having incomplete information [147]. The reasoning at physical level can be divided into prediction, inferences, and causal. The physical level reasoning use DL modeling to outcome physical trajectories, dynamics, and objects properties. In inferencing, the observed (e.g., size, color, and shape), latent properties (e.g., mass, friction, velocity, displacement), and inter related in the dynamics such as collisions, bouncing balls, and objects moving to utilize to reason and inference. The generation of events, movements, dynamics, and unseen future frames can be forecasted given the initial dynamics.
The machine cognition and reasoning with uncertain physical and incomplete physical world and dynamics, a bayesian symbolic framework (BSP) has been proposed [148]. The BSP approach combines the sample efficiency of symbolic methods with the accuracy and generalization of data-driven approaches to incorporate the learning and reasoning (like a human) from incomplete information. In this regard, parsetree-guided reasoning network achieved global reasoning on a dependency tree parsed from the question capable of building an interpretable VQA system that gradually derives image cues following question-driven parse-tree reasoning. The existing methods in intuitive reasoning do not generalize from one scenario to another, for instance, the generalization of two objects collision to multiple objects' collision simultaneously. Similarly, in intuitive physics, the inferencing of latent properties (mass, density, friction, energy, velocity, displacement, etc.) is challenging to extract features from the observed properties (size, shape, and color) by human intervention. Thus, the mapping or derivation of latent properties from the physical world need to be modeled. The inferencing can be modeled by separating visual properties from dynamic properties in the physical world. Similarly, for inferencing, a bayesian symbolic framework has been proposed which combines inferencing with symbolic regression to address the uncertainties in physical dynamics [148].
For future studies of reasoning and inferencing specifically in intuitive physics and cognition, understanding dynamics (e.g., objects over time, collision, direction, bouncing, gravity, etc.,) and their proper representation (e.g., graph [149]) are inevitable [150]. To understand the physics of a scene, it is necessary to reveal the compositional hidden properties, such as mass and charge, and to utilize these factors to improve the symbolic reasoning. In the physical world, the information may be incomplete to model dynamics. Therefore, to understand an environment, all the newtonian forces, physics laws (learnable symbolic expressions using meaningful law), formulation, and entities forces on each other, together with data driven (DL) can be integrated and reasoned about incomplete or intuitive information.
Most of the current reasoning approaches have been applied on images and videos which can be extended to 3D scenarios with sophisticated algorithms. Similarly, neural network can be used for reasoning considering its strong backbone for reasoning such as dynamic computational graph [51]. The reasoning is naturally linked to language, which makes natural language processing (NLP) appropriate for NeSyL. However, whether NLP and NeSyL mutually benefit from each other is still an open question.
Therefore, we proposed a generalized NeSyL based model to intuitively reason and inference from the physical world via symbolic learning toward incomplete, hidden, latent, and intuitive information (Figure 14). The proposed approach receives visual information from physical scenarios and perceives knowledge in a neural network in terms of weights learning. The neural part of the model generates facts given observed information acquired in the physical scenes. Typically, a physical world may contain uncleared and incomplete information for deducing the dynamics. Similarly, a scenario cannot be generalized to other scenarios. The proposed model receives additional information as symbolics in terms of natural language, clauses, ground truths (observed information), and dynamics passed through a parser into a tensor representation. Such representation facilitates the model prediction for intuitive information and also optimizes the weights. The model finally predicts both kind of observed and derived information given the factual visuals. Most importantly, the NeSyL solving approach to incomplete and intuitive information can be extended into medical practices and opthalmology with attaining life saving results.
3) NeSyL for Interactive and Critical Systems: Despite of the wide range of applications and efficiency, DL models can still be fragile and sensitive to given data. In the adversarial attack, adding imperceptibility noise to an image can easily fool a DL based system, such as by classifying a gibbon as panda [151], fooling adversarial network with different lightnings and orientations [152], dodging face detection with stealthy attacks [153], hiding in the human eye as graffiti [154], and sticker and patch adaption based network tricking [155]. Such vulnerabilities of generative models may get worse in a real-time and interactive system where a minor error can lead to huge consequences. For instance, a model for autonomous cars trained on the road side visuals may be mislead by observing perturbation on the perceived visuals. Some of the autonomous vehicle established in industries, including Tesla (HydraNet), Waymo (ChauffeurNet), and Ford/Volkswagen Group (Argo AI) are avoiding end-to-end system to better identify and fix problems while dealing with challenging environment.
Therefore, the futuristic autonomous vehicle can be equipped by combining typical machine learning with tracking and commonsense facility. A number of studies have been attempted to combine sensory information, objects recognition, and tracking with symbolic learning which has improved the object detection from 0.31 intersection-over-Union (IOU) to 0.65 in the simulated environment [156]. Similarly, the interactive systems must consider dynamics such as object projection and interpolation (next step based on the occluded information), identity maintenance (error detection and tracking), objects attribute assumption, commonsense, and counterfactuals (mimicking human cognition). For this purpose, a visual sensemaking framework has been proposed to integrate knowledge representation and computer vision for interactive system [157].
However, NeSyL could be made more robust and efficient via integration of multi-sensory information, multi-agent merging, contextual knowledge, and situational information in modular framework. One of the key areas that need to pay more attention is the medical practices including opthalmology to produce a lifesaving outcome. 4) NeSyL Integration and Embedding: Considering the integration of symbolic learning into the neural network algorithms, varieties of integration methods have been employed following the essence of the problem [74]. However, there is no systematic, generalized, and standard approach to integrate symbolic (high level) and subsymbolic (neural network level) representations. A taxonomy of NeSyL is viewed from two perspectives: the coupling magnitude (strong, tight, and weak) and coupling types (pre/postprocessing, subprocessing, coprocessing, and metaprocessing). In this section, we elaborate some existing approaches and propose futuristic NeSyL architectures to infuse domain knowledge with neural network learning. The following sections illustrate related concepts.
Graph and Hypergraph based Integration: The integration via graph and hypergraph provides an effective capturing relationship between symbolic and sub-symbolic (neural network level), independency on programming language, and most importantly, preserves the topological dependency of information [158]. Graph representations are inherently compositional in nature and allow us to capture entities, attributes and relations in a scalable manner. The study [159] proposed a multimodel graph neural network to induce a factor matrix for question answering session and to solve the problem of compositional generalization for visual reasoning.
Knowledge graphs, for example, frequently display numerous hierarchies at the same time [160]. A spatial network, such as hyperbolic space, may be used to depict a data structure that is a continuous imitation of discrete trees, allowing it to be used to model hierarchical and graphical data. The advantages of hyperbolic-based embedding include fewer dimensions and promising results in downstream applications.
Layer-wise Integration and Attention: In the study [41], varieties of stressing (attention) mechanisms are employed between the symbolic and subsymbolic levels. Similarly, different layers of abstraction have been utilized to integrate high and low level information in terms of symbolic and subsymbolic [158]. The attention mechanism among these layers (L 0 , L 1 , ..., L n ) should be cognitively synergized among all the layers. Thus, sophisticated methods are needed to deal with attentions originating from symbolic and subsymbolic systems and should be assured their coordination, synergization, harmonization, and coherence in interactions. Attention mechanism has high significance of modeling NeSyL, such as particular neurons activation in the neural system by perceiving certain symbols [161].
The study [158] proposed a deep fusion based reasoning engine (DFRE) to represent the knowledge into four layers, where L 0 is close to the raw sensor data obtained from physical system. This study attempts to represent, integrate, and reason knowledge mimicking human approach. The layer-wise inte- gration has also been divided into physical, inferencing, and causal level [147].
Position-wise Integration of Domain Knowledge In [162], the domain knowledge inclusion has been studied at different positions such as input level, in the loss function, and as bias and weights parameters. The objective is to equip ethics, fairness, and explainability toward a responsible and stable neural network given the domain knowledge. If the input contains raw features with details retrieved from the domain, then the input would be transformed (propositionalisation) and fed to a DNN model. The domain knowledge can also be infused into the loss function as penalty term reflecting domain knowledge based guidance and directives. These terms may include the regularization-term, embeddings (low-dimensional learned continuous vector representations of discrete variables), and semantic domain constraint (specific restrictions on the prediction). Similarly, the domain knowledge can be incorporated into a deep network by introducing constraints on the model parameters (weights) or by making a design choice of its structure. Such incorporation can be explicitly performed in Bayesian formulation [163] as prior knowledge in the latent representation [126], [127]. The domain knowledge or symbolic information can be transferred to DL model using transfer-learning as prior distribution over the model parameters [164]. Furthermore, domain-knowledge encoded as a set of propositional rules and domain constraints must be applied on the structure of the neural networks to achieve the intended goal [165], [166], [93]. Similarly, domain knowledge can also be represented by metapaths as the integration of biological objects in a latent space.
Translational Integration: There is no systematic way to translate logical constraints as symbolic representation while translating to neural network. The logical symbolic representations are not differentiable where the weights learning and propagation gradients similar to DL models training may not be carried out [167]. To introduce the cycling dependencies in the symbolic representation, special attention, such as hypergraph, may need to translate into neural network. However, practically it is challenging to employ numerical constraints. The domain knowledge incorporation as loss term is not straightforward and not suitable for critical scenarios as well as special mathematical tool is needed to carry out the optimization. Furthermore, DL models confined to symbolic learning may not be scalable to large datasets.
Discussion on NeSyL Integration and Embedding: The translation and infusion of domain knowledge must be confined with logical constraints prior to translating into the neural network. There is no standard framework for logical constraints that DL can consume which needs special attention prior the integration modeling. The symbolic logics are not differentiable like neural network which lead to the emphasis on making a unified modeling of domain knowledge with neural network. Moreover, there is no straightforward way to construct loss term regarding critical and safety circumstances given domain knowledge.
The graph-based approaches can enable smoother scaling and better manage complex fine-grained multimodal reasoning, and they can be used in applications such as robotic control and autonomous navigation. The inclusion of heterogeneous graph based networks may overcome the challenges of fairness in the treatment of complex relationships and interpretability [143].
This study introduces four methods to integrate domain knowledge as symbolics into neural network ( Figure 15). In the first method ( Figure 15(A)), a complex environment of symbols can be presented in graph and hypergraph structure to transfer into the neural network architecture. Different levels of graphs can be generated given the input information where a cumulative graph sum up at the end prior to parsing. For instance, a physical world can be divided into different level of understanding, such as level-1 (G 1 ) can be included objects shape, appearance, design, and color, level-2 may include dynamics (G 2 ), where level-3 may include caption, source, and text. The dynamic world can further be divided into rolling, floating, trajectory, and bouncing. Graph and hypergraph can better represent such complex structures. A parser can be used to extract features and to proceed for reasoning and inferencing purposes.
The domain knowledge can also be integrated into a layerwise neural network architecture (Figure 15(B)). Each layer separately receives labeled information in terms of caption where specific layer has been pointed out via attention mechanism. The caption information has been received from language modeling. The layer based knowledge is then translated into vectorial representation comprising objects, attributes, dimensions, and message passing mechanism among layers. Moreover, the domain knowledge can be integrated at different position such as at input level, as biase, weights, and loss term ( Figure 15). The domain knowledge can also be embedded by carrying out translating symbolics into neural network architecture ( Figure 15). This paved the road to a systematic symbolic translation, addressing differentiability, and logical constraints representation into the neural network world. 5) Specialized DL Networks of NeSyL: Typically, the Ne-SyL uses symbolic learning (reasoning level) on top of neural learning (closely to physical world). The neural leaning can be reconstructive, generative, reinforcement learning model, and so on. However, little attention has been given to specialized DL algorithms for NeSyL. In the sudy [168], unsupervised model based on VAE framework has been proposed, which encodes domain specific symbolic information or domain symbolic language coupled with neural network encoder to make the latent space interpretable and factorized.
The key idea is to use neural network and deep learning to generate the knowledge, and then the reasoning can be done in symbolic AI framework. And the knowledge dataset is very important for the symbolic learning. So, it is a big challenge on how to make the AI learns the knowledge as much as possible with as few as possible necessary manual input of knowledge data. The encoding of domain knowledge into latent space by making cluster could be more challenging for multiple clusters and their simultaneous interpretation. The coupling of symbolic domain knowledge with standard encoder can enable human to interpret latent space corresponding to different applications such healthcare, sports analytics, and autonomous driving. However, such coupling should be avoided from any human attack as human are involved in data curation, domain knowledge designing, and interpretation.
Special attention is needed while designing DL model to embrace the novel features of symbolic learning in terms of integration, translation, reasoning, and application. So, the future DL models could be equipped with features of both incorporating domain knowledge and reasoning (temporal, structural, relational). Besides from the application perspectives, the pure image computational tasks, such as, image generation (synthesis), reconstruction, restoration, inpainting, registration, enhancement, and super-resolution, are also need to be modeled via NeSyL. Similarly, revisiting image processing scenarios including lacking annotation (zero-shot learning), absence of perfect and high quality as ground truth, and so on, with innovative approaches via NeSyL.
NeSyL can be incorporated into DL model such as VAE to reason via the enlightenment of the VAEs, the generation of the knowledge can be easily carried out. With the VAEs framework, we can build this knowledge-based data more easily in two ways. The first one is to use the logic inference, which can achieve a precise kind of knowledge, where the second one is to find the similarity and relation within knowledge, which might be approximated but also accurate to some extent. We can use the distribution to create knowledge. Using these two methods together can develop the initial knowledge (existing knowledge and a large amount of scattered but categorized elements), and reproduce a more abundant knowledge base. One of the generalized idea for the proposed model has been depicted in Figure 16.
In the first step, we use a small amount of existing knowledge about the visuals and elements (entities). The received existing knowledge and objects can be embedded and represented into corresponding spaces. A neural network extracts patterns from the entities like A and a1/a2, and their corresponding distributions. The key idea is to find a similar distribution of each class of entity and to enable referencing within a confidence coefficient (CC) such as 90% CC level. For instance, the pattern A > B can be used to expand our base knowledge into broadened knowledge: a1 > b2. The acquired broadened knowledge to facilitate logic reference processing and expansion with a larger amount of knowledge.
In the next step, VAE model generates the broadened knowledge into strengthen broadened knowledge using variational encoding and decoding to map the similarities and the relations between the knowledge spaces. The model may create knowledge on its own by comparing the distributions of entities in the latent space. However, this is just a supposition without mathematical proof. We need some fundamental foundation with the mathematical tools and some experiments in the future study. After a round of iteration, the strengthened broadened knowledge can be generated either artificially or manually. Finally, the strengthened broadened knowledge can be used for the upcoming reasoning procedure within the symbolic AI.
6) NeSyL for Video Captioning and Internet of Things: In the study [169], neurosymbolic structure (via bias induction) has been proposed to caption visuals from videos by figuring out the key roles (predicates). The model receives videos coupled text descriptions and learns their cross-model relationship (via key attention-aware) to caption multi-model structures.
To figure out the key roles in image and video captioning, further optimization is needed in roles division and selection in order to reason at the key role-level and sub-role level. This will allow humans to imitate captioning for a visual scenes corresponding to the query.
NeSyL based alerting and safety models [161] for different domains and applications such as homes, offices, stations, and buildings. Alerting model can be applied to critical environments such as safety, security, and forensic scenes, where symbol like gun, fire, knife, unusual movements and behaviors can be detected and captioned. Similarly, in the future, the autonomous system should be adapted to the alert model based on the human behaviors and emotions such as sleeping of a driver after long drive, monitoring driver health, and monitoring people involved in critical scenarios.

V. DISCUSSION
AI faces a number of challenges, including emulating human learning, intuition, and reasoning abilities while also including interpretability into current AI [170]. The basic plasticity principles that regulate the process by which environmental cues are converted into synaptic updates, however, are unclear. Many theories have been proposed, ranging from Hebbianstyle processes that are physiologically reasonable but have yet to be demonstrated to tackle challenging real-world learning problems to a learning strategy that is successful but has multiple biologically dubious features. It is difficult to say whether precise measurements of activation patterns over time, synaptic qualities, or paired-neuron input-output correlations will allow for quantitatively accurate predictions of whether the findings are more compatible with one or another learning rule. This is a substantial difficulty by itself, because it is impossible to determine which patterns of neuronal changes originated from given learning rules on solely theoretical grounds without evaluating the overall network architecture and loss function of the learning system [171]. Rather than viewing NeSyL as universal improvements over deep learning, bias, robustness, and generalization, the architecture of NeSyL requires extensive research and international collaboration. NeSyL is one step closer to create a human brain mimic algorithm with two systems, which will open a door for novel discoveries and exciting applications.
Neural networks have been rapidly expanding in recent years, with novel strategies and applications. However, challenges such as interpretability, explainability, robustness, safety, trust, and sensibility remain unsolved in neural network technologies, although they may unavoidably be addressed for critical applications. Attempts have been made to overcome the challenges in neural network computing, such as ML, by representing and embedding domain knowledge in terms of symbolic representations. Thus, neuro-symbolic learning (Ne-SyL) notion emerged, which incorporates aspects of symbolic representation and bringing common sense into neural networks (NeSyL). In domains where interpretability, reasoning, and explainability are crucial, such as video and image captioning, question-answering and reasoning, health informatics, and genomics, NeSyL has shown promising outcomes. This review presents a comprehensive survey on the state-of-the-art NeSyL approaches, their principles, advances in machine and deep learning algorithms, applications, and most importantly future perspectives.

VI. CONCLUSION
The widespread uses of neural networks are indispensable; nonetheless, the critical challenges such as black-box, interpretablility, and explainability limit its applications to certain scenarios including medical imaging and clinical practices. On the other hand, besides the white-box features and domain rich knowledge, symbolic learning has not yet advanced to the computational level of neural learning. Thus, it would be indispensable to infuse symbolics into neural learning as NeSyL to attain human intuitive features such as transperancy, interpretability, explainability, and reasoning in AI learning algorithms. In this study, we introduced relevant ideas, principles, and applications of NeSyL, so that a beginner reader can thoroughly understand the background knowledge before to conducting further research. We also revisited the existing CNN algorithms in novel directions infused with symbolic learning to pave the way for novel and robust NeSyL modalities. Most importantly, we introduced futuristic perspectives of NeSyL by advancing ML and DL modalities, and introducing viable NeSyL approaches for critical environments such as clinical practices, ocular diseases diagnosing, autonomous vehicles, health, and informatics. ACKNOWLEDGMENT