Our research places the focus on developing high-performing learning networks, for which deep learning processors has now become the state-of-the-arts and have virtually replaced most traditional signal processors in speech and image processing applications. Back-propagation (BP), the current de facto training paradigm for deep learning models, is only useful for parameter learning but offers no role in finding an optimal network structure. We need to go beyond BP in order to derive an optimal network, both in stricture and in parameter.
 
For structural learning, the node/layer importance played a vital role. If we have an effective mechanism to do (1) node-ranking, and (2) layer ranking, then it will greatly facilitate the process of network pruning, i.e. deep compression. To this end, a key ingredient is to develop a joint parameter/structural gradient-type method to gradually polish the network towards an optimal structure, while traversing a comprehensive solution space covering likely candidates for optimal designs in size, power, speed, and accuracy.
 
To facilitate node/layer ranking, we develop an internal learning paradigm, making a good use of (1) internal teacher labels (ITL); and (2) internal optimization metrics (IOM), i.e. DI, for evaluating hidden layers/nodes. In other words, we have  incorporated a notion of Internal Neuron's Learnablility (INL) into the traditional external learning paradigm (i.e. BP) and create a  new generation of neural networks, called  Explainable Neural Network (XNN). Mathematically, we adopt a new IOM, called discriminant information (DI) which offers an effective metric for ranking the nodes/layer in a network. It can be shown that by simply removing redundant and harmful nodes based on DI tended can greatly enhance the model’s robustness. This allows us to develop a joint parameter/structural gradient-type method for deep compression.
 
In addition, the new XNN model opens up a promising machine learning research front for Internal Neuron's Explainablility (INE), a key ingredient in DARPA’s Explainable AI (sometimes referred to as AI3.0). Briefly speaking, it is vital to support end-user-adaptive label(s) so that a learning model may be repurposed to an new and active learning environments. For example, the end user may suddenly need to reclassify a subset of original classes and/or to verify/reject certain nature of a newly observed object: e.g. a drone or not a drone. This calls for explainable learning, i.e. X-learning as in XAI. By tailoring the design of the DI-based X-learning scheme, we can rapidly pin-point the relevant internal nodes/channels in the network. This will in turn allow us to retrieve vital information, either numerically or visually, to make a critical decision in real time.