- Ph.D., Stanford University, 1977
- M.S., Electrical Engineering, University of Rochester, 1974
- B.S., Electrical Engineering, National Taiwan University, 1971
Our research places the focus on developing high-performing learning networks, for which deep learning processors has now become the state-of-the-arts and have virtually replaced most traditional signal processors in speech and image processing applications. Back-propagation (BP), the current de facto training paradigm for deep learning models, is only useful for parameter learning but offers no role in finding an optimal network structure. We need to go beyond BP in order to derive an optimal network, both in stricture and in parameter.
For structural learning, the node/layer importance played a vital role. If we have an effective mechanism to do (1) node-ranking, and (2) layer ranking, then it will greatly facilitate the process of network pruning, i.e. deep compression. To this end, a key ingredient is to develop a joint parameter/structural gradient-type method to gradually polish the network towards an optimal structure, while traversing a comprehensive solution space covering likely candidates for optimal designs in size, power, speed, and accuracy.
To facilitate node/layer ranking, we develop an internal learning paradigm, making a good use of (1) internal teacher labels (ITL); and (2) internal optimization metrics (IOM), i.e. DI, for evaluating hidden layers/nodes. In other words, we have incorporated a notion of Internal Neuron's Learnablility (INL) into the traditional external learning paradigm (i.e. BP) and create a new generation of neural networks, called Explainable Neural Network (XNN). Mathematically, we adopt a new IOM, called discriminant information (DI) which offers an effective metric for ranking the nodes/layer in a network. It can be shown that by simply removing redundant and harmful nodes based on DI tended can greatly enhance the model’s robustness. This allows us to develop a joint parameter/structural gradient-type method for deep compression.
In addition, the new XNN model opens up a promising machine learning research front for Internal Neuron's Explainablility (INE), a key ingredient in DARPA’s Explainable AI (sometimes referred to as AI3.0). Briefly speaking, it is vital to support end-user-adaptive label(s) so that a learning model may be repurposed to an new and active learning environments. For example, the end user may suddenly need to reclassify a subset of original classes and/or to verify/reject certain nature of a newly observed object: e.g. a drone or not a drone. This calls for explainable learning, i.e. X-learning as in XAI. By tailoring the design of the DI-based X-learning scheme, we can rapidly pin-point the relevant internal nodes/channels in the network. This will in turn allow us to retrieve vital information, either numerically or visually, to make a critical decision in real time.
- S. Y. Kung, Zejiang Hou and Yuchen Liu, "Methodical Design and Trimming of Deep Learning Networks: Enhancing External BP learning with Internal Omnipresent-Supervision Training Paradigm", Proceedings, IEEE ICASSP SS-L12.1.
- S. Y. Kung, "Compressive Privacy: From Information/Estimation Theory to Machine Learning”, Invited Lecture-Note, pp. 94-112, IEEE Signal Processing Magazine, Jan. 2017.
- S.Y. Kung, "Discriminant component analysis for privacy protection and visualization of big data", J. of Multimedia Tools and Application, 2017.
- S.Y.Kung, "Kernel Methods and Machine Learning", Cambridge University Press, 615 pages, 2014.
- S.Y. Kung, "VLSI Array Processors," Prentice-Hall Information and System Science Series, (T. Kailath, Series Editor), 667 pages, 1987.
Honors and Awards:
- Life Fellow of IEEE, for contribution to VLSI signal processing and neural networks, 2016
- The Third Millennium Medal, IEEE, 2000
- Best Paper Award, IEEE Signal Processing Society 1996
- Distinguished Lecturer, IEEE Signal Processing Society, 1994
- Honorary Professorship, Central China Science & Technology University, 1994
- Technical Achievement Award, IEEE Signal Processing Society, 1992
- Fellow of IEEE, for contribution to VLSI signal processing and neural networks, 1988
- Sino-US Exchange Scientist, National Academy of Science, 1987