- Ph.D., Stanford University, 1977
- M.S., Electrical Engineering, University of Rochester, 1974
- B.S., Electrical Engineering, National Taiwan University, 1971
Our research places the focus on developing high-performing learning networks, for which deep learning processors has now become the state-of-the-arts and have virtually replaced most traditional signal processors in speech and image processing applications. Back-propagation (BP), the current de facto training paradigm for deep learning models, is only useful for parameter learning but offers no role in finding an optimal network structure. We need to go beyond BP in order to derive an optimal network, both in stricture and in parameter.
For structural learning, the node/layer importance played a vital role. If we have an effective mechanism to do (1) node-ranking, and (2) layer ranking, then it will greatly facilitate the process of network pruning, i.e. deep compression. To this end, a key ingredient is to develop a joint parameter/structural gradient-type method to gradually polish the network towards an optimal structure, while traversing a comprehensive solution space covering likely candidates for optimal designs in size, power, speed, and accuracy.
To facilitate node/layer ranking, we develop an internal learning paradigm, making a good use of (1) internal teacher labels (ITL); and (2) internal optimization metrics (IOM), i.e. DI, for evaluating hidden layers/nodes. In other words, we have incorporated a notion of Internal Neuron's Learnablility (INL) into the traditional external learning paradigm (i.e. BP) and create a new generation of neural networks, called Explainable Neural Network (XNN). Mathematically, we adopt a new IOM, called discriminant information (DI) which offers an effective metric for ranking the nodes/layer in a network. It can be shown that by simply removing redundant and harmful nodes based on DI tended can greatly enhance the model’s robustness. This allows us to develop a joint parameter/structural gradient-type method for deep compression.
In addition, the new XNN model opens up a promising machine learning research front for Internal Neuron's Explainablility (INE), a key ingredient in DARPA’s Explainable AI (sometimes referred to as AI3.0). Briefly speaking, it is vital to support end-user-adaptive label(s) so that a learning model may be repurposed to an new and active learning environments. For example, the end user may suddenly need to reclassify a subset of original classes and/or to verify/reject certain nature of a newly observed object: e.g. a drone or not a drone. This calls for explainable learning, i.e. X-learning as in XAI. By tailoring the design of the DI-based X-learning scheme, we can rapidly pin-point the relevant internal nodes/channels in the network. This will in turn allow us to retrieve vital information, either numerically or visually, to make a critical decision in real time.
- S. Y. Kung, ``Compressive Privacy: From Information/Estimation Theory to Machine Learning”, Invited Lecture-Note, IEEE Signal Processing Magazine, pp. 94-112, Volume 34, Issue 1, Jan. 2017.
- S.Y. Kung, "Discriminant component analysis for privacy protection and visualization of big data", Journal of Multimedia Tools and Application, pp. 3999-4034, Volume 76, Issue 3, Springer/Nature, February 2017.
- S.Y. Kung and Zejiang Hou, “Augment Deep BP-Parameter Learning with Local XAI-Structural Learning”, Kailath Special Issue, Journal of Communications in Information and Systems, Vol. 20, No. 3 (2020), pp. 319-352, 2020.
- S. Y. Kung: “XNAS: A Regressive/Progressive NAS for Deep Learning”. ACM Transactions on Sensor Networks . Vol. 18, No. 4, Article 57, November 2022.
- CHEX: CHannel EXploration for CNN Model Compression, Zejiang Hou, Minghai Qin, Fei Sun, Xiaolong Ma, Kun Yuan, Yi Xu, Yen-Kuang Chen, Rong Jin, Yuan Xie, S. Y. Kung, In IEEE CVPR 2022.
- Liu, Yuchen, Zhixin Shu, Yijun Li, Zhe Lin, Richard Zhang, and S. Y. Kung. "3D-FM GAN: Towards 3d-controllable face manipulation." In European Conference on Computer Vision, pp. 107-125. Springer, Cham, 2022.
Honors and Awards:
- Life Fellow of IEEE, for contribution to VLSI signal processing and neural networks, 2016
- The Third Millennium Medal, IEEE, 2000
- Best Paper Award, IEEE Signal Processing Society 1996
- Distinguished Lecturer, IEEE Signal Processing Society, 1994
- Honorary Professorship, Central China Science & Technology University, 1994
- Technical Achievement Award, IEEE Signal Processing Society, 1992
- Fellow of IEEE, for contribution to VLSI signal processing and neural networks, 1988
- Sino-US Exchange Scientist, National Academy of Science, 1987