Model and Data Efficiency in Deep Learning.

Model and Data Efficiency in Deep Learning
Aug 30, 2022, 10:00 am12:00 pm



Event Description

Deep learning has achieved great and broad breakthroughs in numerous real-world applications
with the advancement of larger size models and the explosive growth and availability of data. How-
ever, the deep learning models usually have excessive computational and memory cost that are not
friendly to practical deployment on mobile or edge devices. Moreover, the deep learning models face
challenges in learning and adapting rapidly from only a few examples to solve new tasks. Hence,
this thesis proposes techniques to learn computationally efficient model architectures and methods
to improve the few-shot learning ability.

We start with subspace analysis methods with application to the feature selection problem. We
then extend these methods to deep neural network structural learning (SL), with the objective
of reducing the redundant parameters to obtain the optimal down-sized model that can retain
or even improve the accuracy. More efficient SL method based on the hybrid pruning-regrowing
technique and more generalized SL method that can reduce the model across many more dimensions
are also introduced. Going beyond static model designs, we also present dynamic neural network
approaches that can adapt the model weights and architectures to different inputs on-the-fly during
inference to control the computation efficiency and improve the representation ability. Apart from
model efficiency, we also present techniques to train models that can rapidly generalize from a few
examples. We propose a few-shot architecture adaption method to customize task-specific model
structure for diverse few-shot tasks by meta-learning a task-aware architecture controller. Different
from the traditional NAS methods that require a separate search cost on each new task, our method
directly generates the task-specific model structure from the dataset in GPU minutes after a one-
time meta-training cost. Finally, we propose a cross-modality self-supervised learning framework
by masked image pretraining on language assisted representations. The resulting models produce
high quality transferable representations that advance the accuracy on numerous computer vision
tasks and demonstrate strong robustness to adversarial/out-of-distribution samples. Moreover, the
resulting models are amenable to structural learning for greater computation efficiency gains and to
low-resource task adaptation for better data efficiency.