Nowadays, training a classifier requires a large amount of training data, which is likely to be gathered from multiple sources. However, due to several concerns, such as privacy issues and bandwidth limitation, it may not be possible to directly share the data to perform the training. In addition, due to data availability, each source may not be able to gather data for allclasses of interest, which further complicates the training procedure. In this work, we present an approach called Unifying Heterogeneous Classifiers (UHC). In UHC, we let each source train its own classifier (teacher) with its available classes, then a central node gathers all the classifiers and uses them to train a new classifier (student) over all classes using only unlabelled data. To train the student classifier, we derive a probabilistic relationship between the outputs of all classifiers and transfer the knowledge from the teachers to the student using knowledge distillation. Our approach does not require any classifiers to be differentiable or have the same architecture, and does not require sharing labelled data between sources and central node.  Our extensive experiments on ImageNet, LSUN, and Places265 show that our approach significantly outperform a naive extension of knowledge distillation and can achieve almost the same accuracy as classifiers trained directly with labelled data from the sources.

Jayakorn Vongkulbhisal received the BEng degree in Information and Communication Engineering from Chulalongkorn University in 2011, the MSc degree in Electrical and Computer Engineering from Carnegie Mellon University in 2016, and the PhD degree in Electrical and Computer Engineering  (dual-degree program) from Carnegie Mellon University and Instituto Superior Técnico in 2018. He is currently a research scientist in the Accessibility Group, IBM Research-Tokyo. His research interests arein the areas of computer vision, machine learning, and optimization.