Towards Source-Free Domain Adaptation of Neural Networks in an Open World

Shiqi Yang

Dr. Joost van de Weijer

CVC

Despite their great success, deep neural networks often require a large amount of labeled data for training. However, collecting labeled data is often laborious and expensive. Therefore, it would be ideal if the knowledge gained from label-rich datasets could be transferred to unlabeled data. However, deep networks are weak at generalizing to unseen domains, even when the differences between datasets are only subtle. In real-world situations, a typical factor that impairs the generalization ability of the model is the change in distribution between data from different domains, which is a long-standing problem often referred to as (unsupervised) domain adaptation. A crucial requirement in the methodology of these domain adaptation methods is that they require access to the data from the source domain during the adaptation process to the target domain. The accessibility of the source data of a trained source model is often impossible in real-world applications, for example, when deploying domain adaptation algorithms on mobile devices where computational capacity is limited or in situations where data privacy rules limit access to source domain data. . Without access to source domain data, existing methods suffer from inferior performance. Thus, in this thesis, we investigate domain adaptation without source data (called sourceless domain adaptation) in multiple different scenarios that focus on image classification tasks. We first study the problem of sourceless domain adaptation in a closed environment, where the label space of different domains is identical. By accessing only the pretrained source model, we propose to approach sourceless domain adaptation from the perspective of unsupervised clustering. We achieve this by relying on nearest-neighborhood clustering. In this way, we can transfer the difficult task of sourceless domain adaptation to a type of clustering problem. The final optimization objective is an upper bound containing only two simple terms, which can be explained as discriminability and diversity. We show that this allows us to relate several other domain adaptation methods, unsupervised clustering, and contrastive learning from the perspective of discriminability and diversity. Following the sourceless domain adaptation setup, we also investigate the catastrophic forgetting problem after adaptation, where the adapted model should maintain good performance on the source or all trained domains. To address the forgetting problem, we propose to use randomly generated domain attention masks to regularize the model updating during adaptation. This manages to maintain knowledge in old domains without affecting the adaptation to new target domains. In real-world applications, there might be some unseen categories in the target data; without additional processing, the model cannot handle these open classes. To prepare the method to generalize to target environments where there may be unseen categories, we propose an elegant and simple solution by inserting an additional dimension in the head of the classifier. Together with additional cross-entropy loss during source pre-training, the model has strong open recognition performance, which could be directly used for target adaptation and excels at distinguishing open classes during adaptation.

In this report, we have investigated a new paradigm of domain adaptation, called source-free domain adaptation, which aims to adapt the pretrained source model to a new unlabeled target domain without access to the labeled source data. We also studied situations where, after adaptation, the model is expected to not forget on the source domain. Finally, we investigated the case where the label spaces between different domains are not identical. The proposed method could be deployed to almost all real-world application scenarios, since nowadays it is inevitable that the pretrained machine learning model from a company needs to be adapted/transferred to a new environment without any labeled target data, and there may exist data privacy issue (for example in hospital record system, when a pretrained model needs to adapt to a new hospital environment, the model during adaptation should not access the old data due to the patient data leakage privacy risk), potential emerging open classes and catastrophic forgetting issue (the model after adapting to new tasks should keep good performance on all old tasks). With our methods, the pretrained model could easily be adapted to plenty of different new application environments, and since the proposed methods are general, the application tasks could be any computer vision task such as classification, segmentation, point-cloud classification and video action recognition.

Labeled Data; Training; Unlabeled Data; Knowledge Transfer; Generalization; Unseen Domains; Distribution Change; Unsupervised Domain Adaptation; Source Data Access; Mobile Devices; Computational Capacity; Data Privacy Rules; Sourceless Domain Adaptation; Image Classification Tasks; Closed Environment; Label Space; Pretrained Source Model; Unsupervised Clustering; Nearest-Neighborhood Clustering; Optimization Objective; Discriminability; Diversity; Contrastive Learning; Catastrophic Forgetting Problem; Randomly Generated Domain Attention Masks; Regularization; Unseen Categories; Open Classes; Classifier Head; Additional Dimension; Cross-Entropy Loss; Source Pre-training; Open Recognition Performance; Target Adaptation.