R-divergence for Estimating Model-oriented Distribution Discrepancy
Zhilin Zhao, Longbing Cao. NeurIPS, 2023.

Real-life data are non-IID owing to complex distributions and interactions, while the sensitivity to the distribution of samples varies for learning models. Accordingly, for a supervised or unsupervised model, a fundamental question is whether the probability distributions of two given datasets can be treated as identical. Here, we propose R-divergence, which is used to evaluate the model-oriented distribution discrepancy, to address the above question. The insight is that two distributions are likely identical if their optimal hypothesis has the same expected risk on each distribution. Accordingly, to estimate the distribution discrepancy between two given datasets, R-divergence learns a minimum hypothesis on their mixture data and then estimates the empirical risk difference between them. We evaluate the test power of R-divergence on various unsupervised and supervised tasks, where R-divergence achieves state-of-the-art performance. Additionally, we apply R-divergence to train robust neural networks on samples with noisy labels.