Zhuo Ouyang, Kaiwen Hu, Qi Zhang, Yifei Wang, Yisen Wang
International Conference on Learning Representations (ICLR) 2025
We develop a new theoretical understanding for the role of the projection head in contrastive learning from the information-theoretic perspective. Mathematically, we rigorously derive both lower and upper bounds for the downstream performance of the features preceding the projection head.
Our findings indicate theoretical principles for designing an effective projection head: it should act as an information bottleneck, filtering out the irrelevant information and preserving the essential information for the contrastive objective.
Based on theoretical principles, we propose two categories of methods to improve projection head design, namely training regularization and structural regularization, that outperform previous approaches across a range of datasets and contrastive methods.
Zhuo Ouyang, Kaiwen Hu, Qi Zhang, Yifei Wang, Yisen Wang
International Conference on Learning Representations (ICLR) 2025
We develop a new theoretical understanding for the role of the projection head in contrastive learning from the information-theoretic perspective. Mathematically, we rigorously derive both lower and upper bounds for the downstream performance of the features preceding the projection head.
Our findings indicate theoretical principles for designing an effective projection head: it should act as an information bottleneck, filtering out the irrelevant information and preserving the essential information for the contrastive objective.
Based on theoretical principles, we propose two categories of methods to improve projection head design, namely training regularization and structural regularization, that outperform previous approaches across a range of datasets and contrastive methods.