SolarCLIP: A Vision-Language Model for the Sun
Solar event classification from multi-wavelength solar images is essential for understanding solar activity and supporting space-weather forecasting, yet annotated data for supervised machine learning is extremely limited. Recent advances in vision-language models, such as CLIP, perform well on natural image-text tasks, but their ability to generalize to solar physics remains unexplored. Solar images capture diverse events that often appear visually similar, and captions rely on specialized terminology that is rare in CLIP’s training data. Labeled pairs are scarce and expensive, making direct application of CLIP ineffective. Consequently, vanilla CLIP performs poorly on zero-shot event classification, and full fine-tuning overwrites useful pretrained features. We present SolarCLIP, a vision-language model adapted to solar physics by selectively fine-tuning projection heads and specific visual layers while freezing the backbone. This approach preserves general features, mitigates overfitting, and improves alignment with domain-specific concepts. On the DeepSDO dataset, SolarCLIP outperforms vanilla and fully fine-tuned CLIP for zero-shot event classification. These results show that selective fine-tuning is an effective strategy for adapting large vision-language models to data-scarce scientific domains.