Cluster-aware upcycling expands pretrained foundation models into specialized mixture-of-experts without retraining from scratch
https://www.eurekalert.org/news-releases/1139465 "upcycling machine vision learning duplicates identical weights or assigns routing randomly, overcome extracting underlying semantic structure from token activations in pretrained layers grouping similar features into distinct clusters from which expert weight matrices/ routing centroids initialized... outperforms standard sparse upcycling across image-text retrieval/ multiple visual classifications... scalable, preserves existing AI assets, reduces computational overhead... large-scale multimodal AI"