Efficient encoder fine-tuning through configuration search and sparse low-rank adaptation
Machine Learning with Applications, cilt.26, 2026 (ESCI, Scopus)
- Yayın Türü: Makale / Tam Makale
- Cilt numarası: 26
- Basım Tarihi: 2026
- Doi Numarası: 10.1016/j.mlwa.2026.101024
- Dergi Adı: Machine Learning with Applications
- Derginin Tarandığı İndeksler: Emerging Sources Citation Index (ESCI), Scopus
- Anahtar Kelimeler: Configuration search, Inference profiling, Low-rank adaptation, Matched-budget evaluation, Parameter-efficient fine-tuning, Pareto optimization
- İstanbul Gelişim Üniversitesi Adresli: Evet
Özet
Low-Rank Adaptation (LoRA) reduces the task-specific optimization burden of transformer fine-tuning, but its effectiveness remains sensitive to rank, placement, scaling, dropout, and training choices. This study examines whether automated configuration search provides a consistent advantage when tuning effort is controlled. Genetic Algorithm (GA), Slime Mould Algorithm (SMA), Optuna, and Pareto-GA were evaluated in a shared mixed search space under a common 25-candidate validation budget, followed by five-seed retraining of selected configurations. DistilBERT experiments covered Emotion, AG News, and Financial PhraseBank, with RoBERTa-Base/SST-2 used as a secondary cross-model check. No optimizer consistently outperformed tuned references. On AG News, full fine-tuning and GA+SU-LoRA reached 94.4 ± 0.2% and 94.3 ± 0.2% accuracy. On Financial PhraseBank, the corresponding values were 86.0 ± 1.0% and 86.2 ± 0.9%, a descriptive difference within the observed retraining variability. Emotion ablations showed that SU-only was numerically comparable to GA+SU-LoRA, whereas Sparse-only and GA+Sparse LoRA differed by 0.1 percentage point. For GA+Sparse LoRA, fixed unstructured masks suppressed an average of 51.6% of LoRA-factor coefficients; the dense trainable allocation remained approximately 0.706M because tensor shapes were unchanged, and merged dense-kernel inference showed no latency benefit. Pareto selection exposed a clearer validation performance-support trade-off than the original scalar compactness penalty. The results emphasize matched baseline tuning, explicit component controls, analytically consistent parameter accounting, and deployment-aware evaluation.