Reliability-Weighted Spectral Allocation with Full Spectral Cores for Parameter-Efficient Visual Fine-Tuning

<title>Abstract</title> <p>Parameter-efficient fine-tuning (PEFT) reduces the cost of adaptingpretrained visual models by optimizing a compact task-specific parameterset while keeping most backbone parameters frozen. However, many PEFTmethods still rely on user-specified ranks, adapter widths, insertionlocations, or layer-wise allocation rules. We introduce areliability-weighted spectral allocation method that estimatestask-relevant gradient subspaces and scores candidate spectral modes usingpooled gradient energy and consistency between two deterministiccalibration partitions. A model-wide evidence distribution then determinesthe retained mode count, participating tensors, and layer-specific rankswithout requiring a user-specified rank budget. Within the selected leftand right spectral subspaces, full spectral cores capture cross-modeinteractions, and the learned updates can be algebraically merged into thecorresponding backbone tensors. Experiments across convolutional andTransformer backbones, single-label classification, multilabelrecognition, and semantic segmentation show that the method improves overlinear probing in most evaluated settings while optimizing1,165--54,610 spectral-core coordinates, equivalent to approximately0.004--0.307% of the evaluated model parameters. On Flowers-102 withViT-B/16, it improves Acc@1 over linear probing by 4.85 percentage points.Although specialized PEFT baselines or full fine-tuning remain stronger insome settings, the reported ablation supports full spectral cores overdiagonal spectral scaling, and algebraic merging removes the activemethod-specific inference branch.</p>