Note that there is an intrinsic trade-off between minimizing file sizes and making access efficient. Something like delta-encoding is highly efficient but makes efficient access impossible.
Most file formats are designed with the assumption that line number information will only be needed rarely (e.g. when explicitly breaking into a debugger, or annotating a backtrace in case of a crash) so they prioritize simplicity and compactness over efficient access.