While the efficiency of organic solar cells (OSCs) has increased considerably in recent years, there remains a significant gap between the experimental open-circuit voltage (VOC) and the theoretical limit. Understanding the origin of this energy loss is important for the future development of OSCs, with temperature-dependent measurement of VOC an important approach to help unlock the underlying physics. Interestingly, previous studies have observed a reduction in VOC at low temperature that has been attributed by different studies to different phenomena. To resolve this issue, herein the temperature dependence of VOC of various polymer-based OSC systems covering a range of acceptor types (fullerene, polymer, and non-fullerene small molecule) as well as device architectures (conventional, inverted, blend and bilayer) is studied. Across all systems studied, VOC reduction at low temperatures is associated with high parasitic leakage current, providing a universal explanation for this phenomenon in OSCs. Moreover, it is shown that leakage current, which causes complexity in the analysis and raises reliability concerns in potential applications, can be suppressed by varying device architecture, providing an effective approach for analyzing the true temperature dependence of VOC.