The effect of incorporating Si and Ge atoms in the conjugated backbone of semiconductor polymers is investigated using transient absorption spectroscopy and quantum chemical calculations to uncover the heavy-atom impact on the excited-state dynamics in neat films and polymer/fullerene blends. The singlet and triplet exciton dynamics of the copolymers are resolved and the time constant of intersystem crossing (ISC) of dithienosilole is found to be 6.98 ± 0.45 ps and nearly 4 times longer than that of dithienogermole. This result indicates that factors other than the heavy-atom effect govern the ISC rates and the overall excited-state dynamics in the copolymers. Our quantum chemical calculations and estimates of the ISC rates based on the semiclassical derivation for the electron-transfer processes in the nonadiabatic limit reveal that the main driver for the increased ISC time constant in BuSiDT is the reduction in planarity and increased torsional out-of-plane vibration of the Si-bridged thiophenes in the dithienosilole compared to the dithienogermole, leading to 8.3 times higher spin–orbit coupling and consequently a higher ISC rate. In the polymer/fullerene blends, charge generation yields are estimated. The results from this study indicate that the incorporation of heavy atoms in a bridge position within the conjugated polymer backbone can be used as a synthetic strategy to fine-tune excited-state properties.