The complex microstructure of organic semiconductor mixtures continues to obscure the connection between the active layer morphology and photovoltaic device performance. For example, the ubiquitous presence of mixed phases in the active layer of polymer/fullerene solar cells creates multiple morphologically distinct interfaces which are capable of exciton dissociation or charge recombination. Here, it is shown that domain compositions and fullerene aggregation can strongly modulate charge photogeneration at ultrafast timescales through studies of a model system, mixtures of a low band-gap polymer, poly[(4,4′-bis(2-ethylhexyl)dithieno[3,2-b:2′,3′-d]germole)-2,6-diyl-alt-(2,1,3-benzothia-diazole)-4,7-diyl], and [6,6]-phenyl-C71-butyric acid methyl ester. Structural characterization using energy-filtered transmission electron microscopy (EFTEM) and resonant soft X-ray scattering shows similar microstructures even with changes in the overall film composition. Composition maps generated from EFTEM, however, demonstrate that compositions of mixed domains vary significantly with overall film composition. Furthermore, the amount of polymer in the mixed domains is inversely correlated with device performance. Photoinduced absorption studies using ultrafast infrared spectroscopy demonstrate that polaron concentrations are highest when mixed domains contain the least polymer. Grazing-incidence X-ray scattering results show that larger fullerene coherence lengths are correlated to higher polaron yields. Thus, the purity of the mixed domains is critical for efficient charge photogeneration because purity modulates fullerene aggregation and electron delocalization.