We have employed quasicontinuous fluorescence and phosphorescence spectroscopy within a temperature range between 10 and 500 K to monitor the spectral diffusion of singlet and triplet excitons in a series of π-conjugated polymers. We investigated (i) how spectral diffusion is controlled by the degree of energetic disorder present in the amorphous film (that is reflected by the inhomogeneous broadening of the photoluminescence spectra) and (ii) how this process depends on the range of the electronic coupling (by comparing singlet exciton diffusion via long-range Förster transfer against triplet exciton diffusion by short-range Dexter transfer). For singlets, we find that the fluorescence spectra bear out a bathochromic shift upon cooling the sample down to a critical temperature below which the shift saturates. This bathochromic shift is a signature of spectral relaxation. Random-walk theory applied to excitation transport within a Gaussian density-of-states distribution is both necessary and sufficient to rationalize the experimental results in a quantitative fashion. The same behavior is observed for triplets in weakly disordered systems, such as in a polymer containing platinum in the main chain and a ladder-type polyphenylene. In contrast we observe a hypsochromic shift of the phosphorescence spectra below a characteristic temperature for triplets in systems with at least moderate energetic disorder. The hypsochromic shift proves that triplet exciton relaxation becomes frustrated because thermally activated exciton jumps that otherwise promote spectral diffusion become progressively frozen out. The frustration effect is controlled by the jump distance and thus it is specific for triplet excitations that migrate via short-range coupling among strongly localized states as compared to singlet excitons.