This article explores the origin of anomalous spectral features that obscure known material resonances in terahertz time-domain spectroscopy (TDS-TDS) of granular samples. Ray-tracing techniques were applied to predict ray scattering by impurities in a granular sample. Pulsed peaks of exiting rays were predicted in the time domain, and a Fourier transform to the frequency domain indicated spurious spectral features such as peaks of loss. Experimental measurements with Borofloat microspheres of various sizes and concentrations in a PTFE matrix validated the ray tracing prediction. This technique provides a new theoretical framework to enhance the utility of THz-TDS to nondestructively measure the size and concentration of contaminants in granular samples.