Integrating Multiple Modalities Into Deep Learning Networks