Efficient Marginalization of Discrete and Structured Latent Variables via Sparsity