Exact, efficient and computationally feasible inference: Application to discrete data problems