The conventional approach to building physics models is based on physical intuition gained in prior studies of similar systems. Unfortunately, intuition is often faulty. We show that a recently developed technique from information science, compressive sensing (CS), provides a simple, efficient, and systematically improvable way of constructing models in a numerically robust and conceptually simple way. CS is a new paradigm for model building in physics - its models are sparse and just as robust or better than those built by current state-of-the-art approaches. They can be constructed at a fraction of the computational cost and user effort. We will illustrate the general idea and highlight applications to alloys, protein folding energetics, and lattice dynamics. We will also introduce a general formalism for obtaining localized ("compressed") solutions to a class of problems in mathematical physics, which can be recast as variational optimization problems. This class includes the important cases of the Schrödinger’s equation in quantum mechanics and electromagnetic equations for light propagation in photonic crystals. These ideas can also be applied to develop a spatially localized basis that spans the eigenspace of a differential operator, for instance, the Laplace operator, generalizing the concept of plane waves to an orthogonal real-space basis with multi-resolution capabilities.