Reality-based modeling is is used to create virtual environments based on real-world interactions. The model can be a database of recorded responses to various stimuli, an empirical input-output mapping, or a set of physics-based equations whose parameters are fit with experimental data. In this talk, a framework for the reality-based modeling approach will be described and demonstrated for two types of tool-tissue interactions: cutting with scissors and needle insertion. For cutting with scissors, we use the database approach and a physics-based fracture mechanics model, both of which can be integrated with a larger deformable tissue model. For needle insertion, we use a simple empirical input-output mapping and physics-based models. We also consider the relative importance of modeling factors such as geometry, boundary conditions, and constitutive model in an FEM simulation of the response of tissues to needle insertion. Finally, we describe a promising application of needle insertion modeling: planning and control with steerable needles.