Rigidity regulates the integrity and function of many physical and biological systems. In this work, we propose that ’energetic rigidity,’ in which all non-trivial deformations raise the energy of a structure, is a more useful notion of rigidity in practice than two more commonly used rigidiy tests: Maxwell-Calladine constraint counting (first-order rigidity) and second-order rigidity. We find that constraint counting robustly predicts energetic rigidity only when the system has no states of self stress. When the system has states of self stress, we show that second-order rigidity can imply energetic rigidity in systems that are not considered rigid based on constraint counting, and is even more reliable than shear modulus. We also show that there may be systems for which neither first nor second-order rigidity imply energetic rigidity. We apply our formalism to examples in two dimensions: random regular spring networks, vertex models, and jammed packings of soft disks. Spring networks and vertex models are both highly under-constrained and first-order constraint counting does not predict their rigidity, but second-order rigidity does. In contrast, jammed packings are over-constrained and thus first-order rigid, meaning that constraint counting is equivalent to energetic rigidity as long as prestresses in the system are sufficiently small. The formalism of energetic rigidity unifies our understanding of mechanical stability and also suggests new avenues for material design.