To quantify ship capsizing, from the energy perspective, the safety degree of a ship in waves is estimated based on stochastic Melnikov function and phase space transport theory. Considering the influence of nonlinear damping moment, nonlinear restoring moment, as well as the random waves, a nonlinear single degree of freedom differential equation for ship rolling is established. Transform the random wave moment from time domain to frequency domain by fast Fourier transformation, the random Melnikov function and rate of phase flux are extended to include the effects of navigation speed and heading angle and the safety degree of ship capsizing is quantified according to its statistical characteristics. Through an example, the accuracy of Melnikov function and phase space transport theory are verified and the effects of ship speed and heading angle on phase space transport rate are also quantified. This method is demonstrated properly to quantify the safety degree of ship capsizing and some valuable reference can be provided for the further research on ship stability criteria.