Total organic carbon (TOC) estimation is very important for shale gas reservoir characterization. There are many introduced methods for TOC prediction in organic-rich shales. However, there are still some weaknesses with the most of the methods. This paper proposes a new method using machine learning, Gaussian Process Regression, which is expert in processing high-dimension, small samples, and non-linear problems. Compare to the Neural Network, and Support Vector Machine, Gaussian Process Regression has adaptation and generalization ability. This paper takes Zhangjiatan shale of the Yanchang Formation of the Triassic period in the south-eastern Ordos Basin as an example. A total of 7 kernel functions are applied to build the regression model. As a result, the Cauchy kernel is chosen due to lowest error. Then, feature selection is carried on based on the weights which calculated from 4 weights algorithms. Finally, compared the Gaussian Process Regression results to the traditional methods, (e.g., Passey and Schmoker methods), we found that Gaussian Process Regression works well for TOC estimation.