Abstract
Poor reproducibility of published findings is a problem that is plaguing most scientific fields. Low statistical power, insufficient sample size, and poor experimental design are often cited as contributing factors to irreproducibility. The majority of scientific research in the oil and gas industry is not designed to readily validate the hypothesis being generated. In a typical study, data is collected without an explicit expectation and the results are analyzed. In these cases, the outcome becomes known and hypotheses are derived afterwards with the benefit of hindsight. Though this process is valuable in scientific induction and hypothesis generation, it leaves most published “conclusions” without true validation. Additionally, due to the well-documented effects of “hindsight bias”, it is difficult to accurately gauge the success of a hypothesis when the results are already known. To truly demonstrate the veracity of the hypothesis, then, it is necessary to predict future behavior and validate these blind predictions with an adequate sample set. This study provides an example of blind validation using a multi-variate regression in the Midland Basin.
Using wells with known outcomes of oil performance, a hypothesis was generated to explain cumulative oil production in the form of a multiple linear equation. This hypothesis was then tested by generating blind predictions of well performance for the next 100+ wells - before they were drilled. Now, with the collection of significant production history on this well set, the accuracy of the blind predictions has been evaluated.
The findings suggest that 1) a relatively large sample set is required for validation 2) it is possible to blindly predict well performance more accurately than chance or stationarity and 3) withholding data during the model tuning process overestimates the success of the model when compared to blind, forward predictions.
The novelty of this study is in the collection of a blind sample set of newly drilled wells after the predictions were generated. This allows for the quantification of the model's predictivity loss when used in a truly forward sense. The significance of this observation is that the error found in a forward sense is not the same as is found when simply withholding a portion of the data for testing. The application of this learning is far-reaching. The majority of published models are validated in a backwards sense, using existing observations. However, most of these studies are aimed at predicting and manipulating future behavior, for example, improving well performance through completions design. Without taking the next step to prove that this future expectation was achieved with a significant sample set, it cannot necessarily be expected that a model will be useful for its intended application.
