A vast amount of oil and gas data has been collected over many years, which can be classified into different groups, including numerical, categorical, historical, real-time, structured, and unstructured. Therefore, analysis and visualization cannot be efficiently accomplished using traditional software platforms. Companies in the oil and gas industry need new technologies and processes to capture and transform data into strategic business intelligence to enhance exploration and production, improve refining and manufacturing efficiency, and optimize global operations while ensuring safety and environmental protection. With the advent of machine learning, data mining, optimization, and cloud computing, it is now possible to manage the large amount of data as never before.
Data-driven analytics, in this concept, offers new capabilities to the engineer in exploration and production to learn from data. Analyzing patterns in historical data and identifying the events in real-time data are two methods used for improving operations of activities and assets. The application of data-driven analytics in the oil and gas industry, therefore, potentially provides opportunities for greater insight and efficiency gains. The goal for data-driven analytics software is to provide the strategy required to optimize business outcomes and decision making while helping to minimize risk. The interaction between the users and the information through software tools results in a decision process that helps determine success of a project. The development of a data-driven software platform involves combining a variety of advanced machine learning and data-mining techniques, as well as data management strategies, to provide fit-for-purpose descriptive, predictive, and prescriptive models. Therefore, to be successful, the development team should have a deep understanding of both the underlying data science and the business model as well as software engineering principles.
This paper presents a workflow for an intelligent data-driven software platform development. The proposed workflow is composed of data management, data analytics and visualization, and predictive modeling. It covers best practices on database design, object-oriented design patterns and principles, predictive model building and deployment, user-interface development, and test-driven software development in an agile software development structure.