Abstract
Reinforcement learning is a novel approach for artificial lift in which optimal control policies are learned through interactions with the environment. This paper reports the first implementation of reinforcement learning for plunger-lifted wells using only historical data in an offline approach. The use of offline reinforcement learning allows production engineers to remain in control during the exploration phase, instead of allowing the agent to explore set-points that could result in unsafe operation. This paper explores the application of the Conservative Q-Learning (CQL) algorithm to maximize production in a small-scale plunger-lifted well model that mimics a low-pressure gas well controlled by the "minimum on-time" framework. The well was designed to receive a new open-trigger value for the motor valve from the CQL agent at the beginning of every cycle. The reinforcement learning problem was formulated to maximize gas production (reward) by adjusting the "casing-line" open trigger value (action) given the volume of fluid in the volume after the well has settled (state). Experimental results demonstrate that a dynamic policy that modulates the open trigger set point based on slug volume improves the average production rate by 35% compared with a static casing-line pressure trigger. The optimal policy suggests that using higher casing pressure open triggers for large slug volume cycles improves long-term production in plunger lift.
