We present an efficient workflow that combines multiscale (MS) forward simulation and stochastic gradient computation - MS-StoSAG - for the optimization of well controls applied to waterflooding under geological uncertainty. The Iterative Multiscale Finite Volume (i-MSFV), a mass conservative reservoir simulation strategy, is employed as the forward simulation strategy. The i-MSFV method has the ability to accurately capture fine scale heterogeneities, and thus the fine scale physics of the problem. This allows for accurate and efficient simulations with fine-scale error estimates in a more computationally efficient coarse-scale simulation grid. The Stochastic Simplex Approximate Gradient (StoSAG) method, a stochastic gradient technique, inspired from Ensemble Optimization (EnOpt), is employed to compute the gradient of the objective function. Stochastic methods have recently been shown to be very efficient in terms of gradient quality and computational cost especially for robust optimization problems. They rely on the reservoir simulator as a black-box and perform a large number of forward simulations to approximate a gradient. In our numerical experiments, we investigate the impact of MS parameters such as coarsening ratio and heterogeneity contrast on the quality of the approximate MS-StoSAG gradient. Our experiments illustrate that i-MSFV simulations provide accurate forward simulation responses for the gradient computation, with the advantage of speeding up the workflow due to faster simulations. Speedups up to a factor of five on the forward simulation were achieved for the examples considered in the paper. The combination of speed and accuracy of MS forward simulation with the flexibility of the StoSAG technique (several different parameters/responses can be easily considered) allows for a flexible and efficient optimization strategy suitable for large-scale problems.