Autonomous Charging of Electric Vehicle Fleets to Enhance Renewable Generation Dispatchability

As suggested earlier, fitted $Q$ -iteration is chosen as the approach to solve the RL problem in this study. Basically, the optimal solution of an MDP is acquired by maximizing the discounted sum of the rewards. The discount rate, $\gamma$ , determines the present value of future rewards: a reward received $k$ time steps in the future is worth only ${\gamma }^{k-1}$ times what it would be worth if it were received immediately. At the time $t$ , the value of taking action $a_t$ at state $s_t$ is denoted by $Q(s_t,a_t)$ and is calculated according to Bellman’s optimality equation as stated in (10).

(10) $$Q(s_t,a_t)=R(s_t,a_t)+\gamma \underset{a_{t+1}\in {\mathscr{H}}_{t+1}}{\max }Q(s_{t+1},a_{t+1})$$

Based on (10), the action value function at each state is the sum of the immediate reward at that state plus the maximum achievable action value at the subsequent state, for all feasible actions at time $t+1$ . The procedure of $Q$ -iteration used for solving the problem is presented in Algorithm 1. This algorithm takes the discount factor ( $\gamma$ ), the maximum number of simulation days (D), the maximum number of iterations ( $K_{\max }$ ), action and control step size ( ${\delta }_{\mathrm{s}}$ and ${\delta }_{\mathrm{c}}$ , respectively), the initial policy ( ${\pi }_0$ ) and the target ( ${ϵ}_0$ ) for the explorer as inputs; while returning the action value function as output. At the initializing step, the variables for day and time indices are set to zero and the sets containing batch samples for each day ( $𝒥$ ) and the whole simulation period ( $ℱ$ ) are pre-allocated.

Line 2 of Algorithm 1 asserts that for the initial day ( $d=0$ ), the actions are acquired based on the initial policy, i.e. ${\pi }_0(s_t)$ . The preset policy is used only for the first day, and we have implemented a random policy for this purpose where actions at each state are chosen randomly from the feasible set of actions, ${\mathscr{H}}_t$ . For the rest of the days, the actions are executed at time steps for each ${\delta }_s$ minutes; however, it is only at control time steps where $mod(t,{\delta }_c)=0$ that new actions are chosen. According to line 5, the best action is first obtained from the optimal action value function. Then based on line 6, the action at each time step is chosen by the explorer. To this end, we have implemented the $ϵ$ -greedy exploration method, where optimal action is acquired as shown in (11) and (12).

(11) $$ϵ=\max \{{ϵ}_0,1-c_{0}d\}$$

(12) $$a_t=\{\begin{array}{cc}{a}^{*},\hfill & ϵ\le \rho \hfill \\ \hfill & \hfill \\ \text{random}({\mathscr{H}}_t),\hfill & ϵ>\rho \hfill \end{array}$$

A linear $ϵ$ -greedy method is selected here, where according to (11), the $ϵ$ curve is a line decreasing from 1 to ${ϵ}_0$ with a slope equal to $c_0$ . The action at each step is then extracted based on (12), where $\rho$ is a random number between 0 and 1. This equation means with the probability of $1-ϵ$ , action at each time step is set to the optimal action returned from the action value function, and with the probability of $ϵ$ , actions are chosen randomly from the feasible action set. In lines 7 and 8 of Algorithm 1, the chosen action is forwarded to the environment and the pair $(s_{t+1},r_t)$ , representing the next state and reward for taking action $a_t$ is returned. Then, the set of batch experiences is updated.

At the end of each day, set $ℱ$ , which has the total simulation experiences so far, is updated. Then through an iterative procedure as indicated in lines 17–24, input and output data are fed to the regression function. Finally, the optimal action value function estimator which is used at line 8 for calculating the optimum action choice is set to be equal to the approximator function of the last iteration. To achieve an estimate of convergence, the factor convergence rate is introduced in (13). $C_d^k$ is the learning convergence for day $d$ at iteration $k$ , and is a measure to indicate how the best action-value function at each iteration performs compared to the previous iteration. The mean of best action value is denoted by $\bar{Q}$ .

(13) $$C_d^k={({\bar{Q}}_d^{k+1}-{\bar{Q}}_d^k)}^2/{\bar{Q}}_d^{k^2}$$

The most important element of the fitted $Q$ -iteration algorithm is the regression function. At the end of each epoch, the data from the environment is fed to the regression function to acquire an approximation of the true Q values. Also, the agent depends on the regression function for choosing actions at each step. Some regression methods that are used by different authors include but are not limited to neural network approximators, multi layer perceptrons, support vector machines, decision trees, and random forests. The fitted $Q$ -iteration algorithm requires the fitting of any arbitrary (parametric or non-parametric) approximation architecture to the Q-function with no bias on the regression function.

In this work, we follow the path of the original fitted $Q$ -iteration algorithm, which made use of an ensemble of decision trees. [ 23 ]. In the search for a desirable regression algorithm that is able to model any Q-function, tree-based models are found to offer great flexibility, meaning they can be used for predicting any type of Q-function. They are non-parametric, i.e. they do not need repeated occasions of trial and error to tune the parameters. Tree based models are also computationally efficient and despite some other methods, with the increase in problem dimensions, the computation burden of tree based methods does not increase exponentially. Among tree based models, we opted for random forests. Random forests are created by putting together various decision trees. At each step, the random forest algorithm selects a random subset of features. That is why this method is robust to outliers. The convergence of tree based regression for fitted $Q$ -iteration is investigated in [ 23 ]. All in all, they are a great tool for estimating a priori unknown shaped Q-functions with satisfactory performance.