Recently, the research result on the cooperative control of electric vehicles and photovoltaics based on reinforcement learning by Zhejiang University - University of Illinois Urbana-Champaign Institute (ZJUI) was accepted by the flagship journal of the electrical field, IEEE Transactions on Industry Applications (IEEE TIA). Sun Fangyuan, a doctoral student at ZJUI, is the first author of the paper, and the sole corresponding author is Assoc Prof. Diao Ruisheng of ZJUI. Other authors include Lan Tu, a doctoral student at ZJUI, and others. IEEE TIA is the flagship journal of the IEEE Industry Applications Society (IAS), covering technical topics in the theory and engineering applications of electrical and electronic fields, and has a high influence in the fields of industrial automation, electric machinery, and power electronics.
With the fast-growing penetration level of renewable energy, the uncertainty of the power system has significantly increased, putting pressure on the balance of power generation and consumption. The large number of electric vehicles connected to the grid can play a similar role to energy storage devices. If the charging and discharging process of electric vehicles can be reasonably controlled under the premise of meeting the charging demand, the power fluctuation of the grid will be reduced. On the other hand, for large open-air electric vehicle charging stations, installing photovoltaics on the sunshade roof will effectively save electricity costs. The research group comprehensively considered factors such as grid regulation needs, user charging needs, and photovoltaic power generation, and reasonably planned the charging process of each electric vehicle in the charging station, and effectively reduced the operating cost of the charging station and increased revenue by responding to grid needs.
There are problems with the control of electric vehicle charging and discharging, such as high control dimensions and strong uncertainty of electric vehicle arrival. The cooperation with photovoltaics further increases the control difficulty. In response to these problems, the group proposed a reinforcement learning-based electric vehicle charging control method, which comprehensively considers factors such as electricity price changes in response to grid needs, changes in photovoltaic output, and the randomness of electric vehicle arrival/departure time and power needs.
▲ System architecture of EV-PV coordination control
The group first designed a coordinated charging control method for electric vehicles inside the station, which automatically allocates the overall charging power of the station to each electric vehicle, ensuring that the charging needs of each electric vehicle are met before the preset departure time, and prioritizes charging for electric vehicles with short remaining stay time to reduce random risk. This method can also reduce the dimension of decision variables to one, that is, the total charging power of the charging station, thereby significantly reducing the decision difficulty.
Then, the group used the SAC algorithm to solve the optimal charging power of the charging station in real time. This algorithm considers factors such as forecast information and the current number of free charging piles in the station for optimal decision-making. The forecast information includes electricity prices, photovoltaic output, and the number of electric vehicles entering and exiting the station. The overall scheme of the proposed method is as follows.
▲ The overall scheme of the proposed method
The results show that the method can reasonably plan the charging power of the station according to various operating states. At the same time, in addition to basic short-term forecast information, the reinforcement learning agent also automatically analyzes the change process of various uncertain information within a day, so it has higher accuracy than traditional model-based methods. In the test scenario, the revenue of the charging station under the proposed algorithm increased by 13.7%.
▲ The algorithm performance in the test environment; (above) photovoltaic output and charging station charging power within a day; (bottom left) the number of charging piles occupied in the station within a day; (bottom right) charging power control signal
▲ Performance comparison of different RL algorithms
ZJUI’s research group on New Energy Power System Simulation and Intelligent Control
The New Energy Power System Simulation and Intelligent Control research group was established in 2022. The main research directions include high-fidelity power system modeling and simulation technology, high-proportion new energy grid-connected technology, stability mechanism and control of power electronification power system, high-performance computing and artificial intelligence applications in power systems, online safety assessment and decision-making of the power grid, aggregation and management of distributed resources, etc.
Prof. Diao Ruisheng obtained his B.S. in Electrical Engineering from Zhejiang University in 2004, M.S. degree in Electric Power Systems & Automation from Zhejiang University in 2006 (Advisor: Prof. Zheng Xu, IEEE Fellow), and Ph.D. degree in Electrical Engineering from Arizona State University in 2009 (Advisor: Prof. Vijay Vittal, Member of the U.S. National Academy of Engineering, IEEE Fellow). His research areas include high-fidelity power system modeling, simulation and analysis, application of HPC, optimization and control, and artificial intelligence in power systems. He served as a team lead, Advanced Grid Analytics, and a deputy program manager at the Pacific Northwest National Laboratory (affiliated with the U.S. Department of Energy) from July 2009 to May 2018 (funding level: $2-3M/year). Then, he joined GEIRI North America as Deputy Department Head in charge of Research and Development in AI & System Analytics. In 2022, Assoc Prof. Diao Ruisheng joined ZJUI. In 2023, Assoc Prof. Ruisheng Diao was elected as a Fellow of the Institution of Engineering and Technology（IET Fellow). Multiple research prototypes developed by Prof. Diao’s team were commercialized by vendors, which are now widely used by power companies in North America and China.
