Home Astrodynamics Article
PDF (14.5 MB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript
Research Article | Open Access

Reinforced Lyapunov controllers for low-thrust lunar transfers

Te Pūnaha Ātea – Space Institute, University of Auckland, Auckland 1010, New Zealand
Surrey Space Centre, University of Surrey, Guildford GU2 7XH, UK
Show Author Information

Graphical Abstract

View original image Download original image

Abstract

Future missions to the Moon and beyond are likely to involve low-thrust propulsion technologies due to their propellant efficiency. However, these still present a difficult trajectory design problem, owing to the near continuous thrust, lack of control authority and chaotic dynamics. Lyapunov control laws can generate sub-optimal trajectories for such missions with minimal computational cost and are suitable for feasibility studies and as initial guesses for optimisation methods. In this work a Reinforced Lyapunov Controller is used to design optimal low-thrust transfers from geostationary transfer orbit towards lunar polar orbit. Within the reinforcement learning (RL) framework, a dual-actor network setup is used, one in each of the Earth- and Moon-centred inertial frames respectively. A key contribution of this paper is the demonstration of a forwards propagated trajectory, removing the need to define a patch point a priori. This is enabled by an adaptive patch distance and extensive initial geometry exploration during the RL training. Results for both time- and fuel-optimal transfers are presented, along with a Monte Carlo analysis of the robustness to disturbances for such transfers. Phasing is introduced where necessary to aid rendezvous with the Moon. The results demonstrate the potential for such techniques to provide a basis for the design and guidance of low-thrust lunar transfers.

Keywords

low-thrustreinforcement learning (RL)Lyapunov controlLunar spiral transfers

References

[1]
Peterson, P. Y., Herman, D. A., Kamhawi, H., Frieman, J. D., Huang, W., Verhey, T., Dinca, D., Boomer, K., Pinero, L., Criswell, K., et al. Overview of NASA’s solar electric propulsion project. In: Proceedings of the 36th International Electric Propulsion Conference, 2019: IEPC-2019-836.
[2]
Racca, G. D., Foing, B. H., Coradini, M. SMART-1: The first time of Europe to the Moon. In: Earth–Moon Relationships. Barbieri, C., Rampazzi, F., Eds. Springer Dordrecht, 2001: 379–390.
Crossref
[3]
Thomas, V. C., Makowski, J. M., Brown, G. M., McCarthy, J. F., Bruno, D., Cardoso, J. C., Chiville, W. M., Meyer, T. F., Nelson, K. E., Pavri, B. E., et al. The Dawn spacecraft. In: The Dawn Mission to Minor Planets 4 Vesta and 1 Ceres. Russell, C., Raymond, C., Eds. Springer New York, 2011: 175–249.
Crossref
[4]
Kawaguchi, J. I., Fujiwara, A., Uesugi, T. K. The ion engines cruise operation and the Earth swingby of ‘Hayabusa’ (MUSES-C). In: Proceedings of the 55th International Astronautical Congress of the International Astronautical Federation, the International Academy of Astronautics, and the International Institute of Space Law, 2005.
[5]

Tsuda, Y., Yoshikawa, M., Abe, M., Minamino, H., Nakazawa, S. System design of the Hayabusa 2—Asteroid sample return mission to 1999 JU3. Acta Astronautica, 2013, 91: 356–362.

Crossref Google Scholar
[6]

Benkhoff, J., van Casteren, J., Hayakawa, H., Fujimoto, M., Laakso, H., Novara, M., Ferri, P., Middleton, H. R., Ziethe, R. BepiColombo—Comprehensive exploration of Mercury: Mission overview and science goals. Planetary and Space Science, 2010, 58(1–2): 2–20.

Crossref Google Scholar
[7]

Gao, Y., Li, X. Optimization of low-thrust many-revolution transfers and Lyapunov-based guidance. Acta Astronautica, 2010, 66(1–2): 117–129.

Crossref Google Scholar
[8]

Shannon, J. L., Ozimek, M. T., Atchison, J. A., Christine, M. Q-law aided direct trajectory optimization for the high-fidelity, many-revolution low-thrust orbit transfer problem. Advances in the Astronautical Sciences, 2019, 168: 781–800.

Google Scholar
[9]
Shannon, J., Ozimek, M., Atchison, J., Hartzell, C. Rapid design and exploration of high-fidelity low-thrust transfers to the moon. In: Proceedings of the IEEE Aerospace Conference, 2020: 1–12.
Crossref
[10]
Shannon, J. L., Ellison, D., Hartzell, C. M. Analytical partial derivatives of the Q-law guidance algorithm. In: Proceedings of the 31st Space Flight Mechanics Meeting, 2021: AAS 21-274.
[11]

Jagannatha, B. B., Bouvier, J. B. H., Ho, K. Preliminary design of low-energy, low-thrust transfers to halo orbits using feedback control. Journal of Guidance, Control, and Dynamics, 2019, 42(2): 260–271.

Crossref Google Scholar
[12]

Epenoy, R., Pérez-Palau, D. Lyapunov-based low-energy low-thrust transfers to the Moon. Acta Astronautica, 2019, 162: 87–97.

Crossref Google Scholar
[13]
Shannon, J. L., Ellison, D., Hartzell, C. M. Exploration of low-thrust lunar swingby escape trajectories. In: Proceedings of the 31st Space Flight Mechanics Meeting, 2021: AAS 21-273.
[14]
Peterson, J. T., Singh, S. K., Junkins, J. L., Taheri, E. Lyapunov guidance in orbit element space for low-thrust cislunar trajectories. In: Proceedings of the AAS Guidance, Navigation and Control Conference, 2020: AAS 20-115.
[15]
Lee, S., Petropoulos, A. E., von Allmen, P. Low-thrust orbit transfer optimization with refined Q-law and multi-objective genetic algorithm. In: Proceedings of the AAS/AIAA Astrodynamics Specialist Conference, 2005: AAS 05-393.
[16]
Varga, G. I., Pérez, J. S. Many-revolution low-thrust orbit transfer computation using equinoctial Q-law including J2 and eclipse effects. In: Proceedings of the 6th International Conference on Astrodynamics Tools and Techniques, 2016: 29–42.
[17]

Holt, H., Armellin, R., Baresi, N., Hashida, Y., Turconi, A., Scorsoglio, A., Furfaro, R. Optimal Q-laws via reinforcement learning with guaranteed stability. Acta Astronautica, 2021, 187: 511–528.

Crossref Google Scholar
[18]
Kwon, H., Oghim, S., Bang, H. Autonomous guidance for multi-revolution lowthrust orbit transfer via reinforcement learning. In: Proceedings of the AAS/AIAA Astrodynamics Specialist Conference, 2021: AAS 21-315.
[19]
Miller, D., Englander, J. A., Linares, R. Interplanetary low-thrust design using proximal policy optimization. In: Proceedings of the AAS/AIAA Astrodynamics Specialist Conference, 2019: AAS 19-779.
[20]

Zavoli, A., Federici, L. Reinforcement learning for robust trajectory design of interplanetary missions. Journal of Guidance, Control, and Dynamics, 2021, 44(8): 1440–1453.

Crossref Google Scholar
[21]
Yanagida, K., Ozaki, N., Funase, R.. Exploration of long time-of-flight three-body transfers using deep reinforcement learning. In: Proceedings of the AIAA Scitech Forum, 2020: AIAA 2020-0460.
Crossref
[22]
Miller, D., Linares, R. Low-thrust optimal control via reinforcement learning. In: Proceedings of the 29th AAS/AIAA Space Flight Mechanics Meeting, 2019: 1817–1834.
[23]
LaFarge, N. B., Miller, D., Howell, K. C., Linares, R. Guidance for closed-loop transfers using reinforcement learning with application to libration point orbits. In: Proceedings of the AIAA Scitech Forum, 2020: AIAA 2020-0458.
Crossref
[24]

LaFarge, N. B., Miller, D., Howell, K. C., Linares, R. Autonomous closed-loop guidance using reinforcement learning in a low-thrust, multi-body dynamical environment. Acta Astronautica, 2021, 186: 1–23.

Crossref Google Scholar
[25]
LaFarge, N. B., Howell, K. C., Linares, R. A hybrid closed-loop guidance strategy for low-thrust spacecraft enabled by neural networks. In: Proceedings of the 31st AAS/AIAA Spaceflight Mechanics Meeting, 2021: AAS 21-302.
[26]
Sullivan, C. J., Bosanac, N. Using reinforcement learning to design a low-thrust approach into a periodic orbit in a multi-body system. In: Proceedings of the AIAA Scitech Forum, 2020: AIAA 2020-1914.
Crossref
[27]
LaFarge, N. B., Howell, K. C., Folta, D. C. An autonomous stationkeeping strategy for multi-body orbits leveraging reinforcement learning. In: Proceedings of the AIAA SCITECH Forum, 2022: AIAA 2022-1764.
Crossref
[28]
Bonasera, S., Elliott, I., Sullivan, C. J., Bosanac, N., Ahmed, N., McMahon, J. Designing impulsive station-keeping maneuvers near a Sun–Earth l2 halo orbit via reinforcement learning. In: Proceedings of the 31st AAS/AIAA Space Flight Mechanics Meeting, 2021: AAS 21-216.
[29]
Bosanac, N., Bonasera, S., Sullivan, C., McMahon, J., Ahmed, N. Reinforcement learning for reconfiguration maneuver design in multi-body systems. In: Proceedings of the AAS/AIAA Astrodynamics Specialist Conference, 2021.
[30]

Sutton, R. S., Barto, A. G. Reinforcement Learning: An Introduction. MIT Press, 2018.

[31]

Shirobokov, M., Trofimov, S., Ovchinnikov, M. Survey of machine learning techniques in spacecraft control design. Acta Astronautica, 2021, 186: 87–97.

Crossref Google Scholar
[32]

Izzo, D., Blazquez, E., Ferede, R., Origer, S., De Wagter, C., de Croon, G. C. Optimality principles in spacecraft neural guidance and control. Science Robotics, 2024, 9(91): eadi6421.

Crossref Google Scholar
[33]

Malyuta, D., Yu, Y., Elango, P., Açıkmeşe, B. Advances in trajectory optimization for space vehicle control. Annual Reviews in Control, 2021, 52: 282–315.

Crossref Google Scholar
[34]

Scorsoglio, A., Furfaro, R., Linares, R., Massari, M. Relative motion guidance for near-rectilinear lunar orbits with path constraints via actor-critic reinforcement learning. Advances in Space Research, 2023, 71(1): 316–335.

Crossref Google Scholar
[35]
Federici, L., Zavoli, A., Furfaro, R. Comparative analysis of reinforcement learning algorithms for robust interplanetary trajectory design. In: Studies in Computational Intelligence. Ieracitano, C., Mammone, N., Di Clemente, M., Mahmud, M., Furfaro, R., Morabito, F. C., Eds. Springer Nature Switzerland, 2023: 133–149.
Crossref
[36]
Holt, H., Baresi, N., Armellin, R. Towards optimal Lyapunov controllers for low-thrust lunar transfers via reinforcement learning. In: Proceedings of the AAS Astrodynamics Specialist Conference, 2021: AAS 21-615.
[37]
Holt, H., Bernardini, N., Baresi, N., Armellin, R. Reinforced Lyapunov controllers and convex optimisation for low-thrust lunar transfers. In: Proceedings of the AAS Astrodynamics Specialist Conference, 2023: AAS 23-247.
[38]
Petropoulos, A. E. Simple control laws for low-thrust orbit transfers. In: Proceedings of the AAS/AIAA Astrodynamics Specialists Conference, 2003.
[39]

Petropoulos, A. E. Refinements to the Q-law for the low-thrust orbit transfers. Advances in the Astronautical Sciences, 2005, 120: 963–982.

Google Scholar
[40]
Koon, W. S., Lo, M. W., Marsden, J. E., Ross, S. D. Dynamical systems, the three-body problem and space mission design. In: Equadiff 99. World Scientific Publishing Company, 2000: 1167–1181.
Crossref
[41]
Wakker, K. F. Fundamentals of Astrodynamics. TU Delft, 2015.
[42]
Holt, H. J. Trajectory design using Lyapunov control laws and reinforcement learning. Doctoral Dissertation. University of Surrey, 2023.
[43]

Pontani, M., Pustorino, M. Nonlinear Earth orbit control using low-thrust propulsion. Acta Astronautica, 2021, 179: 296–310.

Crossref Google Scholar
[44]
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O. Proximal policy optimization algorithms. arXiv preprint, 2017, arXiv: 1707.06347.
[45]
Scorsoglio, A. Adaptive ZEM/ZEV feedback guidance for rendezvous in lunar NRO with collision avoidance. Master Thesis. Politecnico Di Milano, University of Arizona, 2018.
[46]
Miller, D. D. M. Low-thrust spacecraft guidance and control using proximal policy optimization. Doctoral Dissertation. Massachusetts Institute of Technology, 2020.
[47]

Grondman, I., Busoniu, L., Lopes, G. A. D., Babuska, R. A survey of actor-critic reinforcement learning: Standard and natural policy gradients. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 2012, 42(6): 1291–1307.

Crossref Google Scholar
[48]
Sutton, R. S., McAllester, D., Singh, S., Mansour, Y. Policy gradient methods for reinforcement learning with function approximation. In: Proceedings of the 12th International Conference on Neural Information Processing Systems, 1999: 1057–1063.
[49]
Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., Riedmiller, M. Deterministic policy gradient algorithms. In: Proceedings of the International Conference on Machine Learning, 2014: 387–395.
[50]
Locoche, S. An analytical method for evaluation of low-thrust multi-revolutions orbit transfer with perturbations and power constraint. In: Proceedings of the Conference on Guidance, Navigation and Control Systems, 2018: 10–20
[51]

DeMars, K. J., Jah, M. K. Probabilistic initial orbit determination using Gaussian mixture models. Journal of Guidance, Control, and Dynamics, 2013, 36(5): 1324–1335.

Crossref Google Scholar
[52]

Betts, J. T., Erb, S. O. Optimal low thrust trajectories to the Moon. SIAM Journal on Applied Dynamical Systems, 2003, 2(2): 144–170.

Crossref Google Scholar
[53]

Shannon, J. L., Ozimek, M. T., Atchison, J. A., Hartzell, C. M. Rapid design of high-fidelity low-thrust transfers to the Moon. Journal of Spacecraft and Rockets, 2022, 59(5): 1522–1535.

Crossref Google Scholar
[54]
Ozaki, N., Hatakeyama, A., Ito, S., Chikazawa, T., Akiyama, Y., Yamamoto, T. Low-thrust many-revolution trajectory design under operational uncertainties. In: Proceedings of the AAS/AIAA Astrodynamics Specialist Conference, 2023: AAS 23-222.
[55]
Naasz, B. J. Classical element feedback control for spacecraft orbital maneuvers. Master Thesis. Virginia Tech, 2002.
[56]
Lantukh, D. V., Ranieri, C. L., DiPrinzio, M. D., Edelman, P. J. Enhanced Q-law Lyapunov control for low-thrust transfer and rendezvous design. In: Proceedings of the AAS/AIAA Astrodynamics Specialist Conference, 2017: AAS 17-589.
[57]
Locoche, S., Lagadec, K., Erb, S., Valles, C. Y. Reducing operation cost with autonomous guidance for electrical orbit raising. In: Proceedings of the 8th International Conference on Astrodynamics Tools and Techniques, 2021.
[58]

Narayanaswamy, S., Damaren, C. J. Equinoctial Lyapunov control law for low-thrust rendezvous. Journal of Guidance, Control, and Dynamics, 2023, 46(4): 781–795.

Crossref Google Scholar
Astrodynamics
Volume 8 Issue 4,
December 2024
Pages 633-656
DOI: 10.1007/s42064-024-0212-x
Cite this article:
Holt H, Baresi N, Armellin R. Reinforced Lyapunov controllers for low-thrust lunar transfers. Astrodynamics, 2024, 8(4): 633-656. https://doi.org/10.1007/s42064-024-0212-x
Metrics & Citations  
Article History
Copyright
Rights and Permissions
Return