Multi-agent Q-Learning control of spacecraft formation flying reconfiguration trajectories

This paper presents a novel approach based on multi-agent reinforcement learning for spacecraft formation flying reconfiguration tracking problems. In this scheme, spacecrafts learn the control strategy via transfer learning. For this matter, a new generalized discounted value function is introduced for the tracking problems. Due to the digital nature of spacecraft computer systems, local optimal controllers are developed for the spacecrafts in discrete-time. The stability of the controller is proven. Two Q-learning algorithms are proposed, in each of which the optimal control solution is learned on-line without knowledge about the system dynamics. In the first algorithm, each agent learns the optimal control independently. In the second one, each agent shares the learned information with other agents. Next, the collision avoidance capability is provided. The effectiveness of the presented schemes is verified through simulations and compared with each other.