Multi-agent Q-Learning control of spacecraft formation flying reconfiguration trajectories

This paper presents a novel approach based on multi-agent reinforcement learning for spacecraft formation flying reconfiguration tracking problems. In this scheme, spacecrafts learn the control strategy via transfer learning. For this matter, a new generalized discounted value function is introduced for the tracking problems. Due to the digital nature of spacecraft computer systems, local optimal controllers are developed for the spacecrafts in discrete-time. The stability of the controller is proven. Two Q-learning algorithms are proposed, in each of which the optimal control solution is learned on-line without knowledge about the system dynamics. In the first algorithm, each agent learns the optimal control independently. In the second one, each agent shares the learned information with other agents. Next, the collision avoidance capability is provided. The effectiveness of the presented schemes is verified through simulations and compared with each other.     

Locally generalised multi-agent reinforcement learning for demand and capacity balancing with customised neural networks

Reinforcement Learning (RL) techniques are being studied to solve the Demand and Capacity Balancing (DCB) problems to fully exploit their computational performance. A locally generalised Multi-Agent Reinforcement Learning (MARL) for real-world DCB problems is proposed. The proposed method can deploy trained agents directly to unseen scenarios in a specific Air Traffic Flow Management (ATFM) region to quickly obtain a satisfactory solution. In this method, agents of all flights in a scenario form a multi-agent decision-making system based on partial observation. The trained agent with the customised neural network can be deployed directly on the corresponding flight, allowing it to solve the DCB problem jointly. A cooperation coefficient is introduced in the reward function, which is used to adjust the agent’s cooperation preference in a multi-agent system, thereby controlling the distribution of flight delay time allocation. A multi-iteration mechanism is designed for the DCB decision-making framework to deal with problems arising from non-stationarity in MARL and to ensure that all hotspots are eliminated. Experiments based on large-scale high-complexity real-world scenarios are conducted to verify the effectiveness and efficiency of the method. From a statistical point of view, it is proven that the proposed method is generalised within the scope of the flights and sectors of interest, and its optimisation performance outperforms the standard computer-assisted slot allocation and state-of-the-art RL-based DCB methods. The sensitivity analysis preliminarily reveals the effect of the cooperation coefficient on delay time allocation.     

基于强化学习的多无人机避碰计算制导方法

针对大量固定翼无人机在有限空域内的协同避碰问题,提出了一种基于多智能体深度强化学习的计算制导方法。首先,将避碰制导过程抽象为序列决策问题,通过马尔可夫博弈理论对其进行数学描述。然后提出了一种基于深度神经网络技术的自主避碰制导决策方法,该网络使用改进的Actor-Critic模型进行训练,设计了实现该方法的机器学习架构,并给出了相关神经网络结构和机间协调机制。最后建立了一个实体数量可变的飞行场景模拟器,在其中进行"集中训练"和"分布执行"。为了验证算法的性能,在高航路密度场景中进行了仿真实验。仿真结果表明,提出的在线计算制导方法能够有效地降低多无人机在飞行过程中的碰撞概率,且对高航路密度场景具有很好的适应性。     

基于冲突分类与消解的多智能体路径规划算法设计CSCD

多智能体路径规划应用广泛但求解困难。为更好地处理多智能体路径规划中的路径冲突问题,提高求解效率,将冲突进一步分类为相向顶点冲突和交叉顶点冲突,并提出了对应的消解方式。相向顶点冲突的消解方法采用提前添加约束的方式,避免在消解其冲突的过程中产生另一个可预见的冲突;交叉顶点冲突的消解方法采用寻找最佳等待时间的方式,在消解其冲突的同时消解其他存在的冲突。两种冲突消解方法均可减小约束树的规模,在一定程度上减少算法的计算量。并提出了基于冲突搜索算法的高层节点冲突搜索算法。实验结果表明,所提出的冲突分类及消解方式有效地减小了算法高层中约束树的规模,降低了算法计算量,并在智能体密集的环境下表现出更大的优势。     

Adaptive event-triggered distributed optimal guidance design via adaptive dynamic programming

In this paper, the multi-missile cooperative guidance system is formulated as a general nonlinear multi-agent system. To save the limited communication resources, an adaptive eventtriggered optimal guidance law is proposed by designing a synchronization-error-driven triggering condition, which brings together the consensus control with Adaptive Dynamic Programming(ADP) technique. Then, the developed event-triggered distributed control law can be employed by finding an approximate solution of eve...     

具有动态领导者的多智能体系统分组一致性控制

对一类具有动态领导者的非线性多智能体系统的一致性进行研究，解决了领导-跟随一致性控制问题。首先，考虑到多目标任务中广泛存在的合作-竞争机制和领导者的控制输入非零的情况，基于邻居智能体的相对状态信息，设计了一类分布式的分组一致性控制器。通过在所设计的控制器中增加补偿项，解决了由领导者非零控制输入引出的问题。引入了一类Lipschitz-like条件，解决了非线性项在实现分组一致时所带来的困难。其次，利用图论和构造Lyapunov函数，通过求解代数Riccati方程得出了系统实现分组一致性的充分条件，并基于所设计的控制器实现了多智能体系统动态领导-跟随的分组一致性控制。最后，通过数值仿真和结果分析验证了所提出控制方案的有效性和可行性。     

Improving multi-target cooperative tracking guidance for UAV swarms using multi-agent reinforcement learning

Multi-Target Tracking Guidance(MTTG) in unknown environments has great potential values in applications for Unmanned Aerial Vehicle(UAV) swarms. Although Multi-Agent Deep Reinforcement Learning(MADRL) is a promising technique for learning cooperation, most of the existing methods cannot scale well to decentralized UAV swarms due to their computational complexity or global information requirement. This paper proposes a decentralized MADRL method using the maximum reciprocal reward to learn cooper...     

集群多机器人系统建模研究的发展与展望

凭借高效、鲁棒、应用广泛的特性,集群多机器人系统已经成为当今研究的热点课题之一,具有重要的实用价值。首先简述了自顶而下和自底而上两种多机器人系统研究思路的当前研究概况。然后从拟生物集群系统模型引入,进而引出一致性系统模型,针对低阶、高阶、异质、时延等一致性模型进行分析总结,单独阐述了多智能体强化学习系统模型的研究情况,并分别讨论了三种集群多机器人系统自组织建模方法的研究现状与各自存在的问题,总结与分析了集群多机器人系统运动的发生机理。最后,分析了现有集群多机器人系统模型尚待解决的关键问题和面临的挑战,并对其未来发展进行了展望。