Relevant experience learning: A deep reinforcement learning method for UAV autonomous motion planning in complex unknown environments

Unmanned Aerial Vehicles (UAVs) play a vital role in military warfare. In a variety of battlefield mission scenarios, UAVs are required to safely fly to designated locations without human intervention. Therefore, finding a suitable method to solve the UAV Autonomous Motion Planning (AMP) problem can improve the success rate of UAV missions to a certain extent. In recent years, many studies have used Deep Reinforcement Learning (DRL) methods to address the AMP problem and have achieved good results. From the perspective of sampling, this paper designs a sampling method with double-screening, combines it with the Deep Deterministic Policy Gradient (DDPG) algorithm, and proposes the Relevant Experience Learning-DDPG (REL-DDPG) algorithm. The REL-DDPG algorithm uses a Prioritized Experience Replay (PER) mechanism to break the correlation of continuous experiences in the experience pool, finds the experiences most similar to the current state to learn according to the theory in human education, and expands the influence of the learning process on action selection at the current state. All experiments are applied in a complex unknown simulation environment constructed based on the parameters of a real UAV. The training experiments show that REL-DDPG improves the convergence speed and the convergence result compared to the state-of-the-art DDPG algorithm, while the testing experiments show the applicability of the algorithm and investigate the performance under different parameter conditions.     

集群无人机队形重构及虚拟仿真验证

队形重构是集群无人机（UAV）控制的重要问题，指无人机按照要求安全、无碰撞地从一个队形变换到另一个队形，其难点在于快速规划最优安全轨迹并控制无人机进行轨迹姿态的高精度跟踪。针对集群无人机队形重构的上述问题，首先，基于CAPT（Concurrent Assignment and Planning of Trajectories）算法，解决了多无人机的目标分配和轨迹生成的实时性问题，实现了集群无人机的最优安全路径规划；其次，提出一种有限时间多变量积分滑模连续控制算法，解决了无人机轨迹姿态的高精度跟踪问题，并通过MATLAB仿真验证了该控制算法的有效性；最后，为了更加真实直观地演示无人机三维仿真效果，建立了基于Gazebo-ROS的无人机仿真平台，实现了12架四旋翼无人机队形重构"建模-仿真-可视化"的一体化仿真演示，验证了上述路径规划算法和轨迹姿态控制算法的有效性。     

Improving multi-target cooperative tracking guidance for UAV swarms using multi-agent reinforcement learning

Multi-Target Tracking Guidance(MTTG) in unknown environments has great potential values in applications for Unmanned Aerial Vehicle(UAV) swarms. Although Multi-Agent Deep Reinforcement Learning(MADRL) is a promising technique for learning cooperation, most of the existing methods cannot scale well to decentralized UAV swarms due to their computational complexity or global information requirement. This paper proposes a decentralized MADRL method using the maximum reciprocal reward to learn cooper...     

基于改进鲸鱼优化算法的无人机航路规划

针对复杂地形环境下的无人机航路规划问题，提出一种基于改进的鲸鱼优化算法的航路规划算法。首先，根据起始点和目标点等信息，通过坐标系旋转将二维航路规划问题转化为D维空间下的寻优问题；然后，将灰狼优化算法中的等级制度和微分进化算法中的贪婪策略引入鲸鱼优化算法提出改进的鲸鱼优化算法。在保证算法收敛速度的同时，所提的改进鲸鱼优化算法有效地提高了开发能力和搜索能力。最后，将提出的改进算法应用于无人机的航路问题求解。仿真结果表明，所提的改进鲸鱼优化算法能够有效的获得一条代价最优的、有效的航路结果，其性能优于传统的优化算法。     

基于深度强化学习的固定翼无人机编队协调控制方法

由于运动学的复杂性和环境的动态性,控制一组无人机遂行任务目前仍面临较大挑战。首先,以固定翼无人机为研究对象,考虑复杂动态环境的随机性和不确定性,提出了基于无模型深度强化学习的无人机编队协调控制方法。然后,为平衡探索和利用,将ε-greedy策略与模仿策略相结合,提出了ε-imitation动作选择策略;结合双重Q学习和竞争架构对DQN（Deep Q-Network）算法进行改进,提出了ID3QN（Imitative Dueling Double Deep Q-Network）算法以提高算法的学习效率。最后,构建高保真半实物仿真系统进行硬件在环仿真飞行实验,验证了所提算法的适应性和实用性。     

通信和测量受限条件下异构多UAV分布式协同目标跟踪方法

研究了通信和测量受限的异构多无人机(UAV)网络化分布式协同目标观测与跟踪问题.该分布式UAV系统采用长机一僚机异构型网络结构,以实现在电子静默和战术隐身条件下扩大探测和打击纵深.提出改进的一致性信息滤波(ICF)算法,实现通信和测量范围内各UAV节点的分布式信息融合.由于一致性算法的收敛性与网络拓扑结构的连通性密切相...     

多基地多无人机协同侦察问题研究

 充分考虑侦察目标的侦察分辨率要求和侦察时间窗约束,以及位于不同基地的无人机(UAV)的侦察性能和可用数目,首次建立了更加贴近军事应用实际的多基地多UAV协同侦察问题(M-MUCRP)的数学模型,并提出了解决该模型的多基地多UAV协同侦察进化算法(M-MUCREA)。M-MUCREA的染色体数据结构有效地表达了问题的解,有利于交叉、变异等进化操作;充分利用与目标侦察分辨率要求以及目标位置和时间窗约束相关的启发信息,构造初始种群,避免进化过程收敛太慢;基于Pareto最优概念的选择算子确保解在多个目标上的有效优化;精英策略避免了丢失进化过程中产生的非劣解,加快算法收敛;变异和交叉算子在保证有效解的前提下,实现了解的多样性,避免了算法陷入局部最优。仿真实验验证了算法能够有效解决M-MUCRP。     

切换拓扑下无人机集群系统时变编队控制

针对多无人机(UAV)间通信拓扑可能发生变化的情况,研究了具有二阶积分特性的无人机集群系统的轨迹跟踪与时变编队控制问题。基于一致性方法设计了编队控制器,将编队控制问题转换成闭环系统的稳定性问题,引入了切换拓扑平均驻留时间的概念,并在此基础上利用线性矩阵不等式(LMI)方法,给出了控制器设计步骤。通过构造分段连续Lyapunov函数,证明了切换拓扑下无人机集群系统能够实现对指定轨迹的跟踪并且实现时变编队飞行。以三维空间运动的无人机集群系统为例进行了仿真验证,结果表明本文所提方法能够解决切换拓扑下无人机集群系统的轨迹跟踪与时变编队问题。     

非对称机动能力多无人机智能协同攻防对抗

协同攻防对抗是未来军用无人机的重要作战场景。针对不同机动能力无人机群体间的攻防对抗问题，建立了多无人机协同攻防演化模型，基于多智能体强化学习理论，研究了多无人机协同攻防的自主决策方法，提出了基于执行-评判（Actor-Critic）算法的集中式评判和分布式执行的算法结构，保证算法稳定收敛的同时，提升执行效率。无人机的评判模块使用全局信息评价决策优劣引导策略学习，而执行时只需要依赖局部感知信息进行自主决策，提高了多机攻防对抗的效能。仿真结果表明，所提的多无人机强化学习方法具备较强的自进化属性，赋予了无人机一定智能，即稳定的自主学习能力，通过不断演化，能自主学习提升协同对抗的决策效能。     

Virtual target guidance-based distributed model predictive control for formation control of multiple UAVs

The paper proposes a Virtual Target Guidance (VTG)-based distributed Model Predictive Control (MPC) scheme for formation control of multiple Unmanned Aerial Vehicles (UAVs). First, a framework of distributed MPC scheme is designed in which each UAV only shares the information with its neighbors, and the obtained local Finite-Horizon Optimal Control Problem (FHOCP) can be solved by swarm intelligent optimization algorithm. Then, a VTG approach is developed and integrated into the distributed MPC scheme to achieve trajectory tracking and obstacle avoidance. Further, an event-triggered mechanism is proposed to reduce the computational burden for UAV formation control, which takes into consideration the predictive state errors as well as the convergence of cost function. Numerical simulations show that the proposed VTG-based distributed MPC scheme is more computationally efficient to achieve formation control of multiple UAVs in comparison with the traditional distributed MPC method.     

基于机动动作库的实时轨迹生成与仿真研究

论述了基于机动动作库的实时轨迹生成方法,根据环境和任务要求进行在线决策,选取机动动作库中合适的机动动作,最终实时生成无人机所需要的机动轨迹。仿真结果表明,这种方法具有实时性强、生成轨迹易于跟踪控制等优点。     

固定翼无人机编队集结控制算法研究

针对固定翼无人机协同作战时的编队集结问题，提出了一种新的路径规划和位置分配方法，并设计了包括航迹跟踪、高度保持和速度控制在内的自动驾驶仪。该路径规划算法通过矩阵迭代得到一组较优的目标点分配方案，满足总航程较小和同时到达约束。根据得到的各无人机飞向目标点的航迹，算出无人机编队集结的代价矩阵。在每架无人机确定了应飞航路后，开始沿航路飞向目标点，在此过程中，纵向采用高度保持自动驾驶仪，横向采用航迹跟踪自动驾驶仪，控制无人机按规定航迹飞行。速度调节自动驾驶仪可根据速度指令调节油门大小加减速，跟踪上目标速度，进而实现编队集结。仿真结果验证了所提出的编队集结控制方法的有效性和可行性。     

增广误差模型算法在目标跟踪中的应用

针对目标机动运行过程中,滤波模型与机动状态模型失配的问题,提出了一种新的增广状态误差滤波模型。不同于现有增广方案,该模型从模型失配所致状态滤波误差的角度出发,将状态估计误差增广为一状态量,通过滤波估计后用其校正原状态量。算法分析表明,该增广滤波模型具有自适应调节多重渐消因子的等效特性,增强了对目标的跟踪能力。基于该增广状态误差滤波模型,给出了滤波算法设计并进行了仿真实验。实验结果表明,基于该模型的滤波算法在对机动目标进行跟踪时具有更强的鲁棒性。     

An online ensemble semi-supervised classification framework for air combat target maneuver recognition

Online target maneuver recognition is an important prerequisite for air combat situation recognition and maneuver decision-making. Conventional target maneuver recognition methods adopt mainly supervised learning methods and assume that many sample labels are available. However, in real-world applications, manual sample labeling is often time-consuming and laborious. In addition, airborne sensors collecting target maneuver trajectory information in data streams often cannot process information in real time. To solve these problems, in this paper, an air combat target maneuver recognition model based on an online ensemble semi-supervised classification framework based on online learning, ensemble learning, semi-supervised learning, and Tri-training algorithm, abbreviated as Online Ensemble Semi-supervised Classification Framework (OESCF), is proposed. The framework is divided into four parts: basic classifier offline training stage, online recognition model initialization stage, target maneuver online recognition stage, and online model update stage. Firstly, based on the improved Tri-training algorithm and the fusion decision filtering strategy combined with disagreement, basic classifiers are trained offline by making full use of labeled and unlabeled sample data. Secondly, the dynamic density clustering algorithm of the target maneuver is performed, statistical information of each cluster is calculated, and a set of micro-clusters is obtained to initialize the online recognition model. Thirdly, the ensemble K-Nearest Neighbor (KNN)-based learning method is used to recognize the incoming target maneuver trajectory instances. Finally, to further improve the accuracy and adaptability of the model under the condition of high dynamic air combat, the parameters of the model are updated online using error-driven representation learning, exponential decay function and basic classifier obtained in the offline training stage. The experimental results on several University of California Irvine (UCI) datasets and real air combat target maneuver trajectory data validate the effectiveness of the proposed method in comparison with other semi-supervised models and supervised models, and the results show that the proposed model achieves higher classification accuracy.     

A memetic algorithm for path planning of curvature-constrained UAVs performing surveillance of multiple ground targets

The problem of generating optimal paths for curvature-constrained unmanned aerial vehicles （UAVs） performing surveillance of multiple ground targets is addressed in this paper. UAVs are modeled as Dubins vehicles so that the constraints of UAVs＇ minimal turning radius can be taken into account. In view of the effective surveillance range of the sensors equipped on UAVs, the problem is formulated as a Dubins traveling salesman problem with neighborhood （DTSPN）. Considering its prohibitively high computational complexity, the Dubins paths in the sense of terminal heading relaxation are introduced to simplify the calculation of the Dubins distance, and a boundary-based encoding scheme is proposed to determine the visiting point of every target neighborhood. Then, an evolutionary algorithm is used to derive the optimal Dubins tour. To further enhance the quality of the solutions, a local search strategy based on approximate gradient is employed to improve the visiting points of target neighborhoods. Finally, by a minor modification to the individual encoding, the algorithm is easily extended to deal with other two more sophisticated DTSPN variants （multi-UAV scenario and multiple groups of targets scenario）. The performance of the algorithm is demonstrated through comparative experiments with other two state-of-the-art DTSPN algorithms identified in literature. Numerical simulations exhibit that the algorithm proposed in this paper can find high-quality solutions to the DTSPN with lower computational cost and produce significantly improved performance over the other algorithms.     

Study on the resolution of multi-aircraft flight conflicts based on an IDQN

With the rapid growth of flight flow, the workload of controllers is increasing daily, and handling flight conflicts is the main workload. Therefore, it is necessary to provide more efficient conflict resolution decision-making support for controllers. Due to the limitations of existing methods, they have not been widely used. In this paper, a Deep Reinforcement Learning(DRL) algorithm is proposed to resolve multi-aircraft flight conflict with high solving efficiency. First, the characteristics ...     

基于零空间方法的四旋翼无人机避障与协同编队控制

针对四旋翼无人机在编队飞行执行任务时可能遭遇障碍物问题,考虑多无人机避障及机间避撞的需求,提出 1种基于零空间方法的四旋翼无人机避障与协同编队控制算法。首先,建立四旋翼无人机动力学模型,并建立虚拟控制量简化控制模型;其次,基于零空间方法进行避障与协同编队控制算法研究,将无人机任务执行分解为目标趋向任务、避障避撞任务和协同编队任务,并根据优先级进行任务融合得到期望速度;再次,基于 PID方法设计控制律;最后,通过仿真验证所提控制算法的有效性。所提方法可保证四旋翼无人机在编队飞行中遭遇障碍物时的飞行安全。     

城市风场环境中的无人机快速航迹规划方法

密集的城市障碍环境以及复杂的城市风场干扰对航迹规划的实时性和航迹跟踪的准确性提出了严格要求,为此提出一种城市风场环境中的小型无人机(UAVs)快速航迹规划方法。首先,为了保证航迹规划的高效性,对固定翼无人机运动学方程进行了合理简化。其次,由于障碍环境中的最优航迹难以直接完全塑造,因此根据状态受限的最优控制理论给出了可以使用螺旋线与直线构建近似最优航迹的结论,并据此提出了一种针对城市环境的三维航迹规划方法。然后,通过对无人机运动学模型的分析,从规划角度提出了风场干扰下的航迹设计准则。仿真实验中,首先通过算法对比实验,验证了航迹规划方法的高效性;然后使用六自由度(DOF)飞机模型分别在无风场干扰和有风场干扰的环境下进行了航迹跟踪实验,实验结果证明了风场干扰下航迹设计准则的有效性。     

基于罚函数序列凸规划的多无人机轨迹规划

多无人机（UAVs）轨迹规划是具有非线性运动约束和非凸路径约束的最优控制问题。引入序列凸规划思想,将非凸最优控制问题近似为一系列凸优化子问题,并利用成熟的凸优化算法进行求解,以更好地权衡最优性和时效性。首先,建立了多无人机协同轨迹规划的非凸最优控制模型。然后,利用离散化和凸近似方法将其转换为凸优化问题,包括对无人机运动模型的线性化,以及对威胁规避约束和无人机碰撞约束的凸化。同时,提出了一种离散点间的威胁规避方法,保证无人机在离散轨迹点间的飞行安全。在凸优化模型的基础上,给出了基于罚函数序列凸规划求解多无人机轨迹规划的具体框架。最后,通过数值仿真验证了方法的有效性,结果表明该方法在多机轨迹规划结果的最优性和时效性都要优于伪谱法,而且优势随编队数量的增加而增大。     

面向城市飞行安全的无人机离散型多路径规划方法

为了提高无人机（UAV）在城市环境中运行的安全性,且能生成多条备选路径,提出一种离散型城市环境下基于无人机飞行安全的多路径规划方法。根据定义的城市环境模型、无人机的飞行规则和安全性原则,建立无人机飞行安全性分析模型和离散型多路径规划问题的数学模型。为提高算法的收敛速度和解的优质性,以及使算法能够同时输出多条路径,针对蚁群（ACO）算法的运行机制,设计聚类算子,提出改进聚类蚁群（CIACO）算法。实验结果表明,所提方法能够快速的收敛输出多条风险值较低的飞行路径。