current position:Home>Autopilot Lidar, Camera, Millimeter Wave Radar Fusion Algorithm

Autopilot Lidar, Camera, Millimeter Wave Radar Fusion Algorithm

2022-08-06 06:35:48HIT_Vanni

Driverless cars the rest of the multi sensor data fusion algorithm research

【论文】详见知网链接ELSEVIER链接IEEE链接,【开源项目】详见github链接

【编译运行】详见DWD_sensor_fusion编译运行

0. 主要成果

1.Based on laser radar、摄像头、The multi-sensor redundancy of the millimeter wave radar system,Inspection end using laser radar objectCNN_SEG、Object detection cameraYOLOV4And generating millimeter wave objectBox-Muller算法,On track again for each sensor object USES the kalman filter combined with the Hungarian matchingSORT算法.
2.Based on kalman filter variance of multi-sensor redundancy weight distribution system,The system is picked up from the grind ofDWD算法,The system USES a consider the distribution of weight average variance of deviation and the variance of the average rate of change of the weight,And supplemented by weight drop sensor and sensor exit mechanism,Finally finally each sensor is obtained by weighting normalization weights,And check the weight distribution curve through the experiment,Can see the weight drop mechanism and sensor during exit mechanism
3.Based on multisensor dynamic redundancy weight perception of multi-sensor fusion system,该系统利用DWDAlgorithm for the weight of each sensor allocation effect on to provide a basis for the fusion of multi-sensor fusion end,A weighting sensor is higher,The sensor fusion results in the process of integration of the higher the decisive role,反之,The lower the weighting sensor,The sensor fusion results in the process of fusion of the lower the decisive role,If the weight sensor is zero,Namely the sensor exit.

1.主要研究内容

1.Prior to redundant multi-sensor perceptual system set up:
Select the 3 d laser radar-毫米波雷达-Visual redundancy, prior to the multi-sensor perceptual system.激光雷达使用CNN_SEG算法,摄像头采用YOLOV4算法,Millimeter wave radar usingBox-Muller算法.
2.Multiple sensor coordinate transform data alignment problems:
USES the camera coordinate system on the basis of,Compared with the color camera coordinate transformation need rotation matrix and translation matrix,With these two matrices can be coordinate system of two sensor coordinate transformation,And other sensors, such as laser radar、Conversion of the coordinates of the millimeter wave radar to camera coordinates.
3.Sensor multi-target tracking state estimation problem:
The essence of the world's nonlinear,For intelligent vehicle,The perception of the outside world is so,Is very necessary to introduce the nonlinear state estimation method of intelligent vehicle in multiple target tracking,The associated good target for filter,Then the sensor target tracking state,In sensor tracking part,采用了SORT(Simple Online And Realtime Tracking)算法,The algorithm including the Hungarian matching data association and kalman filter tracking.
4.Multisensor forward redundancy weight distribution:
Through the variance of the average deviation of each sensors、Variance of the average rate of change,And give each sensor corresponding weight,To simulate the sensor fault,The whole system can still perception of target accurately,This paper developed a dynamic weighting allocation algorithm——DWD(Dynamic Weight Distribution)算法.
5.Multi-sensor target associated with information fusion problem:
For each sensor information fusion is made,For each sensor of the consistency of the same target detected by explain.Will some sensor fails or precision falling target to eliminate or reduce their impact on the fusion results.

在这里插入图片描述

2.Multi-sensor multi-target detection

2.1激光雷达感知cnnseg算法

CNN_SEGAlgorithm is adopted based on a bird's eye view overlooking the direction(Bird View, BV)Point cloud training of neural network is used to detect the obstacles.Its ability to identify the object types have six classes(car, bicycle, track,bus, pedestrian, sign and others).Point cloud data is used in the testKitti开源数据.感谢知乎大神@MouJiaJunapolloPerception algorithm of analyticcnn_seg
cnnseg其实分为两步,The first step is to use a deep learning network forecast five layer information,center offset, objectness, positiveness, object height, class probability,The second step is to use the five layer information clustering[14].具体步骤如下:

  1. 根据objectness层信息,Detect obstacles grid point target
    With images and point cloud to illustrateobjectness层的作用.objectness输出的是一个[0,1]的二维矩阵,With the raster map for point cloud(在cnn_seg中,The point cloud is constructed2.5D栅格,Each grid corresponding to the8维特征).通过对objectnessConfidence in threshold(默认0.5),就可将objectThe pixel detected.According to information below with point cloud reflection intensity are detected area.
    在这里插入图片描述

  2. 根据center offset层信息,To detect obstacles grid points clustering,By clustering goals
    center offsetIs a two-dimensional matrix of two layers of,分别是x轴的offset和y轴的offset,The pair of layers together,Corresponding to each gridoffset,Length in the direction of the arrow and saidoffset.
    在这里插入图片描述
    在这里插入图片描述

  3. 根据positiveness和object height层信息,The background as well as the high point in each point cloud cluster filtering
    positiveness是一个filter信息.From the previous step actually can see,The point cloud cluster,There are a lot of background region.Through the statistics of each point cloud clusterpositiveness的均值,在经过阈值(默认0.1),The clustering can filter out some background.
    在这里插入图片描述

object height也是一个filter信息,Filter out the high point in grid.
没有使用object height信息
使用object height信息

  1. 根据class probability,Classifying each clustering goal,Get the final detection target
    通过以上几步,cnnsegHave made the point cloud segmentation into point cloud cluster,Type are respectively cart,小车,行人,自行车,未知.Through the statistics of each class of each point cloud cluster probability value,Higher scores for their kind,For each point cloud clusters so as to realize the classification of,得到最终的target.
    下图中,Said the car in red,Blue cart,Green said pedestrians.
    在这里插入图片描述
    The figure below shows this algorithm to detect the object,As shown in figure of square box,With the type of the object and the distance from the default of.
    在这里插入图片描述
    Below detection point cloud topic of output after,As you can see by the side of the road vehicle, and to drive a car.Another topic to detect target output,You can see the various types of object detection is right,至此,The part of the algorithm testing done.
    在这里插入图片描述在这里插入图片描述

2.2 Camera perception algorithm

This training adoptsXeon W-2150B/3.00GHz CPUAnd two2080Ti GPU进行训练,使用10000张Kitti数据集,共训练50000次,The training time about68小时,Training categories a total of5类(pedestrian、car、bicycle、truck、bus)The picture on the right for trainingloss值(公式如下)记录:

在这里插入图片描述
在这里插入图片描述
The trained weights fileAP(Average Precision)及mAP(mean Average Precision)Accuracy test,其结果如表2-1,Precision at a higher level:

类别AP(%)
Pedestrian89
Bicycle86
Car94
Truck78
Bus79
mAP85.2
This part of the algorithm is the detection effect of,Use of the input image asKittiThe left side of the data sets the topic of color camera,话题名为/left_color_cam,The test results,Orange box ascar,蓝色框为truck.由图中可以看出YOLOV4The effect of detection of various objects are relatively good.
在这里插入图片描述

2.3.Based on laser object of millimeter waveBox-MullerMulti-objective generation algorithm

Because the sensors to detect target is roughly in a gaussian distribution,So using the up and down or so four edges of laser radar target to join gaussian random number to deal with laser radar target,The results as the millimeter wave radar target,由于C++语言可以用randFunction to generate uniformly distributed random number,So this research intends to adoptBox-Muller算法[16],The algorithm can generate gaussian distribution according to the uniform distribution,具体算法见知网链接

经过测试,Decided to adopt the scope for(-30,30)的高斯分布,This requires an expectation for0,方差为10的高斯分布,即Y~N(0,10).The actual output below,This distribution for an expectation for-1.891986493192650,方差为9.79255102687986的高斯分布.
在这里插入图片描述

3.Sensor coordinate transformation and multiple target tracking

3.1 Laser radar and the millimeter wave radar target coordinate transformation

Coordinate transformation method is adoptedGeiger, A等对kittiData sets coordinate transformation method,Eventually transform laser point cloud to camera coordinates,And generate the millimeter wave radar target.Coordinate change codegithub有.
在这里插入图片描述
在这里插入图片描述
Object scatter plot
在这里插入图片描述

3.2 Multi-sensor multi-target tracking

Kalman filter module's main function is through kalman filter algorithm on to the next frame to the identified to target state prediction and optimal estimation.卡尔曼滤波过程:
在这里插入图片描述
SORT(Simple Online And Realtime Tracking)算法[19]Including Hungary matching and kalman filter.
First of all to show the Hungarian matching,上下两图分别为T1时刻和T2时刻,假设T1时刻成功跟踪了某个单个物体,ID为1,绘制物体跟踪BBox(紫色).T2时刻物体检测BBox总共有3个(橙色),预测T2时刻物体跟踪的BBox(紫色)有1个,解决紫色物体跟踪BBoxHow the object detection with orangeBBox关联的算法,就是SORTTracking algorithm to solve the core issue of.
在这里插入图片描述
经检验,SORTPart of the program implementationIOU匈牙利匹配,After the match the target location of the output is good for subsequent kalman filter.使用2D MOT 2015 benchmark dataset[20]Open source data set to test the module,Many objects matching index is as follows,Most of the object matching degree is higher,Matching module performance good
在这里插入图片描述
在这里插入图片描述

4.Multisensor dynamic redundancy weight allocation algorithm research

4.1 Summary of dynamic weighting allocation algorithm

In the process of the weight,Study a dynamic weighting allocation(DWD,Dynamic Weight Distribution)算法,Set the initial value variance, first of all, for1,Check multiple object tracking good variance changes as shown in figure.Will track more than20Frame objects to screen out,According to these variance stable value of the object,By calculating arithmetic average to establish a stable value variance centerσ0=0.65514125,It is set as the initial value of variance,初始值σ0Can make the object of variance fast convergence.

在这里插入图片描述在这里插入图片描述
In the process of testing for a long time,All the variances of the tracker output as follows,Figure appeared in some outliers,For these outliers,Should be in the design of variance threshold method eliminate.

在这里插入图片描述
After eliminating outliers,Find all the variance inσmax=0.6730585,σmin=0.639224范围内波动,And most of the points are concentrated inσ0The initial value position,So the maximum threshold is set toσmax,The minimum variance threshold is set toσmin.
在这里插入图片描述

4.2 Considering the dynamic weight average variance of deviation distribution algorithm

The specific algorithm see paper知网链接
对于激光雷达、Millimeter wave radar and camera objects are processed using this method to,平均偏差越大,Means that the sensor detection object that there might be a problem,权重就越小.

在这里插入图片描述

4.3 Considering the variance rate of dynamic weighting allocation algorithm

The specific algorithm see paper知网链接
对于激光雷达、Millimeter wave radar and camera objects are processed using this method to,The greater the average rate of change,Means that the sensor detection object that there might be a problem,权重就越小.
在这里插入图片描述

4.4 Sensors exit decreasing dynamic weighting allocation and test

Automatic weight distribution is primarily to test the system down and sensors exit mechanism does work,When a sensor in the detection and tracking effect is not good situation,The system should be to reduce the weight of the sensor of them,Serious when will make the sensor directly exit.By using large gaussian noise processing per frame to check whether the weight of each sensor will reduce or quit.
在这里插入图片描述
在这里插入图片描述
在这里插入图片描述

4.5 Weight normalized and weight distribution rule

The above consider the variance of the average deviation of weight distribution, and considering the variation of the variance weight distribution,Two parts of each distribution0.5的权重,Normalization is to turn data into(0,1)之间的小数.This research adopts the weights of normalization method for zooming interval method,Zoom in interval method and the trans,The original longer values in the interval scale to(0, 1)Between the smaller after numerical.
According to the previous section described,According to whether sensors exit can get the following eight kinds of situation,OIs not out,X为退出:
在这里插入图片描述
在这里插入图片描述

4.6 The final weight allocation result

DWDAlgorithm by more than the initial value setting、Considering variance average deviation of the weight、Considering variance average rate of change of the weight、Decreasing weight sensor and sensor exit mechanism、权重归一化,Get the final multi-sensor weight,另外,The system not only applies to three sensors' system,If you want to increase or decrease in perceived sensor,Only need to change the rules of sensor dynamic weighting system,总体思想不变,So the system has a strong ductile.
System most of the time in the weight of three sensors distribution,But the system also has some time on the weight distribution of two sensors,例如5-10帧、170-185帧附近,There are some time for the single sensor weight distribution,例如155帧附近、158帧附近、160帧附近、220帧附近、233帧附近,最特殊的,System also has carried on the weight distribution of three sensors exit at the same time,Due to the three sensors are exit weights allocation according to1/3平均分配,So average distribution situation is the three sensors are out of,例如163-170帧、212-216帧.
在这里插入图片描述

5. Based on dynamic redundancy weight of multi-sensor data fusion

The specific algorithm see paper知网链接
下图中C31、L16、R17Become an object,C33、L17、R18Become an object.
在这里插入图片描述

下图中C14、L4、R4Successful fusion object as an object,C11、L5、R5Successful fusion as an object,In addition the frame is a special case,即C18Object only camera tracking success,But no laser radar and the millimeter wave radar for the object tracking,But testing box are output,So go for the fusion of the object is successful.
在这里插入图片描述
Using dynamic redundancy weight and do not use the dynamic redundancy weight directly compared fusion,Can successfully blended into the target number of frames significantly more,对比结果如下.Figure in the green for the use of dynamic weighting fusion results,Blue for not using the results of the dynamic weight,After using dynamic redundancy weight,Number of frames produce fusion object by the182帧提高到228帧,The data set to the total number of frames250帧;Fused object also has obvious improvement,Will object to the success of each frame fusion sum,Fusion successfully the number of objects by the827个提高到1089个,This shows to join dynamic redundancy weight of multi-sensor fusion has a positive role in,Can enhance the success rate of fusion.
在这里插入图片描述
【论文】详见知网链接ELSEVIER链接IEEE链接,【开源项目】详见github链接

copyright notice
author[HIT_Vanni],Please bring the original link to reprint, thank you.
https://caren.inotgo.com/2022/218/202208060519211890.html

Random recommended