No localization with Livox Lidar

[WeilongXia]

Thanks a lot for opening source such a great work, but when I run the code, I found that the odometry output pose has always been the origin and has not changed. I have checked the data topics of lidar and imu, and there is nothing wrong. The frequency of lidar is 10Hz, and the frequency of imu is 200Hz. What are the possible reasons for the above case? Looking forward to your reply~

[kennyjchen]

Yeah I haven't tested Livox support out since I don't have any data. Think you could upload a bag for me to play around with? Thanks!

[WeilongXia]

This is the dataset of Livox-mid360: https://1drv.ms/u/s!AjkIEg1kWUNeiAIZHIDBTzXmtHqf?e=cWek2u
And the extrinsic between the Lidar frame and IMU frame is

[Joosoo1]

Yes, it seems that we are experiencing the same problem, may I ask if you are using it as a navigation for robots? Drones or unmanned vehicles?Maybe we can communicate that i have recently encountered some difficulties.

[WeilongXia]

Yeah I 'm going to use it as a localization module for drones, and maybe we can use the g-mail to communicate: [email protected]

Yes, it seems that we are experiencing the same problem, may I ask if you are using it as a navigation for robots? Drones or unmanned vehicles?Maybe we can communicate that i have recently encountered some difficulties.↳

[Joosoo1]

ok, after my test, livox-mid360 can run DLIO successfully, just change the point cloud publishing form "xfer_format" to "2" in livox_ros_driver2, then it can run DLIO successfully. DLIO can be run successfully, as follows:

[Joosoo1]

But the results do not seem to be very good, odom data z-axis jitter will be relatively large, may be due to the reason of not tuning parameters, but also may be due to the built-in imu noise is too large, the follow-up we can exchange about how to make it better and more stable, in addition, I am also intended to apply it to the top of the drone, used as navigation and positioning.

[WeilongXia]

Thanks to your reply, the dimension of the acceleration data from the build-in imu in Livox MID-360 is g(gravity) instead of m/s^2, maybe you should multiply the data with a constant value like 9.8?

[Joosoo1]

Thanks to your reply, the dimension of the acceleration data from the build-in imu in Livox MID-360 is g(gravity) instead of m/s^2, maybe you should multiply the data with a constant value like 9.8?

ok,i will try!If you have better results, please let me know too!

[WeilongXia]

I have tried to set xfer_format t o 2, and the result seems great!

[Joosoo1]

wow! Your results look great, the position estimation is accurate and the built point cloud map has no drift, but my results are as bad as ever, still with a large offset on the z-axis, can you share your configuration parameters file? Also, about the z-axis, my livox has a z-axis accel bias of -5, while yours is 0. I think there is something wrong here.

[kennyjchen]

Hi @WeilongXia and @Joosoo1 -- thanks for the great discussion you two have started here. Looks like the Livox branch is working! It just needs to be published in the standard format.

Yeah something is definitely wrong if the IMU biases are that large. Have you checked your extrinsics? That's typically an issue, especially since the convention for IMU and LiDAR are different.

Edit: it's working, but deskewing is not (note the deskewed points: 0 in the terminal screenshots above). Looks like a per-point timestamp issue with Livox sensors.

[kennyjchen]

@Joosoo1 Yes, change the units of the accelerometer to m/s^2 as per the advice above.

Could someone record me a dataset in the new xfer_format? Thanks.

[Joosoo1]

Well, it's true that because of the acceleration units, it has to be m/s^2 for it to work well.
I provide the dataset：https://1drv.ms/u/s!AiyqJY-cvK5varx4IwdfshuTfeE?e=TtbAdI
pointcloud_topic:/livox/lidar
imu_topic:/livox/imu (the acceleration unit is g, so you need to modify the code in DLIO)
The parameters are:
! livox-mid360 (built-in imu) extranet

[dxdydz3dt]

Can you translate in English Please!. I am trying to run Livox but its not working

[Joosoo1]

Can you translate in English Please!. I am trying to run Livox but its not working
You can take a look at this: #5 (comment)

It is necessary to additionally convert the acceleration units of the /livox/imu topic output to m/s^2,then the DLIO will work well, of course all this needs to be under the source code of the livox branch.

[Joosoo1]

@kennyjchen I found that when livox outputs the standard point cloud, its feild.name is not "offset_time", so according to the code "deskew_= false", deskewing is not wrok!

[kennyjchen]

Hmm, yeah unfortunately it looks like in the livox_ros_driver2 package, only x y z and intensity fields are copied over when publishing using the xfer_format: 2 value.

https://github.com/Livox-SDK/livox_ros_driver2/blob/master/src/lddc.cpp#L437

It looks like with their custom message, they have per-point timestamps . However, it looks like both timestamp and offset_time are used, which is a bit confusing since their CustomPoint.msg contains offset_time, but the data above (the first link) has a timestamp field.

If only timestamps are available using their custom point format, we'll need to build support for that (unless they add timestamps to the PCL format). I'll keep digging.

[kennyjchen]

Ok -- I think I might have figured this out. For future reference, this is how different sensors timestamp their points:

• Ouster
  • field t of type std::uint32_t, which is the time relative to the beginning of the scan in nanoseconds
• Velodyne
  • field time of type float, which is the time relative to the beginning of the scan in seconds
• Livox
  • xfer_format: 0: field timestamp of type double, which is the absolute time of the point since epoch in nanoseconds
  • xfer_format: 1: field offset_time of type std::uint32_t, which is the time relative to the beginning of the scan in nanoseconds
• Hesai
  • field timestamp of type double, which is the absolute time of the point since epoch in seconds

I pushed a few changes to the feature/livox-support branch. Please test this out using the original xfer_format value so that there are timestamp fields.

@WeilongXia -- I played around with your data, and it looks like the LiDAR and IMU measurements are not time synchronized. Is there an option in the Livox driver to synchronize them somehow?

EDIT 6/28: updated the list above to be more accurate and include both xfer_format: 0 and xfer_format: 1. As of the time of this writing, seems like there is a bug in timestamping with xfer_format: 0.

[WeilongXia]

Ok, I found a timesync choice in the config file like this

and I'll try to record a new rosbag.

[juliangaal]

@WeilongXia could you rename the issue to something more meaningful, if you don't mind, for future reference? Something like "No localization with Livox Lidar"

[WeilongXia]

I tested the feature/livox-support branch with the original xfer_format(value 1) , but there is no localization output. @Joosoo1 Have you tested the new branch with xfer_format value is 1?
In livox_ros_driver2, when I run the msg_MID360.launch file, the lidar topic type is livox_ros_driver2/CustomMsg rather than sensor_msgs::PointCloud2, so I modify some code in livox_ros_driver2 like this:

else if (kLivoxCustomMsg == transfer_format_)
        {
            // PublishCustomPointcloud(p_queue, index);
            PublishPointcloud2(p_queue, index);
        }

else if (kLivoxCustomMsg == transfer_format_)
        {
            // **pub = cur_node_->GetNode().advertise<livox_ros_driver2::CustomMsg>(name_str, queue_size);
            // DRIVER_INFO(*cur_node_, "%s publish use livox custom format, set ROS publisher queue size %d", name_str,
            //             queue_size);
            **pub = cur_node_->GetNode().advertise<sensor_msgs::PointCloud2>(name_str, queue_size);
            DRIVER_INFO(*cur_node_, "%s publish use PointCloud2 format, set ROS publisher queue size %d", name_str,
                        queue_size);
        }

else if (kLivoxCustomMsg == msg_type)
    {
        // DRIVER_INFO(*cur_node_, "%s publish use livox custom format", topic_name.c_str());
        // return cur_node_->create_publisher<CustomMsg>(topic_name, queue_size);
        DRIVER_INFO(*cur_node_, "%s publish use PointCloud2 format", topic_name.c_str());
        return cur_node_->create_publisher<PointCloud2>(topic_name, queue_size);
    }

[yijing2050]

ld multiply the data with a constant value like 9.8?

@Joosoo1 I used xfer_format = 0,and there was no point cloud. My wechat:yijingxs, you can add me and it is convenient to communicate with each other

[Joosoo1]

Yes, I tested the new branch and the DLIO only has odom output when xfer_format=2, but the result is completely wrong, so I have been using the original DLIO now, the original DLIO works well, I tested it on the drone and the positioning is very good. But deskwing is not working and will have some effect on DLIO，So I look forward to the author's help in solving this problem.

[Joosoo1]

ld multiply the data with a constant value like 9.8?

@Joosoo1 I used xfer_format = 0,and there was no point cloud. My wechat:yijingxs, you can add me and it is convenient to communicate with each other
xfer_format = 2, old DLIO could working!

[Joosoo1]

ld multiply the data with a constant value like 9.8?

@Joosoo1 I used xfer_format = 0,and there was no point cloud. My wechat:yijingxs, you can add me and it is convenient to communicate with each other
xfer_format = 2, old DLIO could working!

Also, when you use imu with acceleration units of m/s^2, you should change the https://github.com/vectr-ucla/direct_lidar_inertial_odometry/tree/feature/livox-support/src/ under the livox branch Lines 1373 to 1375 of the code in dlio/odom.cc should be changed to Eigen::Vector3f lin_accel(imu_raw->linear_acceleration.x * gravity_, imu_raw->linear_acceleration.y * gravity_, imu_raw->linear_acceleration.z * gravity_); recompile！

[yijing2050]

Thanks for your reply and kind advice. I have another problem when compiling, the first is the version of cmake. As my ros version is melodic,and the cmake version is 3.10.2, so I modify #VERSION 3.12.4 cmake_minimum_required(VERSION 3.10.2) project(direct_lidar_inertial_odometry) Could this modification do influence the output of the code? &nbsp;it have no add_compile_definitions.Then I comment out the section:# Not all machines have <cpuid.h&gt; available #set(CMAKE_TRY_COMPILE_TARGET_TYPE STATIC_LIBRARY) #file(WRITE ${CMAKE_BINARY_DIR}/test_cpuid.cpp "#include <cpuid.h&gt;") #try_compile(HAS_CPUID ${CMAKE_BINARY_DIR} ${CMAKE_BINARY_DIR}/test_cpuid.cpp) #file(REMOVE ${CMAKE_BINARY_DIR}/test_cpuid.cpp) #if(HAS_CPUID) #&nbsp; add_compile_definitions(HAS_CPUID) #endif() after done above, I compiled successfully.

…

------------------&nbsp;原始邮件&nbsp;------------------ 发件人: ***@***.***&gt;; 发送时间: 2023年6月26日(星期一) 中午11:54 收件人: ***@***.***&gt;; 抄送: ***@***.***&gt;; ***@***.***&gt;; 主题: Re: [vectr-ucla/direct_lidar_inertial_odometry] No localization with Livox Lidar (Issue #5) ld multiply the data with a constant value like 9.8? @Joosoo1 I used xfer_format = 0,and there was no point cloud. My wechat:yijingxs, you can add me and it is convenient to communicate with each other xfer_format = 2, old DLIO could working! Also, when you use imu with acceleration units of m/s^2, you should change the https://github.com/vectr-ucla/direct_lidar_inertial_odometry/tree/feature/livox-support/src/ under the livox branch Lines 1373 to 1375 of the code in dlio/odom.cc should be changed to Eigen::Vector3f lin_accel(imu_raw-&gt;linear_acceleration.x * gravity_, imu_raw-&gt;linear_acceleration.y * gravity_, imu_raw-&gt;linear_acceleration.z * gravity_); recompile！ — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***&gt;

[Joosoo1]

Thanks for your reply and kind advice. I have another problem when compiling, the first is the version of cmake. As my ros version is melodic,and the cmake version is 3.10.2, so I modify #VERSION 3.12.4 cmake_minimum_required(VERSION 3.10.2) project(direct_lidar_inertial_odometry) Could this modification do influence the output of the code?  it have no add_compile_definitions.Then I comment out the section:# Not all machines have <cpuid.h> available #set(CMAKE_TRY_COMPILE_TARGET_TYPE STATIC_LIBRARY) #file(WRITE ${CMAKE_BINARY_DIR}/test_cpuid.cpp "#include <cpuid.h>") #try_compile(HAS_CPUID ${CMAKE_BINARY_DIR} ${CMAKE_BINARY_DIR}/test_cpuid.cpp) #file(REMOVE ${CMAKE_BINARY_DIR}/test_cpuid.cpp) #if(HAS_CPUID) #  add_compile_definitions(HAS_CPUID) #endif() after done above, I compiled successfully.
…
------------------ 原始邮件 ------------------ 发件人: @.>; 发送时间: 2023年6月26日(星期一) 中午11:54 收件人: @.>; 抄送: @.>; @.>; 主题: Re: [vectr-ucla/direct_lidar_inertial_odometry] No localization with Livox Lidar (Issue #5) ld multiply the data with a constant value like 9.8? @Joosoo1 I used xfer_format = 0,and there was no point cloud. My wechat:yijingxs, you can add me and it is convenient to communicate with each other xfer_format = 2, old DLIO could working! Also, when you use imu with acceleration units of m/s^2, you should change the https://github.com/vectr-ucla/direct_lidar_inertial_odometry/tree/feature/livox-support/src/ under the livox branch Lines 1373 to 1375 of the code in dlio/odom.cc should be changed to Eigen::Vector3f lin_accel(imu_raw->linear_acceleration.x * gravity_, imu_raw->linear_acceleration.y * gravity_, imu_raw->linear_acceleration.z * gravity_); recompile！ — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

I don't know much about this, but if it compiles successfully, you should be able to run the program successfully, so you can test it as soon as possible!

[kennyjchen]

@yijing2050 Unfortunately DLIO only supports ROS Noetic, but it sounds like you got it working. Removing those lines won't have any affect on the output; it's just to build support for ARM architectures.

@WeilongXia @Joosoo1 The xfer_format should be 0 so that it's a message of type PointCloud2 but contains the timestamps. I will look into this again later this week. Sorry to ask again -- could someone record me a new bag that ideally time synchronizes the Mid-360's LiDAR and IMU (if not already), using xfer_format: 0? Thank you. In the meanwhile, you can turn deskewing off, but note that (especially at 10Hz) the point clouds will be quite distorted.

[yijing2050]

@kennyjchen
livox_repub.zip
Uploading livox_repub.zip…

[yijing2050]

Actually, I have compiled successfully although, but i haven't got the odom.. The configuration is as the below.

[kennyjchen]

Ok -- after digging through the Livox driver and also looking at this dataset, it seems like there is a bug for per-point timestamps when using xfer_format: 0(i.e., Livox pointcloud2(PointXYZRTLT) pointcloud format). The point cloud and IMU timestamps look synchronized, but the point timestamps are off and do not fall within the sweep window for some reason. This might be just the provided dataset, but either way they don't look lined-up.

I looked at other Mid-360 packages (e.g., this), and it seems like most people use xfer_format: 1 (Livox's custom pointcloud format) so that the offset_time field is available and contains the time since the beginning of the sweep. This requires a subscription to Livox's custom pointcloud format though.

So new plan: please try the updated branch but with xfer_format: 1. Your topic of type livox_ros_driver::CustomMsg should be called /robot/livox, which will call this new callback. This is all assuming that offset_time is in nanoseconds relative to the beginning of the sweep (which I think is the case).

(Please be careful with the units of gravity. I removed the previous multiplication with gravity.)

Note that I have not tested this at all, because I do not have a Livox sensor or any bags with xfer_format: 1. If someone could record me one that would be great so I can test it on my side. Otherwise, let me know if it works or not!

[Joosoo1]

Sorry for the late reply, and thank you @kennyjchen author for your efforts to solve this problem. Here is the dataset I recorded: accel in units is g；and xfer_format: 1
https://onedrive.live.com/?cid=6FAEBC9C8F25AA2C&id=6FAEBC9C8F25AA2C%21120&parId=root&o=OneUp

[kennyjchen]

Ok, I think I got it working. Credit to this Tweet for supplying the data I played around with.

Please use the newer livox_ros_driver2 package with xfer_format: 1. The latest feature/livox-support branch seems to be working with deskewing. Let me know.

[Joosoo1]

Ok, I think I got it working. Credit to this Tweet for supplying the data I played around with.

Please use the newer livox_ros_driver2 package with xfer_format: 1. The latest feature/livox-support branch seems to be working with deskewing. Let me know.

Doesn't seem to work：

[WeilongXia]

Ok, I think I got it working. Credit to this Tweet for supplying the data I played around with.↳

Please use the newer livox_ros_driver2 package with xfer_format: 1. The latest feature/livox-support branch seems to be working with deskewing. Let me know.↳

Doesn't seem to work： ↳

You should modify livox_topic to "/livox/lidar", and pointcloud_topic to another one.
Hi author @kennyjchen , with deskew set to "false", it can work with Livox-MID360 now as below!

And the offset_time units in livox_ros_driver2 is microseconds.

[WeilongXia]

Hi author, when i set deskew to true, the program is not working properly, this is the dataset i used: https://1drv.ms/u/s!AjkIEg1kWUNeiArwedvaRRYd5hWB?e=BaII3v
Could you please help to figure this out?

[kennyjchen]

@WeilongXia -- Your dataset works for me with deskewing.

I'm using the latest feature/livox-support branch without any modifications and the following command:

rosbag play 2023-06-28-10-00-30.bag /livox/imu:=/robot/imu /livox/lidar:=/robot/livox

What are the errors that you're seeing?

[Joosoo1]

@@yES! DLIO is now working with deskewing！just modify livox_topic to "/livox/lidar"

[kennyjchen]

Good to hear!

[yijing2050]

DLIO is now working with deskewing in Ubuntu 18.04, but it seems not stable and the path is not smooth as fast_lio2. And the map will fly in some time later.

[kennyjchen]

That max comp time looks pretty awful, which can result in these sorts of behaviors if too many frames are skipped. Are extrinsics correct?

[juliangaal]

@Joosoo1 what are the extrinsics of the dataset you provided?

[Joosoo1]

@Joosoo1 what are the extrinsics of the dataset you provided?
Please use this dataset, I tested it and it works great!：https://1drv.ms/u/s!AjkIEg1kWUNeiArwedvaRRYd5hWB?e=BaII3v
The extrinsics are as follows：

[CharlieV5]

May I ask you guys what is the meaning of dlio/pointcloud/dekew in dlio.yaml? Does it mean that this program will do the deskewing for pointcloud, or this program will consider the input points has been deskewed so it will no do the deskewing? What will happen when dlio/pointcloud/dekew is set as false? Thank you!

Ok, I think I got it working. Credit to this Tweet for supplying the data I played around with.↳

Please use the newer livox_ros_driver2 package with xfer_format: 1. The latest feature/livox-support branch seems to be working with deskewing. Let me know.↳

Doesn't seem to work： ↳

You should modify livox_topic to "/livox/lidar", and pointcloud_topic to another one. Hi author @kennyjchen , with deskew set to "false", it can work with Livox-MID360 now as below! And the offset_time units in livox_ros_driver2 is microseconds.

[kennyjchen]

It will undistort the point cloud when set to true.

[CharlieV5]

It will undistort the point cloud when set to true.

Thanks

[85256638]

@Joosoo1 what are the extrinsics of the dataset you provided?
Please use this dataset, I tested it and it works great!：https://1drv.ms/u/s!AjkIEg1kWUNeiArwedvaRRYd5hWB?e=BaII3v
The extrinsics are as follows：

Hi, can you kindly let me know where to update those extrinsic values? I guess in the dlio.yaml?
can you kindly paste yours please? I would like to know how to adapt Livox Horizon's extrinsic from the datasheet to the dlio.yaml configuration.

dlio:

  version: 1.1.1

  adaptive: true

  pointcloud:
    deskew: true
    voxelize: true

  imu:
    calibration: true
    intrinsics:
      accel:
        bias: [ 0.0, 0.0, 0.0 ]
        sm:   [ 1.,  0.,  0.,
                0.,  1.,  0.,
                0.,  0.,  1. ]
      gyro:
        bias: [ 0.0, 0.0, 0.0 ]

  extrinsics:
    baselink2imu:
      t: [ 0.006253, -0.011775, 0.007645 ]
      R: [ 1.,  0.,  0.,
           0.,  1.,  0.,
           0.,  0.,  1. ]
    baselink2lidar:
      t: [ 0.,  0.,  0. ]
      R: [ 1.,  0.,  0.,
           0.,  1.,  0.,
           0.,  0.,  1. ]

LIvox Horizon extrinsic:

[85256638]

@Joosoo1 what are the extrinsics of the dataset you provided?
Please use this dataset, I tested it and it works great!：https://1drv.ms/u/s!AjkIEg1kWUNeiArwedvaRRYd5hWB?e=BaII3v
The extrinsics are as follows：

Hi, can you kindly let me know where to update those extrinsic values? I guess in the dlio.yaml? can you kindly paste yours please? I would like to know how to adapt Livox Horizon's extrinsic from the datasheet to the dlio.yaml configuration.

dlio:

  version: 1.1.1

  adaptive: true

  pointcloud:
    deskew: true
    voxelize: true

  imu:
    calibration: true
    intrinsics:
      accel:
        bias: [ 0.0, 0.0, 0.0 ]
        sm:   [ 1.,  0.,  0.,
                0.,  1.,  0.,
                0.,  0.,  1. ]
      gyro:
        bias: [ 0.0, 0.0, 0.0 ]

  extrinsics:
    baselink2imu:
      t: [ 0.006253, -0.011775, 0.007645 ]
      R: [ 1.,  0.,  0.,
           0.,  1.,  0.,
           0.,  0.,  1. ]
    baselink2lidar:
      t: [ 0.,  0.,  0. ]
      R: [ 1.,  0.,  0.,
           0.,  1.,  0.,
           0.,  0.,  1. ]

LIvox Horizon extrinsic:

@kennyjchen @Joosoo1 this issue has been closed, I would love to hear your advice :) thanks!