Merged commit includes the following changes:

616745794 by Waymo Research: Modify config for the 2024 WOSAC challenge. -- 616215113 by Waymo Research: Internal changes. -- 616204684 by Waymo Research: Small text and typo fixes. -- 616193525 by Waymo Research: License header updates. -- 616192816 by Waymo Research: Updating pip package and README file. -- 616065453 by Waymo Research: Fix tutorial and api for womd -- 615749226 by Waymo Research: Internal change. -- 615478267 by Waymo Research: Internal changes. -- 614597908 by Waymo Research: Add z-stretching to the computation of distances to polylines. "Z-stretch" is scaling factor for vertical distances used when finding the closest segment to a query point. This prevents wrong associations in cases with under- and over-passes. -- 614597569 by Waymo Research: Modify distance to polylines function to use altitude. Altitudes i.e. z-coordinates are used to find the closest segment to a query point. The distances returned remain 2d distances in the XY plane. -- 614597267 by Waymo Research: Batch computation of distances to polylines. Instead of iterating on polylines one by one, they are stacked with the same padded length and processed all at once. -- 611184527 by Waymo Research: Automated Code Change -- 610359862 by Waymo Research: Automated Code Change -- 607768062 by Waymo Research: Internal fixes. -- 607661995 by Waymo Research: Use central difference instead of forward difference for derivative fields used in Sim Agents metrics. -- 607067387 by Waymo Research: Normalizes the heading difference angle to (-PI, PI) in the motion challenge metrics trajectory classifier. This also adjusts some of the classifier thresholds to improve classification. This CL also adds multiple unit tests to validate the new logic. -- 606237500 by Waymo Research: Modify object distance computation to enable representing objects with rounded rectangles. Set default distance computation to use a rounding factor of `0.7`. -- 605292127 by Waymo Research: Adding simulated collision and offroad rates for sim agents. -- 604720923 by Waymo Research: Update colab, fix pytype errors, reformat occupancy-flow files. -- 602441918 by Waymo Research: Change paper bibtex -- 601226921 by Waymo Research: fix typo api name in tutorial -- 601222323 by Waymo Research: Add tutorial of merging WOMD camera data scenario protos with existing dataset, and extracting camera embeddings from camera tokens. -- 601220113 by Waymo Research: add util functions to load camera data for WOMD -- 600650427 by Waymo Research: Add camera tokens field in scenario proto for WOMD dataset -- 598915141 by Waymo Research: Automated Code Change -- 596061334 by Waymo Research: internal change -- PiperOrigin-RevId: 616745794
waymo-research · Mar 18, 2024 · 62b97ad · 62b97ad
1 parent 3652937
commit 62b97ad
Show file tree

Hide file tree

Showing 58 changed files with 2,430 additions and 1,401 deletions.
diff --git a/README.md b/README.md
@@ -26,6 +26,28 @@ compliance with the Waymo Dataset License Agreement for Non-Commercial Use. See
 [wdl_limited/sim_agents_metrics/](src/waymo_open_dataset/wdl_limited/sim_agents_metrics),
 respectively, for details.
 
+## March 2024 Update
+This update contains several changes/addition to the datasets:
+
+Perception dataset (v1.4.3 and v2.0.1):
+- We made improvements in the 3D semantic segmentation ground truth labels, especially for the class of motorcyclist.
+
+Motion dataset (v1.2.1):
+- The 1.2.1 WOMD release now provides camera data, including front, front-left, front-right, side-left, side-right, rear-left, rear-right, and rear sensors. Similar to the Lidar data, the camera data of the training, validation and testing sets cover the first 1 second of each of the 9 second windows. Instead of releasing raw camera images, we release the image tokens and image embedding extracted from a pre-trained VQ-GAN model.
+- The initial release of the WOMD camera data contained misalignment between LiDAR data and roadgraph inputs for some frames. The 1.2.1 release provides new timestamps for the lidar data with an updated pose transformation matrix per time step.
+
+We also provide the following changes to the code supporting the challenges.
+
+Motion prediction:
+- We have improved the logic behind the behavior bucketing used for mAP.
+
+Sim Agents:
+- We have improved the quality of the kinematic metrics by using smoother estimates of speeds and accelerations.
+- We have fixed an edge case for offroad computation with over-passes.
+- We have re-calibratred the metric configuration and composite metrics weights.
+- We report simulated collision and offroad rates (not likelihoods).
+
+
 ## December 2023 Update
 We released v1.6.1 version of the pip package with fixes for the WOSAC metrics:
 - Fixing a bug in validity checking for collision and offroad.
@@ -148,12 +170,12 @@ This code repository contains:
   pages={9710-9719}
 }
 
-@article{Kan_2023_arxiv,
+@InProceedings{Kan_2024_icra,
   author={Chen, Kan and Ge, Runzhou and Qiu, Hang and Ai-Rfou, Rami and Qi, Charles R. and Zhou, Xuanyu and Yang, Zoey and Ettinger, Scott and Sun, Pei and Leng, Zhaoqi and Mustafa, Mustafa and Bogun, Ivan and Wang, Weiyue and Tan, Mingxing and Anguelov, Dragomir},
   title={WOMD-LiDAR: Raw Sensor Dataset Benchmark for Motion Forecasting},
-  month={April},
-  journal={arXiv preprint arXiv:2304.03834},
-  year={2023}
+  month={May},
+  booktitle= Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)},
+  year={2024}
 }
 
 ## Dataset Metadata

diff --git a/src/.bazelrc b/src/.bazelrc
@@ -1,5 +1,5 @@
 build -c opt
-build --cxxopt="-std=c++17" --cxxopt=-Wno-unused-variable
+build --cxxopt="-std=c++17" --host_cxxopt="-std=c++17" --cxxopt=-Wno-unused-variable
 build --auto_output_filter=subpackages
 build --copt="-Wall" --copt="-Wno-sign-compare"
 build --linkopt="-lrt -lm -lpthread"

diff --git a/src/WORKSPACE b/src/WORKSPACE
@@ -2,11 +2,31 @@ workspace(name = "waymo_open_dataset_rules")
 
 load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
 
+http_archive(
+    name = "rules_proto",
+    sha256 = "dc3fb206a2cb3441b485eb1e423165b231235a1ea9b031b4433cf7bc1fa460dd",
+    strip_prefix = "rules_proto-5.3.0-21.7",
+    urls = [
+        "https://github.com/bazelbuild/rules_proto/archive/refs/tags/5.3.0-21.7.tar.gz",
+    ],
+)
+
 http_archive(
     name = "rules_python",
-    sha256 = "ffc7b877c95413c82bfd5482c017edcf759a6250d8b24e82f41f3c8b8d9e287e",
-    strip_prefix = "rules_python-0.19.0",
-    url = "https://github.com/bazelbuild/rules_python/releases/download/0.19.0/rules_python-0.19.0.tar.gz",
+    sha256 = "9d04041ac92a0985e344235f5d946f71ac543f1b1565f2cdbc9a2aaee8adf55b",
+    strip_prefix = "rules_python-0.26.0",
+    url = "https://github.com/bazelbuild/rules_python/releases/download/0.26.0/rules_python-0.26.0.tar.gz",
+)
+
+load("@rules_python//python:repositories.bzl", "py_repositories")
+
+py_repositories()
+
+http_archive(
+    name = "rules_cc",
+    sha256 = "2037875b9a4456dce4a79d112a8ae885bbc4aad968e6587dca6e64f3a0900cdf",
+    strip_prefix = "rules_cc-0.0.9",
+    urls = ["https://github.com/bazelbuild/rules_cc/releases/download/0.0.9/rules_cc-0.0.9.tar.gz"],
 )
 
 http_archive(
@@ -41,10 +61,10 @@ http_archive(
 http_archive(
     name = "com_google_glog",
     build_file = "//waymo_open_dataset:bazel/glog.BUILD",
-    sha256 = "f28359aeba12f30d73d9e4711ef356dc842886968112162bc73002645139c39c",
-    strip_prefix = "glog-0.4.0",
+    sha256 = "375106b5976231b92e66879c1a92ce062923b9ae573c42b56ba28b112ee4cc11",
+    strip_prefix = "glog-0.7.0",
     urls = [
-        "https://github.com/google/glog/archive/v0.4.0.tar.gz",
+        "https://github.com/google/glog/archive/refs/tags/v0.7.0.tar.gz",
     ],
 )
 
@@ -63,7 +83,6 @@ http_archive(
     sha256 = "94aef187f688665dc299d09286bfa0d22c4ecb86a80b156dff6aabadc5a5c26d",
     strip_prefix = "abseil-cpp-273292d1cfc0a94a65082ee350509af1d113344d",
     urls = [
-        "https://storage.googleapis.com/mirror.tensorflow.org/github.com/abseil/abseil-cpp/archive/273292d1cfc0a94a65082ee350509af1d113344d.tar.gz",
         "https://github.com/abseil/abseil-cpp/archive/273292d1cfc0a94a65082ee350509af1d113344d.tar.gz",
     ],
 )
@@ -95,17 +114,6 @@ bind(
     actual = "@six_archive//:six",
 )
 
-SHA = "84aec9e21cc56fbc7f1335035a71c850d1b9b5cc6ff497306f84cced9a769841"
-
-VERSION = "0.23.1"
-
-http_archive(
-    name = "rules_python",
-    sha256 = SHA,
-    strip_prefix = "rules_python-0.23.1".format(VERSION),
-    url = "https://github.com/bazelbuild/rules_python/releases/download/{}/rules_python-{}.tar.gz".format(VERSION, VERSION),
-)
-
 http_archive(
     name = "bazel_skylib",
     sha256 = "b8a1527901774180afc798aeb28c4634bdccf19c4d98e7bdd1ce79d1fe9aaad7",
@@ -121,14 +129,17 @@ bazel_skylib_workspace()
 
 http_archive(
     name = "com_google_protobuf",
-    sha256 = "cfcba2df10feec52a84208693937c17a4b5df7775e1635c1e3baffc487b24c9b",
-    strip_prefix = "protobuf-3.9.2",
+    sha256 = "f66073dee0bc159157b0bd7f502d7d1ee0bc76b3c1eac9836927511bdc4b3fc1",
+    strip_prefix = "protobuf-3.21.9",
     urls = [
-        "https://storage.googleapis.com/mirror.tensorflow.org/github.com/protocolbuffers/protobuf/archive/v3.9.2.zip",
-        "https://github.com/protocolbuffers/protobuf/archive/v3.9.2.zip",
+        "https://github.com/protocolbuffers/protobuf/archive/v3.21.9.zip",
     ],
 )
 
+load("@com_google_protobuf//:protobuf_deps.bzl", "protobuf_deps")
+
+protobuf_deps()
+
 http_archive(
     name = "deeplab2",
     build_file = "//waymo_open_dataset:bazel/deeplab2.BUILD",
@@ -153,9 +164,9 @@ http_archive(
 # package.
 http_archive(
     name = "org_tensorflow",
-    sha256 = "e52cda3bae45f0ae0fccd4055e9fa29892b414f70e2df94df9a3a10319c75fff",
-    strip_prefix = "tensorflow-2.11.0",
-    urls = ["https://github.com/tensorflow/tensorflow/archive/v2.11.0.zip"],
+    sha256 = "af0584df1a4e28763c32c218b39f8c4f3784fabb6a8859b00c02d743864dc191",
+    strip_prefix = "tensorflow-2.12.0",
+    urls = ["https://github.com/tensorflow/tensorflow/archive/v2.12.0.zip"],
 )
 
 ###########################################

diff --git a/src/waymo_open_dataset/BUILD b/src/waymo_open_dataset/BUILD
@@ -35,6 +35,7 @@ all_proto_library(
 
 compile_pip_requirements(
     name = "requirements",
+    timeout = "moderate",
     extra_args = [
         "--resolver=backtracking",
         "--allow-unsafe",

diff --git a/src/waymo_open_dataset/dataset.proto b/src/waymo_open_dataset/dataset.proto
@@ -55,6 +55,9 @@ message CameraName {
     FRONT_RIGHT = 3;
     SIDE_LEFT = 4;
     SIDE_RIGHT = 5;
+    REAR_LEFT = 6;
+    REAR = 7;
+    REAR_RIGHT = 8;
 
   }
 }

diff --git a/src/waymo_open_dataset/metrics/BUILD b/src/waymo_open_dataset/metrics/BUILD
@@ -168,6 +168,7 @@ cc_library(
     hdrs = ["motion_metrics_utils.h"],
     deps = [
         "//waymo_open_dataset/common:status",
+        "//waymo_open_dataset/math:math_util",
         "//waymo_open_dataset/math:polygon2d",
         "//waymo_open_dataset/math:vec2d",
         "//waymo_open_dataset/protos:motion_metrics_cc_proto",

diff --git a/src/waymo_open_dataset/metrics/motion_metrics_utils.cc b/src/waymo_open_dataset/metrics/motion_metrics_utils.cc
@@ -17,6 +17,7 @@ limitations under the License.
 
 #include <set>
 
+#include "waymo_open_dataset/math/math_util.h"
 #include "waymo_open_dataset/math/vec2d.h"
 #include "waymo_open_dataset/protos/motion_metrics.pb.h"
 #include "waymo_open_dataset/protos/motion_submission.pb.h"
@@ -29,9 +30,9 @@ absl::optional<TrajectoryType> ClassifyTrack(int prediction_step,
                                              const Track& track) {
   // Parameters for classification.
   static constexpr float kMaxSpeedForStationary = 2.0;                 // (m/s)
-  static constexpr float kMaxDisplacementForStationary = 5.0;          // (m)
-  static constexpr float kMaxLateralDisplacementForStraight = 5.0;     // (m)
-  static constexpr float kMinLongitudinalDisplacementForUTurn = -5.0;  // (m)
+  static constexpr float kMaxDisplacementForStationary = 3.0;          // (m)
+  static constexpr float kMaxLateralDisplacementForStraight = 2.5;     // (m)
+  static constexpr float kMinLongitudinalDisplacementForUTurn = 0.0;  // (m)
   static constexpr float kMaxAbsHeadingDiffForStraight = M_PI / 6.0;   // (rad)
 
   // Find the last valid point in the track.
@@ -58,7 +59,8 @@ absl::optional<TrajectoryType> ClassifyTrack(int prediction_step,
   const double x_delta = end_state.center_x() - start_state.center_x();
   const double y_delta = end_state.center_y() - start_state.center_y();
   const double final_displacement = std::hypot(x_delta, y_delta);
-  const float heading_diff = end_state.heading() - start_state.heading();
+  const float heading_diff =
+      NormalizeAngle(end_state.heading() - start_state.heading());
   const Vec2d normalized_delta =
       Vec2d(x_delta, y_delta).Rotated(-start_state.heading());
   const float start_speed =
@@ -81,8 +83,8 @@ absl::optional<TrajectoryType> ClassifyTrack(int prediction_step,
     return dy < 0 ? TrajectoryType::STRAIGHT_RIGHT
                   : TrajectoryType::STRAIGHT_LEFT;
   }
-  if (heading_diff < -kMaxAbsHeadingDiffForStraight && dy < 0) {
-    return normalized_delta.x() < kMinLongitudinalDisplacementForUTurn
+  if (dy < 0) {
+    return dx < kMinLongitudinalDisplacementForUTurn
                ? TrajectoryType::RIGHT_U_TURN
                : TrajectoryType::RIGHT_TURN;
   }

diff --git a/src/waymo_open_dataset/metrics/motion_metrics_utils_test.cc b/src/waymo_open_dataset/metrics/motion_metrics_utils_test.cc
@@ -15,6 +15,8 @@ limitations under the License.
 
 #include "waymo_open_dataset/metrics/motion_metrics_utils.h"
 
+#include <vector>
+
 #include <glog/logging.h>
 #include "google/protobuf/text_format.h"
 #include "google/protobuf/util/message_differencer.h"
@@ -84,6 +86,17 @@ Track RotateTrack(const Track& track, const double angle) {
   return result;
 }
 
+// Returns the given set of points rotated by a given angle in radians.
+std::vector<Vec2d> RotatePoints(const std::vector<Vec2d>& points,
+                                double angle_radians) {
+  std::vector<Vec2d> result;
+  result.reserve(points.size());
+  for (const auto& point : points) {
+    result.push_back(point.Rotated(angle_radians));
+  }
+  return result;
+}
+
 void TestTrackClassification(const std::vector<Vec2d>& points,
                              const TrajectoryType expected_type) {
   const Track track = CreateTrackFromPoints(points);
@@ -97,6 +110,14 @@ void TestTrackClassification(const std::vector<Vec2d>& points,
   track_rotated = RotateTrack(track, -1.0);
   type = ClassifyTrack(1, track_rotated);
   EXPECT_EQ(*type, expected_type);
+
+  // Rotate the trajectory points through a range of angles.
+  for (double angle = -M_PI; angle <= M_PI; angle += M_PI / 32) {
+    const std::vector<Vec2d> rotated_points = RotatePoints(points, angle);
+    const Track track = CreateTrackFromPoints(rotated_points);
+    auto type = ClassifyTrack(1, track);
+    EXPECT_EQ(*type, expected_type);
+  }
 }
 
 TEST(MotionMetricsUtils, ClassifyTrackStationary) {
@@ -109,6 +130,12 @@ TEST(MotionMetricsUtils, ClassifyTrackStraight) {
   TestTrackClassification(points, TrajectoryType::STRAIGHT);
 }
 
+TEST(MotionMetricsUtils, ClassifyTrackSemiStraightAcrossPiBoundary) {
+  std::vector<Vec2d> points = {
+      {0, 0}, {-1, 0.05}, {-2, 0}, {-3, -0.5}, {-4, -0.5}};
+  TestTrackClassification(points, TrajectoryType::STRAIGHT);
+}
+
 TEST(MotionMetricsUtils, ClassifyTrackStraightLeft) {
   std::vector<Vec2d> points = {{0, 0},    {0, 2},   {0, 4},     {-.5, 6},
                                {-1.5, 8}, {-2, 10}, {-2.5, 12}, {-3, 14},
@@ -144,6 +171,48 @@ TEST(MotionMetricsUtils, ClassifyTrackLeftUTurn) {
   TestTrackClassification(points, TrajectoryType::LEFT_U_TURN);
 }
 
+TEST(MotionMetricsUtils, ClassifyTrackLeftUTurn180Deg) {
+  std::vector<Vec2d> points = {{0, 0},   {-2, 0},  {-4, 0},  {-6, -2},
+                               {-6, -4}, {-4, -6}, {-2, -8}, {0, -10},
+                               {2, -10}, {4, -10}, {6, -10}};
+  TestTrackClassification(points, TrajectoryType::LEFT_U_TURN);
+}
+
+TEST(MotionMetricsUtils, ClassifyTrackLeftUTurnGT180Deg) {
+  std::vector<Vec2d> points = {{0, 0},   {-2, 0},  {-4, 0},  {-6, -2},
+                               {-6, -4}, {-4, -6}, {-2, -8}, {0, -10},
+                               {2, -10}, {4, -9},  {6, -8}};
+  TestTrackClassification(points, TrajectoryType::LEFT_U_TURN);
+}
+
+TEST(MotionMetricsUtils, ClassifyTrackLeftUTurnLT180Deg) {
+  std::vector<Vec2d> points = {{0, 0},   {-2, 0},  {-4, 0},  {-6, -2},
+                               {-6, -4}, {-4, -6}, {-2, -8}, {0, -10},
+                               {2, -10}, {4, -10}, {6, -11}};
+  TestTrackClassification(points, TrajectoryType::LEFT_U_TURN);
+}
+
+TEST(MotionMetricsUtils, ClassifyTrackRightUTurn180Deg) {
+  std::vector<Vec2d> points = {{0, 0},  {-2, 0}, {-4, 0}, {-6, 2},
+                               {-6, 4}, {-4, 6}, {-2, 8}, {0, 10},
+                               {2, 10}, {4, 10}, {6, 10}};
+  TestTrackClassification(points, TrajectoryType::RIGHT_U_TURN);
+}
+
+TEST(MotionMetricsUtils, ClassifyTrackRightUTurnGT180Deg) {
+  std::vector<Vec2d> points = {{0, 0},  {-2, 0}, {-4, 0}, {-6, 2},
+                               {-6, 4}, {-4, 6}, {-2, 8}, {0, 10},
+                               {2, 10}, {4, 9},  {6, 8}};
+  TestTrackClassification(points, TrajectoryType::RIGHT_U_TURN);
+}
+
+TEST(MotionMetricsUtils, ClassifyTrackRightUTurnLT180Deg) {
+  std::vector<Vec2d> points = {{0, 0},  {-2, 0}, {-4, 0}, {-6, 2},
+                               {-6, 4}, {-4, 6}, {-2, 8}, {0, 10},
+                               {2, 10}, {4, 11}, {6, 12}};
+  TestTrackClassification(points, TrajectoryType::RIGHT_U_TURN);
+}
+
 TEST(MotionMetricsUtils, ClassifyTrackRightUTurn) {
   std::vector<Vec2d> points = {{0, 0},   {2, 2},   {2, 4},   {4, 4},
                                {6, 2},   {7, 0},   {8, -2},  {9, -4},

diff --git a/src/waymo_open_dataset/pip_pkg_scripts/BUILD b/src/waymo_open_dataset/pip_pkg_scripts/BUILD
@@ -173,7 +173,7 @@ py_wheel(
         "Topic :: Software Development :: Libraries :: Python Modules",
     ],
     description_file = "pypi_description.md",
-    distribution = "waymo-open-dataset-tf-2-11-0",
+    distribution = "waymo-open-dataset-tf-2-12-0",
     homepage = "https://github.com/waymo-research/waymo-open-dataset",
     python_tag = "py3",
     # Should be in-sync with //requirements.in
@@ -184,7 +184,7 @@ py_wheel(
         "google-auth==2.16.2",
         "immutabledict==2.2.0",
         "matplotlib==3.6.1",
-        "numpy==1.21.5",
+        "numpy==1.23",
         "openexr==1.3.9",
         "pandas==1.5.3",
         "pillow==9.2.0",
@@ -193,13 +193,13 @@ py_wheel(
         "scikit-image==0.20.0",
         "scikit-learn==1.2.2",
         "setuptools==67.6.0",
-        "tensorflow==2.11",
+        "tensorflow==2.12",
         "tensorflow_graphics==2021.12.3",
         "tensorflow_probability==0.19.0",
         "visu3d==1.5.1",
         "dacite==1.8.1",
     ],
-    version = "1.6.1",
+    version = "1.6.2",
     deps = [
         ":all_python_modules",
     ],

diff --git a/src/waymo_open_dataset/protos/BUILD b/src/waymo_open_dataset/protos/BUILD
@@ -84,6 +84,7 @@ all_proto_library(
 all_proto_library(
     src = "scenario.proto",
     deps = [
+        ":camera_tokens_proto",
         ":compressed_lidar_proto",
         ":map_proto",
     ],
@@ -152,3 +153,10 @@ all_proto_library(
         "//waymo_open_dataset:dataset_proto",
     ],
 )
+
+all_proto_library(
+    src = "camera_tokens.proto",
+    deps = [
+        "//waymo_open_dataset:dataset_proto",
+    ],
+)