Skip to content

Commit

Permalink
Merged commit includes the following changes:
Browse files Browse the repository at this point in the history 
374523138  by Waymo Research:

    Fix DATA_FIELDS incorrectly named DATASET_FORMAT

--
374513922  by Waymo Research:

    Add camera trigger and readout done times; fix typo in rolling_shutter_direction frame util

--
374436695  by Waymo Research:

    Fix the python wrapper for motion overlap metrics.

--
374433936  by Waymo Research:

    Update colab tutorial to reflect the overlap rate metrics fix.

--
373723752  by Waymo Research:

    Fix typo in quick start tutorial. Make it clear that packages are for linux only.

--
373204603  by Waymo Research:

    Make metrics deterministic.

--
373102935  by Waymo Research:

    Internal change

--
372065017  by Waymo Research:

    Improve logging handling and compatibility with internal log processing for latency evaluation tool.

--
370808953  by Waymo Research:

    Skip invalid object type

--

PiperOrigin-RevId: 374523138
  • Loading branch information
Waymo Research authored and Yuning Chai committed May 19, 2021
1 parent a19ba04 commit 1f75252
Show file tree
Hide file tree
Showing 16 changed files with 634 additions and 255 deletions.
8 changes: 3 additions & 5 deletions docs/quick_start.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,9 @@ for a quick demo of the installation and data format.

As a first step, please visit https://www.waymo.com/open to gain access to the Waymo Open Dataset.

## Use pre-compiled pip/pip3 packages
## Use pre-compiled pip/pip3 packages for Linux

We only pre-compiled the package for Python 3.6, 3.7, 3.8. If you need the lib
for a different python version, follow steps in pip_pkg_scripts to build pip
package on your own.
We only pre-compiled the package for Python 3.6, 3.7, 3.8 for Linux. If you need the lib for a different python version, follow steps in pip_pkg_scripts to build pip package on your own.

``` bash
pip3 install --upgrade pip
Expand All @@ -22,7 +20,7 @@ pip3 install --upgrade pip
### tf 2.4.0.

``` bash
pip3 install waymo-open-dataset-tf-2-3-0 --user
pip3 install waymo-open-dataset-tf-2-4-0 --user
```

### tf 2.3.0.
Expand Down
2 changes: 2 additions & 0 deletions tutorial/BUILD
View file
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,8 @@ py_binary(
deps = [
"//waymo_open_dataset:dataset_proto_py_pb2",
"//waymo_open_dataset:label_proto_py_pb2",
"//waymo_open_dataset/metrics/ops:gen_metrics_ops",
"//waymo_open_dataset/metrics/ops:motion_metrics_ops",
"//waymo_open_dataset/metrics/ops:py_metrics_ops",
"//waymo_open_dataset/metrics/python:config_util_py",
"//waymo_open_dataset/protos:breakdown_proto_py_pb2",
Expand Down
193 changes: 132 additions & 61 deletions tutorial/tutorial_motion.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -44,8 +44,19 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 1,
"metadata": {
"executionInfo": {
"elapsed": 401,
"status": "ok",
"timestamp": 1620951571164,
"user": {
"displayName": "",
"photoUrl": "",
"userId": ""
},
"user_tz": 420
},
"id": "xdEcN6WilcBn"
},
"outputs": [],
Expand All @@ -61,8 +72,19 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 2,
"metadata": {
"executionInfo": {
"elapsed": 2591,
"status": "ok",
"timestamp": 1620951573762,
"user": {
"displayName": "",
"photoUrl": "",
"userId": ""
},
"user_tz": 420
},
"id": "M5gzSlBTlTiS"
},
"outputs": [],
Expand Down Expand Up @@ -233,8 +255,19 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 3,
"metadata": {
"executionInfo": {
"elapsed": 81,
"status": "ok",
"timestamp": 1620951573865,
"user": {
"displayName": "",
"photoUrl": "",
"userId": ""
},
"user_tz": 420
},
"id": "TpEZq1EMtXV9"
},
"outputs": [],
Expand All @@ -255,8 +288,19 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 7,
"metadata": {
"executionInfo": {
"elapsed": 6331,
"status": "ok",
"timestamp": 1620939490073,
"user": {
"displayName": "",
"photoUrl": "",
"userId": ""
},
"user_tz": 420
},
"id": "utTE9Mtgx3Fq"
},
"outputs": [],
Expand Down Expand Up @@ -537,18 +581,15 @@
" decoded_example = tf.io.parse_single_example(value, features_description)\n",
"\n",
" past_states = tf.stack([\n",
" decoded_example['state/past/x'],\n",
" decoded_example['state/past/y'],\n",
" decoded_example['state/past/length'],\n",
" decoded_example['state/past/width'],\n",
" decoded_example['state/past/x'], decoded_example['state/past/y'],\n",
" decoded_example['state/past/length'], decoded_example['state/past/width'],\n",
" decoded_example['state/past/bbox_yaw'],\n",
" decoded_example['state/past/velocity_x'],\n",
" decoded_example['state/past/velocity_y']\n",
" ], -1)\n",
"\n",
" cur_states = tf.stack([\n",
" decoded_example['state/current/x'],\n",
" decoded_example['state/current/y'],\n",
" decoded_example['state/current/x'], decoded_example['state/current/y'],\n",
" decoded_example['state/current/length'],\n",
" decoded_example['state/current/width'],\n",
" decoded_example['state/current/bbox_yaw'],\n",
Expand All @@ -559,8 +600,7 @@
" input_states = tf.concat([past_states, cur_states], 1)[..., :2]\n",
"\n",
" future_states = tf.stack([\n",
" decoded_example['state/future/x'],\n",
" decoded_example['state/future/y'],\n",
" decoded_example['state/future/x'], decoded_example['state/future/y'],\n",
" decoded_example['state/future/length'],\n",
" decoded_example['state/future/width'],\n",
" decoded_example['state/future/bbox_yaw'],\n",
Expand Down Expand Up @@ -627,16 +667,19 @@
"class SimpleModel(tf.keras.Model):\n",
" \"\"\"A simple one-layer regressor.\"\"\"\n",
"\n",
" def __init__(self, num_states_steps, num_future_steps):\n",
" def __init__(self, num_agents_per_scenario, num_states_steps,\n",
" num_future_steps):\n",
" super(SimpleModel, self).__init__()\n",
" self._num_agents_per_scenario = num_agents_per_scenario\n",
" self._num_states_steps = num_states_steps\n",
" self._num_future_steps = num_future_steps\n",
" self.regressor = tf.keras.layers.Dense(num_future_steps * 2)\n",
"\n",
" def call(self, states):\n",
" states = tf.reshape(states, (-1, self._num_states_steps * 2))\n",
" pred = self.regressor(states)\n",
" pred = tf.reshape(pred, [-1, self._num_future_steps, 2])\n",
" pred = tf.reshape(\n",
" pred, [-1, self._num_agents_per_scenario, self._num_future_steps, 2])\n",
" return pred\n",
"\n",
"\n",
Expand All @@ -645,71 +688,75 @@
"\n",
" def __init__(self, config):\n",
" super().__init__()\n",
" self._ground_truth_trajectory = []\n",
" self._ground_truth_is_valid = []\n",
" self._prediction_trajectory = []\n",
" self._prediction_score = []\n",
" self._ground_truth_trajectory = []\n",
" self._ground_truth_is_valid = []\n",
" self._prediction_ground_truth_indices = []\n",
" self._prediction_ground_truth_indices_mask = []\n",
" self._object_type = []\n",
" self._metrics_config = config\n",
"\n",
" def reset_state():\n",
" self._ground_truth_trajectory = []\n",
" self._ground_truth_is_valid = []\n",
" self._prediction_trajectory = []\n",
" self._prediction_score = []\n",
" self._ground_truth_trajectory = []\n",
" self._ground_truth_is_valid = []\n",
" self._prediction_ground_truth_indices = []\n",
" self._prediction_ground_truth_indices_mask = []\n",
" self._object_type = []\n",
"\n",
" def update_state(self, prediction_trajectory, prediction_score,\n",
" ground_truth_trajectory, ground_truth_is_valid, object_type):\n",
" ground_truth_trajectory, ground_truth_is_valid,\n",
" prediction_ground_truth_indices,\n",
" prediction_ground_truth_indices_mask, object_type):\n",
" self._prediction_trajectory.append(prediction_trajectory)\n",
" self._prediction_score.append(prediction_score)\n",
" self._ground_truth_trajectory.append(ground_truth_trajectory)\n",
" self._ground_truth_is_valid.append(ground_truth_is_valid)\n",
" self._prediction_ground_truth_indices.append(\n",
" prediction_ground_truth_indices)\n",
" self._prediction_ground_truth_indices_mask.append(\n",
" prediction_ground_truth_indices_mask)\n",
" self._object_type.append(object_type)\n",
"\n",
" def result(self):\n",
" # [batch_size, steps, 2].\n",
" # [batch_size, num_preds, 1, 1, steps, 2].\n",
" prediction_trajectory = tf.concat(self._prediction_trajectory, 0)\n",
" # [batch_size].\n",
" # [batch_size, num_preds, 1].\n",
" prediction_score = tf.concat(self._prediction_score, 0)\n",
" # [batch_size, gt_steps, 7].\n",
" # [batch_size, num_agents, gt_steps, 7].\n",
" ground_truth_trajectory = tf.concat(self._ground_truth_trajectory, 0)\n",
" # [batch_size, gt_steps].\n",
" # [batch_size, num_agents, gt_steps].\n",
" ground_truth_is_valid = tf.concat(self._ground_truth_is_valid, 0)\n",
" # [batch_size].\n",
" # [batch_size, num_preds, 1].\n",
" prediction_ground_truth_indices = tf.concat(\n",
" self._prediction_ground_truth_indices, 0)\n",
" # [batch_size, num_preds, 1].\n",
" prediction_ground_truth_indices_mask = tf.concat(\n",
" self._prediction_ground_truth_indices_mask, 0)\n",
" # [batch_size, num_agents].\n",
" object_type = tf.cast(tf.concat(self._object_type, 0), tf.int64)\n",
"\n",
" # We are predicting more steps than needed by the eval code. Subsample.\n",
" interval = (\n",
" self._metrics_config.track_steps_per_second //\n",
" self._metrics_config.prediction_steps_per_second)\n",
" prediction_trajectory = prediction_trajectory[:, (interval - 1)::interval]\n",
"\n",
" # Prepare these into shapes expected by the metrics computation.\n",
" #\n",
" # [batch_size, top_k, num_agents_per_joint_prediction, pred_steps, 2].\n",
" # top_k is 1 because we have a uni-modal model.\n",
" # num_agents_per_joint_prediction is also 1 here.\n",
" prediction_trajectory = prediction_trajectory[:, tf.newaxis, tf.newaxis]\n",
" # [batch_size, top_k].\n",
" prediction_score = prediction_score[:, tf.newaxis]\n",
" # [batch_size, num_agents_per_joint_prediction, gt_steps, 7].\n",
" ground_truth_trajectory = ground_truth_trajectory[:, tf.newaxis]\n",
" # [batch_size, num_agents_per_joint_prediction, gt_steps].\n",
" ground_truth_is_valid = ground_truth_is_valid[:, tf.newaxis]\n",
" # [batch_size, num_agents_per_joint_prediction].\n",
" object_type = object_type[:, tf.newaxis]\n",
" prediction_trajectory = prediction_trajectory[...,\n",
" (interval - 1)::interval, :]\n",
"\n",
" return py_metrics_ops.motion_metrics(\n",
" config=self._metrics_config.SerializeToString(),\n",
" prediction_trajectory=prediction_trajectory,\n",
" prediction_score=prediction_score,\n",
" ground_truth_trajectory=ground_truth_trajectory,\n",
" ground_truth_is_valid=ground_truth_is_valid,\n",
" prediction_ground_truth_indices=prediction_ground_truth_indices,\n",
" prediction_ground_truth_indices_mask=prediction_ground_truth_indices_mask,\n",
" object_type=object_type)\n",
"\n",
"\n",
"model = SimpleModel(11, 80)\n",
"model = SimpleModel(128, 11, 80)\n",
"optimizer = tf.keras.optimizers.Adam(learning_rate=1e-3)\n",
"loss_fn = tf.keras.losses.MeanSquaredError()\n",
"metrics_config = _default_metrics_config()\n",
Expand All @@ -720,36 +767,56 @@
"\n",
"def train_step(inputs):\n",
" with tf.GradientTape() as tape:\n",
" # Collapse batch dimension and the agent per sample dimension.\n",
" # Mask out agents that are never valid in the past.\n",
" sample_is_valid = inputs['sample_is_valid']\n",
" states = tf.boolean_mask(inputs['input_states'], sample_is_valid)\n",
" gt_trajectory = tf.boolean_mask(inputs['gt_future_states'], sample_is_valid)\n",
" gt_is_valid = tf.boolean_mask(inputs['gt_future_is_valid'], sample_is_valid)\n",
" # [batch_size, num_agents, D]\n",
" states = inputs['input_states']\n",
"\n",
" # Predict. [batch_size, num_agents, steps, 2].\n",
" pred_trajectory = model(states, training=True)\n",
"\n",
" # Set training target.\n",
" prediction_start = metrics_config.track_history_samples + 1\n",
" gt_targets = gt_trajectory[:, prediction_start:, :2]\n",
" weights = tf.cast(gt_is_valid[:, prediction_start:], tf.float32)\n",
" pred_trajectory = model(states, training=True)\n",
"\n",
" # [batch_size, num_agents, steps, 7]\n",
" gt_trajectory = inputs['gt_future_states']\n",
" gt_targets = gt_trajectory[..., prediction_start:, :2]\n",
"\n",
" # [batch_size, num_agents, steps]\n",
" gt_is_valid = inputs['gt_future_is_valid']\n",
" # [batch_size, num_agents, steps]\n",
" weights = (\n",
" tf.cast(inputs['gt_future_is_valid'][..., prediction_start:],\n",
" tf.float32) *\n",
" tf.cast(inputs['tracks_to_predict'][..., tf.newaxis], tf.float32))\n",
"\n",
" loss_value = loss_fn(gt_targets, pred_trajectory, sample_weight=weights)\n",
" grads = tape.gradient(loss_value, model.trainable_weights)\n",
" optimizer.apply_gradients(zip(grads, model.trainable_weights))\n",
"\n",
" object_type = tf.boolean_mask(inputs['object_type'], sample_is_valid)\n",
" # [batch_size, num_agents, steps, 2] -\u003e\n",
" # [batch_size, num_agents, 1, 1, steps, 2]\n",
" pred_trajectory = pred_trajectory[:, :, tf.newaxis, tf.newaxis]\n",
"\n",
" # Fake the score since this model does not generate any score per predicted\n",
" # trajectory.\n",
" pred_score = tf.ones(shape=tf.shape(pred_trajectory)[:-2])\n",
" pred_score = tf.ones(shape=tf.shape(pred_trajectory)[:3])\n",
"\n",
" # Only keep `tracks_to_predict` for evaluation.\n",
" tracks_to_predict = tf.boolean_mask(inputs['tracks_to_predict'],\n",
" sample_is_valid)\n",
" # [batch_size, num_agents].\n",
" object_type = inputs['object_type']\n",
"\n",
" motion_metrics.update_state(\n",
" tf.boolean_mask(pred_trajectory, tracks_to_predict),\n",
" tf.boolean_mask(pred_score, tracks_to_predict),\n",
" tf.boolean_mask(gt_trajectory, tracks_to_predict),\n",
" tf.boolean_mask(gt_is_valid, tracks_to_predict),\n",
" tf.boolean_mask(object_type, tracks_to_predict))\n",
" # [batch_size, num_agents].\n",
" batch_size = tf.shape(inputs['tracks_to_predict'])[0]\n",
" num_samples = tf.shape(inputs['tracks_to_predict'])[1]\n",
"\n",
" pred_gt_indices = tf.range(num_samples, dtype=tf.int64)\n",
" # [batch_size, num_agents, 1].\n",
" pred_gt_indices = tf.tile(pred_gt_indices[tf.newaxis, :, tf.newaxis],\n",
" (batch_size, 1, 1))\n",
" # [batch_size, num_agents, 1].\n",
" pred_gt_indices_mask = inputs['tracks_to_predict'][..., tf.newaxis]\n",
"\n",
" motion_metrics.update_state(pred_trajectory, pred_score, gt_trajectory,\n",
" gt_is_valid, pred_gt_indices,\n",
" pred_gt_indices_mask, object_type)\n",
"\n",
" return loss_value\n",
"\n",
Expand All @@ -759,6 +826,7 @@
"dataset = dataset.batch(32)\n",
"\n",
"epochs = 2\n",
"num_batches_per_epoch = 10\n",
"\n",
"for epoch in range(epochs):\n",
" print('\\nStart of epoch %d' % (epoch,))\n",
Expand All @@ -774,6 +842,9 @@
" (step, float(loss_value)))\n",
" print('Seen so far: %d samples' % ((step + 1) * 64))\n",
"\n",
" if step \u003e= num_batches_per_epoch:\n",
" break\n",
"\n",
" # Display metrics at the end of each epoch.\n",
" train_metric_values = motion_metrics.result()\n",
" for i, m in enumerate(\n",
Expand Down
4 changes: 3 additions & 1 deletion waymo_open_dataset/latency/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,12 +39,14 @@ Converting from Frame protos to usable point clouds/images can be non-trivially
* `<CAMERA_NAME>_POSE_TIMESTAMP`: float32 scalar with the timestamp in seconds for the image (i.e. the timestamp that `<CAMERA_NAME>_POSE` is valid at).
* `<CAMERA_NAME>_ROLLING_SHUTTER_DURATION`: float32 scalar with the duration of the rolling shutter in seconds. See the documentation for `CameraImage.shutter in [dataset.proto](https://github.com/waymo-research/waymo-open-dataset/blob/eb7d74d1e11f40f5f8485ae8e0dc71f0944e8661/waymo_open_dataset/dataset.proto#L268-L283) for details.
* `<CAMERA_NAME>_ROLLING_SHUTTER_DIRECTION`: int64 scalar with the direction of the rolling shutter, expressed as the int value of a `CameraCalibration.RollingShutterReadOutDirection` enum.
* `<CAMERA_NAME>_CAMERA_TRIGGER_TIME`: float32 scalar with the time when the camera was triggered.
* `<CAMERA_NAME>_CAMERA_READOUT_DONE_TIME`: float32 scalar with the time when the last readout finished. The difference between this and the trigger time includes the exposure time and the actual sensor readout time.

See the `LaserName.Name` and `CameraName.Name` enums in [dataset.proto](https://github.com/waymo-research/waymo-open-dataset/blob/eb7d74d1e11f40f5f8485ae8e0dc71f0944e8661/waymo_open_dataset/dataset.proto#L48-L69) for the valid lidar and camera name strings.

To request a field from the previous frame, add `_1` to the end of the field name; for example, `TOP_RANGE_IMAGE_FIRST_RETURN_1` is the range image for the top lidar from the previous frame. Likewise, to request a field from two frames ago, add `_2` to the end of the field name (e.g. `TOP_RANGE_IMAGE_FIRST_RETURN_2`). Note that only two previous frames (in addition to the current frame, which does not require a subscript) can be requested.

Users specify which of these arrays they would like their models to receive using the `DATASET_FORMAT` list of strings in their `wod_latency_submission` module. The requested arrays will then be passed to `run_model` as keyword arguments with the original names (e.g. `TOP_RANGE_IMAGE_FIRST_RETURN`).
Users specify which of these arrays they would like their models to receive using the `DATA_FIELDS` list of strings in their `wod_latency_submission` module. The requested arrays will then be passed to `run_model` as keyword arguments with the original names (e.g. `TOP_RANGE_IMAGE_FIRST_RETURN`).

Note that the `convert_frame_to_dict` function in [utils/frame_utils.py](https://github.com/waymo-research/waymo-open-dataset/blob/ae21353bf721bf36e197654e67d482b5619a2302/waymo_open_dataset/utils/frame_utils.py#L215) will convert a Frame proto into a dictionary with the same keys and values defined above. However, it will not add the `_1` or `_2` suffix to the keys from earlier frames for multi-frame input.

Expand Down
Loading

0 comments on commit 1f75252

Please sign in to comment.