KITTI Carla evaluation
Model pre-trained on KITTI odometry was evaluated on KITTI-CARLA dataset converted into KITTI odometry format.
Code:
Prediction errors:
| Iteration | Mean Translation Error (cm) | Mean Rotation Error (°) | Median Translation Error (cm) | Median Rotation Error (°) | Std. Translation Error (cm) | Std. Rotation Error (°) |
|---|---|---|---|---|---|---|
| 0 | 144.8048 | 18.9558 | 149.2977 | 19.4363 | 41.7349 | 5.5941 |
| 1 | 133.1928 | 10.2541 | 115.1579 | 7.6585 | 68.6467 | 7.5684 |
| 2 | 117.8647 | 8.7967 | 117.6092 | 4.8870 | 46.6583 | 8.5631 |
| 3 | 100.7304 | 8.5700 | 99.7916 | 4.1654 | 23.4888 | 8.8601 |
| 4 | 88.9915 | 8.8693 | 85.0354 | 4.5777 | 20.4002 | 8.8486 |
| 5 | 86.5924 | 8.9997 | 83.8602 | 4.5892 | 21.3216 | 8.7617 |
The errors in prediction are very large in comparison to KITTI odometry evaluation results.
Input:
Prediction (after 5th iteration):
Ground truth:
Logs
Iteration 0:
Translation Error && Rotation Error:
Iteration 0: Mean Translation Error: 144.8048 cm Mean Rotation Error: 18.9558 °
Iteration 0: Median Translation Error: 149.2977 cm Median Rotation Error: 19.4363 °
Iteration 0: Std. Translation Error: 41.7349 cm Std. Rotation Error: 5.5941 °
Translation Error XYZ:
Iteration 0: Mean Translation X Error: 71.0694 cm Median Translation X Error: 38.0695 cm Std. Translation X Error: 59.7768 cm
Iteration 0: Mean Translation Y Error: 106.2923 cm Median Translation Y Error: 97.9751 cm Std. Translation Y Error: 18.7499 cm
Iteration 0: Mean Translation Z Error: 68.2074 cm Median Translation Z Error: 83.7281 cm Std. Translation Z Error: 57.0892 cm
Rotation Error RPY:
Iteration 0: Mean Rotation Roll Error: 7.9155 ° Median Rotation Roll Error: 3.9154 ° Std. Rotation Roll Error: 7.6591 °
Iteration 0: Mean Rotation Pitch Error: 15.9658 ° Median Rotation Pitch Error: 15.7860 ° Std. Rotation Pitch Error: 2.5118 °
Iteration 0: Mean Rotation Yaw Error: 8.1643 ° Median Rotation Yaw Error: 7.6444 ° Std. Rotation Yaw Error: 2.3003 °
Iteration 1:
Translation Error && Rotation Error:
Iteration 1: Mean Translation Error: 133.1928 cm Mean Rotation Error: 10.2541 °
Iteration 1: Median Translation Error: 115.1579 cm Median Rotation Error: 7.6585 °
Iteration 1: Std. Translation Error: 68.6467 cm Std. Rotation Error: 7.5684 °
Translation Error XYZ:
Iteration 1: Mean Translation X Error: 96.8216 cm Median Translation X Error: 122.9046 cm Std. Translation X Error: 48.2293 cm
Iteration 1: Mean Translation Y Error: 29.5586 cm Median Translation Y Error: 4.0672 cm Std. Translation Y Error: 44.8311 cm
Iteration 1: Mean Translation Z Error: 58.7128 cm Median Translation Z Error: 53.4680 cm Std. Translation Z Error: 16.7352 cm
Rotation Error RPY:
Iteration 1: Mean Rotation Roll Error: 4.8284 ° Median Rotation Roll Error: 2.9688 ° Std. Rotation Roll Error: 3.4398 °
Iteration 1: Mean Rotation Pitch Error: 10.6400 ° Median Rotation Pitch Error: 2.0681 ° Std. Rotation Pitch Error: 15.9607 °
Iteration 1: Mean Rotation Yaw Error: 6.7819 ° Median Rotation Yaw Error: 5.1707 ° Std. Rotation Yaw Error: 6.5251 °
Iteration 2:
Translation Error && Rotation Error:
Iteration 2: Mean Translation Error: 117.8647 cm Mean Rotation Error: 8.7967 °
Iteration 2: Median Translation Error: 117.6092 cm Median Rotation Error: 4.8870 °
Iteration 2: Std. Translation Error: 46.6583 cm Std. Rotation Error: 8.5631 °
Translation Error XYZ:
Iteration 2: Mean Translation X Error: 45.7237 cm Median Translation X Error: 48.4606 cm Std. Translation X Error: 29.3951 cm
Iteration 2: Mean Translation Y Error: 98.9958 cm Median Translation Y Error: 105.7197 cm Std. Translation Y Error: 51.7050 cm
Iteration 2: Mean Translation Z Error: 51.5480 cm Median Translation Z Error: 56.7265 cm Std. Translation Z Error: 41.5423 cm
Rotation Error RPY:
Iteration 2: Mean Rotation Roll Error: 3.3595 ° Median Rotation Roll Error: 2.7443 ° Std. Rotation Roll Error: 2.8336 °
Iteration 2: Mean Rotation Pitch Error: 12.4442 ° Median Rotation Pitch Error: 3.9686 ° Std. Rotation Pitch Error: 14.9886 °
Iteration 2: Mean Rotation Yaw Error: 3.4786 ° Median Rotation Yaw Error: 3.6270 ° Std. Rotation Yaw Error: 3.3538 °
Iteration 3:
Translation Error && Rotation Error:
Iteration 3: Mean Translation Error: 100.7304 cm Mean Rotation Error: 8.5700 °
Iteration 3: Median Translation Error: 99.7916 cm Median Rotation Error: 4.1654 °
Iteration 3: Std. Translation Error: 23.4888 cm Std. Rotation Error: 8.8601 °
Translation Error XYZ:
Iteration 3: Mean Translation X Error: 59.4481 cm Median Translation X Error: 55.8068 cm Std. Translation X Error: 42.7639 cm
Iteration 3: Mean Translation Y Error: 59.7416 cm Median Translation Y Error: 47.7140 cm Std. Translation Y Error: 26.3071 cm
Iteration 3: Mean Translation Z Error: 38.9230 cm Median Translation Z Error: 48.7838 cm Std. Translation Z Error: 18.4401 cm
Rotation Error RPY:
Iteration 3: Mean Rotation Roll Error: 3.6508 ° Median Rotation Roll Error: 3.1085 ° Std. Rotation Roll Error: 1.4058 °
Iteration 3: Mean Rotation Pitch Error: 9.9906 ° Median Rotation Pitch Error: 0.5662 ° Std. Rotation Pitch Error: 16.5532 °
Iteration 3: Mean Rotation Yaw Error: 3.2017 ° Median Rotation Yaw Error: 2.6915 ° Std. Rotation Yaw Error: 2.9724 °
Iteration 4:
Translation Error && Rotation Error:
Iteration 4: Mean Translation Error: 88.9915 cm Mean Rotation Error: 8.8693 °
Iteration 4: Median Translation Error: 85.0354 cm Median Rotation Error: 4.5777 °
Iteration 4: Std. Translation Error: 20.4002 cm Std. Rotation Error: 8.8486 °
Translation Error XYZ:
Iteration 4: Mean Translation X Error: 54.8688 cm Median Translation X Error: 40.0891 cm Std. Translation X Error: 59.5795 cm
Iteration 4: Mean Translation Y Error: 45.7366 cm Median Translation Y Error: 33.7925 cm Std. Translation Y Error: 27.7975 cm
Iteration 4: Mean Translation Z Error: 39.8040 cm Median Translation Z Error: 53.7304 cm Std. Translation Z Error: 31.9920 cm
Rotation Error RPY:
Iteration 4: Mean Rotation Roll Error: 3.8337 ° Median Rotation Roll Error: 3.6347 ° Std. Rotation Roll Error: 1.0633 °
Iteration 4: Mean Rotation Pitch Error: 11.1607 ° Median Rotation Pitch Error: 1.7004 ° Std. Rotation Pitch Error: 16.6367 °
Iteration 4: Mean Rotation Yaw Error: 2.8803 ° Median Rotation Yaw Error: 0.9007 ° Std. Rotation Yaw Error: 3.5052 °
Iteration 5:
Translation Error && Rotation Error:
Iteration 5: Mean Translation Error: 86.5924 cm Mean Rotation Error: 8.9997 °
Iteration 5: Median Translation Error: 83.8602 cm Median Rotation Error: 4.5892 °
Iteration 5: Std. Translation Error: 21.3216 cm Std. Rotation Error: 8.7617 °
Translation Error XYZ:
Iteration 5: Mean Translation X Error: 58.9295 cm Median Translation X Error: 43.4976 cm Std. Translation X Error: 60.0300 cm
Iteration 5: Mean Translation Y Error: 39.5618 cm Median Translation Y Error: 25.6842 cm Std. Translation Y Error: 26.3772 cm
Iteration 5: Mean Translation Z Error: 43.7144 cm Median Translation Z Error: 59.6551 cm Std. Translation Z Error: 33.1639 cm
Rotation Error RPY:
Iteration 5: Mean Rotation Roll Error: 3.9663 ° Median Rotation Roll Error: 3.9002 ° Std. Rotation Roll Error: 1.0498 °
Iteration 5: Mean Rotation Pitch Error: 11.1202 ° Median Rotation Pitch Error: 1.6900 ° Std. Rotation Pitch Error: 16.9425 °
Iteration 5: Mean Rotation Yaw Error: 3.0535 ° Median Rotation Yaw Error: 1.9188 ° Std. Rotation Yaw Error: 2.9503 °- Converted KITTI-CARLA dataset into KITTI format using this script
- Ran this version of LCCNet on the converted simulation dataset.
Executing task: cd /home/jay/LCCNet && python evaluate_calib.py
WARNING - LCCNet-evaluate-iterative - No observers have been added to this run
INFO - LCCNet-evaluate-iterative - Running command 'main'
INFO - LCCNet-evaluate-iterative - Started
Ground truth poses are not avaialble for sequence 00.
VAL SET: Using this file: /home/jay/datasets/KITTI-CARLA-KITTI-format/sequences/val_RT_left_seq00_20.00_1.50.csv
5000
0%| | 0/5000 [00:00<?, ?it/s]Init worker 0 with seed 134410049
Init worker 1 with seed 134410050
Init worker 2 with seed 134410051
Init worker 3 with seed 134410052
Init worker 4 with seed 134410053
Init worker 5 with seed 134410054
17%|███████████████████▍ | 850/5000 [02:55<14:15, 4.85it/s]/home/jay/mambaforge/envs/lcc/lib/python3.6/site-packages/torch/nn/functional.py:2423: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
"See the documentation of nn.Upsample for details.".format(mode))
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=405 error=11 : invalid argument
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5000/5000 [17:06<00:00, 4.87it/s]
Iterative refinement:
Iteration 0: Mean Translation Error: 208.6760 cm Mean Rotation Error: 24.7035 °
Iteration 0: Median Translation Error: 208.6760 cm Median Rotation Error: 24.7035 °
Iteration 0: Std. Translation Error: nan cm Std. Rotation Error: nan °
Iteration 0: Mean Translation X Error: 119.9736 cm Median Translation X Error: 127.7005 cm Std. Translation X Error: 13.5035 cm
ERROR - LCCNet-evaluate-iterative - Failed after 0:17:12!
Traceback (most recent calls WITHOUT Sacred internals):
File "evaluate_calib.py", line 564, in main
print(f"Iteration {i}: \tMean Translation Y Error: {errors_t2[i][1].mean():.4f} cm "
IndexError: list index out of rangeInstead of generated all overlappings of images and point clouds with predicted extrinsics, it only generated one: 0850.png.
