Object tracking is the process of detecting and following moving entities in subsequent observations from multiple sensors. Accurate tracking in complex urban scenarios is crucial for safety applications such as accident mitigation, predictive traffic control and design of safer infrastructure. Despite a well-established theoretical framework for object tracking, various challenges need to be addressed in a real context, including sensor noise, occlusions and asynchronous measures from multiple sensor.
Viscando uses proprietary stereovision sensors to extract detailed traffic movement information in critical urban settings, and algorithms to produce insights on mobility risks, communicate real-time safety information and study traffic scenarios and behavior models. One key feature of these vision tools is the ability to correctly identify and locate objects, and track their movement in consecutive measurements or partially overlapping views. The aim of the present work is to provide an algorithm to unify detections into tracks, with a focus on accuracy and smoothness of the output, as well as providing a flexible implementation that may easily be modified and re-used.
- Provided data is a collection of information from 4 stereovision sensors, with detections corresponding to approximately 750 frames or one minute of activity, of a roundabout in Kolliken, Switzerland. Data has been previously processed to provide points in 3D as centers of detected objects, and bounding boxes containing those objects. In the particular case, the data was collected in Switzerland where video collection for development purposes was not prohibited at the time of collection. Collected video (consisting of image frames) is solely used for research and improvement of object detection and tracking algorithms.
- It can happen that an object detected in a certain frame is occluded in the following ones, resulting in uncertainties about the position of the object. In addition to this, the sensors may fail in associating only one detection per frame to an object. The problem of multiple detections is very common in the data set, and can cause several tracks to be associated to a single object. It is thus important to understand which detections to retain.
- A first solution to the problem of occlusion and multiple detections was found by eliminating detections from areas where sensors could not be considered reliable. This areas differ for each sensor, and usually depend on the presence of obstacle or on the distance from the sensor itself. The detections in zones indicated in the following pictures are removed from the dataset, considering the fact that they are always better captured from a better positioned sensor.
- Lastly, we implemented a rough separation of the surfaces belonging to pavement and road, in order to distinguish between detections likely related to pedestrians and thw ones originated by vehicles. This approach can also be useful for a posteriori analysis of curves
The tracker represents the main part of the algorithm, which receives as input all the accepted detections and returns the tracks connecting the same object across the frames. The main protagonist of the tracker is the Kalman Filter, which is used to predicted the next steps of a track from the previous position and is able to filter the noise due to errors in the detections. Its tuning and building consists of an important part of the project.
- At first detections from different cameras are joined according to an internal clock, considering as synchronous detections collected in an interval of 0.08 seconds. IN order to join the detections related to the same object from different sensors, we center a gaussian distribution in each object, using the bounding box to determine covariance axis, and compute the probability of representing the same object for each couple of detections.
- For each detection, we compute the distance with the last instance of every track. This distance is given by the weighted sum of euclidean distance and of 1 - Intersection over Union of the bounding boxes. The latter is particularly useful during the first stage of movement, where the Kalman Filter has not yet converged and the bounding boxes provide a good intuition of the trajectory.
-
The association between the detections and tracks is then performed through a visiting algorithm, typical of matching problems. The specifics of the algorithm can be found in greater details in the final report, and theoretically assure the best possible matching for the tracks.
-
Tracks are then classified according to their status. For each matched track the filtered or predicted position is then saved as the track true position.
-
Active and Pending (represented object that are likely to be temporarily occluded) tracks are propagated using the filter, with a simple but effective motion model with constant acceleration.
The final result of the algorith consisted in 154 identified tracks, with 198 detections linked in the longest one. 210 tracks were instead removed, since deemed irrelevant by the tracker. In the following picture there is an overview of the obtained tracks, where the shape of the roundabout is clearly visible.
Two examples of tracks are reported in the figure below, one corresponding to a vehicle moving around all the roundabout, the second to a pedestrian crossing the road. Both tracks are a good example of the tracking properties of the algorithm. More insights about these tracks, and analyses of the velocities are provided in the report.
The main challenge consist in the presence of large vehicles, which are usually detected multiple times by the same sensor, leading in this way to multiple tracks, as shown in the picture below.
The comparison between the final result of our online tracker and the offline benchmark proposed by the company is interesting. In general our tracker is better at reducing the noise in the center of the roundabout, where no tracks should exist but a lot of noise is generated by the statues. The tracks produced by the offline algorithm are of course more polished, but this is due to the possibility of applying smoothing techniques in a second moment. In one of the highlighted examples, we can notice that the longest track predicted by our algorithm is actually wrongly divided in three smaller tracks by the benchmark algorithm.
One possible notable improvement consists in developing a merging algorithm, able to detect complete tracks related to the same object and join them. We developed such an algorithm, but we had the time to implement it only as an offline instrument. The merging algorith works by propagating all tracks for 20 frames, thanks to the Kalman Filter, and then comparing all the couple of tracks in terms of mean distance, mean difference iin velocity and mean angle (computed via scalar product of velocities). Tracks respecting particular conditions on this values are then merged. One example of proposed merging is shown in the last figure.
More analysis on the results of our algorithm, its pros and cons and further improvements are presented in the report.
Main.ipynbis the Jupyter Notebook containing all the work of the project. The results are displayed, but the code cannot be used since the data are private.Final Reportis the pdf of the final report of the project. It contains detailed information on every step of the algorithm, a more accurate analysis of challenges and results and some theoretical insights.