I've found the streaming parser to be too slow, mostly because it's maintaining a complicated state machine, and generating lots of objects during the parsing.
I think it's better to just slurp in the data, first, before dealing with the logic within julia. Here's an example:
With this package:
julia> MAP_FILENAME = "tech_square.osm"
"tech_square.osm"
julia> @time nodes, hwys, builds, feats = getOSMData(MAP_FILENAME);
1.001549 seconds (340.25 k allocations: 1.888 GB, 8.55% gc time)
Under my proposal (implemented here):
julia> @time nodes1 = parseNodes(MAP_FILENAME);
0.032125 seconds (163.66 k allocations: 10.621 MB, 17.01% gc time)
julia> @time ways1 = parseWays(MAP_FILENAME);
0.022908 seconds (141.77 k allocations: 8.979 MB)
The differences become significant when processing the following osm file in julia v0.4:
Before:
julia> @time nodes, hwys, builds, feats = getOSMData(MAP_FILENAME);
2161.987085 seconds (570.21 M allocations: 3.354 TB, 13.85% gc time)
After:
julia> @time nodes = OpenStreetMapParser.parseNodes(MAP_FILENAME);
49.350339 seconds (244.94 M allocations: 14.512 GB, 9.32% gc time)
julia> @time ways = OpenStreetMapParser.parseWays(MAP_FILENAME);
50.805148 seconds (209.75 M allocations: 11.937 GB, 23.16% gc time)
I've found the streaming parser to be too slow, mostly because it's maintaining a complicated state machine, and generating lots of objects during the parsing.
I think it's better to just slurp in the data, first, before dealing with the logic within julia. Here's an example:
With this package:
Under my proposal (implemented here):
The differences become significant when processing the following osm file in julia v0.4:
Before:
After: