Merging diff files

[joto]

To fully replace Osmosis, we need to merge minutely diff files into hourly and daily diffs and I am trying to figure out exactly how this should be done.

In normal operation the program will run

• once an hour (shortly after the full hour) to assemble 60 minutely diffs into one hourly diff and
• once a day (shortly after midnight) to assemble 24 hourly diffs into one daily diff.

So far that's easy enough. But the question is: What should happen in special situations, say when the program didn't run for some reason and there are older minutelies lying around etc.

What I am bit unclear about is how osmdbt should figure out which minutelies to assemble into an hourly. If if just takes the next 60 it hasn't processed yet, the result might not correspond to full hours, because there might be minutelies missing. But it can't really look at the timestamps either, because its a bit fuzzy how they are generated. The Osmosis code is really hard to read so I haven't figured out yet what it is doing exactly.

In addition to that I would expect something like the following behaviour:

• For every hour fully in the past, if there is at least one minutely, create the hourly from all minutelies in that hour. And the analogue for the daylies.
• Independently of when it is run, hours that are not "done" should not be processed. So if the program is run at half past, it should ignore the 30 minutely files lying around. Same for days.
• Don't create empty diff files. This could happen if the server is down for some extended period.
• It should be possible to "reset" the program to some earlier point in time, so that if we have created defective hourly or daily files for some reason, we can re-generate them easily.
• The diff and state files should be generated outside the directory tree and synced before being moved in place, so that if the program dies for some reason, no partial output is generated. The next run should ignore partial work done before and do the right thing.

Any opinions on these issues?

[mmd-osm]

The Osmosis code is really hard to read so I haven't figured out yet what it is doing exactly.

Maybe @brettch still remembers some of those details, or has some good pointers to the relevant bits in the code to get started.

[brettch]

Yep, it looks like I wrote most of this code back in 2009 so my memory is a bit hazy on the details.

Osmosis does use timestamps heavily. While timestamps in the state files may be a bit fuzzy, there are some guarantees available. The timestamp in a replication state file is greater than or equal to all data inside that changeset and those preceding it (I hope I have this right, it's important). Similarly, data in subsequent replication files may also be less than this timestamp due to late arriving data. It's eventually consistent. You can use the timestamps to determine when to start replicating and know that you'll eventually see all data created after that timestamp.

Osmosis doesn't count files, or even really care what interval size you're using, it's generic in that sense. You can specify interval lengths in seconds if you wish, and that's what's I was striving for when I first wrote it. Osmosis is setup to create minute files initially because that's the minimal interval in cron, but it is possible to create a file every second if you wish. --receive-replication and the --send-replication-xxxx tasks were written to support always connected streaming but I never got around to getting them up and running or iron out the remaining kinks.

If at least one interval of data is available then Osmosis starts processing. It calculates a maximum interval prior to processing that is a multiple of the interval length based on what's available on the server. If a partial interval is available it won't process. If 1.5 intervals are available, it will process one interval and stop.

Osmosis using the local state file at the root of the directory tree to figure out what the current local interval number (and associated timestamp) is. Therefore resetting is a simple matter of copying an earlier state file over the top and it will start from there again.

If you want to understand the Osmosis date logic, the two main classes to look at are:

• BaseReplicationDownloader - It has a method called calculateMaximumTimestamp which determines when to stop downloading replication files. It limits to data available on the server and also has a maximum bound that avoids downloading half of the Internet in a single run if processing hasn't been run for a long time. Under normal operation this limit won't get hit. The basic ReplicationDownloader class (--read-replication-interval) used for streaming changes inherits this basic functionality.
• ReplicationFileMerger - This is the implementation of --merge-replication-files and a sub-class of BaseReplicationDownloader. It imposes additional timestamp constraints for file merging. It augments the calculateMaximumTimestamp method by further limiting the maximum timestamp to align to file boundaries. It has all the logic to produce multiple output files if multiple intervals of data are available.

Hope that helps.