Btrfs subvolumes/snapshots, UUID and file move detection

[taam]

Currently the SnapRAID support for Btrfs is probably not ideal:

• SnapRAID is using the filesystem UUID as unique identifier (also in relation to the inode numbers), which in the case of Btrfs is maybe incorrect, since a Btrfs filesystem can have various subvolumes. There can be identical inode numbers across different subvolumes for different files. ⇒ Given the current logic, it might be more correct to not use the filesystem UUID, but the subvolume UUID instead.
• SnapRAID is unable to get the filesystem UUID, if the disk path is a pointing to a not explicitly mounted subvolume, since the device number does not appear in /proc/self/mountinfo. This could be an actual distinct subvolume, but it would also be typical for a snapshot, which you usually don't mount explicitly. ⇒ Btrfs allows mounting subvolumes, at which point the current UUID detection "works" and returns the filesystem UUID, independently on which subvolume is mounted, but that's basically a bad hack and not practical.
• (SnapRAID does not support nested subvolumes in a disk directory. ⇒ Supporting that would probably be more complicated, also not necessary for my use case.)

Why does it matter? Using snapshots is a nice way to mitigate some SnapRAID issues. If you snapshot a Btrfs subvolume, the inode numbers of the snapshot are apparently cloned from the parent¹ subvolume, so file move detection should be supported.

Therefore the UUID detection should be fixed to also work for not explicitly mounted subvolumes. The devuuid function already has the actual path and should use it. Some possible solutions regarding which UUID to use:

• [A] Keep using the filesystem UUID for better backwards compatibility. (See BTRFS_IOC_FS_INFO ioctl. Regarding fsid vs metadata_uuid, see the comment below.)
• [B] Instead of using the filesystem UUID, use the subvolume UUID. For snapshots use the parent¹ UUID. (See BTRFS_IOC_GET_SUBVOL_INFO ioctl. Caveat: Have seen from ext4 converted filesystems, where the root subvolume had an empty/all-zero UUID, did not try to reproduce.)
• [C] Like [B], but for snapshots search for the parent¹ subvolumes recursively (as there could be snapshots of snapshots), and use the top subvolume's UUID. (Maybe the most correct method, but looks way more tricky than the previous options. Starting point might be libbtrfsutil. And yes, I did read some of the source code, looks non-trivial...)

All variants should fix the snapshot issue and allow file move detection.

(Side node: The TODO file says "Keep track of the subvolumeid, along with the inode, and use it when searching/comparing inodes.", however, I think this is probably not the way to go. First, the numeric subvolume ID is not unique, quite the contrary. Second, it would prevent robust move detection using snapshots, or even worse result in random behavior, e.g. if you don't keep the old snapshot(s) and sometimes end up with the same subvolume ID.)

Footnotes

1. Original snapshotted subvolume, so referring to the parent UUID, contrary to the parent ID, which has a different meaning. ↩ ↩² ↩³

[MasterCATZ]

I also use BTRFS

I was wondering why Moving Files did not always work and it gets picked up as a completely new file

I also use https://github.com/automorphism88/snapraid-btrfs
I am unsure how you can even use snapraid without it ? or else your in a endless scan ?

[taam]

Regarding fsid vs metadata_uuid: Btrfs actually has two (potentially different) fsids, but the documentation there is more confusing than helpful.

As I understand it, initially the filesystem has one common fsid, stored in all metadata blocks, and also in the superblock field fsid, but they can be changed e.g. using btrfstune. I spare you the details there, but the important point is, that the superblock fsid can be changed separately. When this is the case, the original fsid (which is still stored in the metadata blocks) is copied into the superblock field metadata_uuid, hence the name I guess, it's the metadata fsid/UUID. You can manually set the fsid to the metadata fsid/UUID, then the metadata_uuid field is not required anymore and set to zero.

Filesystem tools like blkid/lsblk/wipefs apparently use the superblock's fsid, so this is what SnapRAID is currently picking up as well. Using the above mentioned ioctl call and the BTRFS_FS_INFO_FLAG_METADATA_UUID flag one can get both fsids. However, while using the conceptually probably more permanent metadata fsid/UUID would be again maybe more correct in comparison to the superblock fsid, if a breaking change is an option, I see little benefit over the above mentioned subvolume UUIDs, so I guess the options stay the same.

[amadvance]

I worked on the Btrfs support, and now the plan is to have full support, including the ability to create snapshots at sync time and use them during scrub, check, and fix. Essentially, it does what the existing scripts are doing but with better integration.

You can get a preview of the documentation here:
https://github.com/amadvance/snapraid/blob/master/doc/snapraid.txt#L1505

Regarding the UUID, I'll keep using the filesystem UUID since snapshots retain the same inodes as the original copy. Sharing the same UUID allows for an easy switch from one snapshot to another. In case of false positives, the code already checks the size and timestamp, so a real false positive match is highly unlikely.

[taam]

Sounds good, some questions/comments/ideas regarding the description in the documentation preview:

• By storing the .content file inside the snapshotted filesystem, the stable snapshot preserves an older version of the content file, and the metadata within it, alongside the corresponding version of the data.

  Does that imply, that the stable content file is not simply a backup, but used as data source during recovery? If so, is that effectively a better alternative to snapshotting the parity (which has implications regarding the required disk space), or would it still be beneficial to snapshot that as well?

  • Could there be complications, when a sync is interrupted, and then resumed not with the pending snapshots, but a new snapshot set? Or is that not allowed in general?

• It says the content file should be put "inside the data disks that are being snapshotted", but where exactly? So far the typical setup probably is to use separate subvolumes for data and content files, should this be kept?

  During this promotion, if a content file is stored on the data disk, the final version of the content file is cloned (via reflink) into the snapshot.

  If it's copied/cloned afterwards, it should be in a separate subvolume to start with I guess?

• Design-wise the "into the snapshot" part does not seem ideal to me, wouldn't it be cleaner to actually start with a read-only snapshot and really not touch it? The content file can still be preserved separately.

  • If you want the content file copy to be read-only in the end, you could e.g. create a subvolume "[...pending...]", put the actual data subvolume snapshot in a subdirectory there, and the content file next to it. That way the data snapshot can always be read-only, and the (path-wise) parent subvolume with the content file (and theoretically further metadata) can be made read-only during the promotion to stable.

• It does not mention, how the diff command interacts with the snapshots, so I assume currently it's live-only? At least for me it would still be important, that one can create snapshots, evaluate the situation with diff, and only then sync using exactly those snapshots. Therefore:

  • Since the existence of pending snapshots fundamentally alters the behavior of other commends, probably there should be a stage before pending, similar to the snapraid-btrfs diff snapshots, promoted to pending only after a sync actually started. (That might be better even for snapraid "alone" considering the gui/daemon related arguments, since snapraid already optionally checks for certain conditions before the actual sync starts.)
  • There should be a way to explicitly create a set of snapshots.
  • There should probably also be a way to cleanup/remove (non-stable) snapshot sets?
  • There should be a way to tell SnapRAID which snapshot set to use where applicable. (E.g. for diff you might want to start with a new set, or use the last diff set, for sync you might want to start with a new set, the last diff set, or the current pending set for resuming, if any of this is even allowed.)

[amadvance]

  Does that imply, that the `stable` content file is not simply a backup, but used as data source during recovery? If so, is that effectively a better alternative to snapshotting the parity (which has implications regarding the required disk space), or would it still be beneficial to snapshot that as well?

Yes. The stable snapshot contains exactly what was used to compute the parity. Therefore, there is no need to snapshot the parity disks themselves.

  * Could there be complications, when a `sync` is interrupted, and then resumed not with the pending snapshots, but a new snapshot set? Or is that not allowed in general?

Every time a sync starts, a new pending snapshot is created, and any existing pending snapshot is deleted. This maintains current behavior. If you restart an interrupted sync, the new sync will capture the current state of the live filesystem, including all changes that occurred while the sync was interrupted.

* It says the content file should be put "inside the data disks that are being snapshotted", but where exactly? So far the typical setup probably is to use separate subvolumes for data and content files, should this be kept?
  If it's copied/cloned afterwards, it should be in a separate subvolume to start with I guess?

The content file must be located within the same subvolume as the data.
The cloning process is necessary because the pending snapshot is created at the start of the sync, but the final content file is written at the end. Consequently, the initial snapshot contains an outdated copy. SnapRAID clones the latest content file into the existing pending snapshot, which also frees up the space used by the outdated version.

* Design-wise the "into the snapshot" part does not seem ideal to me, wouldn't it be cleaner to actually start with a read-only snapshot and really not touch it? The content file can still be preserved separately.

The snapshot is made writable specifically to allow the latest content file to be cloned into it. While it is possible to toggle the read-only flag before and after writing the content file, I don't see a strict need for it. The snapshot resides in a hidden pending/ subdirectory known only to SnapRAID, and SnapRAID does not write to it during the sync itself.
Using a separate subvolume for the content file would add unnecessary complication. Keeping both the data and the content file in the same subvolume mimics the current behavior of storing content files on data disks.

* It does not mention, how the `diff` command interacts with the snapshots, so I assume currently it's live-only? At least for me it would still be important, that one can create snapshots, evaluate the situation with `diff`, and only then `sync` using exactly those snapshots. 

The diff command operates on the live filesystem. Only sync, scrub, check, and fix utilize snapshots.

  * Since the existence of `pending` snapshots fundamentally alters the behavior of other commands, probably there should be a stage before `pending`, similar to the snapraid-btrfs `diff` snapshots, promoted to `pending` only after a sync actually started. (That might be better even for snapraid "alone" considering the gui/daemon related arguments, since snapraid already optionally checks for certain conditions before the actual sync starts.)

A pending snapshot only exists if a sync is interrupted. This is a corner case. It makes sense to use this snapshot only for commands that utilize parity, as the parity is synchronized with the files in that specific snapshot.

  * There should be a way to explicitly create a set of snapshots.

I don't see an advantage to this. Currently, snapshots are entirely transparent and require no manual management. You can use SnapRAID exactly as before, but with added protection.

  * There should probably also be a way to cleanup/remove (non-`stable`) snapshot sets?

This is already handled automatically. When a sync completes, the previous stable snapshot is deleted and the pending one is promoted to stable. Thus, there is only ever one stable snapshot. If a sync is interrupted, a pending snapshot remains, but it is deleted and replaced at the start of the next sync.

  * There should be a way to tell SnapRAID which snapshot set to use where applicable. (E.g. for `diff` you might want to start with a new set, or use the last diff set, for `sync` you might want to start with a new set, the last diff set, or the current pending set for resuming, if any of this is even allowed.)

The only scenario where this might be needed is if a disk fails during a sync. In that case, you have two snapshots: stable and pending. A fix would use the pending snapshot, but a full recovery might require a second fix against the stable snapshot.
I am still evaluating how to address this. The ideal solution would be to make fix capable of transparently using files from both snapshots as needed.

In my view, snapshots serve as a reference point synchronized with the parity files. For all other commands, the live filesystem is what matters and should be used, just as it is in a setup without snapshots.

[taam]

  Does that imply, that the `stable` content file is not simply a backup, but used as data source during recovery? If so, is that effectively a better alternative to snapshotting the parity (which has implications regarding the required disk space), or would it still be beneficial to snapshot that as well?

Yes. The stable snapshot contains exactly what was used to compute the parity. Therefore, there is no need to snapshot the parity disks themselves.

But in case of an interrupted sync, the parity should be split, no? I would expect the unmodified parts to match the old content file, the already overwritten parts to follow the new content file, and if the interruption was not clean, probably a part in some broken state. (Later you wrote "a full recovery might require a second fix", which seems to confirm this? What I meant by "used as data source during recovery" sounds like the "ideal solution" you mentioned there.)

  * Could there be complications, when a `sync` is interrupted, and then resumed not with the pending snapshots, but a new snapshot set? Or is that not allowed in general?

Every time a sync starts, a new pending snapshot is created, and any existing pending snapshot is deleted. This maintains current behavior. If you restart an interrupted sync, the new sync will capture the current state of the live filesystem, including all changes that occurred while the sync was interrupted.

If my above assumption is true, couldn't that potentially delete information necessary to recover some of the data, by modifying the content file and deleting files only present in the snapshot, that are now part of the parity? If so, it seems one would need either every set of content file + snapshot used for any part of the parity (which I think snapraid-btrfs is effectively doing, at least for the snapshot part), or a stable snapshot of the parity.

* It says the content file should be put "inside the data disks that are being snapshotted", but where exactly? So far the typical setup probably is to use separate subvolumes for data and content files, should this be kept?
  If it's copied/cloned afterwards, it should be in a separate subvolume to start with I guess?

The content file must be located within the same subvolume as the data. The cloning process is necessary because the pending snapshot is created at the start of the sync, but the final content file is written at the end. Consequently, the initial snapshot contains an outdated copy. SnapRAID clones the latest content file into the existing pending snapshot, which also frees up the space used by the outdated version.

Ok, could live with that I guess, but why is it necessary to put it in the same subvolume? You can reflink between subvolumes, there are just certain restrictions with older kernels and multiple mountpoints (https://btrfs.readthedocs.io/en/latest/Reflink.html). I would assume you'd use something like the UUID detection to match content files and data disks, since you've chosen the filesystem route there, that should work with different subvolumes as well. So shouldn't it just work either way? Seems much cleaner to me keeping it outside of the data subvolume.

The diff command operates on the live filesystem. [...] Currently, snapshots are entirely transparent and require no manual management. You can use SnapRAID exactly as before, but with added protection. [...]

So my understanding of your approach would be you want to use snapshots to fix existing problems, but you don't want to enhance the functionality (for now). This is certainly valid, and I can understand it, because the later could make it much more complicated, but it also means you can't have some of the snapraid-btrfs stuff like dsync. E.g. I'm using a balancer script, that moves files between disks automatically based on certain rules, in order for that to work safely, snapshots are used, so the script can check what exactly would be synced, that would not be possible when working only on the live filesystem.

Therefore the currently suggested snapshot implementation seems to be a good step, but I'd probably continue to manage snapshots externally then. Maybe in the end it can be combined somehow, let's see. This ticket was primarily about the UUID part, for which I've seen you already added a commit, thanks a lot!

[amadvance]

If my above assumption is true, couldn't that potentially delete information necessary to recover some of the data, by modifying the content file and deleting files only present in the snapshot, that are now part of the parity? If so, it seems one would need either every set of content file + snapshot used for any part of the parity (which I think snapraid-btrfs is effectively doing, at least for the snapshot part), or a stable snapshot of the parity.

Yes. But if you get an interrupted sync, you are supposed to fix your array and not delete files on it. Recovering from two snapshots is already complex enough. Supporting recovery from even more snapshots seems like an extreme corner case.

Note that, in my understanding, snapraid-btrfs does not snapshot the content file, so you won't be able to recover from any past snapshot, only from the latest. In fact, I don't think snapraid-btrfs can handle the case of an interrupted sync due to a disk failure.

Ok, could live with that I guess, but why is it necessary to put it in the same subvolume? ... Seems much cleaner to me keeping it outside of the data subvolume.

Having the .content file in the same subvolume isn't due to a Btrfs limitation, but because it seems like the most natural approach. The .content file describes what is in the subvolume, and having it in the same snapshot makes their bond clear.

So my understanding of your approach would be you want to use snapshots to fix existing problems, but you don't want to enhance the functionality (for now).

Indeed. The goal is to improve data safety, not to allow the use of snapshots for other purposes. In fact, I don't think that would even make sense. You have only one parity, so SnapRAID can only secure one snapshot. To truly support multiple data snapshots, you would also need to snapshot the parity, as the btrfs-snapraid script does.

This is certainly valid, and I can understand it, because the later could make it much more complicated, but it also means you can't have some of the snapraid-btrfs stuff like dsync.

The new SnapRAID sync (with snapshots enabled) is like dsync. It creates a new snapshot and computes parity on it. It lacks the diff and manual confirmation, but those are not features inherently related to snapshots.

[amadvance]

Just to report that I've removed the requirement to snapshot content files.

The new content file format now includes metadata for previous versions of files. This ensures that even in the event of a disk failure during a sync process, the fix command can access historical file data from the snapshots. This allows SnapRAID to correctly resolve the parity equations and reconstruct the lost disk, even if the required data blocks belong to files that were updated or deleted after the last successful sync.

[taam]

Just to report that I've removed the requirement to snapshot content files.

The new content file format now includes metadata for previous versions of files. This ensures that even in the event of a disk failure during a sync process, the fix command can access historical file data from the snapshots. This allows SnapRAID to correctly resolve the parity equations and reconstruct the lost disk, even if the required data blocks belong to files that were updated or deleted after the last successful sync.

Nice, that sounds good. I assume that information is only stored until a sync completes? (For my last comment I actually wrote a paragraph suggesting exactly this, but then deleted it, as I deemed it to be too unrealistic.)

Is that only going to work in combination with the integrated snapshots, or is the information always kept (independently from filesystem and snapshots)? I'm asking because it might o.c. be relevant, if you manage the snapshots yourself. And to name another corner case, it might be also relevant, if you don't have a snapshot to start with, but are able to restore the relevant data from another source.

Note that, in my understanding, snapraid-btrfs does not snapshot the content file, so you won't be able to recover from any past snapshot, only from the latest. In fact, I don't think snapraid-btrfs can handle the case of an interrupted sync due to a disk failure.

So far not by itself indeed. I therefore keep a history of the content files separately to hopefully mitigate this, not great I know, but maybe that's not necessary then anymore.

The new SnapRAID sync (with snapshots enabled) is like dsync. It creates a new snapshot and computes parity on it. It lacks the diff and manual confirmation, but those are not features inherently related to snapshots.

Sorry, but I must disagree here. It is more like snapraid-btrfs sync, but not like dsync, there's no equivalent for --use-snapshot-all=diff. The separation of diff and sync is key here, that's the part I'm missing. (But it's fine, if I can continue managing the snapshots externally. If your latest change improves the situation there as well, even better.)

[amadvance]

I assume that information is only stored until a sync completes?
Yes. The information is written at the start of the sync and removed when the sync completes.

Is that only going to work in combination with the integrated snapshots, or is the information always kept (independently from filesystem and snapshots)?
It's only for the integrated snapshots, and only during sync. In fact, sync is the only command that scans for differences and saves a new content file.

Sorry, but I must disagree here. It is more like snapraid-btrfs sync, but not like dsync, there's no equivalent for --use-snapshot-all=diff.
What is missing is the possibility of manually verifying the differences on the same snapshot that will be used by sync. I agree on that. But the automated checks that SnapRAID Daemon has, like limits on the number of deleted or updated files, are going to operate on the same snapshot used by sync.