The Quest of Finding Out: "How to find out the actual byte-limit of xattrs on /that/ filesystem?"

[pjotrek-b]

Good question.
Does anyone know a way to query "available xattr bytesize" for filesystem-handle XY (file/folder/object) - in code?
Is there anything in Python, like: get_max_xattr_size() or get_free_xattr_space(filename)?

Currently all I can find are short hints about "4k", "64k" or unlimited - depending on the filesystem and the parameters it was formatted/mounted with.
And I get an "out of diskspace" error when I try to write the IPTC international metadata image standard sample data to ext4.

Any insights or links to sources are greatly appreciated :)

TODO:
Put links here:

• ext4 (kernel) docs: 4k (keys and/or values?)
• zfs docs: unlimited xattrs! O.O (claim)
• btrfs (kernel) docs:
• Apple filesystems docs (HFS, APFS): awesome unlimited resource forks! (claim)

[pjotrek-b]

Oh, and of course:
How does Samba do/handle it?
Is ZFS over Samba still unlimited?

[pjotrek-b]

ZFS over Samba is not even supporting 64kB per value:

When trying to read large xattrs over Samba, assigned on ZFS directly, I get:
getfattr: bigdata.test: Input/output error
😭

[ross-spencer]

It is an interesting question.

Intrigued (and easily distracted today) I looked at the Linux kernel and FreeBSD this morning.

Linux kernel

The Linux Kernel seems quite clean in naming conventions, and uses XATTR_SIZE_MAX to describe this value: https://github.com/torvalds/linux/blob/5c8c229261f14159b54b9a32f12e5fa89d88b905/include/uapi/linux/limits.h#L15-L17

And in different file system modules you can see how that value is modified as necessary, e.g. in OrangeFS, it seems there are Xattrs, but limited to ~8000 bytes: https://github.com/torvalds/linux/blob/5c8c229261f14159b54b9a32f12e5fa89d88b905/fs/orangefs/protocol.h#L187-L188

NB. I also wondered how many values could be added, and it seems like that's answered here too, another 65536 by default, unless otherwise modified in code.

FreeBSD

FreeBSD seems less clear. It mixes both:

EXTATTR_ and XATTR_ in its naming conventions (and possible DXATTR_ for ZFS).

• e.g. ZFS DXATTR_: https://github.com/freebsd/freebsd-src/blob/e3898d552c9807b523ab3b0a66056520aa2e252a/sys/contrib/openzfs/include/sys/zfs_sa.h#L133-L134
• e.g. EXTATTR (NAMELEN only): https://github.com/freebsd/freebsd-src/blob/e3898d552c9807b523ab3b0a66056520aa2e252a/sys/sys/extattr.h#L60

FreeBSD doesn't seem to have clearer "this is the size" statements -- some clues to size being limitless, but beyond what i can reasonably read in the source.

For Linux it describes the 65536 limit: https://github.com/freebsd/freebsd-src/blob/e3898d552c9807b523ab3b0a66056520aa2e252a/sys/compat/linux/linux_xattr.c#L45-L47

Notes

Interesting comment in linux kernel ext4fs: https://github.com/torvalds/linux/blob/5c8c229261f14159b54b9a32f12e5fa89d88b905/fs/ext4/xattr.h#L72-L81

You asked about get_free_xattr_space(filename) <-- I didn't see anything specific, but I do think this sounds like it would be a good addition to xattr interfaces in future if it doesn't already exist.

NB. while researching, I found this promoting the use of Xattrs for access, might be a good reference in future: https://docs.couchbase.com/sync-gateway/current/access-control-how-use-xattrs-for-access-grants.html#why-use-xattrs

[pjotrek-b]

Awesome! Thanks for the links.
I was surprised to see "FreeBSD" here, but mistook it for "OpenBSD" - where (according to Wikipedia):

Support for extended file attributes was removed from the OpenBSD source code in 2005 due to a lack of interest in Access Control Lists.

[ross-spencer]

No worries!

Also, I meant to say, I got in my head a little bit about the file system itself and fs-specs controlling the limits and I am sure it does happen at that level of-course, but I guess it makes sense that ultimate responsibility for the limits being respected and their safely being applied is at the operating system level? which seems like the easiest lens to look at them?

[pjotrek-b]

Yesterday, someone mentioned pathconf as potential candidate for providing information like that.

[pjotrek-b]

Yesterday, btw i successfully stored a full JPG image (base64 encoded) as xattr on ZFS:
https://github.com/pjotrek-b/mercs/blob/main/helpers/data/Makefile

The code in the Makefile shows a full-roundtrip example with MD5 validation 🌻

btw: If the xattr value is >64kB on ZFS, the error thrown by setfattr is:

setfattr: bigdata.test: Argument list too long

(ext4 and btrfs say "out of diskspace")

[pjotrek-b]

btw the xattr value size that counts when using setfattr is after base64 encoding.

For example:

The file itself has:
cat file_in.bin | wc -c = 46503 Bytes

But to use setfattr I send it through base64 before:
base64 file_in.bin | wc -c = 62820 Bytes (<-- THAT value counts)

[pjotrek-b]

Since the xattrs themselves don't actually (AFAIK) worry about encoding and just treat the values binary-safe, I'm wondering if that overhead is still an issue when using eg Python libs to write binary xattrs directly?
(Haven't tried yet)

[ross-spencer]

There's a little information on MacOS and OSX here: https://stackoverflow.com/a/51518859 the arg list too long error here for MacOS seems to parallel XFS: https://apple.stackexchange.com/a/88760

Ah, that led me to the Darwin Kernel (I think) which I didn't realize was open source (til). Both XATTR max size (and name length) and Apple Doublefork lengths are described here: https://github.com/apple-oss-distributions/xnu/blob/8d741a5de7ff4191bf97d57b9f54c2f6d4a15585/bsd/sys/xattr.h#L72-L77

[pjotrek-b]

Great links!

Here's also more information about Mac and filesystem-metadata:
https://eclecticlight.co/tag/metadata/

Here's an interesting quote from APFS: Extended attributes revisited

For smaller extended attributes up to 3,804 bytes, their data is stored with the xattr in the file system metadata. Larger extended attributes are stored as data streams, with separate records, but still separately from the file data. Apple doesn’t give a limit on the maximum size of xattrs, but they can certainly exceed 200 KB, and each file and folder can have an effectively unlimited number of xattrs.

[pjotrek-b]

Update on BTRFS

Thanks to replies on this thread by developers writing the linux-kernel docs for btrfs, the current insights are:

1. "It is possible to mkfs with a node size of 64k, but not change it on an existing fs." (@Forza-tng)
2. Current default is ~16kB.
3. Current maximum limit is ~64kB: in total for key + value + btrfs structure bytes (see details below)

Details (thx @adam900710) on how much bytes /exactly/ can be used, are:

"There is a macro calculating the max XATTR size inside kernel, BTRFS_MAX_XATTR_SIZE(), when fully expanded it is:
nodesize - sizeof(struct btrfs_header) - sizeof(struct btrfs_item) - sizeof(btrfs_dir_item)
Under default mkfs options, it's 16K - 101 - 25 - 30 , which is 16228 bytes.

But please keep in mind, this limit is shared by both the key and data part.
E.g. with user.testdata as key, you only have 16228 - 13 bytes left for data. (No terminating '\0')"

Formatting with larger nodesize:

$ mkfs.btrfs -n 64K <device>