Can support striped directory to increase metadata performance of very large directories?

[hmings888]

In current implementation of beegfs metadata server, all the files under one directory are stored on the same metadata server that their parent directory is .
Image a very large directory containing tens of thousands or even more files, which would be visited frequently, then the load of the metadata server storing this directory would be very very high, evenly response timeout. Could the files under one directory be hashed to multiple shards and then stored across multiple metadata servers? I think this would avoid the performance issue. As I known, Lustre supports this feature.

[iamjoemccormick]

@hmings888,

I agree this would be useful for very large directories. While we currently do not have plans to implement such functionality, I'd like to keep this issue open to gauge community interest and better understand use cases where this might be beneficial.

It’s important to note that sharding or striping directories across multiple metadata servers would introduce its own performance trade-offs. For instance, the current architecture avoids distributed locks for many operations. As an example, moving or renaming a file within the same directory today involves only a single metadata server. With a sharded directory, such operations might require coordination across multiple metadata servers, adding complexity and potential overhead.

This isn’t to say that such a feature wouldn’t improve performance in some scenarios. However, I’m curious how often this feature is actively used in Lustre, compared to simply advising users to avoid creating very large directories. This is a best practice for most file systems I’m aware of, including Lustre.