Issue on page /grain.experimental.html

[AakashKumarNain]

I have been part of the Google OSS ML stack for almost a decade, so I do understand the meaning of experimental very clearly. I do not understand why the documentation for grain is so poor that too in the LLM era where assisted documentation writing is a thing.

There are functionalities that I am interested in using and showcasing to other people, but the lack of documentation means I spend more time looking at the source code to figure out the right way to use an API. For example, look at the documentation of ParquetIterDataset. Does it answer any of the following questions:

1. What are the read_kwargs?
2. What if I want to lazy load the shards?
3. What if I want to lazy load, but also want to ensure that next shard is already read so that I do not create data pipeline bubble during training?
4. How to maximize the performance when you have small/large number of shards?

I have been waiting for things to improve on the documentation for almost 2 years now. There are a few of us who can also help, but even for that we need to know the API well (chicken-egg problem here!)

[Davidnet]

I would like to join, that trying to follow official documents and then you get hit by things like array-record is yanked and we cannot seem to run any of the fast examples :(.

[AakashKumarNain]

Exactly! Worse case is when so many documentation pages are either outdated or give 404. Not sure if someone is seriously looking into it, but if the project is not abandoned, these things should be the top priority

[yk5]

Hi all,

we appreciate your feedback. The documentation over experimental features are indeed getting less attention as we focus on more popular use cases.

The mentioned ParquetIterDataset is a thin wrapper of the pyarrow.parquet.ParquetFile. As it's infeasible to keep our API up-to-date to Parquet API, we chose to pass through all keyword args transparently to ParquetFile as documented there. Would it clarify if we add a link to the ParquetFile mentioned in the docstring (this one perhaps)?

At least 404 links shouldn't happen. We fixed the dangling links, and the fixed links will be available in the latest now (and moved to stable once released).

Thank you.

[AakashKumarNain]

as we focus on more popular use cases.

Sorry to be that person here @yk5 , but the documentation even for stable features feel incomplete to me. Going through the documentation as of now feels discovering "what does this feature do?" instead of answering "why does this feature exist, and what are the ideal use cases for it?" A utility is only as good as the description of the scenario it is suited for