Arrow-csv does not update null flag in `Field` on inference

[realonbebeto]

Describe the bug

The Format infer_schema does not update the null flag (by default it's true).

To Reproduce

Run the following on any csv file and the Fields will be always null even when they are not supposed to.

use std::fs::File;
use arrow::csv::Format;

 let mut file = File::open(path)?;
 let (schema, _) = Format::default()
            .with_header(source.metadata.has_header)
            .infer_schema(&mut file, Some(source.metadata.num_rows))?;

The line of code in infer_schema defaulting to true

        // build schema from inference results
        let fields: Fields = column_types
            .iter()
            .zip(&headers)
            .map(|(inferred, field_name)| Field::new(field_name, inferred.get(), true))
            .collect();

Expected behaviour

On running on a csv, the null flag should be updated accordingly if the field inferred is actually null or not. Besides, there is a disconnect to use the null regex and the null flag always being true.

Additional context

[alamb]

What should happen if the CSV reader subsequently encounters a row with a null in a field it determined to be non nullable?

If we inferred Non nullable i think it would error

So by inferring Nullable the CSV writer is being tolerant of more inputs and will error on fewer of them

We could add some flag to change this behavior if it was warranted

Another alternative would be for your application to review the data that was parsed and if there are no nulls ajust the schema accordingly

[realonbebeto]

Well, it is an intentional design to tolerate nulls downstream, and by default,t it's a good choice.

My alternative approach and suggestion would be to make strict inference an option for catching nulls early on, and to use relaxed inference as the default.

[alamb]

What do you think about the idea for the application to check for nulls rather than the CSV reader?

[realonbebeto]

The idea is good for the application to handle nulls.

As an enhancement, what do you think of having the CSV reader have an option for strict inference, where the values read determine the Schema in its entirety, including the null flag?