Expose the File's `from_std` API

[OpenTritium]

我注意到 File 的 from_std 方法竟然不是公开的，如果使用 from_handle 和 from_fd 又需要自己处理跨平台，作者这样设计是有什么考虑吗？

[Berrysoft]

Windows 的 socket 默认是 overlapped，但是 file 不是。如果来自 std 的 File 在创建的时候没有加上额外的 flag，是没办法异步的。因此你基本上总是需要有一些跨平台处理。

[Hellomik2002]

Do you have any ready solution how take file make clone and wrap it in new CompioFile ?

[Berrysoft]

The type File of compio implements Clone. Do you mean that?

[Hellomik2002]

Ohh you have already. I try use file to read and write between few threads,I try clone it insead of opening File in new thread again. Can you documenation or example how correclty clone file for using in another thread ? or I need open it again ?

[Berrysoft]

It's straightforward to clone the File and send it to other threads. If rustc complains about Send or Sync, you need to enable the feature "sync". After it is sent to another thread, make sure to call any async fns inside a compio runtime. Then it should work.

There are some other corner cases on Windows: https://compio.rs/docs/compio/driver/iocp . TL;DR: the File is attached to the runtime that creates it. It's OK to send it to other threads, but the operation is still scheduled on the original runtime. (If the original runtime is dropped, it will be an undocumented behavior of Windows.) Unless you enable the feature "iocp-global" of compio-driver manually.

[Hellomik2002]

Can you give any example of code with thread::spawn please

[Hellomik2002]

i tried this one It works in in macos but not in windows

use compio::io::AsyncReadAt;
use compio::runtime::Runtime;
use compio_fs::OpenOptions;

use std::io::Result;

use std::thread;

#[compio::main]
async fn main() -> Result<()> {
    tracing_subscriber::fmt()
        .with_max_level(tracing::Level::INFO)
        .init();

    // 1. ОТКРЫТИЕ РОДИТЕЛЯ (ТОЛЬКО ЧТЕНИЕ!)
    let parent_path: &str = r"hello.txt";
    let file = OpenOptions::new()
        .read(true)
        .write(true)
        .open(parent_path)
        .await?;
    let mut buf = Vec::<u8>::with_capacity(8);
    unsafe {
        buf.set_len(8);
    }

    let (_, buf) = file.read_at(buf, 0).await.unwrap();
    println!("Parent: Data: {:?}", buf);

    let file1 = file.clone();
    let handle = thread::spawn(move || {
        let rt = Runtime::new().unwrap();

        rt.block_on(async move {
            println!("Worker:  ID: {:?}", thread::current().id());

            let mut buf = Vec::<u8>::with_capacity(8);
            unsafe {
                buf.set_len(8);
            }
            let (_, buf) = file1.read_at(buf, 0).await.unwrap();

            println!("Worker: Data: {:?}", buf);
        });
    });

    handle.join().unwrap();
    Ok(())
}

[Berrysoft]

Yes, it is expected behavior. handle.join() occupies the main thread and the runtime cannot poll the IOCP. That's why the new thread doesn't receive the completed event. A small fix:

• Replace thread::spawn with compio::runtime::spawn_blocking, and
• Replace handle.join().unwrap() with handle.await.unwrap().

You're free to use any other thread pools, but should wait for the thread in an asynchronous way. If there's no async join, consider using an oneshot async channel.

[Berrysoft]

Another simple (but less efficient) solution is to enable "iocp-global" directly...

[George-Miao]

@Hellomik2002 And also I want to point out your usage of set_len is not correct. All bytes from index 0 to n should be initialized when calling set_len(n). It's not really necessary here since read_at will read into all available space no matter if they're marked as initialized or not.

[Hellomik2002]

I am currently working on a VHDX implementation, specifically dealing with differencing disks (where one file contains changes and references a parent file).

I had to create a custom Struct to handle slicing with recursion. Standard slicing resulted in deeply nested types like Slice<Slice<...>>, which became unmanageable. My custom implementation avoids this nesting types while allowing recursive reads across the disk chain. That's why I can have few threads that can read and write in file.
I am newbie in rust, sorry.
If you have any advise u are welcome
I just tried to create vector without fiiling with 0(it takes time maybe?), that's why used set_len(). I don't care at beggining what is in vector cause it will override

#[derive(Debug, Clone, Copy)]
pub struct UnsafeSlice {
    pub ptr: *mut u8,
    pub len: usize,      
    pub capacity: usize, 
}
...
pub async fn read_at_raw(
        &self,
        slice: UnsafeSlice,
        offset: u64,
        len: usize,
    ) -> {...

@Hellomik2002 And also I want to point out your usage of set_len is not correct. All bytes from index 0 to n should be initialized when calling set_len(n). It's not really necessary here since read_at will read into all available space no matter if they're marked as initialized or not.

[Berrysoft]

Well, it's OK to call set_len, but you called it at the wrong place. It has been called by read_at. For example,

let mut buf = Vec::<u8>::with_capacity(8);

let (n, buf) = file.read_at(buf, 0).await.unwrap();

assert_eq!(n, buf.len());

I don't understand what did you mean about "unmanageable". The Slice type impls IntoInner, so you can get the inner buffer calling into_inner.

[Berrysoft]

I would like to mention that read and read_at already set the lengths of the buffer so actually you don't need to call set_len at all. I have modified the above example.