Spawn not executing when creating multiple runtimes

[manishrjain]

Following from issue #335 , I created multiple runtimes encapsulated in a singleton struct, so I can spawn the right functions on the right runtime. However, I quickly encountered that these functions weren't getting executed, because the runtime is scheduling on the main thread as well, which isn't available. The fix was to skip the main thread. See this commit in my fork of zio:

manishrjain@2c945ad

[lalinsky]

@manishrjain how do you run the multiple runtimes?

[lalinsky]

The cleanest option would be to spawn a new thread, create the additional runtime, spawn your task on it and then join it.

I've removed Runtime.run, thinking it would no longer be needed, but maybe I could add it back. That would allow you to spawn/join the task from another runtime.

[manishrjain]

With this code. Pool.init gets called from main:

    // Runtime pool singleton — created first so all runtimes are available.
    const pool = try runtime.Pool.init(allocator, .{});
    defer pool.deinit();

I do have to call this as well, to run appMain.

    var rt = try zio.Runtime.init(
        allocator,
        .{ .executors = .exact(1) },
    );
    defer rt.deinit();

    appMain(&raft_rt, &zdb_srv, node, &args);

Once inside appMain, I call things like this:

    var group: zio.Group = .init;
    defer group.cancel();
    group.spawn(runServer, .{zdb_srv}) catch return;

/// Spawn the client TCP accept loop on the query runtime pool.
/// Must be called from a zio coroutine. Blocks until cancelled.
fn runServer(s: *Server) void {
    var handle = runtime.pool().get(.query).spawn(
        Server.runClientLoop,
        .{s},
    ) catch |err| {
        log.err("runServer err: {}", .{err});
        return;
    };
    handle.join();
}

runtime.Pool code:

const std = @import("std");
const zio = @import("zio");

pub const Runtime = enum(u2) {
    raft = 0,
    query = 1,
    background = 2,
};

pub const Config = struct {
    /// Number of executor threads per runtime.
    /// 0 = auto-detect based on CPU count.
    raft_threads: u6 = 2,
    query_threads: u6 = 0,
    background_threads: u6 = 2,
};

var g_pool: ?*Pool = null;

/// Get the global Pool singleton. Panics if not initialized.
pub fn pool() *Pool {
    return g_pool orelse @panic("runtime.Pool not initialized");
}

pub const Pool = struct {
    runtimes: [3]*zio.Runtime,
    allocator: std.mem.Allocator,

    pub fn init(allocator: std.mem.Allocator, cfg: Config) !*Pool {
        if (g_pool != null) @panic("runtime.Pool already initialized");

        var runtimes: [3]*zio.Runtime = undefined;
        const counts = [3]u6{ cfg.raft_threads, cfg.query_threads, cfg.background_threads };
        var initialized: usize = 0;
        errdefer for (runtimes[0..initialized]) |rt| rt.deinit();

        for (counts, 0..) |n, i| {
            runtimes[i] = try zio.Runtime.init(allocator, .{
                .executors = @enumFromInt(n),
            });
            initialized += 1;
        }

        const p = try allocator.create(Pool);
        p.* = .{ .runtimes = runtimes, .allocator = allocator };
        g_pool = p;
        return p;
    }

    pub fn deinit(self: *Pool) void {
        var i: usize = self.runtimes.len;
        while (i > 0) {
            i -= 1;
            self.runtimes[i].deinit();
        }
        if (g_pool == self) g_pool = null;
        self.allocator.destroy(self);
    }

    pub fn get(self: *Pool, which: Runtime) *zio.Runtime {
        return self.runtimes[@intFromEnum(which)];
    }
};

[lalinsky]

This is not doing what you expect. You can only instanciate one Runtime per thread. For multiple runtimes, you would need to spawn threads yourself.

I could probably add a special option for Runtime.init to NOT setup the main task, only spawn additional executors, then it would be doable.

[manishrjain]

Yes, that'd be nice -- to not set up the main task and spawn as many new executors as provided. It's very likely that one of the runtime coroutines wants to hand off a task to the other runtime coroutine. That's possible with this singleton by just doing a Pool.get.spawn.