binaries: rpc-compat/

[hinto-janai]

What

Adds a binary for testing RPC compatibility between monerod and cuprated.

See binaries/rpc-compat/README.md.

Where

cuprate-rpc-compat = { path = "/binaries/rpc-compat" }

[Boog900]

Would this be better as an integration test rather than a separate binary?

[hinto-janai]

The testing space is endpoints.for_each(|e| e.common_input_set()). For practical purposes this testing harness must cover:

• All sets
• Selective sets on demand

In the write code -> test -> fix loop I will be going through for each endpoint, I think I will need something like ./cuprate-rpc-compat --endpoint "$ENDPOINT" to make progress at a reasonable pace.

If the purpose of the integration test is to run all tests in CI it could be done with:

• a separate CI workflow that does something like cargo r --bin cuprate-rpc-compat -- --all-tests
• a #[test] that can be modified at runtime with env vars e.g. ENDPOINTS="$array_of_endpoints" (a bit hacky)

Although running the tests in CI also runs into a few problems:

• it will take a long time (and not be needed at most times, already the case for some of our CI runs)
• some tests need access to a synced monerod and cuprated

I guess the second could be solved with something like ./cuprate-rpc-compat --monerod-ip $x --cuprated-ip $y, although CI passing now relies on those nodes and cargo test will fail without them, thoughts?

[hinto-janai]

Discussed in monero-project/meta#1176 monero-project/meta#1185.

Rough plan for cuprate-rpc-compat (this PR)

what

cuprate-rpc-compat is a binary for testing RPC compatibility between cuprated and monerod.

For a set of defined f(Request) -> Response, it can assert some or all fields of Response are equal in type and value for both cuprated and monerod.

This binary must:

• Run necessary tests in CI
• Cover a wide set of inputs (all) or at least key ones with known change (e.g. certain heights)
• Be configurable on the endpoints/methods for selective testing (e.g. test only get_blocks)
• Be optimized to run as many tests in parallel as possible (or at least finish in a reasonable amount of time)

exceptions

Exceptions to type/value equality assertions:

• Some fields may ignored due to being difficult to replicate in an automated way (e.g. p2p related)
• Non-standard or edge-case behavior may be asserted to be maintained within cuprated, espescially if the behavior is required for wallet2 operation
• Practical compatability may be good enough (in contrast to bit-for-bit equality), for example, as long as JSON fields are proper JSON (e.g. as_json in get_transactions), the formatting inside can be different.

cli api

* = required

--monerod    <ADDR>         * The `monerod` node to use.
--cuprated   <ADDR>         * The `cuprated` node to use.
--test-set   <TEST_SET>     The set of RPC tests to run. Default = Full.
--endpoints  <ENDPOINTS>    Endpoints to run tests on. Default = All.
--methods    <METHODS>      JSON-RPC methods to run tests on. Default = All.

ci

name: cuprate-rpc-compat

on:
  # when there are changes to relevent directories

jobs:
  ci:
    runs-on: ubuntu-latest

    # ... all required pre-steps

    - name: Test
      run: |
        cargo run --release --bin cuprate-rpc-compat -- --test-set ci --monerod $ADDR --cuprated $ADDR2

remaining questions/problems

• we need access to unrestricted RPC for some data (e.g. https://github.com/monero-project/monero/blob/977dedce2cf773e76bf02ed03f2ecdbb1cacf4d5/src/rpc/core_rpc_server.cpp#L3400) that is available in CI (although maybe CI won't run these tests?)
• generic setup/testing across {/json_rpc + serde_json, /* + serde_json, /*.bin + epee}
• is cuprate_rpc_types's deserialization working enough to assert type correctness? are there edge-cases?

internal signatures

Not working code, just the general shape.

//! base data structures + test generation code

struct Config {/*...*/}
enum Endpoint {/*...*/}
enum Method   {/*...*/}

enum TestSet {
    /// Specific subset of tests meant for CI.
    Ci,
    /// Subset of important tests (e.g. key blocks for `get_block`).
    Core,
    /// All tests possible (e.g. all blocks for `get_block`).
    Full,
}

/// TODO: generic handling for {json,epee} such that
/// serde and epee is handled outside of any test.
///
/// `type Request; type Response` could be added to `RpcCall`
struct Test {
    request: Request,
    /// (monerod, cuprated) responses
    test_fn: fn(Response, Response),
}

impl Config {
    fn endpoint_enabled(&self, Endpoint) -> bool;
    fn method_enabled(&self, Method) -> bool;

    /// generates the tests for the given test-set
    fn generate_tests(&self, top_height: u64) -> HashSet<Test> {
        // this is where we define test cases (can be anywhere,
        // although we must define the creation of these somewhere)
        //
        // it can be something like:
        (0..top_height)
            .map(|height| format!(r#"{"jsonrpc":"2.0","method":"get_block_count","params":{"height":{height}}}"#))
            .collect();
        // or a 1-off special case or anything in-between.

        // TODO: maybe define the full set of tests, then define
        // the other sets by selectively `.take()`ing.
        match self.test_set {
            Self::Ci => /* define tests for ci */,
            Self::Core => /* ... */,
            Self::Full => /* define all tests */,
        }
    }
}

//! value assertion test fn examples

/// example test fn for `get_block`
fn get_block(monerod: Response, cuprated: Response) {
    use serde_json::from_str;
    use cuprate_types::json::Block;

    let block_monerod = from_str::<Block>(&monerod.json).unwrap();
    let block_cuprated = from_str::<Block>(&cuprated.json).unwrap();
    assert_eq!(block_monerod, block_cuprated);

    /* ... test other fields ... */
}

/// example test fn for `calc_pow`
fn calc_pow(monerod: Response, cuprated: Response) {
    assert_eq!(monerod.result, cuprated.result);
}

/// example test fn for `get_info`
fn get_info(monerod: Response, cuprated: Response) {
    assert_eq!(monerod.adjusted_time, cuprated.adjusted_time);

    // ignored
    // assert_eq!(monerod.busy_syncing, cuprated.busy_syncing);
    // assert_eq!(monerod.database_size, cuprated.database_size);

    /* ... test other fields ... */
}

//! rpc client

struct Block {/*...*/}

struct RpcClient {
    config: Config,
    /* ... */
}

// -> (monerod, cuprated)
impl RpcClient {
    fn new() -> Self;
    fn top_height(self) -> (u64, u64);
    fn call(&self, Request) -> (Response, Response);

    fn test(&self, test: Test) {
        // TODO: logging:
        {
            struct Log {
                test_name: Option<String>, // if None, just the request or endpoint?
                request: Request,
                endpoint: EndpointOrMethod,
            }
            let log: Log = todo!();
            println!("{log}");
        }

        let (monerod, cuprated) = self.call(test.request);
        test.test_fn(monerod, cuprated);
    }
}

fn main() {
    let config = todo!();

    let mut rpc_client = RpcClient::new()

    if todo!("any test requires blocks") {
        rpc_client.get_blocks();
    }

    let tests = config.generate_tests();

    for test in tests {
        rpc_client.test(test);
    }
}