Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/dev/logging.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ logger.fail("Test failure or similar issue")
You can adjust the log level when running pytest with the `--eest-log-level` option:

```bash
consume engine --input=latest@stable --eest-log-level=VERBOSE -s --sim.limit=".*chainid.*"
consume engine --input=tests@v20.0.0 --eest-log-level=VERBOSE -s --sim.limit=".*chainid.*"
```

The argument accepts both log level names (e.g., "DEBUG", "VERBOSE", "INFO") and numeric values.
Expand Down
104 changes: 46 additions & 58 deletions docs/running_tests/consume/cache.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,50 +3,47 @@
The `consume cache` command can be used to resolve, download and cache fixture releases:

```console
consume cache --input=stable@v4.5.0
consume cache --input=tests@v20.0.0
```

All `consume` subcommands have an `--input` argument, which implements the same functionality as `consume cache` to download and cache fixtures, respectively obtain downloaded fixtures from the cache.

## Example: Two-liner to Download the Latest Fixture Release

Releases can be downloaded using EEST tooling without (manually) cloning and installing the @ethereum/execution-specs tools as following:
Releases can be downloaded without (manually) cloning and installing the @ethereum/execution-specs tools as following:

1. Install `uv` (a fast, rust-based Python package manager):

```console
curl -LsSf https://astral.sh/uv/install.sh | sh
```

2. Run EEST's `consume cache` command via `uv` and request the latest ["stable" fixture release](../releases.md):
2. Run the `consume cache` command via `uv` and request the latest [mainnet `tests` release](../releases.md):

```console
uvx --from "git+https://github.com/ethereum/execution-specs.git#subdirectory=packages/testing" \
consume cache --input=stable@latest
consume cache --input=latest
```

<!-- TODO: Re-capture this example output from the new repo; the command works but the transcript still shows the legacy repo identity. -->
Expected output, as of `v4.5.0`:
<!-- TODO: Re-capture this example output; the transcript below is constructed. -->
Expected output, as of `tests@v20.0.0`:

```console
Built ethereum-execution-testing @ git+https://github.com/ethereum/execution-specs.git@a48e0b381d5225a6c3de2d06cd9ee7ae0b6ca9bb#subdirectory=packages/testing
Installed 70 packages in 15ms

Path: /home/dtopz/.cache/ethereum-execution-spec-tests/cached_downloads/ethereum/execution-spec-tests/v5.4.0/fixtures_stable/fixtures
Input: https://github.com/ethereum/execution-spec-tests/releases/download/v5.4.0/fixtures_stable.tar.gz
Release page: https://github.com/ethereum/execution-spec-tests/releases/tag/v5.4.0
Path: /home/dtopz/.cache/ethereum-execution-spec-tests/cached_downloads/ethereum/execution-specs/tests%40v20.0.0/fixtures/fixtures
Input: https://github.com/ethereum/execution-specs/releases/download/tests%40v20.0.0/fixtures.tar.gz
Release page: https://github.com/ethereum/execution-specs/releases/tag/tests%40v20.0.0
```

**Note:** Use direct URLs to avoid GitHub API calls (better for CI environments). Version specifiers like `stable@latest` will always use the GitHub API to resolve versions. More details on the arguments to `--input` are provided below.
**Note:** Use direct URLs to avoid GitHub API calls (better for CI environments). Version specifiers like `tests@latest` will always use the GitHub API to resolve versions. More details on the arguments to `--input` are provided below.

**Explanation:** `uv` creates a local Python virtual environment in `~/.cache/uv/`, installs EEST and executes the `consume cache` command to resolve and download the release, which gets cached at `~/.cache/ethereum-execution-spec-tests`. Subsequent commands will use the cached version of the fixtures.
**Explanation:** `uv` creates a local Python virtual environment in `~/.cache/uv/`, installs the testing package and executes the `consume cache` command to resolve and download the release, which gets cached at `~/.cache/ethereum-execution-spec-tests`. Subsequent commands will use the cached version of the fixtures.

## The `--input` Flag to Specify Fixtures

All `consume` sub-commands take an `--input=<fixture_path>|<release_spec>|<url>` flag to specify which fixtures should be used for the command, `<fixtures>` may be:

1. **A local directory**: Fixtures from your local file system.
2. **A release specification**: An EEST release tag or "release specification" `stable@latest`, `fusaka-devnet-1@v1.0.0`, etc.
2. **A release specification**: A fixture release tag or "release specification" `tests@latest`, `bal-devnet@v7.0.0`, etc.
3. **A URLs**: A full URL to a custom hosted release or a Github release.

### Release Specifications
Expand All @@ -55,40 +52,41 @@ A release specification has the format `<release_name>@<version>`.

**Supported release names:**

- `stable`: Latest stable fork release.
- `develop`: Latest development fork release.
- Custom release names: e.g., `pectra-devnet-4`, `eip7692`.
- `tests`: The mainnet release, all tests for all forks up to and including the latest mainnet fork. A bare `latest` or `vX.Y.Z` input is shorthand for `tests@latest`, respectively `tests@vX.Y.Z`.
- `<feat>-devnet`: Devnet releases, e.g. `bal-devnet`, `glamsterdam-devnet`.
- Other features: e.g. `benchmark`, `zkevm`.

Any release name is also accepted with its `tests-` git tag prefix, e.g. `tests-bal@v7.3.2`.

**Supported version formats:**

- `latest`: Most recent release for the specified name.
- `latest`: Highest version for the specified name (publish time only breaks ties).
- `v1.2.3`: Specific semantic version.

### Examples

Examples using a release specification:

```bash
# Latest standard, full stable release (all forks up to and including the latest deployed mainnet fork)
uv run consume engine --input stable@latest

# Latest standard, full development release (all forks up to and including the latest development fork)
uv run consume rlp --input develop@latest

# Standard, full releases by tag
uv run consume engine --input stable@v4.1.0
uv run consume rlp --input develop@v4.2.1

# Pre-release tags
uv run consume cache --input pectra-devnet-6@v1.0.0
uv run consume direct --input eip7692@latest --bin ../go-ethereum/build/bin/evm
# Latest mainnet (tests) release
uv run consume engine --input latest
uv run consume rlp --input tests@latest

# Mainnet release by version
uv run consume engine --input v20.0.0
uv run consume rlp --input tests@v20.0.0

# Feature releases, with or without the tests- tag prefix
uv run consume cache --input bal-devnet@v7.0.0
uv run consume cache --input glamsterdam-devnet@latest
uv run consume direct --input tests-bal@v7.3.2 --bin ../go-ethereum/build/bin/evm
```

Examples using a URL, the target must be a `.tar.gz`:

```bash
# GitHub release URL
uv run consume engine --input https://github.com/ethereum/execution-spec-tests/releases/download/v4.1.0/fixtures_develop.tar.gz
uv run consume engine --input https://github.com/ethereum/execution-specs/releases/download/tests%40v20.0.0/fixtures.tar.gz

# Direct archive URL
uv run consume rlp --input https://example.com/custom-fixtures.tar.gz
Expand All @@ -109,13 +107,13 @@ All remote fixture sources are automatically cached to avoid repeated downloads:
You can override this location with the `--cache-folder` flag:

```bash
uv run consume cache --input stable@latest --cache-folder /path/to/custom/cache
uv run consume cache --input latest --cache-folder /path/to/custom/cache
```

Or extract directly to a specific directory (bypasses cache structure):

```bash
uv run consume cache --input fusaka-devnet-2@v1.1.0 --extract-to ./benchmark-fixtures
uv run consume cache --input bal-devnet@v7.0.0 --extract-to ./devnet-fixtures
```

**Cache structure:**
Expand All @@ -124,25 +122,15 @@ uv run consume cache --input fusaka-devnet-2@v1.1.0 --extract-to ./benchmark-fix
❯ tree ~/.cache/ethereum-execution-spec-tests/ -L 5
/home/dtopz/.cache/ethereum-execution-spec-tests/
├── cached_downloads
│   ├── ethereum
│   │   └── execution-spec-tests
│   │   ├── pectra-devnet-5%40v1.0.0
│   │   │   └── fixtures_pectra-devnet-5
│   │   ├── pectra-devnet-6%40v1.0.0
│   │   │   └── fixtures_pectra-devnet-6
│   │   ├── v4.0.0
│   │   │   └── fixtures_develop
│   │   ├── v4.1.0
│   │   │   └── fixtures_develop
│   │   ├── v4.2.0
│   │   │   ├── fixtures_develop
│   │   │   ├── fixtures_eip7692
│   │   │   └── fixtures_stable
│   │   ├── v4.3.0
│   │   │   └── fixtures_develop
│   │   └── v4.5.0
│   │   └── fixtures_stable
│   └── other
│ ├── ethereum
│ │ └── execution-specs
│ │ ├── tests%40v20.0.0
│ │ │ └── fixtures
│ │ ├── tests-bal%40v7.3.2
│ │ │ └── fixtures_bal
│ │ └── tests-glamsterdam-devnet%40v6.1.0
│ │ └── fixtures_glamsterdam-devnet
│ └── other
└── release_information.json
```

Expand All @@ -155,7 +143,7 @@ The [`fill` command](../../filling_tests/index.md) generates a JSON file `<fixtu
When using direct GitHub release URLs (instead of version specifiers), the consume command automatically avoids unnecessary GitHub API calls to prevent rate limiting in CI environments:

```console
consume cache --input=https://github.com/ethereum/execution-spec-tests/releases/download/v4.5.0/fixtures_stable.tar.gz
consume cache --input=https://github.com/ethereum/execution-specs/releases/download/tests%40v20.0.0/fixtures.tar.gz
```

**API Call Behavior:**
Expand All @@ -167,8 +155,8 @@ Examples:

```console
# No API calls - direct download
consume cache --input=https://github.com/ethereum/execution-spec-tests/releases/download/v4.5.0/fixtures_stable.tar.gz
consume cache --input=https://github.com/ethereum/execution-specs/releases/download/tests%40v20.0.0/fixtures.tar.gz

# API calls required - version resolution
consume cache --input=stable@latest
# API calls required - version resolution
consume cache --input=latest
```
14 changes: 7 additions & 7 deletions docs/running_tests/hive/common_options.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,12 @@ All execution-specs (EELS) Hive simulators share common command-line options and

While they may be omitted, it's recommended to specify the `fixtures` and `branch` simulator build arguments when running execution-specs simulators.

For example, this runs "stable" fixtures from the v4.3.0 [latest stable release](../releases.md#standard-releases) and builds the simulator at the v4.3.0 tag:
For example, this runs fixtures from the [`tests@v20.0.0` release](../releases.md#test-release-types) and builds the simulator at a pinned ref:

```bash
./hive --sim ethereum/eels/consume-engine \
--sim.buildarg fixtures=stable@v4.3.0 \
--sim.buildarg branch=v4.3.0 \
--sim.buildarg fixtures=tests@v20.0.0 \
--sim.buildarg branch=forks/amsterdam \
--client go-ethereum
```

Expand All @@ -29,8 +29,8 @@ The `collectonly:` prefix can be used to inspect which tests would match an expr

```bash
./hive --sim ethereum/eels/consume-engine \
--sim.buildarg fixtures=stable@v4.3.0 \
--sim.buildarg branch=v4.3.0 \
--sim.buildarg fixtures=tests@v20.0.0 \
--sim.buildarg branch=forks/amsterdam \
--docker.output \
--sim.limit="collectonly:.*eip4844.*"
```
Expand All @@ -41,8 +41,8 @@ The `id:` prefix can be used to select a single test via its ID (this will autom

```console
./hive --sim ethereum/eels/consume-engine \
--sim.buildarg fixtures=stable@v4.3.0 \
--sim.buildarg branch=v4.3.0 \
--sim.buildarg fixtures=tests@v20.0.0 \
--sim.buildarg branch=forks/amsterdam \
--docker.output \
--sim.limit "id:tests/cancun/eip4844_blobs/test_blob_txs.py::test_sufficient_balance_blob_tx"
```
Expand Down
2 changes: 1 addition & 1 deletion docs/running_tests/hive/dev_mode.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ This avoids running the simulator in a dockerized environment and has several ad

```bash
uv run consume engine --input ./fixtures -k "test_chainid"
uv run consume rlp --input stable@latest
uv run consume rlp --input latest
```

## Hive Dev Setup on macOS
Expand Down
1 change: 1 addition & 0 deletions docs/running_tests/releases.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,7 @@ The [`consume cache`](./consume/cache.md) command resolves EELS release and pre-
to release URLs and downloads them. For example:

```bash
uv run consume cache --input=latest # shorthand for tests@latest
uv run consume cache --input=tests@latest
uv run consume cache --input=bal-devnet@v7.0.0
```
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -260,7 +260,7 @@ def from_release_spec(
extract_to: Optional[Path] = None,
) -> "FixturesSource":
"""
Create a fixture source from a release spec (e.g., develop@latest).
Create a fixture source from a release spec (e.g., tests@latest).
"""
if cache_folder is None:
cache_folder = CACHED_DOWNLOADS_DIRECTORY
Expand Down Expand Up @@ -376,8 +376,10 @@ def pytest_addoption(parser: pytest.Parser) -> None: # noqa: D103
"Specify the JSON test fixtures source. Can be a local "
"directory, a URL pointing to a fixtures.tar.gz archive, a "
"release name and version in the form of `NAME@v1.2.3` "
"(`stable` and `develop` are valid release names, and `latest` "
"is a valid version), or the special keyword 'stdin'. "
"(e.g. `tests@v20.0.0` or `bal-devnet@v7.0.0`, with or "
"without the `tests-` tag prefix, and `latest` is a valid "
"version), a bare `latest` or `vX.Y.Z` which resolves the "
"mainnet `tests` release, or the special keyword 'stdin'. "
f"Defaults to the following local directory: '{default_input()}'."
),
)
Expand Down
Loading
Loading