Skip to content

Commit 8b2a0c5

Browse files
authored
Remake SCALE (#33)
* epic: total remake Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * variant Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * fixed integer Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * classic compact integer Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * jam compact integer Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * enum Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * sptr upts refwrap Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * fix: custom decomposable Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * draft Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * docs and polishing Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * docs and polishing Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * fix: boost variant test Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * tagged types Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * fixes Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * fixes Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * update: qtils Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * update: ci os Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * fix: lvalue/rvalue ambiguous Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * hotfix Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * hotfix Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * feature: decode into lvalue and rvalue fix: enum macros fix: decomposition over rvalue Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * feature: macro for using base-class properties in custom decomposition Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * feature: support of decoding into immutable collection Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * update: doc&test Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * refactor: decomposable Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * refactor: remove of using ScaleEncode/ScaleDecode concepts Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * clean: remove commented code Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * fix: clang warn for immutable collection Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * feature: scale::impl::memory::encoded_size fix: scale::impl::memory::encoded Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * fix: always forward encode's value by cons-lvalue-ref Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * refactor: encode for EncodeOpaqueValue Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * clean: remove commented code feature: expose ForCount implementation Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * fix: typo Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * fix: installation of scale_append Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * fix: dtor for encoder/decoder Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * refactor: assert for abnormal aggregate Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * refactor: jump-table Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * refactor: Decoder::read without copying Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * remake: BitVector/SmallBitVector Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * fix: review issues Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * feature: single-byte implementation of optional bool Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * update: qtils Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * refactor: use externally provided source and receiver of bytes Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * refactor: optimisation of read/write continuous sequences of bytes Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * fix: case of custom config Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * refactor: unification Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * fix: scale::decode Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * fix: review issue Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> * update: README Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com> --------- Signed-off-by: Dmitriy Khaustov aka xDimon <khaustov.dm@gmail.com>
1 parent db00369 commit 8b2a0c5

74 files changed

Lines changed: 6719 additions & 3063 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/workflows/ci.yml

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -18,10 +18,10 @@ env:
1818
1919
jobs:
2020
MacOS:
21-
runs-on: macos-latest
21+
runs-on: macos-15
2222
steps:
23-
- uses: actions/checkout@v2
24-
- uses: actions/cache@v2
23+
- uses: actions/checkout@v4
24+
- uses: actions/cache@v4
2525
with:
2626
path: ${{ env.CACHE_PATHS }}
2727
key: ${{ github.job }}-${{ env.CACHE_VERSION }}
@@ -38,10 +38,10 @@ jobs:
3838
- name: "Linux: clang"
3939
run: ./scripts/build.sh -DCMAKE_CXX_COMPILER=clang++
4040
name: "${{ matrix.options.name }}"
41-
runs-on: ubuntu-latest
41+
runs-on: ubuntu-24.04
4242
steps:
43-
- uses: actions/checkout@v2
44-
- uses: actions/cache@v2
43+
- uses: actions/checkout@v4
44+
- uses: actions/cache@v4
4545
with:
4646
path: ${{ env.CACHE_PATHS }}
4747
key: ${{ github.job }}-${{ matrix.options.name }}-${{ env.CACHE_VERSION }}
@@ -50,13 +50,13 @@ jobs:
5050

5151
clang-tidy:
5252
name: "Linux: clang-tidy"
53-
runs-on: ubuntu-latest
53+
runs-on: ubuntu-24.04
5454
container: qdrvm/kagome-dev@sha256:2d70246c32418a3dd45c246d3f5c2dd99bdafde145b903271849affe476c4cfc
5555
steps:
56-
- uses: actions/checkout@v2
56+
- uses: actions/checkout@v4
5757
with:
5858
fetch-depth: 0
59-
- uses: actions/cache@v2
59+
- uses: actions/cache@v4
6060
with:
6161
path: ${{ env.CACHE_PATHS }}
6262
key: ${{ github.job }}-${{ env.CACHE_VERSION }}

CMakeLists.txt

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
cmake_minimum_required(VERSION 3.12)
88

99
option(JAM_COMPATIBLE "Build compatible with JAM-codec" OFF)
10-
option(CUSTOM_CONFIG_SUPPORT "Support custom config of streams" OFF)
10+
option(CUSTOM_CONFIG_SUPPORT "Support custom config of coder" OFF)
1111
set(MAX_AGGREGATE_FIELDS 20 CACHE STRING "Max number of aggregates fields (1..1000); for generation")
1212

1313
option(BUILD_TESTS "Whether to include the test suite in build" OFF)
@@ -42,7 +42,7 @@ if(BUILD_TESTS)
4242
endif()
4343
endif()
4444

45-
project(Scale LANGUAGES CXX VERSION 1.1.0)
45+
project(Scale LANGUAGES CXX VERSION 2.0.0)
4646

4747
set(CMAKE_CXX_STANDARD 20)
4848
set(CMAKE_CXX_STANDARD_REQUIRED ON)
@@ -88,6 +88,15 @@ install(TARGETS scale EXPORT scaleConfig
8888
FRAMEWORK DESTINATION ${CMAKE_INSTALL_PREFIX}
8989
)
9090

91+
install(TARGETS scale_append EXPORT scaleConfig
92+
LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
93+
ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR}
94+
RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR}
95+
INCLUDES DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}
96+
PUBLIC_HEADER DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}
97+
FRAMEWORK DESTINATION ${CMAKE_INSTALL_PREFIX}
98+
)
99+
91100
install(
92101
DIRECTORY ${CMAKE_SOURCE_DIR}/include/scale
93102
DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}

README.md

Lines changed: 121 additions & 78 deletions
Original file line numberDiff line numberDiff line change
@@ -1,24 +1,51 @@
11
# SCALE codec C++ implementation
2-
fully meets polkadot specification.\
2+
**SCALE (Simple Concatenated Aggregate Little-Endian)** is a lightweight serialization format commonly used in blockchain applications.
3+
More details in [spec](https://docs.polkadot.com/polkadot-protocol/basics/data-encoding/#scale-codec) from Polkadot.
4+
35
It allows encoding and decoding following data types:
46
* Built-in integer types specified by size:
5-
* ```uint8_t```, ```int8_t```
6-
* ```uint16_t```, ```int16_t```
7-
* ```uint32_t```, ```int32_t```
8-
* ```uint64_t```, ```int64_t```
9-
* bool values
10-
* pairs of types represented by ```std::pair<T1, T2>```
11-
* compact integers represented by CompactInteger type
12-
* optional values represented by ```std::optional<T>```
13-
* as special case of optional values ```std::optional<bool>``` is encoded using one byte following specification.
14-
* collections of items represented by ```std::vector<T>```
15-
* variants represented by ```boost::variant<T...>```
16-
17-
## ScaleEncoderStream
18-
class ScaleEncoderStream is in charge of encoding data
7+
* ```uint8_t```, ```int8_t```
8+
* ```uint16_t```, ```int16_t```
9+
* ```uint32_t```, ```int32_t```
10+
* ```uint64_t```, ```int64_t```
11+
* Multiprecision integer from boost:
12+
* ```uint128_t```, ```int128_t```
13+
* ```uint256_t```, ```int256_t```
14+
* ```uint512_t```, ```int512_t```
15+
* ```uint1024_t```, ```int1024_t```
16+
* boolean values
17+
* pairs, tuples and other structurally bindable types (limited by N members)
18+
* aggregates limited by N field (except array, that coded as collection)
19+
* compact integers represented by CompactInteger type (classic and JAM-compatible). **Unsigned only!**
20+
* optional values represented by ```std::optional<T>``` and ```boost::optional<T>```
21+
* as special case of optional values ```*::optional<bool>``` is encoded using one byte following specification.
22+
* various collections of items
23+
* if item codable
24+
* encodes item by item in order of forward iterator
25+
* decodes items by inserting in order as encoded
26+
* variants represented by ```std::variant<T...>``` and ```boost::variant<T...>```
27+
28+
## encode(value, encoder)
29+
It is function is in charge of encoding of value and store to backend set in encoder
30+
31+
## decode(value, encoder)
32+
It is function is in charge of read and decode data from backend set in encoder and initialize provided variable
33+
34+
## Encoder
35+
class Encoder is in charge of encoding data
36+
It receives values over `encode()` function and store encoded data into EncoderBackend
37+
Additionally it receive values over `<<` operator.
1938

39+
## Decoder
40+
class Decoder is in charge of decoding data
41+
It initializes provided value over `decode()` function by decoded value from encoded source data
42+
Additionally it initialize provided values over `>>` operator.
43+
44+
## Example
2045
```c++
21-
ScaleEncoderStream s;
46+
std::vector<uint8_t> out;
47+
ToBytes encoder(out); // Encoder which used backend 'to bytes'
48+
2249
uint32_t ui32 = 123u;
2350
uint8_t ui8 = 234u;
2451
std::string str = "asdasdasd";
@@ -32,9 +59,16 @@ std::pair<uint8_t, uint32_t> pair{1u, 2u};
3259
std::vector<uint32_t> coll_ui32 = {1u, 2u, 3u, 4u};
3360
std::vector<std::string> coll_str = {"asd", "fgh", "jkl"};
3461
std::vector<std::vector<int32_t>> coll_coll_i32 = {{1, 2, 3}, {4, 5, 6, 7}};
62+
3563
try {
36-
s << ui32 << ui8 << str << raw_str << b << ci << vint;
37-
s << opt_str << opt_bool << pair << coll_ui32 << coll_str << coll_coll_i32;
64+
// functional style
65+
encode(ui32, encoder);
66+
encode(ui8, encoder);
67+
encode(str, encoder);
68+
// combine for one call
69+
encode(std::tie(raw_str, b, ci, vint), encoder);
70+
// stream-style
71+
encoder << opt_str << opt_bool << pair << coll_ui32 << coll_str << coll_coll_i32;
3872
} catch (std::runtime_error &e) {
3973
// handle error
4074
// for example make and return outcome::result
@@ -43,15 +77,12 @@ try {
4377
```
4478
You can now get encoded data:
4579
```c++
46-
ByteArray data = s.data();
80+
auto in = out;;
4781
```
48-
49-
## ScaleDecoderStream
50-
class ScaleEncoderStream is in charge of encoding data
51-
82+
Now you can decode that data back:
5283
```c++
53-
ByteArray bytes = {...};
54-
ScaleEncoderStream s(bytes);
84+
FromBytes decoder(in); // Decoder which used backend 'from bytes'
85+
5586
uint32_t ui32 = 0u;
5687
uint8_t ui8 = 0u;
5788
std::string str;
@@ -65,90 +96,102 @@ std::vector<uint32_t> coll_ui32;
6596
std::vector<std::string> coll_str;
6697
std::vector<std::vector<int32_t>> coll_coll_i32;
6798
try {
68-
s >> ui32 >> ui8 >> str >> b >> ci >> vint;
69-
s >> opt_str >> opt_bool >> pair >> coll_ui32 >> coll_str >> coll_coll_i32;
99+
// functional style
100+
decode(ui32, decoder);
101+
decode(ui8, decoder);
102+
decode(str, decoder);
103+
// combine for one call
104+
decode(std::tie(raw_str, b, ci, vint), decoder);
105+
// stream-style
106+
decoder >> opt_str >> opt_bool >> pair >> coll_ui32 >> coll_str >> coll_coll_i32;
70107
} catch (std::system_error &e) {
71108
// handle error
72109
}
73110
```
111+
Now we have variables initialized by decoded values
74112
75113
## Custom types
76-
You may need to encode or decode custom data types, you have to define custom << and >> operators.
114+
You may need to encode or decode custom data types, you have to define custom `encode()` and `decode()` function.
77115
Please note, that your custom data types must be default-constructible.
78116
```c++
79117
struct MyType {
80118
int a = 0;
81119
std::string b;
120+
121+
friend void encode(const MyType &v, Encoder &encoder) {
122+
encoder << a;
123+
encode(b, encoder);
124+
}
125+
friend void decode(MyType &v, Decoder &decoder) {
126+
decoder >> a;
127+
decode(b, decoder);
128+
}
82129
};
83-
84-
ScaleEncoderStream &operator<<(ScaleEncoderStream &s, const MyType &v) {
85-
return s << v.a << v.b;
86-
}
87-
88-
ScaleDecoderStream &operator>>(ScaleDecoderStream &s, MyType &v) {
89-
return s >> v.a >> v.b;
90-
}
91130
```
92131
Now you can use them in collections, optionals and variants
93132
```c++
94-
std::vector<MyType> v = {{1, "asd"}, {2, "qwe"}};
95-
ScaleEncoderStream s;
133+
std::vector<uint8_t> out;
134+
EncoderToVector encoder(out)
135+
std::vector<MyType> src_vec = {{1, "asd"}, {2, "qwe"}};
96136
try {
97-
s << v;
137+
encoder << src_vec;
98138
} catch (...) {
99139
// handle error
100140
}
101-
```
102-
The same for ```ScaleDecoderStream```
103-
```c++
104-
ByteArray data = {...};
105-
std::vector<MyType> v;
106-
ScaleDecoderStream s{data};
141+
142+
std::vector<uint8_t> in = {...};
143+
DecoderFromSpan decoder(in);
144+
std::vector<MyType> dst_vec;
107145
try {
108-
s >> v;
146+
decode(dst, decoder);
109147
} catch (...) {
110148
// handle error
111149
}
112150
```
113151
114152
## Convenience functions
115-
Convenience functions
153+
Library provides ready well done function to encode/decode in one line. You should just use import it in you namespace:
154+
116155
```c++
117-
template <typename T>
118-
outcome::result<std::vector<uint8_t>> encode(T &&t);
156+
// template <typename T>
157+
// outcome::result<std::vector<uint8_t>> encode(T &&value);
158+
using ::scale::impl::bytes::encode;
119159
120-
template <typename T>
121-
outcome::result<T> decode(const RangeOfBytes auto& span)
160+
SomeClass object = {...};
161+
auto res = encode(object); // <- Just one-line call
162+
if (res.has_value()) {
163+
std::vector<uint8_t> encoded = std::move(res.value()); // Bytes of encoded data
164+
}
122165
123-
template <typename T>
124-
outcome::result<T> decode(ScaleDecoderStream &s)
125-
```
126-
are wrappers over ```<<``` and ```>>``` operators described above.
166+
// template <typename T>
167+
// outcome::result<T> decode(const RangeOfBytes auto& span);
168+
using ::scale::impl::memory::decode;
127169
128-
Encoding data using ```encode``` convenience function looks as follows:
129-
```c++
130-
std::vector<uint32_t> v = {1u, 2u, 3u, 4u};
131-
auto &&result = encode(v);
132-
if (!res) {
133-
// handle error
170+
BytesArray data = {...};
171+
auto res = decode<SomeClass>(data); // <- Just one-line call
172+
if (res.has_value()) {
173+
SomeClass object = std::move(res.value()); // Decoded value
134174
}
135-
```
136175
137-
Decoding data using ```decode``` convenience function looks as follows:
176+
using ::scale::impl::bytes::EncoderToVector;
177+
178+
SomeClass object = {...};
179+
std::vector<uint8_t> out;
180+
EncoderToVector encoder;
181+
try {
182+
encoder << object;
183+
// or encode(object, decoder);
184+
std::vector<uint8_t> encoded = std::move(res.value())
185+
} catch ...
186+
187+
using ::scale::impl::bytes::DecoderFromBytes;
188+
189+
BytesArray in = {...};
190+
DecoderFromSpan decoder(in);
191+
try {
192+
Object object;
193+
decoder >> object;
194+
// or decode(object, decoder);
195+
} catch ...
138196
139-
```c++
140-
ByteArray bytes = {...};
141-
outcome::result<MyType> result = decode<MyType>(bytes);
142-
if (!result) {
143-
// handle error
144-
}
145-
```
146-
or
147-
```c++
148-
ByteArray bytes = {...};
149-
ScaleDecoderStream s(bytes);
150-
outcome::result<MyType> result = decode<MyType>(s);
151-
if (!result) {
152-
// handle error
153-
}
154197
```

cmake/Hunter/config.cmake

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,6 @@
1919

2020
hunter_config(
2121
qtils
22-
URL https://github.com/qdrvm/qtils/archive/16e7c819dd50af2f64e2d319b918d0d815332266.tar.gz
23-
SHA1 71989938b5c8b7650eaf1a8195c2b52c5a8c250b
22+
URL https://github.com/qdrvm/qtils/archive/9a64dfd6ed0226dec29805aa89d4c713a6f81d9f.tar.gz
23+
SHA1 16c0269175018e88c33090f9fbdaa230c8fdb911
2424
)

0 commit comments

Comments
 (0)