Skip to content

Commit cffcef1

Browse files
zihaowang678popematttgregg
authored
Add Support for IonElement (#69)
* add IonElement support * add tests for IonElement * update README.md * Update src/com/amazon/ion/benchmark/IonMeasurableReadTask.java Co-authored-by: Matthew Pope <81593196+popematt@users.noreply.github.com> * Update src/com/amazon/ion/benchmark/IonMeasurableWriteTask.java Co-authored-by: Matthew Pope <81593196+popematt@users.noreply.github.com> * Update src/com/amazon/ion/benchmark/IonMeasurableWriteTask.java Co-authored-by: Matthew Pope <81593196+popematt@users.noreply.github.com> * Update pom.xml Co-authored-by: Matthew Pope <81593196+popematt@users.noreply.github.com> * revert unrelated changes; import ElementLoader * Update src/com/amazon/ion/benchmark/Main.java Co-authored-by: Tyler Gregg <greggt@amazon.com> * Update tst/com/amazon/ion/benchmark/OptionsTest.java Co-authored-by: Tyler Gregg <greggt@amazon.com> * Update tst/com/amazon/ion/benchmark/OptionsTest.java Co-authored-by: Tyler Gregg <greggt@amazon.com> * Add consumption loops to DOM read methods --------- Co-authored-by: Matthew Pope <81593196+popematt@users.noreply.github.com> Co-authored-by: Tyler Gregg <greggt@amazon.com>
1 parent 36bfda9 commit cffcef1

15 files changed

Lines changed: 331 additions & 9 deletions

‎README.md‎

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -101,18 +101,29 @@ ion-java-benchmark write --ion-imports-for-benchmark tables.ion \
101101
Benchmark a full-traversal read of data equivalent to exampleWithImports.10n, which declares the shared
102102
symbol table imports provided by inputTables.ion, re-encoded (if necessary) using the shared symbol
103103
tables provided by benchmarkTables.ion, inputTables.ion, and no shared symbol tables. Produce
104-
results from using both the DOM and IonReader APIs.
104+
results from using the DOM, IonReader and IonElement APIs.
105105

106106
```
107107
ion-java-benchmark read --ion-imports-for-input inputTables.ion \
108108
--ion-imports-for-benchmark benchmarkTables.ion \
109109
--ion-imports-for-benchmark auto \
110110
--ion-imports-for-benchmark none \
111-
--ion-api dom \
112-
--ion-api streaming \
111+
--api dom \
112+
--api streaming \
113+
--api ion_element_dom \
113114
exampleWithImports.10n
114115
```
115116

117+
Benchmark a full-traversal read of `example.10n` using the IonElement API from ion-element-kotlin,
118+
comparing performance against the traditional DOM API.
119+
120+
```
121+
ion-java-benchmark read --api dom \
122+
--api ion_element_dom \
123+
example.10n
124+
```
125+
126+
116127
## Tips
117128

118129
As the JMH output warns: "Do not assume the numbers tell you what you want them to tell." Benchmarking

‎pom.xml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,21 @@
9797
<artifactId>ion-java-path-extraction</artifactId>
9898
<version>1.2.0</version>
9999
</dependency>
100+
<dependency>
101+
<groupId>com.amazon.ion</groupId>
102+
<artifactId>ion-element</artifactId>
103+
<version>[1.3.0, )</version>
104+
</dependency>
105+
<dependency>
106+
<groupId>org.jetbrains.kotlin</groupId>
107+
<artifactId>kotlin-stdlib-jdk8</artifactId>
108+
<version>1.6.20</version>
109+
</dependency>
110+
<dependency>
111+
<groupId>org.jetbrains.kotlinx</groupId>
112+
<artifactId>kotlinx-collections-immutable-jvm</artifactId>
113+
<version>0.3.4</version>
114+
</dependency>
100115
<!-- test dependencies -->
101116
<dependency>
102117
<groupId>junit</groupId>
@@ -117,6 +132,13 @@
117132
<configuration>
118133
<source>8</source>
119134
<target>8</target>
135+
<annotationProcessorPaths>
136+
<path>
137+
<groupId>org.openjdk.jmh</groupId>
138+
<artifactId>jmh-generator-annprocess</artifactId>
139+
<version>1.23</version>
140+
</path>
141+
</annotationProcessorPaths>
120142
</configuration>
121143
</plugin>
122144
<plugin>

‎src/com/amazon/ion/benchmark/API.java‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,5 +13,10 @@ enum API {
1313
/**
1414
* For Ion: the DOM APIs (IonLoader, IonValue, etc.). For JSON (Jackson): JsonNode via ObjectMapper.
1515
*/
16-
DOM
16+
DOM,
17+
18+
/**
19+
* For Ion: the IonElement APIs from ion-element-kotlin. For JSON (Jackson): Not supported.
20+
*/
21+
ION_ELEMENT_DOM
1722
}

‎src/com/amazon/ion/benchmark/CborJacksonMeasurableReadTask.java‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -156,4 +156,14 @@ void fullyReadDomFromFile(SideEffectConsumer consumer) throws IOException {
156156
consumer.consume(iterator.next());
157157
}
158158
}
159+
160+
@Override
161+
public void fullyReadElementFromBuffer(SideEffectConsumer consumer) throws IOException {
162+
throw new UnsupportedOperationException("IonElement API is not supported for CBOR format. Use ion_binary or ion_text format instead.");
163+
}
164+
165+
@Override
166+
public void fullyReadElementFromFile(SideEffectConsumer consumer) throws IOException {
167+
throw new UnsupportedOperationException("IonElement API is not supported for CBOR format. Use ion_binary or ion_text format instead.");
168+
}
159169
}

‎src/com/amazon/ion/benchmark/CborJacksonMeasurableWriteTask.java‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -150,4 +150,9 @@ CBORGenerator newWriter(OutputStream outputStream) throws IOException {
150150
void closeWriter(CBORGenerator generator) throws IOException {
151151
generator.close();
152152
}
153+
154+
@Override
155+
void generateWriteInstructionsElement(Consumer<WriteInstruction<CBORGenerator>> instructionsSink) throws IOException {
156+
throw new UnsupportedOperationException("IonElement API is not supported for CBOR format. Use ion_binary or ion_text format instead.");
157+
}
153158
}

‎src/com/amazon/ion/benchmark/IonMeasurableReadTask.java‎

Lines changed: 37 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,9 @@
77
import com.amazon.ion.system.IonReaderBuilder;
88
import com.amazon.ionpathextraction.PathExtractor;
99
import com.amazon.ionpathextraction.PathExtractorBuilder;
10+
import com.amazon.ionelement.api.AnyElement;
11+
import com.amazon.ionelement.api.IonElementLoader;
12+
import com.amazon.ionelement.api.ElementLoader;
1013

1114
import java.io.IOException;
1215
import java.nio.file.Path;
@@ -25,6 +28,7 @@ class IonMeasurableReadTask extends MeasurableReadTask {
2528
private final byte[] reusableLobBuffer;
2629
private IonReaderBuilder readerBuilder;
2730
private SideEffectConsumer sideEffectConsumer = null;
31+
private IonElementLoader elementLoader;
2832

2933
/**
3034
* Returns the next power of two greater than or equal to the given value.
@@ -67,6 +71,11 @@ private int pathExtractorCallback(IonReader reader) {
6771
} else {
6872
reusableLobBuffer = null;
6973
}
74+
75+
// Initialize IonElement loader for ION_ELEMENT_DOM API
76+
if (options.api == API.ION_ELEMENT_DOM) {
77+
elementLoader = ElementLoader.createIonElementLoader();
78+
}
7079
}
7180

7281
@Override
@@ -240,15 +249,41 @@ public void traverseFromFile(List<String> paths, SideEffectConsumer consumer) th
240249
public void fullyReadDomFromBuffer(SideEffectConsumer consumer) throws IOException {
241250
sideEffectConsumer = consumer;
242251
IonReader reader = readerBuilder.build(buffer);
243-
ionSystem.newLoader().load(reader);
252+
for (com.amazon.ion.IonValue value : ionSystem.newLoader().load(reader)) {
253+
consumer.consume(value);
254+
}
244255
reader.close();
245256
}
246257

247258
@Override
248259
public void fullyReadDomFromFile(SideEffectConsumer consumer) throws IOException {
249260
sideEffectConsumer = consumer;
250261
IonReader reader = readerBuilder.build(options.newInputStream(inputFile));
251-
ionSystem.newLoader().load(reader);
262+
for (com.amazon.ion.IonValue value : ionSystem.newLoader().load(reader)) {
263+
consumer.consume(value);
264+
}
265+
reader.close();
266+
}
267+
268+
@Override
269+
public void fullyReadElementFromBuffer(SideEffectConsumer consumer) throws IOException {
270+
sideEffectConsumer = consumer;
271+
IonReader reader = readerBuilder.build(buffer);
272+
Iterable<AnyElement> elements = elementLoader.loadAllElements(reader);
273+
for (AnyElement element : elements) {
274+
consumer.consume(element);
275+
}
276+
reader.close();
277+
}
278+
279+
@Override
280+
public void fullyReadElementFromFile(SideEffectConsumer consumer) throws IOException {
281+
sideEffectConsumer = consumer;
282+
IonReader reader = readerBuilder.build(options.newInputStream(inputFile));
283+
Iterable<AnyElement> elements = elementLoader.loadAllElements(reader);
284+
for (AnyElement element : elements) {
285+
consumer.consume(element);
286+
}
252287
reader.close();
253288
}
254289
}

‎src/com/amazon/ion/benchmark/IonMeasurableWriteTask.java‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,13 +7,18 @@
77
import com.amazon.ion.IonWriter;
88
import com.amazon.ion.SymbolToken;
99
import com.amazon.ion.Timestamp;
10+
import com.amazon.ionelement.api.AnyElement;
11+
import com.amazon.ionelement.api.IonElementLoader;
12+
import com.amazon.ionelement.api.ElementLoader;
1013

1114
import java.io.IOException;
1215
import java.io.OutputStream;
1316
import java.math.BigDecimal;
1417
import java.math.BigInteger;
1518
import java.nio.file.Path;
19+
import java.util.ArrayList;
1620
import java.util.Iterator;
21+
import java.util.List;
1722
import java.util.function.Consumer;
1823

1924
import static com.amazon.ion.benchmark.Constants.ION_SYSTEM;
@@ -24,6 +29,7 @@
2429
class IonMeasurableWriteTask extends MeasurableWriteTask<IonWriter> {
2530

2631
private final IonUtilities.IonWriterSupplier writerBuilder;
32+
private IonElementLoader elementLoader;
2733

2834
/**
2935
* @param inputPath path to the data to re-write.
@@ -39,6 +45,11 @@ class IonMeasurableWriteTask extends MeasurableWriteTask<IonWriter> {
3945
} else {
4046
throw new IllegalStateException("IonFormatWriter is compatible only with ION_TEXT and ION_BINARY");
4147
}
48+
49+
// Initialize IonElement loader for ION_ELEMENT_DOM API
50+
if (options.api == API.ION_ELEMENT_DOM) {
51+
elementLoader = ElementLoader.createIonElementLoader();
52+
}
4253
}
4354

4455
/**
@@ -195,4 +206,33 @@ public void closeWriter(IonWriter writer) throws IOException {
195206
writer.close();
196207
}
197208

209+
@Override
210+
void generateWriteInstructionsElement(Consumer<WriteInstruction<IonWriter>> instructionsSink) throws IOException {
211+
Iterable<AnyElement> elements;
212+
if (options.limit == Integer.MAX_VALUE) {
213+
try (IonReader reader = IonUtilities.newReaderBuilderForInput(options).build(options.newInputStream(inputFile))) {
214+
elements = elementLoader.loadAllElements(reader);
215+
}
216+
} else {
217+
List<AnyElement> limitedElements = new ArrayList<>();
218+
try (IonReader reader = IonUtilities.newReaderBuilderForInput(options).build(options.newInputStream(inputFile))) {
219+
int count = 0;
220+
while (count++ < options.limit && reader.next() != null) {
221+
limitedElements.add(elementLoader.loadCurrentElement(reader));
222+
}
223+
}
224+
elements = limitedElements;
225+
}
226+
227+
// Convert IonElements to write instructions
228+
int elementCount = 0;
229+
for (AnyElement element : elements) {
230+
instructionsSink.accept(element::writeTo);
231+
elementCount++;
232+
if (options.flushPeriod != null && elementCount % options.flushPeriod == 0) {
233+
instructionsSink.accept(IonWriter::flush);
234+
}
235+
}
236+
instructionsSink.accept(IonWriter::finish);
237+
}
198238
}

‎src/com/amazon/ion/benchmark/JsonJacksonMeasurableReadTask.java‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -145,4 +145,14 @@ void fullyReadDomFromFile(SideEffectConsumer consumer) throws IOException {
145145
consumer.consume(iterator.next());
146146
}
147147
}
148+
149+
@Override
150+
public void fullyReadElementFromBuffer(SideEffectConsumer consumer) throws IOException {
151+
throw new UnsupportedOperationException("IonElement API is not supported for JSON format. Use ion_binary or ion_text format instead.");
152+
}
153+
154+
@Override
155+
public void fullyReadElementFromFile(SideEffectConsumer consumer) throws IOException {
156+
throw new UnsupportedOperationException("IonElement API is not supported for JSON format. Use ion_binary or ion_text format instead.");
157+
}
148158
}

‎src/com/amazon/ion/benchmark/JsonJacksonMeasurableWriteTask.java‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -142,4 +142,9 @@ JsonGenerator newWriter(OutputStream outputStream) throws IOException {
142142
void closeWriter(JsonGenerator generator) throws IOException {
143143
generator.close();
144144
}
145+
146+
@Override
147+
void generateWriteInstructionsElement(Consumer<WriteInstruction<JsonGenerator>> instructionsSink) throws IOException {
148+
throw new UnsupportedOperationException("IonElement API is not supported for JSON format. Use ion_binary or ion_text format instead.");
149+
}
145150
}

‎src/com/amazon/ion/benchmark/Main.java‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -133,10 +133,10 @@ public class Main {
133133
+ " -f --format <type> Format to benchmark, from the set (ion_binary | ion_text | json | "
134134
+ "cbor). May be specified multiple times to compare different formats. [default: ion_binary]\n"
135135

136-
+ " -a --api <api> The API to exercise (dom or streaming). For the ion-binary or "
136+
+ " -a --api <api> The API to exercise (dom, streaming, or ion_element_dom). For the ion-binary or "
137137
+ "ion-text formats, 'streaming' causes IonReader/IonWriter to be used while 'dom' causes IonLoader to be "
138-
+ "used. For Jackson JSON, 'streaming' causes JsonParser/JsonGenerator to be used while 'dom' causes "
139-
+ "ObjectMapper to materialize JsonNode instances. May be specified multiple times to compare both "
138+
+ "used, and 'ion_element_dom' causes IonElement from ion-element-kotlin to be used with dedicated code paths. For Jackson JSON, 'streaming' causes JsonParser/JsonGenerator to be used while 'dom' causes "
139+
+ "ObjectMapper to materialize JsonNode instances. 'ion_element_dom' is not supported for JSON or CBOR formats. May be specified multiple times to compare "
140140
+ "APIs. [default: streaming]\n"
141141

142142
+ " -I --ion-imports-for-input <file> A file containing a sequence of Ion symbol tables, or the string "

0 commit comments

Comments
 (0)