GML-1432: Refactor dataloaders by billshitg · Pull Request #194 · tigergraph/pyTigerGraph

billshitg · 2023-10-10T19:47:57Z

Refactor BaseLoader

parkererickson-tg

Do we have any benchmarks surrounding this or will we get that insight once the neighbor loader is implemented?

billshitg · 2023-10-10T20:12:05Z

good point. We don't have benchmarks. We should probably do it once we have the actual dataloaders.

qe-tigergraph

Unit Test: SUCCESS, e2e Test: FAILURE, Jenkins_job:http://192.168.99.101:30080/job/mlwb_build/1013/

qe-tigergraph

Unit Test: SUCCESS, e2e Test: FAILURE, Jenkins_job:http://192.168.99.101:30080/job/mlwb_build/1015/

qe-tigergraph

Unit Test: FAILURE, Jenkins_job:http://192.168.99.101:30080/job/mlwb_build/1016/

qe-tigergraph

Unit Test: SUCCESS, e2e Test: FAILURE, Jenkins_job:http://192.168.99.101:30080/job/mlwb_build/1017/

qe-tigergraph

Unit Test: SUCCESS, e2e Test: FAILURE, Jenkins_job:http://192.168.99.101:30080/job/mlwb_build/1017/

qe-tigergraph

Unit Test: FAILURE, Jenkins_job:http://192.168.99.101:30080/job/mlwb_build/1018/

qe-tigergraph

Unit Test: FAILURE, Jenkins_job:http://192.168.99.101:30080/job/mlwb_build/1019/

qe-tigergraph

Unit Test: FAILURE, Jenkins_job:http://192.168.99.101:30080/job/mlwb_build/1020/

qe-tigergraph

Unit Test: FAILURE, Jenkins_job:http://192.168.99.101:30080/job/mlwb_build/1032/

qe-tigergraph

Unit Test: FAILURE, Jenkins_job:http://192.168.99.101:30080/job/mlwb_build/1033/

qe-tigergraph

Unit Test: SUCCESS, e2e Test: FAILURE, Jenkins_job:http://192.168.99.101:30080/job/mlwb_build/1034/

qe-tigergraph

Unit Test: SUCCESS, e2e Test: FAILURE, Jenkins_job:http://192.168.99.101:30080/job/mlwb_build/1037/

qe-tigergraph

Unit Test: FAILURE, Jenkins_job:http://192.168.99.101:30080/job/mlwb_build/1038/

qe-tigergraph

Unit Test: SUCCESS, e2e Test: FAILURE, Jenkins_job:http://192.168.99.101:30080/job/mlwb_build/1039/

qe-tigergraph

Unit Test: SUCCESS, e2e Test: SUCCESS, Jenkins_job:http://192.168.99.101:30080/job/mlwb_build/1041/

qe-tigergraph

QE Approved

billshitg requested a review from qe-tigergraph as a code owner October 10, 2023 19:47

parkererickson-tg approved these changes Oct 10, 2023

View reviewed changes

qe-tigergraph requested changes Nov 10, 2023

View reviewed changes

qe-tigergraph requested changes Nov 15, 2023

View reviewed changes

qe-tigergraph requested changes Nov 18, 2023

View reviewed changes

qe-tigergraph requested changes Nov 22, 2023

View reviewed changes

qe-tigergraph requested changes Nov 29, 2023

View reviewed changes

qe-tigergraph requested changes Dec 6, 2023

View reviewed changes

qe-tigergraph requested changes Dec 8, 2023

View reviewed changes

billshitg added 18 commits December 8, 2023 10:35

refactor(BaseLoader): split read_data and parse_data

55e7049

refactor(BaseLoader): consolidate vertex and edge parse

324fea0

refactor(BaseLoader): consolidate attr_to_tensor and reindex

441a0b3

feat(BaseLoader): handle one data point per msg

65cca46

fix(BaseLoader): upate _start_request for the refactor

9eaef0e

refactor(VertexLoader): update gsql template

e4e407e

refactor(VertexLoader): update reader thread

4c42d4a

fix(VertexLoader): fix wrong comma in gsql

41c45f9

fix(BaseLoader): fix missing function decorator

ce95a21

fix(BaseLoader,VertexLoader): change how we shuffle

a1ff95b

feat(BaseLoader,VertexLoader): fix when to stop kafka

e7f3e94

feat(VertexLoader): update shuffle

1dfa8bd

test(VertexLoader): update tests

d5a97df

feat(VertexLoader): move kafka commands into template

0298bc0

feat: move shuffle to gsql for speed and add EdgeLoader

7315071

feat(GraphLoader): update gsql and add subquery

d8b816b

feat(dataloaders): update for GraphLoader

7b2970c

feat(utilities): add function to install multiple files

7f5f584

billshitg added 11 commits December 8, 2023 10:35

test(GraphLoaders): update unit tests

7320237

feat(dataloaders): update NeighborLoader

ffe5b0d

feat: update all about EdgeNeighborLoader

a78f33d

feat: add new hgt loader

c3434b5

feat(gds): update nodepieceloader

001e1ea

test(BaseLoader): rm unneeded tests

2ac345b

fix(dataloaders): fix seeds type issue

189bc29

test(GDS): fix num_batches issue

9a9e78c

fix(Trainer): rm num_batches

17390dc

test(featurizer): fix patch_ver issue

0d03fe5

fix: bool error when num_batches is None

22a02b6

billshitg force-pushed the GML-1432-refactor-baseloader branch from cf68fe8 to 22a02b6 Compare December 8, 2023 18:36

qe-tigergraph requested changes Dec 8, 2023

View reviewed changes

qe-tigergraph requested changes Dec 11, 2023

View reviewed changes

qe-tigergraph requested changes Dec 12, 2023

View reviewed changes

fix(test_HGTLoader): fix error when edge not present

ed6a53c

qe-tigergraph requested changes Dec 13, 2023

View reviewed changes

fix(dataloaders): check edge number returned from DB

3f257a2

qe-tigergraph reviewed Dec 15, 2023

View reviewed changes

qe-tigergraph approved these changes Dec 15, 2023

View reviewed changes

billshitg added 5 commits December 20, 2023 16:24

feat: update baseloader and unit test for new queries

98cdc1f

feat(EdgeNeighborLoader): update loader and gsql

7e0de75

fix(parse_edge_data): error when edge type has _

4d334a1

feat: allow kafka in distributed query

c704e02

tests(EdgeNeighborLoader): test distributed query

0ae8be8

Conversation

billshitg commented Oct 10, 2023

Uh oh!

parkererickson-tg left a comment

Choose a reason for hiding this comment

Uh oh!

billshitg commented Oct 10, 2023

Uh oh!

qe-tigergraph left a comment

Choose a reason for hiding this comment

Uh oh!

qe-tigergraph left a comment

Choose a reason for hiding this comment

Uh oh!

qe-tigergraph left a comment

Choose a reason for hiding this comment

Uh oh!

qe-tigergraph left a comment

Choose a reason for hiding this comment

Uh oh!

qe-tigergraph left a comment

Choose a reason for hiding this comment

Uh oh!

qe-tigergraph left a comment

Choose a reason for hiding this comment

Uh oh!

qe-tigergraph left a comment

Choose a reason for hiding this comment

Uh oh!

qe-tigergraph left a comment

Choose a reason for hiding this comment

Uh oh!

qe-tigergraph left a comment

Choose a reason for hiding this comment

Uh oh!

qe-tigergraph left a comment

Choose a reason for hiding this comment

Uh oh!

qe-tigergraph left a comment

Choose a reason for hiding this comment

Uh oh!

qe-tigergraph left a comment

Choose a reason for hiding this comment

Uh oh!

qe-tigergraph left a comment

Choose a reason for hiding this comment

Uh oh!

qe-tigergraph left a comment

Choose a reason for hiding this comment

Uh oh!

qe-tigergraph left a comment

Choose a reason for hiding this comment

Uh oh!

qe-tigergraph left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants