Remove python folder #449

svij-sc · 2026-01-21T17:33:50Z

Scope of work done

Our source code has lived in GiGL/python for a while.
Although this was fine when the repo was first created, it has since became a headache as we need to make special considerations when handling it in code, when reading bundled yamls, resolution for IDEs, building docker images, et al.

This is also making it harder to bundle static files (non source code) in wheels; thus I am opting to remove it.

Where is the documentation for this feature?: N/A

Did you add automated tests or write a test plan?

Ran testing suite in comments below; addressed fixes as needed.

Updated Changelog.md? NO

Ready for code review?: YES

svij-sc · 2026-01-21T20:53:00Z

/help

github-actions · 2026-01-21T20:53:17Z

GiGL Automation

@ 20:53:17UTC :

🤖 Available PR Commands

You can trigger the following workflows by commenting on this PR:

/help - Checkout code
/unit_test_py - Run Python Unit Tests
/unit_test_scala - Run Scala Unit Tests
/integration_test - Run Integration Tests
/e2e_test - Run E2E Tests
/notebook_tests - Run Example Notebooks Tests
/lint_test - Run Linting Tests

💡 Usage: Simply comment on this PR with any of the commands above (e.g., /unit_test)

⏱️ Note: Commands may take some time to complete. Progress updates will be posted as comments.

…lder

svij-sc · 2026-01-21T21:00:25Z

/unit_test_py

svij-sc · 2026-01-21T21:00:31Z

/integration_test

github-actions · 2026-01-21T21:01:02Z

GiGL Automation

@ 21:01:02UTC : 🔄 Python Unit Test started.

@ 21:14:56UTC : ❌ Workflow failed.
Please check the logs for more details.

github-actions · 2026-01-21T21:01:06Z

GiGL Automation

@ 21:01:06UTC : 🔄 Integration Test started.

@ 21:18:15UTC : ❌ Workflow failed.
Please check the logs for more details.

github-actions · 2026-01-21T21:01:29Z

GiGL Automation

@ 21:01:29UTC : Starting to build base images for CUDA and CPU.
This may take a while, please be patient.
The images will be pushed to the GCP Artifact Registry.
Once done, the workflow will update the dep_vars.env file with the new image names.

…names

github-actions · 2026-01-21T21:11:57Z

GiGL Automation

@ 21:11:56UTC : Built and pushed new images:

CUDA base image: us-central1-docker.pkg.dev/external-snap-ci-github-gigl/public-gigl/gigl-cuda-base:2b270be5b9e9aa0396d9ced365407e9f6a19a262.90.1
CPU base image: us-central1-docker.pkg.dev/external-snap-ci-github-gigl/public-gigl/gigl-cpu-base:2b270be5b9e9aa0396d9ced365407e9f6a19a262.90.1
Dataflow base image: us-central1-docker.pkg.dev/external-snap-ci-github-gigl/public-gigl/gigl-dataflow-base:2b270be5b9e9aa0396d9ced365407e9f6a19a262.90.1
Builder image: us-central1-docker.pkg.dev/external-snap-ci-github-gigl/gigl-base-images/gigl-builder:2b270be5b9e9aa0396d9ced365407e9f6a19a262.90.1

Updated dep_vars.env with new image names.
Updated .github/cloud_builder/run_command_on_pr_cloud_build.yaml with new builder image.

svij-sc · 2026-01-22T03:27:28Z

/unit_test_py

github-actions · 2026-01-22T03:27:43Z

GiGL Automation

@ 03:27:43UTC : 🔄 Python Unit Test started.

@ 04:22:11UTC : ✅ Workflow completed successfully.

svij-sc · 2026-01-22T03:27:46Z

/integration_test

github-actions · 2026-01-22T03:27:57Z

GiGL Automation

@ 03:27:57UTC : 🔄 Integration Test started.

@ 04:20:45UTC : ❌ Workflow failed.
Please check the logs for more details.

…lder

svij-sc · 2026-01-22T06:16:13Z

/integration_test

github-actions · 2026-01-22T06:16:28Z

GiGL Automation

@ 06:16:28UTC : 🔄 Integration Test started.

@ 07:23:00UTC : ✅ Workflow completed successfully.

github-actions · 2026-01-22T06:17:56Z

GiGL Automation

@ 06:17:56UTC : Starting to build base images for CUDA and CPU.
This may take a while, please be patient.
The images will be pushed to the GCP Artifact Registry.
Once done, the workflow will update the dep_vars.env file with the new image names.

…names

github-actions · 2026-01-22T06:26:49Z

GiGL Automation

@ 06:26:49UTC : Built and pushed new images:

CUDA base image: us-central1-docker.pkg.dev/external-snap-ci-github-gigl/public-gigl/gigl-cuda-base:b34c863a2168c8df5a6da1f6385e5d374f0175d2.91.1
CPU base image: us-central1-docker.pkg.dev/external-snap-ci-github-gigl/public-gigl/gigl-cpu-base:b34c863a2168c8df5a6da1f6385e5d374f0175d2.91.1
Dataflow base image: us-central1-docker.pkg.dev/external-snap-ci-github-gigl/public-gigl/gigl-dataflow-base:b34c863a2168c8df5a6da1f6385e5d374f0175d2.91.1
Builder image: us-central1-docker.pkg.dev/external-snap-ci-github-gigl/gigl-base-images/gigl-builder:b34c863a2168c8df5a6da1f6385e5d374f0175d2.91.1

Updated dep_vars.env with new image names.
Updated .github/cloud_builder/run_command_on_pr_cloud_build.yaml with new builder image.

svij-sc · 2026-01-22T16:38:56Z

/e2e_test

github-actions · 2026-01-22T16:39:32Z

GiGL Automation

@ 16:39:32UTC : 🔄 E2E Test started.

@ 18:01:50UTC : ❌ Workflow failed.
Please check the logs for more details.

kmontemayor2-sc · 2026-01-22T17:04:23Z

/unit_test

github-actions · 2026-01-22T17:04:37Z

GiGL Automation

@ 17:04:36UTC : 🔄 Scala Unit Test started.

@ 17:13:14UTC : ✅ Workflow completed successfully.

github-actions · 2026-01-22T17:04:41Z

GiGL Automation

@ 17:04:41UTC : 🔄 Python Unit Test started.

@ 18:18:19UTC : ✅ Workflow completed successfully.

kmontemayor2-sc

Can we verify this works internally too?

Makefile

.gitignore

containers/Dockerfile.builder

containers/Dockerfile.dataflow.src

gigl/src/common/constants/local_fs.py

scala_spark35/common/src/main/scala/snapchat/research/gbml/gbml_config/GbmlConfig.scala

pyproject.toml

tests/unit/main.py

svij-sc · 2026-01-22T17:58:48Z

@kmontemayor2-sc addressed ur comments.

Re:

Can we verify this works internally too?

See: go/gigl-issue/1465

kmontemayor2-sc

Have we verified that we can still release/etc with these changes?

We should probably have a version bump with this (and update CHANGELOG)

Otherwise, LGTM but I'd prefer we can verify everything works internally first :)

tests/unit/utils/__init__.py

svij-sc · 2026-01-22T20:10:49Z

Have we verified that we can still release/etc with these changes?

No, mostly because release (the wheel) is broken right now and this PR is a prerequisite to me fixing the release process.

We should probably have a version bump with this (and update CHANGELOG)

I am planning to have a version bump and subsequent release when the release wheel is fixed - which I am working on right now.

svij-sc · 2026-01-22T20:16:45Z

@kmontemayor2-sc

Otherwise, LGTM but I'd prefer we can verify everything works internally first :)

As more PRs go into GiGL the burden of work here to constantly update and test this scales tremendously as this will conflict with every* PR.

You can track the progress here go/gigl-issue/1465; which should be resolved soon

mkolodner-sc

Thanks Shubham! A few remaining python references from my search: 1, 2, 3, 4, 5, 6, 7

svij-sc · 2026-01-22T20:46:56Z

Thanks for the finds Matt.

Thanks Shubham! A few remaining python references from my search: 1, 2, 3, 4, 5, 6, 7

1: is expected as its on a specific commit/blob
2-7: fixed

github-actions · 2026-01-22T20:47:13Z

GiGL Automation

@ 20:47:13UTC : 🔄 E2E Test started.

@ 22:04:38UTC : ✅ Workflow completed successfully.

mkolodner-sc

Thanks Shubham! Generally LGTM conditioned on all testing passing

kmontemayor2-sc

Synced offline, lgtm provided we're find with addressing any breakages asap.

Makefile

svij-sc added 2 commits January 21, 2026 17:32

remove python folder

c28d519

fixes for path mangling

e243101

Merge remote-tracking branch 'origin/main' into svij/remove-python-fo…

2b270be

…lder

[AUTOMATED] Update dep.vars, and other relevant files with new image …

f5c6589

…names

fix

65ba86e

svij-sc added 6 commits January 22, 2026 03:41

readd dost+partitioner_test

a7c3740

remove added fix

3d7bb7a

revert

fcd4f0f

Merge remote-tracking branch 'origin/main' into svij/remove-python-fo…

f586a5e

…lder

move

44c3cda

fix

b34c863

[AUTOMATED] Update dep.vars, and other relevant files with new image …

563375d

…names

svij-sc marked this pull request as ready for review January 22, 2026 16:37

svij-sc requested a review from kmontemayor2-sc as a code owner January 22, 2026 16:37

svij-sc requested review from mkolodner-sc, xgao4-sc, yliu2-sc and zfan3-sc as code owners January 22, 2026 16:37

svij-sc changed the title ~~[WIP] remove python folder~~ Remove python folder Jan 22, 2026

kmontemayor2-sc reviewed Jan 22, 2026

View reviewed changes

pr comments + formatter

1c2badc

svij-sc mentioned this pull request Jan 22, 2026

Combine the two tests folder tests/ and testing/ #457

Open

nit

c4d721a

kmontemayor2-sc requested changes Jan 22, 2026

View reviewed changes

tests/unit/utils/__init__.py Show resolved Hide resolved

update changelog

0e1eed2

mkolodner-sc reviewed Jan 22, 2026

View reviewed changes

address comments

898fff1

mkolodner-sc approved these changes Jan 22, 2026

View reviewed changes

kmontemayor2-sc approved these changes Jan 22, 2026

View reviewed changes

Makefile Show resolved Hide resolved

svij-sc enabled auto-merge January 22, 2026 23:37

svij-sc added this pull request to the merge queue Jan 22, 2026

Merged via the queue into main with commit e1a3721 Jan 23, 2026
6 checks passed

svij-sc deleted the svij/remove-python-folder branch January 23, 2026 00:59

Remove python folder #449

Remove python folder #449

Conversation

svij-sc commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

svij-sc commented Jan 21, 2026

Uh oh!

github-actions bot commented Jan 21, 2026

GiGL Automation

🤖 Available PR Commands

Uh oh!

svij-sc commented Jan 21, 2026

Uh oh!

svij-sc commented Jan 21, 2026

Uh oh!

github-actions bot commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

GiGL Automation

Uh oh!

github-actions bot commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

GiGL Automation

Uh oh!

github-actions bot commented Jan 21, 2026

GiGL Automation

Uh oh!

github-actions bot commented Jan 21, 2026

GiGL Automation

Uh oh!

svij-sc commented Jan 22, 2026

Uh oh!

github-actions bot commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

GiGL Automation

Uh oh!

svij-sc commented Jan 22, 2026

Uh oh!

github-actions bot commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

GiGL Automation

Uh oh!

svij-sc commented Jan 22, 2026

Uh oh!

github-actions bot commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

GiGL Automation

Uh oh!

github-actions bot commented Jan 22, 2026

GiGL Automation

Uh oh!

github-actions bot commented Jan 22, 2026

GiGL Automation

Uh oh!

svij-sc commented Jan 22, 2026

Uh oh!

github-actions bot commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

GiGL Automation

Uh oh!

kmontemayor2-sc commented Jan 22, 2026

Uh oh!

github-actions bot commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

GiGL Automation

Uh oh!

github-actions bot commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

GiGL Automation

Uh oh!

kmontemayor2-sc left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

svij-sc commented Jan 21, 2026 •

edited

Loading

github-actions bot commented Jan 21, 2026 •

edited

Loading

github-actions bot commented Jan 21, 2026 •

edited

Loading

github-actions bot commented Jan 22, 2026 •

edited

Loading

github-actions bot commented Jan 22, 2026 •

edited

Loading

github-actions bot commented Jan 22, 2026 •

edited

Loading

github-actions bot commented Jan 22, 2026 •

edited

Loading

github-actions bot commented Jan 22, 2026 •

edited

Loading

github-actions bot commented Jan 22, 2026 •

edited

Loading

svij-sc commented Jan 22, 2026 •

edited

Loading

svij-sc commented Jan 22, 2026 •

edited

Loading

github-actions bot commented Jan 22, 2026 •

edited

Loading