troyfeng116/tests

[troyfeng116]

No description provided.

[niansong1996]

@troyfeng116 LMK if you want my input on anything

[niansong1996]

For integration test, one suggestion is that instead of doing what I did in tests/test_inference_pipeline.py, you should follow this doc and run it with Lightning API instead

[troyfeng116]

For integration test, one suggestion is that instead of doing what I did in tests/test_inference_pipeline.py, you should follow this doc and run it with Lightning API instead

Got it, currently working on models (something similar to what I did for datasets if you have time to take a look at that).

Would using Lightning API just be to avoid using os.system?

[niansong1996]

Would using Lightning API just be to avoid using os.system?

Yes, but it seems like this function is not added for the version of the lightning we are using (1.7.4). So let's stall this.

[niansong1996]

But I found another issue: I was trying to use GPUs for the tests but got error like this you've requested GPU [0,1] but only got [0]. I suspect that when you start the runner, there is some env var that sets CUDA_VISIBLE_DEVICES=0 somewhere. Can you check this out?

[troyfeng116]

But I found another issue: I was trying to use GPUs for the tests but got error like this you've requested GPU [0,1] but only got [0]. I suspect that when you start the runner, there is some env var that sets CUDA_VISIBLE_DEVICES=0 somewhere. Can you check this out?

Should have just fixed, left an env vars from SummerTime I think

[niansong1996]

I just merged #40 into main, so please merge from main and fix the conflicts. But definitely feel free to ping me if you need me on anything.

[troyfeng116]

@niansong1996 Running into some problem with spider_reader, seems like example_to_demonstration_sql was deprecated so promptify_function function pointer can't be resolved?

[niansong1996]

Ignore the non-few-shot datasets for now and add an issue as a TODO instead.

[troyfeng116]

@niansong1996 done expanding all tests, takes about 9-10 minutes to run at the moment. Let me know if it look ok for now, ready to merge anytime