Hi, I am trying to use geobeam to ingest a shapefile into BigQuery, and creating the table with a schema from the shapefile if the table does not exist. I came across few issues and questions.
I attempt this using a modified example shapefile_nfhl.py. And ran with this command.
python -m shapefile_nfhl --runner DataflowRunner --project my-project --temp_location gs://mybucket-geobeam/data --region australia-southeast1 --worker_harness_container_image gcr.io/dataflow-geobeam/example --experiment use_runner_v2 --service_account_email service-account@appspot.gserviceaccount.com --gcs_url gs://geobeam/examples/510104_20170217.zip --dataset examples --table output_table
Using get_bigquery_schema_dataflow() from geobeam.util is throwing error due to undefined variable.
NameError: name 'gcs_url' is not defined
I have opened a PR to fix this. #38
Once the function is fixed, it seems that it does not accept a shapefile. Passing in the GCS URL to the zipped shapefile is throwing this error.
Traceback (most recent call last):
File "fiona/_shim.pyx", line 83, in fiona._shim.gdal_open_vector
File "fiona/_err.pyx", line 291, in fiona._err.exc_wrap_pointer
fiona._err.CPLE_OpenFailedError: '/vsigs/geobeam/examples/510104_20170217.zip' not recognized as a supported file format.
Am I using the function in a wrong way or (zipped) shapefile is not support for this? For reference, this is the modified template. Thank you!
Hi, I am trying to use geobeam to ingest a shapefile into BigQuery, and creating the table with a schema from the shapefile if the table does not exist. I came across few issues and questions.
I attempt this using a modified example
shapefile_nfhl.py. And ran with this command.Using
get_bigquery_schema_dataflow()fromgeobeam.utilis throwing error due to undefined variable.NameError: name 'gcs_url' is not definedI have opened a PR to fix this. #38
Once the function is fixed, it seems that it does not accept a shapefile. Passing in the GCS URL to the zipped shapefile is throwing this error.
Am I using the function in a wrong way or (zipped) shapefile is not support for this? For reference, this is the modified template. Thank you!