SVS-1233-SVS-1234-add-udf-and-workspace-provisionig

docs/USER_DATA_FILTER_MGMT.md

@@ -0,0 +1,55 @@
+# GD User Data Filter Management
+
+Tool which helps manage User Data Filters in a GoodData organization.
+
+User Data Filters can be created, updated, and deleted based on CSV input.
+
+## Usage
+
+The tool requires the following arguments on input:
+
+- `filepath` - a path to a csv file defining user data filters, their values, and target workspace
+- `ldm_column_name` - LDM column name
+- `maql_column_name` - MAQL column name in the form `{attribute/dataset.field}`
+
+Some other, _optional_, arguments are:
+
+- `-d | --delimiter` - column delimiter for the csv files. Use this to define how the csv is parsed. Default value is `,`
+- `-q | --quotechar` - quotation character used to escape special characters (such as the delimiter) within the column field value. Default value is `"` If you need to escape the quotechar itself, you have to embed it in quotechars and then double the quotation character (e.g.: `"some""string"` will yield `some"string`).
+- `-p | --profile-config` - optional path to GoodData profile config. If no path is provided, the default profiles file is used.
+- `--profile` - GoodData profile to use. If no profile is provided, `default` is used.
+
+Use the tool like so:
+
+```sh
+python scripts/user_data_filter_mgmt.py path/to/udfs.csv ldm_column_name maql_column_name
+```
+
+If you would like to define custom delimiters, use the tool like so:
+
+```sh
+python scripts/user_data_filter_mgmt.py path/to/udfs.csv ldm_column_name maql_column_name -d ","
+```
+
+To show the help for using arguments, call:
+
+```sh
+python scripts/user_data_filter_mgmt.py -h
+```
+
+## Input CSV file
+
+The input CSV file defines the user data filter values to be managed. All user data filters in all workspaces listed in the input will be overwritten based on the CSV content.
+
+Following format of the csv is expected:
+
+| workspace_id              | udf_id    | udf_value |
+| ------------------------- | --------- | --------- |
+| workspace_with_wdf_values | user_id_1 | 1         |
+| workspace_with_wdf_values | user_id_2 | 2         |
+
+Here, each `workspace_id` is the ID of the workspace where the user data filter applies.
+
+The `user_data_filter_id` identifies the specific User Data Filter you want to assign or update for the given workspace. Should be equal to the ID of the user the UDF is applied to.
+
+The `udf_value` field specifies the value to be set for that User Data Filter.

docs/WORKSPACE_MGMT.md

@@ -0,0 +1,58 @@
+# GD Workspace Management
+
+Tool which helps manage child workspace entities in an GoodData organization.
+
+Workspaces can be created, updated, and deleted. This includes applying Workspace Data Filter values, when provided in input.
+
+## Usage
+
+The tool requires the following argument on input:
+
+- `filepath` - a path to a csv file defining workspace entities, their relevant attributes, workspace data filter configuration, and isActive state
+
+Some other, _optional_, arguments are:
+
+- `-d | --delimiter` - column delimiter for the csv files. Use this to define how the csv is parsed. Default value is `,`
+- `-i | --inner-delimiter` - Workspace Data Filter values column delimiter. Use this to separate the different values defined in the `workspace_data_filter_values` column. Default value is `|`. Note that `--delimiter` and `--inner_delimiter` have to differ.
+- `-q | --quotechar` - quotation character used to escape special characters (such as the delimiter) within the column field value. Default value is `"` If you need to escape the quotechar itself, you have to embed it in quotechars and then double the quotation character (e.g.: `"some""string"` will yield `some"string`).
+- `-p | --profile-config` - optional path to GoodData profile config. If no path is provided, the default profiles file is used.
+- `--profile` - GoodData profile to use. If no profile is provided, `default` is used.
+
+Use the tool like so:
+
+```sh
+python scripts/workspace_mgmt.py path/to/workspace_definitions.csv
+```
+
+If you would like to define custom delimiters, use the tool like so:
+
+```sh
+python scripts/workspace_mgmt.py path/to/workspace_definitions.csv -d "," -i "|"
+```
+
+To show the help for using arguments, call:
+
+```sh
+python scripts/workspace_mgmt.py -h
+```
+
+## Input CSV file
+
+The input CSV file defines the workspace entities which you might want to manage. Note that GD organization workspaces that are not defined in the input will not be modified in any way.
+
+Following format of the csv is expected:
+
+| parent_id           | workspace_id                 | workspace_name               | workspace_data_filter_id | workspace_data_filter_values | is_active |
+| ------------------- | ---------------------------- | ---------------------------- | ------------------------ | ---------------------------- | --------- |
+| parent_workspace_id | workspace_with_wdf_values    | Workspace With WDF Values    | wdf_id                   | 1|2|3              | true      |
+| parent_workspace_id | workspace_without_wdf_values | Workspace Without WDF Values |                          |                              | true      |
+
+Here, each `workspace_id` is the ID of the workspace to manage.
+
+The `parent_id` specifies the parent workspace under which the workspace should be placed.
+
+The `workspace_name` field specifies the display name of the workspace.
+
+The `workspace_data_filter_id` and `workspace_data_filter_values` fields specify Workspace Data Filter configuration. Leave `workspace_data_filter_values` empty if no values should be set.
+
+Lastly, the `is_active` field holds boolean values containing information about whether the workspace should or should not exist in the organization.

requirements.txt

@@ -1,6 +1,6 @@
 # GoodData Python SDK packages
-gooddata_sdk>=1.51
-gooddata-pipelines>=1.51
+gooddata_sdk>=1.52
+gooddata-pipelines>=1.52
 
 # Other dependencies
 # TODO: remove after full transition to GoodData SDK packages

scripts/backup.py

@@ -534,7 +534,7 @@ def main(args: argparse.Namespace) -> None:
     process_batches_in_parallel(sdk, api, org_id, storage, batches)
 
 
-if __name__ == "__main__":
+def backup():
     parser: argparse.ArgumentParser = create_parser()
     args: argparse.Namespace = parser.parse_args()
 
@@ -545,3 +545,7 @@ def main(args: argparse.Namespace) -> None:
         logger.info("Backup completed!")
     except Exception as e:
         logger.error(f"Backup failed: {e}")
+
+
+if __name__ == "__main__":
+    backup()

scripts/custom_fields.py

@@ -11,21 +11,26 @@
 from custom_fields.custom_field_manager import (  # type: ignore[import]
     CustomFieldManager,
 )
+from utils.logger import get_logger, setup_logging  # type: ignore[import]
 from utils.utils import read_csv_file_to_dict  # type: ignore[import]
 
+setup_logging()
+logger = get_logger(__name__)
 
-def main(
-    path_to_custom_datasets_csv: str,
-    path_to_custom_fields_csv: str,
-    check_relations: bool,
-) -> None:
+
+def custome_fields() -> None:
     """Main function to run the custom fields script."""
     # Get host and token from environment variables
     # TODO: add option to load credentials from profile
     # TODO: (refactor) credentials should be handled in one place for the project
     host = os.environ.get("GDC_HOSTNAME")
     token = os.environ.get("GDC_AUTH_TOKEN")
 
+    args: argparse.Namespace = parse_args()
+    path_to_custom_datasets_csv = args.path_to_custom_datasets_csv
+    path_to_custom_fields_csv = args.path_to_custom_fields_csv
+    check_relations: bool = args.check_relations
+
     if not host:
         raise ValueError("GDC_HOSTNAME environment variable is not set.")
     if not token:
@@ -72,8 +77,4 @@ def parse_args():
 
 
 if __name__ == "__main__":
-    args: argparse.Namespace = parse_args()
-    path_to_custom_datasets_csv = args.path_to_custom_datasets_csv
-    path_to_custom_fields_csv = args.path_to_custom_fields_csv
-    check_relations: bool = args.check_relations
-    main(path_to_custom_datasets_csv, path_to_custom_fields_csv, check_relations)
+    custome_fields()

scripts/permission_mgmt.py

@@ -1,6 +1,5 @@
 # (C) 2025 GoodData Corporation
 import argparse
-import logging
 import os
 from pathlib import Path
 
@@ -10,18 +9,18 @@
     PermissionProvisioner,
 )
 from gooddata_sdk.utils import PROFILES_FILE_PATH
-from utils.logger import setup_logging  # type: ignore[import]
+from utils.logger import get_logger, setup_logging  # type: ignore[import]
 from utils.utils import (  # type: ignore[import]
     create_provisioner,
     read_csv_file_to_dict,
 )
 
+setup_logging()
+logger = get_logger(__name__)
+
 
 def create_parser() -> argparse.ArgumentParser:
     parser = argparse.ArgumentParser(description="Management of workspace permissions.")
-    parser.add_argument(
-        "-v", "--verbose", action="store_true", help="Turns on the debug log output."
-    )
     parser.add_argument(
         "perm_csv",
         type=Path,
@@ -109,13 +108,10 @@ def validate_args(args: argparse.Namespace) -> None:
         )
 
 
-if __name__ == "__main__":
+def permission_mgmt():
     parser = create_parser()
     args = parser.parse_args()
 
-    setup_logging(args.verbose)
-    logger = logging.getLogger(__name__)
-
     permissions = read_permissions_from_csv(args)
 
     permission_manager = create_provisioner(
@@ -125,3 +121,7 @@ def validate_args(args: argparse.Namespace) -> None:
     permission_manager.logger.subscribe(logger)
 
     permission_manager.incremental_load(permissions)
+
+
+if __name__ == "__main__":
+    permission_mgmt()

scripts/restore.py

@@ -591,17 +591,25 @@ def create_client(args: argparse.Namespace) -> tuple[GoodDataSdk, GDApi]:
     )
 
 
-def main(args):
-    """Main entry point of the script."""
-    if args.verbose:
-        logger.setLevel(logging.DEBUG)
-
+def validate_args(args: argparse.Namespace) -> None:
+    """Validates the arguments provided."""
     if not os.path.exists(args.ws_csv):
         raise RuntimeError("Invalid path to csv given.")
 
     if not os.path.exists(args.conf):
         raise RuntimeError("Invalid path to backup storage configuration given.")
 
+
+def restore():
+    """Main entry point of the script."""
+
+    parser = create_parser()
+    args = parser.parse_args()
+    validate_args(args)
+
+    if args.verbose:
+        logger.setLevel(logging.DEBUG)
+
     sdk, api = create_client(args)
 
     conf = BackupRestoreConfig(args.conf)
@@ -619,6 +627,4 @@ def main(args):
 
 
 if __name__ == "__main__":
-    parser = create_parser()
-    args = parser.parse_args()
-    main(args)
+    restore()