create a spec for code generation

Author: tswast

.pre-commit-config.yaml

@@ -16,16 +16,16 @@
 # See https://pre-commit.com/hooks.html for more hooks
 repos:
 -   repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v4.0.1
+    rev: v6.0.0
     hooks:
     -   id: trailing-whitespace
     -   id: end-of-file-fixer
     -   id: check-yaml
 -   repo: https://github.com/psf/black
-    rev: 22.3.0
+    rev: 23.7.0
     hooks:
     - id: black
 -   repo: https://github.com/pycqa/flake8
-    rev: 3.9.2 # version-scanner: ignore
+    rev: 6.1.0 # version-scanner: ignore
     hooks:
     - id: flake8

packages/bigframes/bigframes/core/sentinels.py

@@ -0,0 +1,33 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Sentinel values used throughout BigFrames."""
+
+from __future__ import annotations
+
+from enum import Enum
+
+
+class Default(Enum):
+    """Default values used throughout BigFrames.
+
+    When a parameter is set to this, that parameter is explicitly omitted
+    from the SQL text. This allows for NULL (None in Python) to be explicitly
+    passed in to optional parameters.
+    """
+
+    token = 0
+
+
+DEFAULT = Default.token

packages/bigframes/specs/bigframes-bigquery-generator.md

@@ -0,0 +1,62 @@
+# Code generation for bigframes.bigquery
+
+This document describes code generation for the `bigframes.bigquery` modules.
+For detailed specifications on input and output types, refer to
+[Contributing to bigframes.bigquery](./bigframes-bigquery-contributing.md).
+
+## Overview
+
+The script at `packages/bigframes/scripts/generate_bigframes_bigquery.py`
+generates python submodules for the `bigframes.bigquery` module. When run
+without any arguments, it iterates through all yaml files at
+`packages/bigframes/scripts/data/sql-functions/**/*.yaml` to generate the code.
+
+The script at `packages/bigframes/scripts/check_bigframes_bigquery.py` iterates
+through all the same yaml files and checks that the functions have been included
+in the `bigframes.bigquery` module, as the `__init__.py` file requires manual
+updates.
+
+## Generated code organization
+
+The `generate_bigframes_bigquery.py` script generates submodules of
+`bigframes.bigquery._operations`, with the full path reflecting the organization
+of the YAML files. For example, a YAML file at
+`packages/bigframes/scripts/data/sql-functions/aead.yaml` corresponds to a
+generated Python module at `bigframes.bigquery._operations.aead`. Likewise,
+`packages/bigframes/scripts/data/sql-functions/builtins/bit.yaml` corresponds
+to the `bigframes.bigquery._operations.builtins.bit` submodule.
+
+## Generated module implementation
+
+Each generated module has all functions defined in the YAML file converted to
+the equivalent Python definition, including keyword arguments and docstrings.
+
+### Handling optional arguments
+
+When the user calls a Python function without specifying the optional
+argument, that argument is omitted from the SQL text. To allow for explicit
+NULL values to be passed in (None in Python), the default value is specified
+to be a default sentinel value enum `bigframes.core.sentinels.DEFAULT`. For
+example:
+
+```python
+import bigframes.core.sentinels
+
+def current_date(
+    time_zone_expression: str | bigframes.core.sentinels.Default = bigframes.core.sentinels.DEFAULT,
+):
+    ...
+```
+
+### Input and output types
+
+Refer to the table in
+[Contributing to bigframes.bigquery](./bigframes-bigquery-contributing.md).
+
+### Internal bigframes operator
+
+Scalar functions should generate an expression using the `GoogleSqlScalarOp`.
+This keeps the implementation as scalar SQL functions consistent.
+
+Aggregate, analytic, and table-valued functions currently require custom ops. As
+such, those functions are currently out of scope for this generator.