diff --git a/docs/source/citation-attribution.rst b/docs/source/citation-attribution.rst new file mode 100644 index 000000000..40a3f256a --- /dev/null +++ b/docs/source/citation-attribution.rst @@ -0,0 +1,96 @@ +.. _citation-attribution: + +###################### +Citation & Attribution +###################### + +This page provides guidelines for citing dsgrid and datasets produced using +the toolkit. + +Citing the dsgrid Toolkit +========================= + +When using the dsgrid toolkit in your research, please cite: + +.. code-block:: bibtex + + @software{dsgrid, + author = {NREL dsgrid Team}, + title = {dsgrid: Demand-Side Grid Toolkit}, + url = {https://github.com/dsgrid/dsgrid}, + institution = {National Renewable Energy Laboratory} + } + +Citing dsgrid Datasets +====================== + +Each published dataset has its own citation requirements. When using data from +a dsgrid project: + +1. **Cite the specific dataset** using the DOI provided in the + :doc:`published datasets catalog ` + +2. **Cite the dsgrid toolkit** (see above) + +3. **Acknowledge underlying data sources** as appropriate (e.g., ResStock, + ComStock, TEMPO) + +Example Dataset Citation +------------------------ + +.. code-block:: bibtex + + @techreport{dsgrid_standard_scenarios_2024, + title = {Standard Scenarios 2024: Hourly Electricity Load Profiles}, + author = {NREL dsgrid Team}, + institution = {National Renewable Energy Laboratory}, + year = {2024}, + doi = {10.XXXX/XXXXX} + } + +Acknowledging Underlying Data +============================= + +dsgrid datasets typically integrate data from multiple sources. When using +dsgrid data, please also acknowledge the underlying data sources: + +**ResStock** + Residential building energy model data. See + `ResStock documentation `_ for citation + requirements. + +**ComStock** + Commercial building energy model data. See + `ComStock documentation `_ for + citation requirements. + +**TEMPO** + Transportation electrification projections. Contact the dsgrid team for + citation requirements. + +**Other Sources** + Check the documentation for specific datasets for additional attribution + requirements. + +License +======= + +dsgrid is open-source software released under the BSD 3-Clause License. + +Published datasets may have their own licensing terms. Check the documentation +for each dataset for specific license information. + +Questions +========= + +For questions about citation or attribution: + +- :doc:`Contact the dsgrid team ` +- `GitHub Discussions `_ + +See Also +======== + +- :doc:`publications` - Related publications +- :doc:`data/published-datasets` - Available datasets +- :doc:`contact` - Get in touch diff --git a/docs/source/conf.py b/docs/source/conf.py index 7dcdd3440..30f909ab2 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -48,6 +48,7 @@ "sphinx_click", "sphinxcontrib.autodoc_pydantic", "sphinx_tabs.tabs", + "sphinx_design", ] # Add any paths that contain templates here, relative to this directory. @@ -111,5 +112,5 @@ autodoc_pydantic_model_show_json = False autodoc_pydantic_model_show_config_member = False autodoc_pydantic_model_show_config_summary = False -autodoc_pydantic_model_erdantic_figure = True +autodoc_pydantic_model_erdantic_figure = False autodoc_pydantic_model_erdantic_figure_collapsed = False diff --git a/docs/source/contact.rst b/docs/source/contact.rst new file mode 100644 index 000000000..ff96300aa --- /dev/null +++ b/docs/source/contact.rst @@ -0,0 +1,91 @@ +.. _contact: + +####### +Contact +####### + +This page provides information on how to get in touch with the dsgrid team. + +General Inquiries +================= + +For general questions about dsgrid: + +- Visit the `NREL dsgrid website `_ +- Email the dsgrid coordination team (contact information available on the NREL + website) + +Technical Support +================= + +For technical questions and issues: + +**GitHub Issues** + Report bugs, request features, or ask technical questions: + `dsgrid Issues `_ + +**GitHub Discussions** + For general technical discussions and questions: + `dsgrid Discussions `_ + +**Documentation** + Many questions are answered in the documentation: + + - :doc:`Getting Started ` - User-type specific guides + - :doc:`How-To Guides ` - Common workflows + - :doc:`Tutorials ` - Step-by-step examples + - :doc:`Fundamentals ` - Technical deep-dives + +Data Access +=========== + +For questions about accessing dsgrid data: + +- :doc:`Published Datasets ` - Available datasets +- :doc:`Current Projects ` - Active and completed projects +- :doc:`Getting Started for Data Users ` - Quick path to data + +Dataset Submission +================== + +For questions about contributing data to dsgrid: + +- :doc:`Getting Started for Dataset Submitters ` + - Submission workflow +- :doc:`Create and Submit Dataset Tutorial ` + - Step-by-step guide + +Contact your project coordinator for project-specific submission requirements. + +Collaboration +============= + +For collaboration opportunities: + +- **Dataset contributions**: If your organization has relevant datasets +- **Custom projects**: Tailored analyses for specific needs +- **Research partnerships**: Collaborative research and development + +Contact the dsgrid coordination team through the NREL website to discuss +opportunities. + +Feedback +======== + +We welcome feedback on: + +- Documentation improvements +- Feature requests +- Bug reports +- General suggestions + +Use `GitHub Issues `_ or +`GitHub Discussions `_ to +share your feedback. + +See Also +======== + +- :doc:`publications` - Related publications +- :doc:`citation-attribution` - How to cite dsgrid +- `NREL dsgrid website `_ diff --git a/docs/source/data/index.rst b/docs/source/data/index.rst new file mode 100644 index 000000000..b2beab6d6 --- /dev/null +++ b/docs/source/data/index.rst @@ -0,0 +1,113 @@ +.. _data: + +#### +Data +#### + +This section covers everything related to accessing, understanding, and working +with dsgrid data. + +.. grid:: 2 + :gutter: 3 + + .. grid-item-card:: Published Datasets + :link: published-datasets + :link-type: doc + + Browse the catalog of published dsgrid datasets available for download + and analysis. + + .. grid-item-card:: Querying Data + :link: querying-data + :link-type: doc + + Learn how to write queries to extract exactly the data you need from + dsgrid projects. + + .. grid-item-card:: Mapping Data + :link: mapping-data + :link-type: doc + + Transform data across dimensions, geographies, and time conventions. + + .. grid-item-card:: Input Datasets + :link: input-datasets/index + :link-type: doc + + Requirements and formats for contributing datasets to dsgrid projects. + +Data Overview +============= + +dsgrid manages multi-dimensional timeseries data describing electricity demand +and other energy metrics. Data is organized in a hierarchical structure: + +**Projects** + Top-level containers that define scope (geography, time period, sectors) + and coordinate multiple datasets. + +**Datasets** + Individual data contributions from various sources (models, surveys, + historical data) that are mapped to project dimensions. + +**Derived Datasets** + Created by combining, transforming, or querying project datasets. + +Data Dimensions +--------------- + +All dsgrid data is organized along these dimension types: + +- **Geography**: Spatial resolution (counties, states, regions, etc.) +- **Time**: Temporal resolution (hourly, daily, monthly, annual) +- **Sector**: Economic sectors (residential, commercial, industrial, transportation) +- **Subsector**: Detailed categorizations within sectors +- **Metric**: What is being measured (energy, power, emissions, etc.) +- **Scenario**: Future pathways or policy scenarios +- **Model Year**: Projection years for forward-looking analysis +- **Weather Year**: Historical weather patterns used in simulations + +See :doc:`Dimensions ` for detailed documentation. + +Data Access Patterns +==================== + +**For quick analysis:** + Download pre-computed datasets from the :doc:`published datasets ` + catalog. + +**For custom queries:** + Use the dsgrid CLI or Python API to :doc:`query ` specific data subsets. + +**For data contribution:** + Follow the :doc:`input datasets ` guide to prepare and + submit your data. + +Quick Links +=========== + +**Access Data:** + +- :doc:`Published Datasets ` - Download ready-to-use data +- :doc:`Current Projects ` - View active projects +- :doc:`Query Data ` - Extract custom subsets + +**Understand Data:** + +- :doc:`Data Formats ` - File format specifications +- :doc:`Dimensions ` - Dimension types and records +- :doc:`Datasets ` - Dataset structure + +**Contribute Data:** + +- :doc:`Input Datasets ` - Submission requirements +- :doc:`Dataset Submission Guide ` - Step-by-step workflow + +.. toctree:: + :maxdepth: 2 + :hidden: + + published-datasets + querying-data + mapping-data + input-datasets/index diff --git a/docs/source/data/input-datasets/index.rst b/docs/source/data/input-datasets/index.rst new file mode 100644 index 000000000..b138b7bb4 --- /dev/null +++ b/docs/source/data/input-datasets/index.rst @@ -0,0 +1,143 @@ +.. _input-datasets: + +############## +Input Datasets +############## + +This section covers everything you need to know about preparing and submitting +datasets to dsgrid projects. + +Overview +======== + +To contribute a dataset to dsgrid, you need to: + +1. **Format your data** according to dsgrid's data format specifications +2. **Define dimensions** that describe your dataset's structure +3. **Create configuration files** that document your dataset +4. **Validate** that your dataset meets all requirements +5. **Submit** to a project (with dimension mappings if required) + +.. grid:: 2 + :gutter: 3 + + .. grid-item-card:: Data Formats + :link: /reference/dataset_formats + :link-type: doc + + Learn about the one-table and two-table formats for dataset files. + + .. grid-item-card:: Dimensions + :link: /explanations/components/dimensions + :link-type: doc + + Understand dimension types and how to define dimension records. + + .. grid-item-card:: Dataset Configuration + :link: /how_tos/create_dataset_dimensions + :link-type: doc + + Create configuration files for your dataset. + + .. grid-item-card:: Submission Workflow + :link: /getting-started/for-dataset-submitters + :link-type: doc + + Step-by-step guide to the submission process. + +Dataset Requirements +==================== + +All dsgrid datasets must: + +**Define all dimension types:** + Every dataset must define a dimension for each of the dsgrid + :ref:`dimension types `. + Dimensions can be "trivial" (single record) if the dataset doesn't vary + along that dimension. + +**Use a supported data format:** + Data files must use one of the :doc:`supported formats `: + + - **One-table format** - All data in a single table + - **Two-table format** - Separates dimension records from values + +**Pass validation:** + Datasets must pass dsgrid's validation checks for: + + - Configuration file syntax + - Data format compliance + - Dimension record consistency + - Value completeness + +**Provide metadata:** + Datasets must include descriptive metadata: + + - Dataset ID (unique identifier) + - Name and description + - Data source information + - Dimension definitions + +Dimension Types +=============== + +Every dataset must define these dimensions: + +.. list-table:: + :header-rows: 1 + :widths: 20 40 40 + + * - Dimension Type + - Description + - Examples + * - Geography + - Spatial extent + - Counties, states, census regions + * - Time + - Temporal resolution + - Hourly 2018, annual 2020-2050 + * - Sector + - Economic sector + - Residential, commercial, industrial + * - Subsector + - Detailed categorization + - Building types, vehicle classes + * - Metric + - What is measured + - Energy consumption, peak demand + * - Scenario + - Future pathway + - Reference, high electrification + * - Model Year + - Projection year + - 2020, 2030, 2040, 2050 + * - Weather Year + - Weather pattern year + - 2012, TMY3 + +See :doc:`Dimensions ` for complete documentation. + +Getting Started +=============== + +**New to dataset submission?** + Start with the :doc:`Getting Started guide `. + +**Ready to create a dataset?** + Follow the :doc:`Create and submit dataset tutorial `. + +**Need format details?** + Review the :doc:`Data formats reference `. + +**Questions?** + :doc:`Contact the dsgrid team ` for assistance. + +Related Documentation +===================== + +- :doc:`Getting started for submitters ` +- :doc:`Create and submit dataset tutorial ` +- :doc:`Data formats reference ` +- :doc:`Dimensions ` +- :doc:`Dimension mappings ` +- :doc:`Apache Spark guide ` (for large datasets) diff --git a/docs/source/data/mapping-data.rst b/docs/source/data/mapping-data.rst new file mode 100644 index 000000000..bed757823 --- /dev/null +++ b/docs/source/data/mapping-data.rst @@ -0,0 +1,240 @@ +.. _mapping-data: + +############ +Mapping Data +############ + +This guide explains how dimension mappings work in dsgrid and how to use them +to transform data across different dimensional representations. + +Overview +======== + +Dimension mappings are the mechanism dsgrid uses to reconcile differences between +datasets. Because different data sources often use different: + +- Geographic boundaries (census divisions vs. counties vs. utility territories) +- Sector classifications (EIA vs. NREL building types) +- Time conventions (different weather years, time zones) + +dsgrid provides explicit, validated mappings to transform data consistently. + +Why Mappings Matter +=================== + +Consider combining two datasets: + +- **Dataset A**: Residential load by census division, annual +- **Dataset B**: Commercial load by county, hourly + +To combine these, we need mappings to: + +1. Disaggregate census divisions to counties +2. Disaggregate annual data to hourly (if needed) + +dsgrid makes these mappings explicit, documented, and programmatically validated. + +Types of Mappings +================= + +Dataset-to-Project Mappings +--------------------------- + +Transform a dataset's dimensions to match a project's base dimensions: + +.. code-block:: text + + Dataset Dimension → Project Base Dimension + ───────────────────────────────────────────── + Census Division 9 → CA, OR, WA, NV, AZ (with weights) + "Single Family" → "single_family_detached" + 2018 Weather Year → 2018 (pass-through) + +These mappings are defined when submitting a dataset to a project. + +Base-to-Supplemental Mappings +----------------------------- + +Transform a project's base dimensions to supplemental (alternate) dimensions: + +.. code-block:: text + + Base Dimension → Supplemental Dimension + ────────────────────────────────────────── + County FIPS → State abbreviation + Hourly → Monthly + Building type → Sector only + +These mappings are defined when creating a project and used in queries. + +Mapping Operations +================== + +One-to-One +---------- + +Simple relabeling or renaming: + +.. code-block:: text + + from_id to_id + ─────────────────── + res residential + com commercial + Calif CA + +Many-to-One (Aggregation) +------------------------- + +Combine multiple records into one: + +.. code-block:: text + + from_id to_id + ─────────────────── + 06001 CA # Alameda County → California + 06003 CA # Alpine County → California + 06005 CA # Amador County → California + +One-to-Many (Disaggregation) +---------------------------- + +Split one record into multiple, with weighting: + +.. code-block:: text + + from_id to_id from_fraction + ────────────────────────────────────────── + Census_Div_9 CA 0.50 + Census_Div_9 OR 0.20 + Census_Div_9 WA 0.25 + Census_Div_9 AZ 0.03 + Census_Div_9 NV 0.02 + +The ``from_fraction`` column specifies how to distribute values. + +Creating Mapping Files +====================== + +Mapping files are CSV files with specific columns: + +**Simple mapping (one-to-one or many-to-one):** + +.. code-block:: text + + from_id,to_id + res,residential + com,commercial + ind,industrial + +**Weighted mapping (one-to-many):** + +.. code-block:: text + + from_id,to_id,from_fraction + Census_Div_9,CA,0.50 + Census_Div_9,OR,0.20 + Census_Div_9,WA,0.30 + +**Bidirectional mapping:** + +.. code-block:: text + + from_id,to_id,from_fraction,to_fraction + A,X,0.5,0.4 + A,Y,0.5,0.6 + B,X,0.3,0.6 + B,Y,0.7,0.4 + +Mapping Validation +================== + +dsgrid validates mappings to ensure: + +- **Completeness**: All source records have mappings +- **Consistency**: Fractions sum to 1.0 for disaggregation +- **Validity**: Target records exist in target dimension +- **No orphans**: No unmapped records in source or target + +Validation happens during: + +- Mapping registration +- Dataset submission +- Query execution + +Common Mapping Scenarios +======================== + +Geographic Disaggregation +------------------------- + +Distribute regional data to finer geography using population or other weights: + +.. code-block:: text + + # Region to state mapping with population weights + from_id,to_id,from_fraction + Northeast,CT,0.10 + Northeast,MA,0.19 + Northeast,ME,0.04 + Northeast,NH,0.04 + Northeast,NJ,0.26 + Northeast,NY,0.26 + Northeast,PA,0.08 + Northeast,RI,0.03 + Northeast,VT,0.02 + +Time Resolution Changes +----------------------- + +Map between different temporal resolutions: + +.. code-block:: text + + # Annual to monthly (equal distribution) + from_id,to_id,from_fraction + 2030,2030-01,0.0833 + 2030,2030-02,0.0833 + 2030,2030-03,0.0833 + ... + +Sector Harmonization +-------------------- + +Map between different sector taxonomies: + +.. code-block:: text + + # EIA to dsgrid sectors + from_id,to_id + Residential,residential + Commercial,commercial + Industrial,industrial + Transportation,transportation + Other,other + +Best Practices +============== + +**Document your mappings:** + Include comments explaining the rationale for mapping decisions. + +**Use appropriate weights:** + For disaggregation, use relevant metrics (population, floor area, etc.). + +**Validate before use:** + Test mappings with sample data to verify expected behavior. + +**Consider reversibility:** + Think about whether mappings can be reversed if needed. + +**Handle edge cases:** + Explicitly map all records, including "other" or "unknown" categories. + +Related Documentation +===================== + +- :doc:`Dimension mappings concept ` +- :doc:`Map a dataset tutorial ` +- :doc:`Getting started for mappers ` +- :doc:`Dimensions ` diff --git a/docs/source/data/published-datasets.rst b/docs/source/data/published-datasets.rst new file mode 100644 index 000000000..b245e24c8 --- /dev/null +++ b/docs/source/data/published-datasets.rst @@ -0,0 +1,197 @@ +.. _published-datasets: + +#################### +Published Datasets +#################### + +This page catalogs dsgrid datasets available for public use. All datasets are +validated, documented, and citable. + +.. note:: + + This page will be populated as datasets are formally published. Check back + regularly for updates, or :doc:`contact the dsgrid team ` for + information about upcoming releases. + +Available Datasets +================== + +.. note:: + + **Coming Soon** + + The dsgrid team is actively working on publishing datasets. Information + about available datasets will be added here as they become available. + + In the meantime: + + - :doc:`Contact the team ` for early access or collaboration + - Check :doc:`current projects ` for ongoing work + - Review the :doc:`tutorials ` using example data + +Data Access Guide +================= + +When datasets are published, they will be accessible through multiple methods: + +Download Pre-Computed Data +-------------------------- + +Published datasets will provide pre-computed outputs in multiple formats: + +**Available formats:** + +- **Parquet**: Efficient for large datasets, works with Python/R +- **CSV**: Universal compatibility, good for smaller subsets + +**Typical structure:** + +.. code-block:: text + + dataset_name/ + ├── load_data/ # Time series data + │ ├── by_county/ # Geographic partitioning + │ └── by_sector/ # Sector partitioning + ├── dimensions/ # Dimension metadata + ├── metadata.json # Dataset metadata + └── README.md # Documentation + +Query Datasets Directly +------------------------ + +For custom analysis, query datasets directly using the dsgrid toolkit: + +.. code-block:: python + + from dsgrid import RegistryManager + + registry = RegistryManager.load("/path/to/registry") + project = registry.get_project("project-id") + + # Run a query + results = project.run_query(query_config) + results.to_pandas() + +:doc:`Full query guide ` + +Work with Data +-------------- + +**Python example:** + +.. code-block:: python + + import pandas as pd + + # Load Parquet file + df = pd.read_parquet("load_data/by_county/county_001.parquet") + + # Basic operations + annual_load = df.groupby("year")["value"].sum() + peak_load = df.groupby("county")["value"].max() + +**R example:** + +.. code-block:: r + + library(arrow) + + # Load Parquet file + df <- read_parquet("load_data/by_county/county_001.parquet") + + # Basic operations + annual_load <- aggregate(value ~ year, df, sum) + peak_load <- aggregate(value ~ county, df, max) + +Understanding the Data +======================= + +Data Structure +-------------- + +dsgrid datasets are multi-dimensional: + +- **Geography**: Counties, states, utility territories, etc. +- **Time**: Hour of year, time-of-day, season, year +- **Sector**: Residential, commercial, transportation, industrial +- **Subsector**: Building types, vehicle types, etc. +- **End-use**: Heating, cooling, appliances, etc. +- **Scenario**: Reference, high electrification, policy scenarios +- **Metric**: Energy consumption, power demand, etc. + +:doc:`Learn more about dimensions ` + +Data Quality +------------ + +All published datasets undergo: + +- Automated validation checks +- Peer review by project coordinator +- Comparison with historical data where available +- Documentation of known limitations + +Units and Conventions +--------------------- + +**Standard units:** + +- Energy: Kilowatt-hours (kWh) or Megawatt-hours (MWh) +- Power: Kilowatts (kW) or Megawatts (MW) +- Time: Hours (local time) + +**Time conventions:** + +- Hourly data represents the average power during that hour +- Hour 1 = 12:00 AM - 1:00 AM +- All times in local time (adjusted for timezone) + +**Geographic conventions:** + +- FIPS codes for counties +- State abbreviations for states +- Custom geography defined in project + +Dataset Updates +=============== + +Published datasets follow these update patterns: + +- **Major releases**: Comprehensive updates with new model runs +- **Minor updates**: Bug fixes and data corrections +- **Documentation updates**: Improved metadata and guidance + +Subscribe to the dsgrid mailing list or watch the GitHub repository for +release announcements. + +Citation & Attribution +====================== + +When using dsgrid datasets: + +1. Cite the specific dataset (DOI provided with each release) +2. Cite the dsgrid toolkit +3. Acknowledge underlying data sources (ResStock, ComStock, etc.) +4. Follow any additional attribution requirements + +:doc:`Complete citation guide ` + +Support +======= + +**Questions about data:** + :doc:`Contact the dsgrid team ` + +**Technical issues:** + `GitHub Issues `_ + +**General questions:** + `GitHub Discussions `_ + +See Also +======== + +- :doc:`Query data ` - Write custom queries +- :doc:`Data formats ` - Understand data structure +- :doc:`Current projects ` - Active dsgrid work +- :doc:`Data user guide ` - Getting started diff --git a/docs/source/data/querying-data.rst b/docs/source/data/querying-data.rst new file mode 100644 index 000000000..4b75cbea3 --- /dev/null +++ b/docs/source/data/querying-data.rst @@ -0,0 +1,306 @@ +.. _querying-data: + +############## +Querying Data +############## + +This guide explains how to query dsgrid projects and datasets to extract +exactly the data you need for your analysis. + +Overview +======== + +dsgrid queries allow you to: + +- **Filter** data by specific dimension records (e.g., only California, only 2030) +- **Aggregate** across dimensions (e.g., sum all sectors, average across hours) +- **Map** to supplemental dimensions (e.g., different geographic resolutions) +- **Export** results in various formats (Parquet, CSV, pandas DataFrame) + +Query Types +=========== + +Project Queries +--------------- + +Query across all datasets in a project, with automatic dimension mapping: + +.. code-block:: console + + $ dsgrid query project run query_config.json5 + +Project queries: + +- Apply dataset-to-project mappings automatically +- Combine data from multiple datasets +- Support supplemental dimension mappings +- Handle aggregation and filtering + +See :doc:`Query a project tutorial ` for examples. + +Dataset Queries +--------------- + +Query a single dataset directly: + +.. code-block:: console + + $ dsgrid query dataset run query_config.json5 + +Dataset queries: + +- Work with the dataset's native dimensions +- Don't require project context +- Useful for exploring individual data sources + +See :doc:`Query a dataset tutorial ` for examples. + +Query Configuration +=================== + +Queries are defined in JSON5 configuration files: + +.. code-block:: json5 + + { + // Project or dataset to query + "project_id": "standard-scenarios-2024", + + // Dimension filtering + "filters": { + "geography": ["CA", "OR", "WA"], + "model_year": ["2030", "2040"], + "sector": ["residential", "commercial"] + }, + + // Dimension aggregation + "aggregations": { + "subsector": "sum", + "time": "sum" + }, + + // Output configuration + "output": { + "format": "parquet", + "path": "results/" + } + } + +Query Components +================ + +Filtering +--------- + +Limit results to specific dimension records: + +.. code-block:: json5 + + "filters": { + // Include only these states + "geography": ["CA", "TX", "NY"], + + // Include only these years + "model_year": ["2030"], + + // Include only residential sector + "sector": ["residential"] + } + +Aggregation +----------- + +Combine data across dimension records: + +.. code-block:: json5 + + "aggregations": { + // Sum across all subsectors + "subsector": "sum", + + // Sum across time (e.g., annual totals) + "time": "sum", + + // Keep geography disaggregated (don't aggregate) + "geography": null + } + +Supported aggregation methods: + +- ``sum``: Add values together +- ``mean``: Calculate average +- ``max``: Maximum value +- ``min``: Minimum value + +Dimension Mapping +----------------- + +Map to supplemental dimensions defined in the project: + +.. code-block:: json5 + + "dimension_mapping": { + // Map counties to states + "geography": "state_level", + + // Map hourly to monthly + "time": "monthly" + } + +Running Queries +=============== + +Using the CLI +------------- + +.. code-block:: console + + # Query a project + $ dsgrid query project run my_query.json5 -o results/ + + # Query a dataset + $ dsgrid query dataset run my_query.json5 -o results/ + +Using Python +------------ + +.. code-block:: python + + from dsgrid import RegistryManager + + # Load registry + registry = RegistryManager.load("/path/to/registry") + + # Get project + project = registry.get_project("standard-scenarios-2024") + + # Run query + results = project.run_query("my_query.json5") + + # Convert to pandas + df = results.to_pandas() + +Query Examples +============== + +Example 1: State-Level Annual Load +----------------------------------- + +Get annual electricity load by state for 2030: + +.. code-block:: json5 + + { + "project_id": "standard-scenarios-2024", + "filters": { + "model_year": ["2030"], + "metric": ["electricity"] + }, + "aggregations": { + "time": "sum", // Annual totals + "subsector": "sum", // Aggregate subsectors + "sector": null // Keep sectors separate + }, + "dimension_mapping": { + "geography": "state_level" + } + } + +Example 2: Hourly Load Profile +------------------------------- + +Get hourly load profile for a specific region: + +.. code-block:: json5 + + { + "project_id": "standard-scenarios-2024", + "filters": { + "geography": ["CA"], + "model_year": ["2030"], + "scenario": ["reference"] + }, + "aggregations": { + "subsector": "sum", + "sector": "sum" + } + // Keep time disaggregated for hourly profile + } + +Example 3: Peak Load by County +------------------------------- + +Find peak hourly load for each county: + +.. code-block:: json5 + + { + "project_id": "standard-scenarios-2024", + "filters": { + "model_year": ["2030"] + }, + "aggregations": { + "sector": "sum", + "subsector": "sum", + "time": "max" // Peak hour + } + } + +Performance Considerations +========================== + +Large Queries +------------- + +For queries returning large amounts of data: + +- Use Apache Spark for processing +- Consider filtering to reduce data volume +- Aggregate where possible +- Use Parquet output format + +See :doc:`Apache Spark guide ` for configuration. + +Query Optimization +------------------ + +**Filter early:** + Apply filters to reduce data volume before aggregation. + +**Aggregate where possible:** + Aggregating reduces output size and processing time. + +**Use appropriate output format:** + Parquet is more efficient than CSV for large results. + +Troubleshooting +=============== + +Common Issues +------------- + +**Query runs out of memory:** + +- Increase Spark executor memory +- Add more filters to reduce data volume +- Aggregate more dimensions + +**Query takes too long:** + +- Check Spark configuration +- Increase parallelism (shuffle partitions) +- Consider pre-computing results + +**Dimension not found:** + +- Check dimension names match project/dataset +- Verify supplemental dimension is defined + +Related Documentation +===================== + +- :doc:`Query project tutorial ` +- :doc:`Query dataset tutorial ` +- :doc:`Dimensions ` +- :doc:`Apache Spark ` +- :doc:`CLI reference ` diff --git a/docs/source/explanations/components/projects.rst b/docs/source/explanations/components/projects.rst index 55d11d7cf..b7a51e775 100644 --- a/docs/source/explanations/components/projects.rst +++ b/docs/source/explanations/components/projects.rst @@ -18,8 +18,8 @@ defined both at the project and dataset levels. More specifically, there are thr dimensions used in dsgrid. - ``Base Dimension``: Base Dimensions are the core dimensions of a project. They define the - project's dimension expectations. All datasets must adhere to the projet's base dimension - definition or provide an appropriate :ref:`dimension_mapping_overview` to the base dimension. + project's dimension expectations. All datasets must adhere to the project's base dimension + definition or provide an appropriate :doc:`dimension mapping ` to the base dimension. - ``Supplemental Dimension``: These are supplemental dimensions defined by the project to support different queries and data aggregations or disaggregations. diff --git a/docs/source/fundamentals/apache-spark.rst b/docs/source/fundamentals/apache-spark.rst new file mode 100644 index 000000000..58cd51567 --- /dev/null +++ b/docs/source/fundamentals/apache-spark.rst @@ -0,0 +1,765 @@ +.. _apache-spark: + +************ +Apache Spark +************ + +This page describes how to set up and use Apache Spark with dsgrid. + +.. contents:: On This Page + :local: + :depth: 2 + +When Do I Need Spark? +===================== + +Before diving into Spark configuration, determine whether you actually need it. + +You Likely DO Need Spark If +--------------------------- + +- **Your dataset is > 10 GB**: Large datasets require distributed processing +- **Complex dimension mappings**: Operations that disaggregate or duplicate data + (e.g., census divisions to counties) can explode data sizes +- **Hourly data across many years**: Multi-year, sub-hourly data at fine geographic + resolution creates very large tables +- **Multi-dataset queries**: Joining multiple large datasets in a project +- **Creating derived datasets**: Combining large source datasets with transformations + +You Likely DON'T Need Spark If +------------------------------- + +- **Your dataset is < 5 GB**: Small datasets work fine with pandas +- **Validating configuration files**: Config validation is fast without Spark +- **Development and testing**: Use sample data that fits in memory +- **Accessing pre-computed data**: Published datasets don't require Spark to use +- **Simple, single-dataset queries**: Small queries on single datasets + +Decision Flowchart +------------------ + +.. code-block:: text + + Start + │ + ▼ + Dataset > 10 GB? ─── Yes ──▶ Use Spark + │ + No + │ + ▼ + Complex mappings ─── Yes ──▶ Use Spark + (disaggregation)? + │ + No + │ + ▼ + Multi-dataset ────── Yes ──▶ Consider Spark + query? + │ + No + │ + ▼ + Use pandas (no Spark needed) + +**If unsure**: Start without Spark. If operations fail with memory errors or +take too long, switch to Spark. + +Quick Start +=========== + +**On your laptop (local mode):** + +.. code-block:: console + + $ pip install pyspark + $ pyspark # Interactive session + +**On NREL HPC (Kestrel):** + +.. code-block:: console + + $ salloc -t 01:00:00 -N1 --account=dsgrid --partition=nvme --mem=240G + $ pip install "sparkctl[pyspark]" + $ sparkctl configure --start + $ export SPARK_CONF_DIR=$(pwd)/conf + $ export JAVA_HOME=/datasets/images/apache_spark/jdk-21.0.7 + +Windows Users +============= + +Spark does not offer a great user experience in Windows. While ``pyspark`` and +``spark-submit`` work in local mode, running a cluster requires manual configuration. +The developers provide cluster management scripts in bash, and so they do not work +in Windows. When running a cluster on your laptop we recommend that you use dsgrid +in a Windows Subsystem for Linux (WSL2) environment instead of a native Windows +environment. + +If you are running in local mode, you will need Hadoop's ``winutils.exe`` because +windows doesn't support HDFS. If you don't have winutils.exe installed, you will +need to download the winutils.exe and hadoop.dll files from +https://github.com/steveloughran/winutils (select the Hadoop version you are using +as winutils are specific to Hadoop versions). Then copy them into a folder like +``C:\hadoop\bin``, set the environment variable ``HADOOP_HOME`` to, e.g., +``C:\hadoop``, and add ``%HADOOP_HOME%\bin`` to your ``PATH``. + +If you get an error like: ``Python was not found; run without arguments to install +from the Microsoft Store, or disable this shortcut from Settings > Manage App +Execution Aliases.`` try setting this environment variable: ``PYSPARK_PYTHON=python`` +(or set the value to ipython, if you would prefer). + +Conventions +=========== + +This page uses the UNIX conventions for environment variables and running commands +in a shell. If you do run any of these commands in a native Windows environment, +you will have to adjust depending on whether you are running PowerShell or the old +Command shell. + +Explanation of bash functionality used here: + +- ``$VARIABLE_NAME`` or ``${VARIABLE_NAME}``: This uses the environment variable + ``VARIABLE_NAME``. If the variable isn't defined, bash will use an empty string. +- ``export VARIABLE_NAME=x``: This sets the environment variable ``VARIABLE_NAME`` + to the value ``x`` in the current shell. If you want to set this variable for all + future shells as well, add the statement to your shell rc file (``$HOME/.bashrc`` + or ``$HOME/.zshrc``). +- ``export VARIABLE_NAME=$(hostname)``: This sets the environment variable + ``VARIABLE_NAME`` to the string returned by the command ``hostname``. + +In examples listed on this page where you need to enter text yourself, that text is +enclosed in ``<>`` as in ``http://:4040``. + + +Spark Overview +============== + +Cluster Modes +------------- + +Apache provides an overview at https://spark.apache.org/docs/latest/cluster-overview.html + +The most important parts to understand are how dsgrid uses different cluster modes +in different environments. + +- **Local computer in local mode**: All Spark components run in a single process. + This is great for testing and development but is not performant. It will not use + all CPUs on your system. You run in this mode when you type ``pyspark`` in your + terminal. + +- **Local computer with a standalone cluster**: You must install Spark and then + manually start the cluster. Refer to the `Laptop`_ installation instructions. + This enables full performance on your system. It also allows you to debug your + jobs in the Spark UI before or after they complete. + +- **HPC compute node in local mode**: Use this only for quick checks. Same points + above for local computer in local mode apply. Create a standalone cluster for + real work. If you use this, set the environment variable + ``SPARK_LOCAL_DIRS=/tmp/scratch`` so that you have enough space. + +- **HPC compute node(s) with a standalone cluster**: Create a cluster on any number + of compute nodes and then use all CPUs for your jobs. Refer to the `HPC`_ + installation instructions. + +- **AWS EMR cluster**: The EMR scripts in the dsgrid repo at ``/emr`` will create a + Spark cluster on EC2 compute nodes with a cluster manager. The cluster manager + allows multiple users to access a single cluster and offers better job scheduling. + Refer to the README.md in that directory. + + +Run Spark Applications +---------------------- + +There are three ways of running Spark applications in Python. ``spark-submit`` and +``pyspark``, provided by the Spark and pyspark installations, are recommended +because they allow you to fully customize the execution environment. More details +follow in `Tuning Spark Configuration Settings`_. + +1. ``spark-submit`` + + This command will create a SparkSession, make that session available to the + Python process, and run your code. + + .. code-block:: console + + $ spark-submit [options] your_script.py [your_script_options] + +2. ``pyspark`` + + This command will create a SparkSession, make that session available to the + Python process, and leave you in the Python interpreter for an interactive + session. + + .. code-block:: console + + $ pyspark [options] + +3. Inside Python + + .. code-block:: python + + from pyspark.sql import SparkSession + spark = SparkSession.builder.appName("your_app_name").getOrCreate() + +Run pyspark through IPython or Jupyter +-------------------------------------- + +You can configure ``pyspark`` to start ``IPython`` or ``Jupyter`` instead of the +standard Python interpreter by setting the environment variables +``PYSPARK_DRIVER_PYTHON`` and ``PYSPARK_DRIVER_PYTHON_OPTS``. + +IPython +~~~~~~~ + +.. code-block:: console + + $ export PYSPARK_DRIVER_PYTHON=ipython + +Local mode: + +.. code-block:: console + + $ pyspark + +Cluster mode: + +.. code-block:: console + + $ pyspark --master=spark://$(hostname):7077 + +Now you are in IPython instead of the standard Python interpreter. + +Jupyter +~~~~~~~ + +.. code-block:: console + + $ export PYSPARK_DRIVER_PYTHON=jupyter + $ export PYSPARK_DRIVER_PYTHON_OPTS="notebook --no-browser --port=8889 --ip=0.0.0.0" + $ export SPARK_HOME=$(python -c "import pyspark;print(pyspark.__path__[0])") + +Local mode: + +.. code-block:: console + + $ pyspark + +Cluster mode: + +.. code-block:: console + + $ pyspark --master=spark://$(hostname):7077 + +Pyspark will start a Jupyter notebook and you'll see the URL printed in the +terminal. If you're on a remote server, like in an HPC environment, you'll need +to create an ssh tunnel in order to connect in a browser. + +Once you connect in a browser, enter the following in a cell in order to connect +to this cluster: + +.. code-block:: python + + from pyspark.sql import SparkSession + spark = SparkSession.builder.appName("your_app_name").getOrCreate() + +Spark UI +-------- + +The Spark master starts a web application at ``http://:8080``. +Job information is available at port 4040. You can monitor and debug all aspects +of your jobs in this application. You can also inspect all cluster configuration +settings. + +If your Spark cluster is running on a remote system, like an HPC, you may need +to open an ssh tunnel to the master node. Here is how to do that on NREL's +Kestrel cluster. + +On your laptop: + +.. code-block:: console + + $ export COMPUTE_NODE= + $ ssh -L 4040:$COMPUTE_NODE:4040 -L 8080:$COMPUTE_NODE:8080 $USER@kestrel.hpc.nrel.gov + + +Installing a Spark Standalone Cluster +===================================== + +The next sections describe how to install a standalone cluster on a local system +and an HPC. + +Laptop +------ + +.. note:: As stated earlier, the scripts mentioned in this section do not work in + a native Windows environment. You can still start a cluster in Windows; you + just have to run the java commands yourself. + +Download your desired version from https://spark.apache.org/downloads.html and +extract it on your system. + +.. note:: Choose the version that matches dsgrid's pyspark dependency. Major, + minor, and patch versions must match. + +.. code-block:: console + + $ cd # directions below assume $HOME + $ wget https://dlcdn.apache.org/spark/spark-3.3.1/spark-3.3.1-bin-hadoop3.tgz + $ tar -xzf spark-3.3.1-bin-hadoop3.tgz && rm spark-3.3.1-bin-hadoop3.tgz + + +The full instructions to create a cluster are at +http://spark.apache.org/docs/latest/spark-standalone.html. The rest of this +section documents the requirements for dsgrid. + +Set environment variables +~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: console + + $ export SPARK_HOME=$HOME/spark-3.3.1-bin-hadoop3 + $ export PATH=$PATH:$SPARK_HOME/sbin + + +Note that after doing this your system will have two versions of ``pyspark``: + +- In your Python virtual environment where you installed dsgrid (because dsgrid + installs pyspark) +- In ``$HOME/spark-3.3.1-bin/hadoop3/bin`` + +If you use a conda virtual environment, when that environment is activated, its +``pyspark`` will be in your system path. Be sure not to add the spark bin +directory to your path so that there are no collisions. + +**Warning**: Setting ``SPARK_HOME`` will affect operation of your Python +``pyspark`` installation in local mode. That may not be what you want if you +make settings specific to the standalone cluster. + +Customize Spark configuration settings +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: console + + $ cp $SPARK_HOME/conf/spark-defaults.conf.template $SPARK_HOME/conf/spark-defaults.conf + $ cp $SPARK_HOME/conf/spark-env.sh.template $SPARK_HOME/conf/spark-env.sh + +Set ``spark.driver.memory`` and ``spark.driver.maxResultSize`` in +``spark-defaults.conf`` to the maximum data sizes that you expect to pull from +Spark to Python, such as if you call ``df.toPandas()``. ``1g`` is probably +reasonable. + +Set ``spark.sql.shuffle.partitions`` to 1-4x the number of cores in your system. +Note that the default value is 200, and you probably don't want that. + +Set ``spark.executor.cores`` and ``spark.executor.memory`` to numbers that allow +creation of your desired number of executors. Spark will try to create the most +number of executors such that each executor has those resources. For example, if +your system has 16 cores and you assign 16g of memory to the worker (see below), +``spark.executor.cores 3`` and ``spark.executor.memory 5g`` will result in 3 +executors. + +Start the Spark processes +~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: console + + $ $SPARK_HOME/sbin/start-master.sh + +Start a worker with this command. Give the worker as much memory as you can +afford. Executor memory comes from this pool. You can also configure this in +``spark-env.sh``. + +.. code-block:: console + + $ $SPARK_HOME/sbin/start-worker.sh -m 16g spark://$(hostname):7077 + +If you add the ``sbin`` to your ``PATH`` environment variable, here is a +one-liner: + +.. code-block:: console + + $ start-master.sh && start-worker.sh -m 24g spark://$(hostname):7077 + +Stop the Spark processes +~~~~~~~~~~~~~~~~~~~~~~~~ + +Stop all of the processes when you complete your work. + +.. code-block:: console + + $ stop-worker.sh && stop-master.sh + +.. _spark-on-hpc: + +HPC +--- + +This section describes how you can run Spark jobs on any number of HPC compute +nodes. The scripts and examples described here rely on the SLURM scheduling +system and have been tested on NREL's Kestrel cluster. + +Install the Python package ``sparkctl`` to run the scripts described below. You +can install it with this command: + +.. code-block:: console + + $ pip install "sparkctl[pyspark]" + +The `sparkctl documentation `_ has generic +instructions to run Spark in a variety of ways. The rest of this section calls +out choices that you should make to run Spark jobs with dsgrid. + +1. Choose compute node(s) with fast local storage. This example will allocate + one node. Refer to the `Kestrel documentation + `_ + for more information for this type of compute node. + + .. code-block:: console + + $ salloc -t 01:00:00 -N1 --account=dsgrid --partition=nvme --mem=240G + +2. Configure Spark parameters based on the amount of memory and CPU in each + compute node. + + This command must be run on a compute node. The script will check for the + environment variable ``SLURM_JOB_ID``, which is set by ``Slurm``. If you + ssh'd into the compute node, it won't be set and then you have to pass it + as an argument. + + .. code-block:: console + + $ sparkctl configure --start + + Run ``sparkctl configure --help`` to see all options. + + Alternatively, or in conjunction with the above command, customize the Spark + configuration files in ``./conf`` as necessary. + +3. Ensure that the dsgrid application uses the Spark configuration that you + just defined. + + .. code-block:: console + + $ export SPARK_CONF_DIR=$(pwd)/conf + $ export JAVA_HOME=/datasets/images/apache_spark/jdk-21.0.7 + +4. Follow the rest of the HPC instructions in the sparkctl documentation. + + +Tuning Spark Configuration Settings +----------------------------------- + +In general you want to run Spark with as many executors as possible on each +worker node. The Amazon orchestration software along with the cluster manager +*may* take care of that when running on AWS (you will still have to adjust +``spark.sql.shuffle.partitions``). You will have to perform more customizations +when running a standalone cluster on your laptop or an HPC. + +There are multiple ways of setting parameters. These are listed in order of +priority---later methods will override the earlier methods when allowed. + +1. **Global Spark configuration directory**: This is ``$SPARK_HOME/conf`` or + ``$SPARK_CONF_DIR``. You can customize settings in ``spark-defaults.conf`` + and ``spark-env.sh``. Make customizations here if you will use the same + settings in all jobs. + +2. **Spark launch scripts**: Use ``spark-submit`` to run scripts. Use ``pyspark`` + to run interactively. Both scripts offer the same startup options. You can + choose to run in local mode or attach to a cluster. You can override any + setting from #1. Make changes here if you will use different settings across + jobs. Note that some settings must be made before the Spark JVM starts, like + ``spark.driver.memory``, and so this is your last chance to customize those + values. + +3. **SparkSession construction inside a Python process**: You can customize + things like executor settings when you construct the ``SparkSession`` in + Python. For example, this code block will create a session where the job + starts a single executor with a single core that uses all available memory. + + .. code-block:: python + + from pyspark import SparkConf + from pyspark.sql import SparkSession + + conf = SparkConf().setAppName("my_app") + conf.set("spark.executor.cores", 1) + conf.set("spark.executor.memory", "16g") + conf.setMaster(cluster) + spark = SparkSession.builder.config(conf=conf).getOrCreate() + +4. **Dynamic changes**: You can make changes to a limited number of settings + at runtime. You can't change the number of executor cores because those + have already been allocated. You can change the number of shuffle partitions + that Spark will use. You may want to change that value if the sizes of the + dataframes you're working on change dramatically. + + .. code-block:: python + + from pyspark.sql import SparkSession + + spark = SparkSession.getActiveSession() + spark.conf.set("spark.sql.shuffle.partitions", 500) + +Creating a SparkSession with dsgrid +=================================== + +Ensure that the dsgrid software uses the cluster with optimized settings. If +you start the dsgrid Python process with the Spark scripts ``spark-submit`` or +``pyspark`` and set the ``--master`` option, those scripts will create a +SparkSession attached to the cluster and pass it to the Python process. + +You can optionally set the ``SPARK_CLUSTER`` environment variable to the cluster +URL and then dsgrid will connect to it. + +.. code-block:: console + + $ export SPARK_CLUSTER=spark://$(hostname):7077 + +Using ``SPARK_CLUSTER`` is a bit simpler, but you cannot configure settings like +``spark.driver.memory``, which, as stated earlier, must be set before the JVM +is created. + +spark-submit +------------ + +Running dsgrid CLI commands through ``spark-submit`` requires cumbersome syntax +because the tool needs to + +1. Detect that the script is Python (which is why this example uses dsgrid-cli.py + instead of dsgrid). +2. Know the full path to the script (accomplished with the utility ``which``). + +Here's how to do that: + +.. code-block:: console + + $ spark-submit --master spark://$(hostname):7077 \ + $(which dsgrid-cli.py) \ + query project run \ + --registry-path=/scratch/${USER}/.dsgrid-registry \ + query.json + +.. note:: If you want to set a breakpoint in your code for debug purposes, you + cannot use spark-submit. + + +Troubleshooting +=============== + +Get used to monitoring Spark jobs in the Spark UI. The master is at +``http://:8080`` and jobs are at ``http://:4040``. +If a job seems stuck or slow, explore why. Then kill the job, make config changes, +and retry. A misconfigured job will take too long or never finish. + +This section explains some common problems. + +spark.sql.shuffle.partitions +---------------------------- + +The most common performance issue we encounter when running complex queries is +due to a non-ideal setting for ``spark.sql.shuffle.partitions``. The default +Spark value is 200. Some online sources recommend setting it to 1-4x the total +number of CPUs in your cluster. This +`video `_ by a Spark +developer offers a recommendation that has worked out better. + +Use this formula:: + + num_partitions = max_shuffle_write_size / target_partition_size + +You will have to run your job once to determine ``max_shuffle_write_size``. You +can find it on the Spark UI Stages tab in the Shuffle Write column. Your +``target_partition_size`` should be between 128 - 200 MB. + +The minimum partitions value should be the total number of cores in the cluster +unless you want to leave some cores available for other jobs that may be running +simultaneously. + +Running out of space in local mode on the HPC +--------------------------------------------- + +The ``/tmp`` directory on HPC filesystems is very small. If you run Spark local +mode with default settings, it will use that directory for scratch space and +then quickly fill it up and fail. Set the environment variable ``SPARK_LOCAL_DIRS`` +to an appropriate directory. + +.. code-block:: console + + $ export SPARK_LOCAL_DIRS=/tmp/scratch + +The scripts discussed above set this environment variable for standalone clusters +on an HPC. + +Dynamic allocation +------------------ + +This feature is disabled by default on standalone clusters. It is described in +the `Spark documentation +`_. +This section does not necessarily line up with the Spark documentation and will +change as we learn more. + +We have observed two cases where enabling dynamic allocation on a standalone +cluster significantly improves performance: + +- When a Spark job produces a huge number of tasks. This can happen when checking + dimension associations as well as queries that map a dataset to project + dimensions. +- When there is really only enough work for one executor but Spark distributes + it to all executors anyway. The inter-process communication adds tons of + overhead. The executors also appear to do a lot more work. With dynamic + allocation Spark gradually adds executors and these problems don't always occur. + +The second issue sometimes occurs when submitting a dataset to a project in the +registry. We recommend enabling dynamic allocation when doing this. If that is +enabled then dsgrid code will reconfigure the SparkSession to use a single +executor with one core. + +Set these values in your ``spark-defaults.conf``. ``spark.shuffle.service.enabled`` +must be set before you start the workers. + +:: + + spark.dynamicAllocation.enabled true + spark.dynamicAllocation.shuffleTracking.enabled true + spark.shuffle.service.enabled true + spark.shuffle.service.db.enabled = true + spark.worker.cleanup.enabled = true + +We have not observed any downside to having this feature enabled. + +Slow local storage +------------------ + +Spark will write lots of temporary data to local storage during shuffle operations. +If your joins cause lots of shuffling, it is very important that your local +storage be fast. If you direct Spark to use the Lustre filesystem for local +storage, your mileage will vary. If the system is idle, it might work. If it is +saturated, your job will likely fail. + +Too many executors are involved in a job +---------------------------------------- + +Some of our Spark jobs, particularly those that create large, in-memory tables +from dimension records and mappings, perform much better when there is only one +executor and only one CPU. If you think this is happening then set these values +in your ``spark-defaults.conf``:: + + spark.executor.cores 1 + # This value should be greater than half of the memory allocated to the Spark workers, which + # will ensure that Spark can only create one executor. + spark.executor.memory 15g + +One common symptom of this type of problem is that a job run in local mode works +better than in a standalone cluster. The likely reason is that the standalone +cluster has many executors and local mode only has one. + +Partition sizes are wrong +------------------------- + +Spark documentation recommends that Parquet partition files be in the range of +100-200 MiB, with 128 MiB being ideal. A very high compression ratio may change +that. This affects how much data each task reads from storage into memory. + +Check your partition sizes with this command: + +.. code-block:: console + + $ find -name "*.parquet" -exec ls -lh {} + + +Check the total size with this command: + +.. code-block:: console + + $ du -sh + +Divide the total size by the target partition size to get the desired number of +partitions. + +If there are too few partitions currently: + +.. code-block:: python + + df.repartition(target_partitions).write.parquet("data.parquet") + +If there are too many partitions currently: + +.. code-block:: python + + df.coalesce(target_partitions).write.parquet("data.parquet") + +If you are partitioning by a column and find that there are many very small +partition files, repartition like this: + +.. code-block:: python + + df.repartition(column_name).write.partitionBy(column_name).parquet("data.parquet") + +.. _executors-spilling-to-disk: + +Executors are spilling to disk +------------------------------ + +You are observing that the Spark job seems stalled. + +- Most of the work in a stage is complete, but one or two executors are still + running. +- Only 1-2 CPUs out of the entire cluster show any activity. +- You see this log message over and over in the ``stderr`` log files + (e.g., ``spark_scratch/workers/app-20250115165825-0006/1/stderr``): + +.. code-block:: console + + 25/07/15 14:24:06 INFO UnsafeExternalSorter: Thread 60 spilling sort data of 4.6 GiB to disk (201 times so far) + +If this has occurred 201 times over more than an hour, then you might be better +off canceling the job and re-running with a different configuration. + +First, ensure that you have set ``spark.sql.shuffle.partitions`` to a reasonable +value, as discussed above. + +Second, your executor memory may be too small. Kestrel compute nodes have +disproportionate CPUs with respect to memory. By default, our configuration +scripts will allocate 7 GB of memory per executor and this is insufficient for +some queries. The error message above indicates that the executor was not able +to perform a sort in memory, and so spilled to disk. This is very slow. + +Try to double or triple the executor memory. You can do this by setting the +``spark.executor.memory`` value in ``spark-defaults.conf``. You can set that +value directly by editing the file or indirectly by setting ``--executor-cores`` +to 10 or 15 (default is 5), thereby reducing the number of executors (by +assigning each one more of the total available cores) and giving each executor +more memory. + +You can also acquire a bigmem node, which has 2 TB of memory. On these nodes, +our Spark scripts will allocate 70 GB of memory per executor. The debug partition +on Kestrel usually has two bigmem nodes. + +Third, you may be experiencing data skew. This has happened frequently when +performing dimension mapping operations that explode data sizes by disaggregating +or duplicating data. One or two executors end up with significantly more data +than the others, and get stuck. Refer to the next section. + +Data Skew +--------- + +If your query will produce high data skew, such as can happen with a query that +produces results from large and small counties, you can use a salting technique +to balance the data. For example, + +.. code-block:: python + + df.withColumn("salt_column", F.lit(F.rand() * (num_partitions - 1))) \ + .groupBy("county", "salt_column") \ + .agg(F.sum("county")) \ + .drop("salt_column") + +dsgrid will employ this technique for specific mapping types or when you enabled +``handle_data_skew`` in a dataset's mapping plan. + +Related Documentation +===================== + +- :doc:`Fundamentals overview ` +- :doc:`Running dsgrid on Kestrel ` +- :doc:`Spark cluster on Kestrel ` diff --git a/docs/source/fundamentals/index.rst b/docs/source/fundamentals/index.rst new file mode 100644 index 000000000..82e5d1f84 --- /dev/null +++ b/docs/source/fundamentals/index.rst @@ -0,0 +1,129 @@ +.. _fundamentals: + +############ +Fundamentals +############ + +This section provides technical deep-dives into dsgrid's architecture, +components, and the technologies it uses. + +.. grid:: 2 + :gutter: 3 + + .. grid-item-card:: Apache Spark + :link: apache-spark + :link-type: doc + + Setting up and configuring Spark for dsgrid workloads. + + .. grid-item-card:: Architecture + :link: /reference/architecture + :link-type: doc + + Technical architecture of the dsgrid system. + + .. grid-item-card:: CLI Reference + :link: /reference/cli + :link-type: doc + + Complete command-line interface documentation. + + .. grid-item-card:: API Reference + :link: /reference/dsgrid_api/index + :link-type: doc + + Python API documentation. + +Overview +======== + +dsgrid is built on two key technologies: + +**SQL Database (SQLite)** + Holds dsgrid registries---metadata on and relationships between dsgrid + components (dimensions, datasets, dimension mappings, projects, derived + datasets, queries). + +**Big Data Engine (Apache Spark)** + Performs database operations across a cluster of computational nodes, + enabling processing of terabyte-scale datasets. + +Understanding when and how to use these technologies is essential for +effective dsgrid usage. + +When Do I Need Spark? +===================== + +**You likely DO need Spark if:** + +- Your dataset is > 10 GB +- You're performing complex dimension mappings that increase data size +- You're working with hourly data across many years and geographies +- You're running queries that join multiple large datasets +- You're creating derived datasets from large source datasets + +**You likely DON'T need Spark if:** + +- Your dataset is < 5 GB +- You're just validating configuration files +- You're doing development or testing with sample data +- You're accessing pre-computed published datasets + +**Still unsure?** +Start without Spark. If operations fail or take too long, switch to Spark. + +See :doc:`Apache Spark guide ` for setup instructions. + +Computing Environments +====================== + +dsgrid supports two main computing environments: + +Standalone (Local) +------------------ + +Running on a personal laptop or single server: + +- Suitable for development, testing, small datasets +- Limited by local memory and CPU +- Spark runs in local mode or local standalone cluster + +See :doc:`Apache Spark ` for local setup. + +HPC (Kestrel) +------------- + +Running on NREL's high-performance computing cluster: + +- Required for production-scale projects +- Provides distributed computing resources +- Spark runs across multiple compute nodes + +See :doc:`Apache Spark ` for HPC setup. + +Technical Documentation +======================= + +**Architecture & Design:** + +- :doc:`System architecture ` +- :doc:`Data models ` + +**Interfaces:** + +- :doc:`CLI reference ` +- :doc:`CLI fundamentals ` +- :doc:`Python API ` + +**Concepts:** + +- :doc:`Dimensions ` +- :doc:`Datasets ` +- :doc:`Projects ` +- :doc:`Queries ` + +.. toctree:: + :maxdepth: 2 + :hidden: + + apache-spark diff --git a/docs/source/getting-started/for-clients-funders.rst b/docs/source/getting-started/for-clients-funders.rst new file mode 100644 index 000000000..e96aa903e --- /dev/null +++ b/docs/source/getting-started/for-clients-funders.rst @@ -0,0 +1,200 @@ +.. _getting-started-clients-funders: + +######################################## +Getting Started: Clients & Funders +######################################## + +Welcome! This page provides an overview of dsgrid's capabilities, quality +assurance processes, and how to access the data and expertise your team needs. + +What is dsgrid? +=============== + +dsgrid (demand-side grid) is a toolkit that compiles high-resolution, validated +datasets describing future electricity demand. These datasets integrate multiple +data sources---including building stock models, transportation models, and historical +data---into a coherent picture of electricity load across scenarios, geographies, +and time periods. + +**Key capabilities:** + +- **Multi-source integration**: Combines ResStock, ComStock, transportation, and + other datasets into unified load profiles + +- **High resolution**: Hourly timeseries across thousands of geographic units + +- **Scenario coverage**: Explores multiple technology adoption and policy scenarios + +- **Rigorous validation**: Ensures data quality, consistency, and traceability + +- **Flexible access**: Data available in multiple formats for different use cases + +Why Use dsgrid? +=============== + +**For capacity expansion modeling:** + dsgrid provides the detailed load profiles needed to properly value distributed + energy resources and assess grid needs under high electrification scenarios. + +**For production cost modeling:** + Hourly load profiles with proper temporal and spatial resolution enable accurate + modeling of grid operations and renewable integration. + +**For policy analysis:** + Scenario coverage allows exploration of different technology adoption pathways, + policy interventions, and their impacts on electricity demand. + +**For planning studies:** + Integration across sectors (buildings, transportation, industry) provides + comprehensive view of future electricity needs. + +Quality Assurance +================= + +dsgrid ensures data quality through: + +**Validation Framework** + - Automated checks for data completeness and consistency + - Dimension mapping verification + - Unit consistency validation + - Temporal coverage checks + +**Transparent Provenance** + - Clear documentation of data sources + - Versioned datasets with change tracking + - Explicit mapping between data sources and project dimensions + +**Peer Review** + - Project coordinators review all dataset submissions + - Multi-stage validation before publication + - Documented quality assurance processes + +**Open Methodology** + - Open-source toolkit enables independent verification + - Published methodologies for all major datasets + - Community review and feedback + +Published Datasets +================== + +Browse available datasets: + +:doc:`Published Datasets Catalog ` + +Access Methods +============== + +**For your technical team:** + +1. **Direct download**: Pre-computed datasets in common formats (Parquet, CSV) + +2. **Query interface**: Custom queries through Python API or CLI + +3. **NREL support**: Technical support from dsgrid coordination team + +**Getting started:** + +- :doc:`Browse datasets ` - See what's available +- :doc:`Data user guide ` - Technical team starting point +- :doc:`Contact us ` - Connect with coordination team + +Current Projects +================ + +See active and completed dsgrid projects: + +:doc:`Current Projects ` + +Publications & Impact +===================== + +dsgrid-derived datasets support: + +- Academic research (peer-reviewed publications) +- DOE and utility planning studies +- National and state energy policy analysis +- Technology adoption modeling + +:doc:`View publications ` + +Technical Resources +=================== + +Point your technical team to: + +**Quick Start:** + :doc:`Data Users Guide ` - Fastest path to data + +**Installation:** + :doc:`Installation Guide ` - Set up dsgrid toolkit + +**Documentation:** + - :doc:`Data formats ` + - :doc:`Query interface ` + - :doc:`API reference ` + +**Support:** + - :doc:`Contact page ` + - `GitHub repository `_ + +Collaboration Opportunities +============================ + +**Dataset contribution:** + If your organization has relevant datasets (building models, load research data, + transportation models), consider contributing to dsgrid. + + :doc:`Dataset submission guide ` + +**Custom projects:** + The dsgrid team can support custom projects tailored to your specific needs. + + :doc:`Contact us ` to discuss + +Frequently Asked Questions +========================== + +**What makes dsgrid different from other load databases?** + +dsgrid uniquely: + +- Integrates multiple building and transportation models +- Provides hourly resolution across all end uses and geographies +- Offers flexible querying and mapping capabilities +- Maintains rigorous data provenance and validation +- Supports custom scenario development + +**How often are datasets updated?** + +Major datasets (like Standard Scenarios) are updated annually. Regional studies +and custom projects have their own schedules. + +**What's the cost?** + +dsgrid datasets funded by DOE are freely available. Custom project support may +involve cost recovery. + +**How do we cite dsgrid data?** + +See :doc:`Citation Guide ` for specific requirements. + +Next Steps +========== + +**For project leadership:** + +1. Review :doc:`published datasets ` +2. Explore :doc:`current projects ` +3. :doc:`Contact the team ` to discuss your needs + +**For your technical team:** + +1. Start with :doc:`Data Users Guide ` +2. Install the toolkit: :doc:`Installation ` +3. Try a tutorial: :doc:`Query a project ` + +**For more information:** + +- `NREL dsgrid website `_ +- :doc:`Publications ` +- :doc:`Contact us ` diff --git a/docs/source/getting-started/for-data-users.rst b/docs/source/getting-started/for-data-users.rst new file mode 100644 index 000000000..e2910dac5 --- /dev/null +++ b/docs/source/getting-started/for-data-users.rst @@ -0,0 +1,181 @@ +.. _getting-started-data-users: + +############################## +Getting Started: Data Users +############################## + +Welcome! This guide helps you quickly access and use dsgrid datasets for your +power system analysis. + +What You Can Do +=============== + +As a data user, you can: + +- **Access published datasets** describing electricity demand across scenarios, + geographies, and time periods +- **Query existing projects** to extract specific subsets of data +- **Download pre-computed results** for common analysis scenarios +- **Map data** to different time conventions or geographies + +Your Quick Path to Data +======================== + +Follow these steps to get started: + +1. **Browse Available Datasets** + + Start by exploring what datasets are available: + + - :doc:`Published Datasets ` - Catalog of available data + - :doc:`Current Projects ` - Active dsgrid projects + +2. **Understand the Data Structure** + + Learn how dsgrid organizes data: + + - :doc:`Data Formats ` - How data is structured + - :doc:`Dimensions ` - Understanding dataset dimensions + - :doc:`Datasets ` - Conceptual overview + +3. **Access the Data** + + Choose your access method: + + - **Download pre-queried data**: Many projects provide ready-to-use datasets + - **Query yourself**: Write custom queries for your specific needs + (:doc:`querying guide `) + - **Use the API**: Programmatic access (:doc:`API reference `) + +4. **Cite Your Work** + + When using dsgrid data in publications: + + - :doc:`Citation and Attribution ` - Proper citation guidelines + +What You'll Need +================ + +**Minimum requirements:** + +- Python 3.10 or later (for custom queries) +- Basic familiarity with pandas DataFrames (for custom queries) +- Web browser (for browsing published datasets) + +**For custom queries:** + +- Installed dsgrid package (:doc:`installation guide `) +- Understanding of SQL-like query syntax (helpful but not required) + +**You typically do NOT need:** + +- Apache Spark (pre-computed results don't require Spark) +- Deep understanding of dsgrid internals +- Dataset submission access + +Common Use Cases +================ + +Browse Published Datasets +------------------------- + +**Goal:** Find and download a dataset for your analysis + +**Steps:** + +1. Visit :doc:`Published Datasets ` +2. Find a dataset matching your needs (geography, time period, scenarios) +3. Download the dataset or access link +4. Load into your analysis tool (Python, R, Excel, etc.) + +Query a Project +--------------- + +**Goal:** Extract a specific subset of data from a dsgrid project + +**Steps:** + +1. Install dsgrid (:doc:`installation `) +2. Choose your project from :doc:`current projects ` +3. Write a query (:doc:`how-to guide `) +4. Execute and download results + +**Example query:** +"Give me 2030 residential electricity load for California counties, +summed by end-use, at hourly resolution" + +Map Data to Your Geography +--------------------------- + +**Goal:** Convert a dataset to your specific geographic boundaries + +**Steps:** + +1. Identify the dataset's current geography +2. Check available mapping options (:doc:`mapping guide `) +3. Apply the mapping transformation +4. Export mapped data + +Tutorials +========= + +We recommend starting with these tutorials: + +1. :doc:`Query a dsgrid project ` + + Learn to extract custom subsets of data + +2. :doc:`Query a dataset ` + + Understand how to query individual datasets + +Frequently Asked Questions +========================== + +**Do I need Apache Spark to use dsgrid data?** + +No! Spark is only needed if you're: + +- Creating or submitting datasets +- Running queries on very large projects (multi-TB scale) +- Performing complex dimension mappings + +For accessing published datasets or running small queries, Spark is not required. + +**What file formats does dsgrid use?** + +dsgrid primarily uses Parquet format for efficient storage and querying of large +datasets. You can easily convert to CSV or other formats using Python. + +**Can I access data without coding?** + +Yes! Many dsgrid projects provide: + +- Pre-computed CSV files for common scenarios +- Downloadable summary reports + +**How do I cite dsgrid data?** + +See our :doc:`Citation and Attribution Guide ` for +specific citation requirements for each dataset. + +**Where can I get help?** + +- :doc:`Contact page ` - Reach out to the dsgrid team +- `GitHub Issues `_ - Report bugs or request features + +Next Steps +========== + +Ready to dive deeper? + +- :doc:`Browse published datasets ` - Start exploring data +- :doc:`Query a project ` - Write your first query +- :doc:`Installation guide ` - Set up dsgrid +- :doc:`Data querying ` - Learn advanced querying + +Need something different? + +- :doc:`Dataset submission ` - Contribute data +- :doc:`Project coordination ` - Lead a project +- :doc:`Fundamentals ` - Technical deep-dive diff --git a/docs/source/getting-started/for-dataset-mappers.rst b/docs/source/getting-started/for-dataset-mappers.rst new file mode 100644 index 000000000..6cd71aa60 --- /dev/null +++ b/docs/source/getting-started/for-dataset-mappers.rst @@ -0,0 +1,232 @@ +.. _getting-started-dataset-mappers: + +################################## +Getting Started: Dataset Mappers +################################## + +Welcome! This guide helps you understand how to map datasets across different +dimensions, geographies, and time conventions using dsgrid. + +What is Dataset Mapping? +======================== + +Dataset mapping is the process of transforming data from one dimensional representation +to another. This is essential when: + +- Combining datasets with different geographic resolutions (e.g., states to counties) +- Aligning time conventions across datasets (e.g., different weather years) +- Reconciling sector or subsector classifications between data sources +- Translating between different building type taxonomies + +dsgrid provides a robust framework for defining, validating, and applying these mappings +consistently across large datasets. + +What You Can Do +=============== + +As a dataset mapper, you can: + +- **Define dimension mappings** between source and target dimensions +- **Use pre-constructed registries** with common mappings already defined +- **Apply complex transformations** like disaggregation or aggregation +- **Validate mappings** before applying them to ensure consistency +- **Chain multiple mappings** together for multi-step transformations + +Key Concepts +============ + +Dimension Types +--------------- + +dsgrid supports mappings across all dimension types: + +- **Geography**: Counties, states, census regions, utility territories, etc. +- **Time**: Different weather years, time zones, temporal resolutions +- **Sector**: Residential, commercial, industrial, transportation +- **Subsector**: Building types, vehicle types, end-use categories +- **Metric**: Energy, power, emissions, costs +- **Scenario**: Reference cases, policy scenarios, technology pathways +- **Model Year**: Projection years for forward-looking analysis +- **Weather Year**: Historical weather patterns used in simulations + +See :doc:`Dimension types ` for full details. + +Mapping Types +------------- + +dsgrid supports two main mapping types: + +**Dataset-to-Project Mappings** + Transform a dataset's dimensions to match a project's base dimensions. + These are defined when submitting a dataset to a project. + +**Base-to-Supplemental Mappings** + Transform a project's base dimensions to supplemental (alternate) dimensions. + These are defined when creating a project and used in queries. + +See :doc:`Dimension mappings ` for details. + +Mapping Operations +------------------ + +Common mapping operations include: + +- **One-to-one**: Simple renaming or relabeling +- **Many-to-one (aggregation)**: Combining multiple records into one +- **One-to-many (disaggregation)**: Splitting one record into multiple +- **Filtering**: Excluding certain records from the output + +Getting Started +=============== + +Step 1: Understand the Source and Target +----------------------------------------- + +Before creating mappings: + +1. **Identify source dimensions**: What dimensions does your dataset use? +2. **Identify target dimensions**: What dimensions does the project require? +3. **Determine the relationship**: Are records one-to-one, many-to-one, or one-to-many? + +Step 2: Create Mapping Files +----------------------------- + +Dimension mappings are defined in CSV files that specify the relationship between +source and target dimension records. + +**Example: Geography mapping (states to regions)** + +.. code-block:: text + + from_id,to_id + CA,Pacific + OR,Pacific + WA,Pacific + NV,Mountain + AZ,Mountain + ... + +**Example: Disaggregation with fractions** + +.. code-block:: text + + from_id,to_id,from_fraction + Census_Division_9,CA,0.35 + Census_Division_9,OR,0.15 + Census_Division_9,WA,0.20 + Census_Division_9,NV,0.10 + Census_Division_9,AZ,0.20 + +Step 3: Register Mappings +-------------------------- + +Mappings are registered with dsgrid as part of dataset submission or project +configuration. + +See :doc:`Dimension mappings tutorial ` for a complete example. + +Common Mapping Scenarios +========================= + +Geographic Disaggregation +------------------------- + +**Scenario:** Your data is at the state level but the project needs county-level data. + +**Approach:** + +1. Create a mapping file with state-to-county relationships +2. Include population or other weighting factors as ``from_fraction`` +3. dsgrid will distribute the state-level values to counties + +**Example:** + +.. code-block:: text + + from_id,to_id,from_fraction + CA,06001,0.023 # Alameda County + CA,06003,0.001 # Alpine County + CA,06005,0.012 # Amador County + ... + +Time Zone Alignment +------------------- + +**Scenario:** Datasets use different time conventions that need alignment. + +**Approach:** + +1. Define time mappings that account for timezone differences +2. Handle daylight saving time transitions +3. dsgrid applies temporal shifts during queries + +Sector Harmonization +-------------------- + +**Scenario:** One dataset uses EIA building types, another uses NREL building types. + +**Approach:** + +1. Create a crosswalk mapping between the two taxonomies +2. Use many-to-one or one-to-many as appropriate +3. Document any assumptions about partial matches + +Best Practices +============== + +**Document your mappings:** + Explain the rationale for mapping decisions, especially for disaggregation fractions. + +**Validate early:** + Test mappings on a small subset before applying to full datasets. + +**Use existing mappings:** + Check if the project already has relevant mappings defined. + +**Consider edge cases:** + Handle missing or unknown records explicitly rather than ignoring them. + +**Preserve traceability:** + Keep records of source data so mapped results can be traced back. + +Tutorials and How-Tos +===================== + +- :doc:`Map a dataset to project dimensions ` +- :doc:`Create dimension mappings ` +- :doc:`Understanding dimension mappings ` + +Technical Reference +=================== + +- :doc:`Dimension types ` +- :doc:`Data formats ` +- :doc:`CLI reference ` + +Getting Help +============ + +**Questions about mappings:** + +- :doc:`Contact the dsgrid team ` +- `GitHub Discussions `_ + +**Related documentation:** + +- :doc:`Dimension mappings explained ` +- :doc:`Project structure ` + +Next Steps +========== + +Ready to create mappings? + +1. Review :doc:`dimension mappings concepts ` +2. Try the :doc:`mapping tutorial ` +3. Check existing project mappings as examples + +**Other roles:** + +- :doc:`Submit datasets ` - Contribute data +- :doc:`Query data ` - Access existing datasets +- :doc:`Coordinate projects ` - Lead a project diff --git a/docs/source/getting-started/for-dataset-submitters.rst b/docs/source/getting-started/for-dataset-submitters.rst new file mode 100644 index 000000000..c2770b065 --- /dev/null +++ b/docs/source/getting-started/for-dataset-submitters.rst @@ -0,0 +1,273 @@ +.. _getting-started-dataset-submitters: + +###################################### +Getting Started: Dataset Submitters +###################################### + +Welcome! This guide walks you through the process of submitting your dataset to +a dsgrid project. + +What You'll Do +============== + +As a dataset submitter, you will: + +1. **Prepare your dataset** according to dsgrid requirements +2. **Create configuration files** that describe your dataset +3. **Test and validate** your dataset before submission +4. **Submit to a project** (either offline or through the shared registry) +5. **Work with the project coordinator** to ensure successful integration + +The Submission Workflow +======================== + +Step 1: Understand Requirements +-------------------------------- + +Before starting, review: + +- :doc:`Dimensions ` - What dimensions you must define +- :doc:`Data formats ` - One-table or two-table format +- :doc:`Projects ` - Specific requirements from your project coordinator + +**Key questions to answer:** + +- What dimension types does your dataset cover? (geography, time, sector, etc.) +- What resolution is your data? (hourly? annual? county-level? state-level?) +- What scenarios or model years does it include? +- How large is your dataset? (this affects whether you need Spark) + +Step 2: Prepare Your Data +-------------------------- + +Format your dataset according to dsgrid requirements: + +**Choose your format:** + +- **One-table format** - Simpler, good for smaller datasets +- **Two-table format** - More efficient for larger datasets + +See :doc:`Data formats ` for details. + +**Ensure data quality:** + +- No missing values (use appropriate fill values) +- Consistent units (usually kWh or MWh for electricity) +- Valid dimension records (no typos, consistent naming) +- Proper temporal resolution (no gaps in timeseries) + +**Tools:** + +- Use pandas for small/medium datasets +- Use PySpark for large datasets (see :ref:`when-spark` below) + +Step 3: Create Configuration Files +----------------------------------- + +Create the required configuration files: + +1. **Dataset config** - Describes your dataset metadata and dimensions +2. **Dimension records files** - CSV files defining each dimension +3. **Mapping config** (if submitting to a project) - How to map to project dimensions + +:doc:`How to create datasets ` + +**Example minimal dataset config:** + +.. code-block:: json5 + + { + "dataset_id": "my_residential_load_2024", + "dataset_type": "modeled", + "data_source": "ResStock 2024", + "dimensions": { + "geography": { + "dimension_id": "us_states", + "dimension_type": "geography" + }, + "time": { + "dimension_id": "hourly_2018", + "dimension_type": "time" + } + // ... other dimensions + } + } + +Step 4: Validate Your Dataset +------------------------------ + +Test your dataset before submitting: + +.. code-block:: console + + $ dsgrid registry datasets register --help + +This checks: + +- Configuration file syntax +- Data file format compliance +- Dimension consistency +- Record completeness + +**Common validation errors:** + +- Dimension records in data file but not in dimension definition +- Inconsistent dimension record names +- Missing required dimensions +- Data type mismatches + +Step 5: Submit to Project +-------------------------- + +Choose your submission method: + +**Offline submission** (recommended for first-time submitters): + +Work with pre-created registry data locally before submitting to shared registries. + +**Registry submission** (for experienced users): + +Submit directly to the shared registry after validation passes. + +:doc:`Full submission tutorial ` + +.. _when-spark: + +When Do I Need Apache Spark? +============================= + +Use this decision tree to determine if you need Spark: + +**You likely DO need Spark if:** + +- Your dataset is > 10 GB +- You're performing complex dimension mappings that explode data size +- You're working with hourly data across many years and geographies +- You're submitting to a project with many other large datasets + +**You likely DON'T need Spark if:** + +- Your dataset is < 5 GB +- You're just validating configuration files +- You're doing a test submission with sample data +- You're submitting a single small dataset + +**Still unsure?** Start without Spark. If validation/submission fails or takes +too long, switch to Spark. + +:doc:`Apache Spark setup guide ` + +Testing Best Practices +====================== + +Before submitting your full dataset: + +1. **Create a test subset** + + - Subset to 1-2 geographies + - Subset to 1 week or month of data + - Validate this small version first + +2. **Check dimension mappings** + + - Ensure all your dimension records map to project dimensions + - Check for unmapped records + - Verify mapping logic makes sense + +3. **Validate incrementally** + + - Check config files first (fast) + - Then check data format (medium) + - Finally check full consistency (slow) + +4. **Use offline mode first** + + - Test submission offline before using shared registry + - Verify outputs look correct + - Get feedback from project coordinator + +Common Issues and Solutions +============================ + +**Issue:** "Dimension record 'XYZ' not found in dimension definition" + +**Solution:** Check your dimension CSV file. Either: + +- Add the missing record to your dimension file, or +- Remove it from your data file (if it's an error) + +--- + +**Issue:** Validation takes forever or runs out of memory + +**Solution:** You need Apache Spark. See :ref:`when-spark` above. + +--- + +**Issue:** "Cannot map dimension record 'ABC' to project dimension" + +**Solution:** Check the project's base dimensions. You may need to: + +- Add a mapping in your dataset-to-project mapping config +- Work with project coordinator to add mapping to project +- Use a different dimension record that already maps + +--- + +**Issue:** Submission fails with Spark errors + +**Solution:** Common Spark configuration issues: + +- Increase executor memory: ``spark.executor.memory 15g`` +- Increase shuffle partitions: ``spark.sql.shuffle.partitions 1000`` +- See :doc:`Spark troubleshooting ` + +Tutorials +========= + +Step-by-step guides: + +1. :doc:`Create and submit a dataset ` + + Complete walkthrough of the entire process + +2. :doc:`Create dataset dimensions ` + + How to properly define your dimensions + +Getting Help +============ + +**Before submitting:** + +- Review all documentation thoroughly +- Test with a small subset first +- Validate offline before using shared registry + +**If stuck:** + +- Contact your project coordinator +- Post on `GitHub Discussions `_ +- :doc:`Contact dsgrid support ` + +**What to include when asking for help:** + +- Your dataset config file +- Full error message +- What you've tried already +- Dataset size and characteristics + +Next Steps +========== + +Ready to submit your dataset? + +1. :doc:`Review dimensions ` +2. :doc:`Choose your format ` +3. :doc:`Create config files ` +4. :doc:`Submit to project ` + +**Other roles:** + +- :doc:`Query data ` - Access existing datasets +- :doc:`Coordinate projects ` - Lead a project diff --git a/docs/source/getting-started/for-project-coordinators.rst b/docs/source/getting-started/for-project-coordinators.rst new file mode 100644 index 000000000..7aa280c85 --- /dev/null +++ b/docs/source/getting-started/for-project-coordinators.rst @@ -0,0 +1,294 @@ +.. _getting-started-project-coordinators: + +########################################## +Getting Started: Project Coordinators +########################################## + +Welcome! This guide provides an overview of creating and managing dsgrid projects, +coordinating dataset submissions, and publishing comprehensive load data. + +What is a Project Coordinator? +============================== + +A project coordinator is responsible for: + +- **Creating dsgrid projects** that define the scope and structure for load data +- **Coordinating dataset submissions** from multiple contributors +- **Creating derived datasets** by combining and transforming submitted data +- **Writing and running queries** to produce analysis-ready outputs +- **Publishing data** for use by data users and stakeholders + +This is typically the most technical role in the dsgrid ecosystem, requiring +familiarity with the full dsgrid workflow and often Apache Spark. + +Project Coordinator Workflow +============================ + +.. code-block:: text + + 1. Design Project + ↓ + 2. Create Project Configuration + ↓ + 3. Register Project + ↓ + 4. Coordinate Dataset Submissions + ↓ + 5. Create Derived Datasets + ↓ + 6. Run Queries + ↓ + 7. Publish Results + +Step 1: Design Your Project +============================ + +Before creating configuration files, define: + +**Scope:** + +- What geographic region does this project cover? +- What time period? (historical, future projections, or both) +- What sectors? (residential, commercial, transportation, industrial) +- What scenarios will be included? + +**Base Dimensions:** + +Define the project's base dimensions---the common framework all datasets will +map into: + +- :doc:`Geography dimension ` (e.g., counties, states, utility territories) +- Time dimension (e.g., hourly 2018-2050) +- Sector and subsector dimensions +- Scenario and model year dimensions + +**Expected Datasets:** + +- What datasets will be submitted? +- What dimensions will each dataset provide? +- What mappings will be needed? + +Step 2: Create Project Configuration +===================================== + +A dsgrid project requires several configuration components: + +**Project config file:** + +Defines project metadata and dimension requirements. + +.. code-block:: json5 + + { + "project_id": "my_project_2024", + "name": "My Load Study 2024", + "description": "Regional electricity load analysis...", + "dimensions": { + // Base dimension definitions + }, + "datasets": [ + // Expected dataset submissions + ] + } + +**Base dimension files:** + +CSV files defining each base dimension's records. + +**Supplemental dimensions (optional):** + +Alternate dimension resolutions for query flexibility. + +See :doc:`Create a project ` for a complete example. + +Step 3: Register Your Project +============================== + +Register the project with dsgrid: + +.. code-block:: console + + $ dsgrid registry projects register project_config.json5 -r /path/to/registry + +This validates the configuration and adds the project to the registry. + +Step 4: Coordinate Dataset Submissions +======================================= + +Work with dataset contributors to submit their data: + +1. **Share project requirements** with contributors + + - Base dimension definitions + - Required dimension mappings + - Data format specifications + - Validation requirements + +2. **Review submitted datasets** + + - Check mapping completeness + - Validate data quality + - Verify dimension consistency + +3. **Troubleshoot issues** + + - Help with mapping problems + - Resolve dimension mismatches + - Support Spark configuration + +See :doc:`Dataset submission guide ` for +the contributor perspective. + +Step 5: Create Derived Datasets +================================ + +Derived datasets combine and transform submitted datasets: + +**Common uses:** + +- Calculate residuals (e.g., "other" sector by subtraction) +- Apply growth rates to historical data +- Aggregate across dimensions +- Create scenario variations + +.. code-block:: console + + $ dsgrid query project run derived_dataset_query.json5 + +See :doc:`Create derived datasets ` for details. + +Step 6: Run Queries +=================== + +Queries extract data in specific formats for analysis: + +**Query capabilities:** + +- Filter by dimension records (e.g., specific states, scenarios) +- Aggregate across dimensions (e.g., sum sectors, average time) +- Map to supplemental dimensions +- Export to various formats + +.. code-block:: console + + $ dsgrid query project run my_query.json5 + +See :doc:`Query a project ` for examples. + +Step 7: Publish Results +======================= + +Make data available to users: + +- Export to common formats (Parquet, CSV) +- Document data provenance and methodology +- Create data dictionaries +- Publish to appropriate repositories + +Technical Requirements +====================== + +**Computing Resources:** + +Most project coordination tasks require significant computing resources: + +- **Local machine**: Suitable for small test projects and development +- **HPC (Kestrel)**: Required for production-scale projects + +**Apache Spark:** + +Project coordinators typically need Spark for: + +- Processing large dataset submissions +- Creating derived datasets +- Running complex queries +- Handling dimension mappings that explode data size + +See :doc:`Apache Spark guide ` for setup instructions. + +**Registry Access:** + +- Development: Use local registries for testing +- Production: Coordinate with dsgrid team for shared registry access + +Key Tutorials +============= + +Complete these tutorials to learn the full workflow: + +1. :doc:`Create a project ` + + Set up project configuration and dimensions + +2. :doc:`Create and submit a dataset ` + + Understand the contributor experience + +3. :doc:`Map a dataset ` + + Learn dimension mapping mechanics + +4. :doc:`Create a derived dataset ` + + Combine datasets and apply transformations + +5. :doc:`Query a project ` + + Extract data for analysis + +Reference Documentation +======================= + +- :doc:`Projects concept ` +- :doc:`Datasets concept ` +- :doc:`Dimension mappings ` +- :doc:`CLI reference ` +- :doc:`API reference ` + +Best Practices +============== + +**Start small:** + Create a test project with sample data before scaling up. + +**Document thoroughly:** + Future users and contributors will rely on your documentation. + +**Validate early:** + Catch problems in dataset submissions before they propagate. + +**Plan for scale:** + Consider data sizes and Spark requirements from the start. + +**Coordinate closely:** + Maintain close communication with dataset contributors. + +Getting Help +============ + +**Technical questions:** + +- :doc:`Contact the dsgrid team ` +- `GitHub Issues `_ +- `GitHub Discussions `_ + +**Examples:** + +- Review existing dsgrid projects for reference +- Check the tutorials for working examples + +Next Steps +========== + +Ready to create a project? + +1. Review :doc:`project concepts ` +2. Complete the :doc:`create project tutorial ` +3. Set up :doc:`Apache Spark ` +4. :doc:`Contact the team ` for shared registry access + +**Other roles:** + +- :doc:`Submit datasets ` - Contribute data +- :doc:`Query data ` - Access existing datasets +- :doc:`Map datasets ` - Transform dimensions diff --git a/docs/source/getting-started/index.rst b/docs/source/getting-started/index.rst new file mode 100644 index 000000000..93c51cbe7 --- /dev/null +++ b/docs/source/getting-started/index.rst @@ -0,0 +1,75 @@ +.. _getting-started: + +############### +Getting Started +############### + +Welcome to dsgrid! This section helps you find the right starting point based on how +you plan to use the toolkit. + +Choose Your Path +================ + +.. grid:: 2 + :gutter: 3 + + .. grid-item-card:: Clients & Funders + :link: for-clients-funders + :link-type: doc + + Understand capabilities, view published data, and connect with the team. + + .. grid-item-card:: Data Users + :link: for-data-users + :link-type: doc + + Quickly access and analyze dsgrid datasets for your research. + + .. grid-item-card:: Dataset Submitters + :link: for-dataset-submitters + :link-type: doc + + Contribute your datasets to dsgrid projects. + + .. grid-item-card:: Dataset Mappers + :link: for-dataset-mappers + :link-type: doc + + Map datasets across dimensions, geographies, and time conventions. + + .. grid-item-card:: Project Coordinators + :link: for-project-coordinators + :link-type: doc + + Create and manage dsgrid projects, coordinate submissions, and publish data. + +Quick Links +=========== + +**Most Common Tasks:** + +- :doc:`Browse published datasets ` +- :doc:`Install dsgrid ` +- :doc:`Query a project ` +- :doc:`Submit a dataset ` + +**Technical Resources:** + +- :doc:`Apache Spark guide ` +- :doc:`CLI reference ` +- :doc:`API reference ` + +**Need Help?** + +- :doc:`Contact the team ` +- `GitHub Issues `_ + +.. toctree:: + :maxdepth: 1 + :hidden: + + for-clients-funders + for-data-users + for-dataset-submitters + for-dataset-mappers + for-project-coordinators diff --git a/docs/source/index.rst b/docs/source/index.rst index 24c9f6075..3ac9e324e 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -1,269 +1,215 @@ #################### -dsgrid documentation +dsgrid #################### -What is dsgrid? -=============== +Demand-Side Grid Toolkit +========================= -The dsgrid Python package is the central tool for creating, managing, contributing to, and -accessing demand-side grid (dsgrid) toolkit projects. The dsgrid toolkit enables the compilation -of high-resolution load datasets suitable for forward-looking power system and other analyses. -For more information and completed work products, please see -`https://www.nrel.gov/analysis/dsgrid.html `_. +The dsgrid Python package enables the compilation of high-resolution load datasets +suitable for forward-looking power system and energy analyses. The toolkit +standardizes and aligns diverse datasets containing timeseries information about +future energy use, making comprehensive electricity demand modeling more accessible +and reliable. -Documentation Overview +For more information, see `NREL's dsgrid page `_. + +Who Should Use This Documentation? +=================================== + +Different users have different needs. Choose your path: + +.. grid:: 2 + :gutter: 3 + + .. grid-item-card:: Clients & Funders + :link: getting-started/for-clients-funders + :link-type: doc + + **Understand dsgrid's capabilities and access high-quality load data.** + + dsgrid provides rigorously validated datasets describing electricity demand across + scenarios, geographies, time periods, and sectors. + + +++ + :doc:`Browse datasets ` | + :doc:`View projects ` | + :doc:`Contact ` + + .. grid-item-card:: Data Users + :link: getting-started/for-data-users + :link-type: doc + + **Access and analyze dsgrid datasets as quickly as possible.** + + Access published dsgrid datasets for your power system analysis, download pre-queried + results, or write custom queries to extract exactly the data you need. + + +++ + :doc:`Published datasets ` | + :doc:`Query data ` | + :doc:`How to cite ` + + .. grid-item-card:: Dataset Submitters + :link: getting-started/for-dataset-submitters + :link-type: doc + + **Contribute your dataset to a dsgrid project.** + + Follow our clear workflow for testing, validating, and submitting datasets to dsgrid + projects. + + +++ + :doc:`Submission workflow ` | + :doc:`When to use Spark ` + + .. grid-item-card:: Dataset Mappers + :link: getting-started/for-dataset-mappers + :link-type: doc + + **Map your dataset to different dimensions or geographies.** + + Use dsgrid's mapping capabilities to transform your data across time conventions, + geographic boundaries, or sector classifications. + + +++ + :doc:`Mapping overview ` | + :doc:`Dimension mappings ` + + .. grid-item-card:: Project Coordinators + :link: getting-started/for-project-coordinators + :link-type: doc + + **Create and manage comprehensive dsgrid projects.** + + Create dsgrid projects, coordinate dataset submissions, create derived datasets, + write complex queries, and publish comprehensive load data. + + +++ + :doc:`Create projects ` | + :doc:`Apache Spark ` + + .. grid-item-card:: Developers + :link: fundamentals/index + :link-type: doc + + **Deep dive into the toolkit architecture and API.** + + Technical documentation for developers contributing to dsgrid or building + integrations. + + +++ + :doc:`Architecture ` | + :doc:`API reference ` + +Quick Start +=========== + +.. grid:: 3 + :gutter: 2 + + .. grid-item-card:: + :link: data/published-datasets + :link-type: doc + + **Browse Datasets** + ^^^ + + Access published dsgrid datasets + + .. grid-item-card:: + :link: how_tos/installation + :link-type: doc + + **Install dsgrid** + ^^^ + + Set up the dsgrid toolkit + + .. grid-item-card:: + :link: tutorials/index + :link-type: doc + + **Start Tutorial** + ^^^ + + Step-by-step examples + +Documentation Sections ====================== -If you are new to dsgrid, you'll likely want to start by reading the rest of this page, reading -the :ref:`how-to guide on installation `, and then choosing a tutorial that corresponds to how you expect -to be using dsgrid in the near future. - -For general use, the documentation is organized into: - -- :ref:`tutorials`, which provide step-by-step instructions (via examples) for common high-level - tasks; -- :ref:`how-to-guides`, which provide quick reminder recipes for key workflows; -- :ref:`explanations`, which describe concepts and answer "why" questions to facilitate deeper - understanding; and -- :ref:`reference`, which provides complete information on various interfaces (e.g., command line, - data formats, data models, public Python API). - -Please note that for now: - -⚠️ **dsgrid is under active development and does not yet have a formal package release.** ⚠️ - -and **details listed here are subject to change**. Please reach out to the dsgrid coordination -team with any questions or other feedback. - -dsgrid Overview -=============== - -dsgrid is a tool for collecting and aligning datasets containing timeseries information describing -future energy use, especially electricity load, to be used in planning studies. `Datasets`_ are -defined over specific scenario, model year, weather year, geography, time, sector, subsector, and -metric `Dimensions`_ and can range in size from less than 1 megabyte to over 1 terabyte. Typically, -datasets are organized into `Projects`_ with specific base dimensions with the help of -`Dimension Mappings`_. Projects use `Queries`_ to consolidate information into -`Derived Datasets`_ that, together with the standalone datasets, eventually enable a comprehensive -description of the electricity load or other energy use being modeled. Projects can also be -queried to produce output data for ingestion into another model or for direct analysis. - -The people who interact with dsgrid are typically: - -- `Project Coordinators`_, who construct, analyze and publish projects; -- `Dataset Contributors`_, who register datasets with dsgrid and and submit them to projects; or -- `Data Users`_, who access already-queried data or write and run their own custom queries. - -Because the dsgrid data can be quite large (on the order of terabytes), and are compiled from a -variety of data sources, dsgrid uses two key technologies to facilitate its workflows: - -- A SQL database (currently SQLite) to hold dsgrid registries (metadata on and relationships - between dsgrid components, e.g., dimensions, datasets, dimension mappings, projects, derived - datasets, queries) -- A big-data engine (currently Apache Spark) to perform database operations across a cluster of - computational nodes - -Both of these technologies mean that some care must be taken when choosing and setting up a -computational environment for a particular task. The dsgrid coordination team currently supports -two computational environments: - -- `Standalone`_, single-node environments e.g., personal laptops or a single server or virtual - machine, which is suitable for: - - - Small-scale development and testing - - Submitting a single dataset to a project in offline mode - - Very small dsgrid projects (no large datasets and little to no downscaling) - - Note that use on standalone Windows machines is especially limited. -- `NREL High Performance Computing`_, where users can: - - - Directly work on projects through the shared registry - - Develop code and test data and workflows using their own registry - - In either case, users will need to launch Apache Spark clusters with sufficient computational, - memory, and disk resources. - -Components ----------- - -Dimensions -^^^^^^^^^^ -dsgrid datasets and projects are multi-dimensional, and some dimensions are defined over thousands -of elements (e.g., counties in the United States, hours in a year). It is also typical for -different datasets that nominally describe the same thing to use different labels (e.g., -`ResStock building types -`_ -and `EIA Residential Energy Consumption Survey (RECS) building types -`_). - -To manage this complexity while allowing different analysts and modeling teams to use their own -labels (to facilitate transparency, easy maintenance and debugging), dsgrid requires its users to -explicitly define each data dimension by specifying its :ref:`dimension type `, -metadata, and a table listing each dimension record's id and name. dsgrid uses this information -to ensure that submitted data is as-expected. - -.. _dataset_overview: - -Datasets -^^^^^^^^ -A dsgrid dataset describes energy use or another metric (e.g., population, stock of certain -assets, delivered energy service, growth rates) resolved over a number of different dimensions, -e.g., scenario, geography, time, etc. When registering a dataset, the data submitter must define -a dimension for each dsgrid :ref:`dimension type `, but that does not mean that -datasets are required to be resolved (i.e., have multiple entries) for each dimension type--any -dimension can be "trivial", in which case it is defined by a single record (e.g., 'unspecified' -subsector, or a single '2012' weather year) and is not included in the data files. - -Registering a dataset requires a dataset config file, which lists dataset and dimension metadata, -the actual data file(s), and dimension record files (csvs) for any dimensions not already in the -dsgrid registry. The data files must conform to one of the dsgrid :ref:`dataset-formats`, -currently either the :ref:`one-table-format` or the :ref:`two-table-format`. Upon registration, -dsgrid checks the data files, which contain dimension records and numerical values, for -consistency with the specified dimensions. Inconsistent data fails registration to prevent -compounding downstream errors. - -Projects -^^^^^^^^ -A dsgrid project is a collection of datasets that describe energy demand for a specific region -over a specific timeframe. Because datasets describing different sectors' energy use are defined -in different ways, the key task of a dsgrid project is to enable and perform mappings from -datasets' dimensions into the **project base dimensions**. Project base dimensions are defined -in the same way as dataset dimensions (i.e., there is a project base dimension for each dimension -type), however, they are used differently. Whereas dataset dimensions are *descriptive*--they -describe what you will find if you look in a dataset's data files, project base dimensions are -*prescriptive*--they define what dataset submitters must map their dimensions into. dsgrid projects -are also highly prescriptive about what datasets they expect to be submitted, and what data -dimensions they are expecting each dataset to provide (post-mapping, see -:ref:`dimension_mapping_overview`). dsgrid uses all of the prescribed information to check that -submitted datasets are as expected, and throws errors if they are not. - -dsgrid projects are also the starting point for `Queries`_. Queries are the process whereby -datasets are actually mapped into the project base dimensions, concatenated, and further -transformed. The two straightforward applications of queries are: - -1. Output data suitable for use in another model, and -2. Analyze project data. - -Either way, users generally do not want full detail along all dimensions. dsgrid supports this -by letting users specify what *supplemental dimensions* they want their query results in, as well -as if they want any dimension records filtered out and what mathematical operations they want to -use for aggregations. For example, a query can be written to filter out non-electricity energy -use; sum electricity use over all sectors, subsectors, and end-uses; and map to a specific -power system model's geography to create load data for capacity expansion or production cost -modeling. - -.. _dimension_mapping_overview: - -Dimension Mappings -^^^^^^^^^^^^^^^^^^ -While many data sources provide information by, e.g., scenario, geographic place, and sector, -different data sources often define such dimensions differently and/or simply report out at a -different level of resolution. Because dsgrid joins many datasets together to create a coherent -description of energy for a specific place over a spectific timeframe, we need a mechanism for -reconciling these differences. For example: - -- How should census division data be downscaled to counties? -- What's the best mapping between EIA AEO commercial building types and NREL ComStock commercial - building types? -- `Residential`, `res`, and `Res.` should all be interpreted the same way, as referring to - residential energy use or housing stock, etc. - -The mappings that answer these questions are explicitly registered with dsgrid as dimension -mappings. This way they are clearly documented and usable in automated queries. Explicit, -programmatically checked and used dimensions and dimension mappings are key features that help -dsgrid efficiently and reliably assemble detailed datasets of energy demand from a combination of -historical and modeled data. - -Currently, dsgrid supports two different types of mappings: - -1. ``Dataset-to-Project``: These are mappings from a dataset's dimension to a project's base - dimension of the same dimension type. They are declared and registered when a dataset is - submitted to a project. -2. ``Base-to-Supplemental``: These are mappings from a project's base dimensions to its - supplemental dimensions, which are the alternate data resolutions available for use in queries. - Base-to-supplemental dimensions are defined when registering a project. - -Queries -^^^^^^^ -TODO - -Derived Datasets -^^^^^^^^^^^^^^^^ -During the creation of a dsgrid project, one of the key tasks is to use queries to create -derived datasets. As their name implies, derived datasets are dsgrid datasets and must -meet all requirements that status implies, but they are created by combining multiple datasets -already in the project. Because they are formed from datasets already mapped to the project base -dimensions, defining the dimensions of derived datasets is typically straightforward and is either -fully or mostly automated by dsgrid. Derived datasets are the mechanism dsgrid provides to do -things like apply growth rates or calculate residuals. Registering derived datasets with the -project enables dsgrid modelers to bootstrap data into providing a complete and straightforward -accounting of a region's energy demand over a specified timeframe. - -Published Projects -^^^^^^^^^^^^^^^^^^ -TODO - -Tasks ------ - -Project Coordinators -^^^^^^^^^^^^^^^^^^^^ -dsgrid project coordinators :ref:`create projects `, collaborate with -dataset contributors to get datasets added to the project, :ref:`create derived datasets -`, :ref:`write queries `, analyze and -publish data. - -Dataset Contributors -^^^^^^^^^^^^^^^^^^^^ -The role of dataset contributors is primarily to :ref:`create, register, and submit datasets -`. Of course, dataset contributors might also be project -coordinators and/or data users. - -Data Users -^^^^^^^^^^ -Data users might access data provided by a project coordinator through formal publication or other -dissemination channels, or they might :ref:`write their own queries `. - -Computational Environment -------------------------- - -dsgrid is cross-platform software that can be used with any datasets that conform to the metadata -and data formatting requirements. However, the typical dsgrid project involves large, simulated -datasets that are "exploded" to align across multiple dimension types and to produce the variety -of different views needed by different data users and analysts. Thus while some development, -testing, and work with single datasets or very small projects can be performed on `Standalone`_ -machines, currently a lot of dsgrid work requires `NREL High Performance Computing`_. - -Standalone -^^^^^^^^^^ - -TODO: List the key set-up tasks (spin up and connect to -Apache Spark cluster, and generally configure dsgrid to be pointing to all the right places) -and link to the appropriate how-tos - -NREL High Performance Computing -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -TODO: List the key set-up tasks (spin up and connect to -Apache Spark cluster, and generally configure dsgrid to be pointing to all the right places) -and link to the appropriate how-tos - -Indices, Tables, and Contents -============================= +:doc:`Data ` + Access published datasets, query projects, and understand data formats -* :ref:`genindex` -* :ref:`modindex` -* :ref:`search` +:doc:`Projects ` + Learn about dsgrid projects and view current work + +:doc:`Fundamentals ` + Technical deep-dive into architecture, API, and Apache Spark + +:doc:`How-To Guides ` + Quick recipes for common workflows + +:doc:`Tutorials ` + Step-by-step learning through examples + +:doc:`Explanations ` + Understand concepts and design decisions + +:doc:`Reference ` + Complete API and CLI documentation + +.. attention:: -.. TODO: Search page seems to be where search results land, but if you just click on it - it's blank. Can the default page have a search bar on it or some sort of instruction? + **dsgrid is under active development and does not yet have a formal + package release.** Details listed here are subject to change. Please reach + out to the :doc:`dsgrid coordination team ` with any questions or + feedback. + +Additional Resources +==================== + +- :doc:`publications` - Academic papers and reports using dsgrid +- :doc:`citation-attribution` - How to cite dsgrid and its datasets +- :doc:`contact` - Get in touch with the team +- `NREL dsgrid website `_ .. toctree:: :maxdepth: 2 - :caption: Contents: + :hidden: + :caption: Getting Started - tutorials/index + getting-started/index + getting-started/for-clients-funders + getting-started/for-data-users + getting-started/for-dataset-submitters + getting-started/for-dataset-mappers + getting-started/for-project-coordinators + +.. toctree:: + :maxdepth: 2 + :hidden: + :caption: Documentation + + data/index + projects/index + fundamentals/index how_tos/index + tutorials/index explanations/index reference/index - spark_overview + +.. toctree:: + :maxdepth: 1 + :hidden: + :caption: Resources + + publications + citation-attribution + contact + +Indices and Search +================== + +* :ref:`genindex` +* :ref:`modindex` +* :ref:`search` diff --git a/docs/source/projects/current-projects.rst b/docs/source/projects/current-projects.rst new file mode 100644 index 000000000..45b0b84c1 --- /dev/null +++ b/docs/source/projects/current-projects.rst @@ -0,0 +1,101 @@ +.. _current-projects: + +################ +Current Projects +################ + +This page lists active and completed dsgrid projects. + +.. note:: + + This page will be updated as projects are formally published. Check back + regularly for updates, or :doc:`contact the dsgrid team ` for + information about ongoing work. + +Active Projects +=============== + +.. note:: + + **Information Coming Soon** + + The dsgrid team is actively working on several projects. Details will be + added here as projects reach publication status. + + :doc:`Contact the team ` for early access or collaboration + opportunities. + +Completed Projects +================== + +.. note:: + + **Information Coming Soon** + + Historical project information will be documented here. + +Project Categories +================== + +dsgrid projects typically fall into these categories: + +Standard Scenarios +------------------ + +Annual releases of comprehensive U.S. electricity demand projections, +combining multiple building and transportation models across scenarios. + +**Typical scope:** + +- National coverage (contiguous U.S.) +- Multi-decade projections (2020-2050) +- Multiple electrification scenarios +- Hourly temporal resolution +- County-level geographic resolution + +Regional Studies +---------------- + +Focused analyses for specific states, utilities, or regions with +enhanced resolution or scenario coverage. + +**Typical scope:** + +- State or utility service territory +- Detailed end-use breakdowns +- Multiple weather years +- Custom scenarios aligned with regional planning + +Research Projects +----------------- + +Specialized studies supporting academic research, policy analysis, +or technology assessments. + +**Typical scope:** + +- Specific research questions +- Novel methodologies +- Deep-dive on particular sectors or technologies + +Proposing a Project +=================== + +If you're interested in proposing a new dsgrid project: + +1. **Define the scope**: Geographic region, time period, sectors, scenarios +2. **Identify data sources**: What datasets would be included? +3. **Determine resources**: What computing and personnel resources are needed? +4. **Contact the team**: :doc:`Reach out ` to discuss feasibility + +See :doc:`Getting started for project coordinators ` +for more details on the project creation process. + +Related Documentation +===================== + +- :doc:`Projects overview ` +- :doc:`Getting started for coordinators ` +- :doc:`Create a project tutorial ` +- :doc:`Published datasets ` +- :doc:`Publications ` diff --git a/docs/source/projects/index.rst b/docs/source/projects/index.rst new file mode 100644 index 000000000..70b224dde --- /dev/null +++ b/docs/source/projects/index.rst @@ -0,0 +1,159 @@ +.. _projects: + +######## +Projects +######## + +A dsgrid project is a collection of datasets that describe energy demand for a +specific region over a specific timeframe. Projects define the common framework +(base dimensions) that all contributing datasets map into. + +.. grid:: 2 + :gutter: 3 + + .. grid-item-card:: Current Projects + :link: current-projects + :link-type: doc + + View active and completed dsgrid projects. + + .. grid-item-card:: Publications + :link: /publications + :link-type: doc + + Academic papers and reports using dsgrid data. + +What is a dsgrid Project? +========================= + +A dsgrid project: + +- **Defines scope**: Geographic region, time period, sectors covered +- **Establishes base dimensions**: The common dimensional framework +- **Coordinates datasets**: Specifies expected data contributions +- **Enables queries**: Provides unified access to combined data +- **Publishes results**: Makes data available to users + +Key Concepts +============ + +Base Dimensions +--------------- + +Every project defines **base dimensions**---the common framework that all +datasets must map into. This includes: + +- **Geography**: The spatial resolution (e.g., counties, states) +- **Time**: The temporal coverage and resolution +- **Sector/Subsector**: The sectoral breakdown +- **Scenario**: The scenarios to be analyzed +- **Model Year**: The projection years +- **Weather Year**: The weather patterns used + +See :doc:`Dimensions ` for details. + +Dimension Mappings +------------------ + +Because datasets use different dimensional representations, each dataset +submission includes **dimension mappings** that transform the dataset's +dimensions to the project's base dimensions. + +See :doc:`Dimension mappings `. + +Derived Datasets +---------------- + +Projects can create **derived datasets** by combining, transforming, or +calculating from submitted datasets. Common uses include: + +- Calculating residuals (e.g., "other" sector) +- Applying growth rates +- Creating scenario variations +- Aggregating across dimensions + +See :doc:`Derived datasets `. + +Queries +------- + +Projects support **queries** that extract data subsets: + +- Filter by dimension records +- Aggregate across dimensions +- Map to supplemental dimensions +- Export in various formats + +See :doc:`Queries `. + +Project Workflow +================ + +.. code-block:: text + + 1. Project Creation + ├── Define scope and base dimensions + ├── Specify expected datasets + └── Define supplemental dimensions + + 2. Dataset Submission + ├── Contributors prepare datasets + ├── Define dimension mappings + └── Submit and validate + + 3. Data Integration + ├── Create derived datasets + ├── Run validation queries + └── Check data completeness + + 4. Publication + ├── Final quality checks + ├── Generate outputs + └── Publish data and documentation + +Roles +===== + +**Project Coordinators** + Create and manage projects, coordinate submissions, create derived datasets, + run queries, and publish results. + + :doc:`Getting started for coordinators ` + +**Dataset Contributors** + Prepare datasets, create dimension mappings, and submit to projects. + + :doc:`Getting started for submitters ` + +**Data Users** + Access published data through downloads or queries. + + :doc:`Getting started for data users ` + +Related Documentation +===================== + +**Concepts:** + +- :doc:`Projects explained ` +- :doc:`Datasets explained ` +- :doc:`Derived datasets ` +- :doc:`Dimension mappings ` + +**Tutorials:** + +- :doc:`Create a project ` +- :doc:`Submit a dataset ` +- :doc:`Create derived datasets ` +- :doc:`Query a project ` + +**Reference:** + +- :doc:`CLI reference ` +- :doc:`Data models ` + +.. toctree:: + :maxdepth: 1 + :hidden: + + current-projects diff --git a/docs/source/publications.rst b/docs/source/publications.rst new file mode 100644 index 000000000..cfe427692 --- /dev/null +++ b/docs/source/publications.rst @@ -0,0 +1,53 @@ +.. _publications: + +############ +Publications +############ + +This page lists academic papers, reports, and other publications related to +dsgrid and datasets produced using the toolkit. + +.. note:: + + This page will be updated as publications are added. Check back regularly + or :doc:`contact the team ` for information about recent work. + +Citing dsgrid +============= + +If you use dsgrid or dsgrid-derived datasets in your research, please cite +appropriately. See :doc:`citation-attribution` for complete guidelines. + +Related NREL Tools +================== + +dsgrid integrates with and builds upon data from several NREL modeling tools: + +**ResStock** + Residential building stock analysis tool. + `ResStock documentation `_ + +**ComStock** + Commercial building stock analysis tool. + `ComStock documentation `_ + +**TEMPO** + Transportation electrification modeling. + +**ReEDS** + Regional Energy Deployment System for capacity expansion modeling. + `ReEDS documentation `_ + +External Resources +================== + +- `NREL dsgrid website `_ +- `dsgrid GitHub repository `_ + +See Also +======== + +- :doc:`citation-attribution` - How to cite dsgrid +- :doc:`data/published-datasets` - Available datasets +- :doc:`projects/current-projects` - Active projects +- :doc:`contact` - Get in touch diff --git a/docs/source/spark_overview.rst b/docs/source/spark_overview.rst deleted file mode 100644 index ca3a1c5a5..000000000 --- a/docs/source/spark_overview.rst +++ /dev/null @@ -1,609 +0,0 @@ -.. _spark-overview: - -************ -Apache Spark -************ - -This page describes Spark concepts that are important to understand when using dsgrid. - -.. todo:: Split this content up into tutorials, how-tos, explanations, and reference material - -Windows Users -============= -Spark does not offer a great user experience in Windows. While ``pyspark`` and -``spark-submit`` work in local mode, running a cluster requires manual configuration. The developers -provide cluster management scripts in bash, and so they do not work in Windows. When running a -cluster on your laptop we recommend that you use dsgrid in a Windows Subsystem for Linux (WSL2) -environment instead of a native Windows environment. - -If you are running in local mode, you will need Hadoop's ``winutils.exe`` because windows doesn't -support HDFS. If you don’t have winutils.exe installed, you will need to download -the wintils.exe and hadoop.dll files from https://github.com/steveloughran/winutils (select the -Hadoop version you are using as winutils are specific to Hadoop versions). Then copy them into a -folder like ``C:\hadoop\bin``, set the environment variable ``HADOOP_HOME`` to, e.g., -``C:\hadoop``, and add ``%HADOOP_HOME%\bin`` to your ``PATH``. - -If you get an error like: ``Python was not found; run without arguments to install from the -Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases.`` try -setting this environment variable: ``PYSPARK_PYTHON=python`` (or set the value to ipython, if -you would prefer). - -Conventions -=========== -This page uses the UNIX conventions for environment variables and running commands in -a shell. If you do run any of these commands in a native Windows environment, you will have to -adjust depending on whether you are running PowerShell or the old Command shell. - -Explanation of bash functionality used here: - -- ``$VARIABLE_NAME`` or ``${VARIABLE_NAME}``: This uses the environment variable ``VARIABLE_NAME``. - If the variable isn't defined, bash will use an empty string. -- ``export VARIABLE_NAME=x``: This sets the environment variable ``VARIABLE_NAME`` to the value - ``x`` in the current shell. If you want to set this variable for all future shells as well, add - the statement to your shell rc file (``$HOME/.bashrc`` or ``$HOME/.zshrc``). -- `export VARIABLE_NAME=$(hostname)`: This sets the environment variable ``VARIABLE_NAME`` to the - string returned by the command ``hostname``. - -In examples listed on this page where you need to enter text yourself, that text is enclosed in -``<>`` as in ``http://:4040``. - - -Spark Overview -============== - -Cluster Mode ------------- -Apache provides an overview at https://spark.apache.org/docs/latest/cluster-overview.html - -The most important parts to understand are how dsgrid uses different cluster modes in different -environments. - -- Local computer in local mode: All Spark components run in a single process. This is - great for testing and development but is not performant. It will not use all CPUs on - your system. You run in this mode when you type ``pyspark`` in your terminal. - -- Local computer with a standalone cluster: You must install Spark and then manually - start the cluster. Refer to the [installation - instructions](#installing-a-spark-standalone-cluster-on-your-laptop). This - enables full performance on your system. It also allows you to debug your - jobs in the Spark UI before or after they complete. - -- HPC compute node in local mode: Use this only for quick checks. Same points above for local - computer in local mode apply. Create a standalone cluster for real work. If you use this, set - the environment variable ``SPARK_LOCAL_DIRS=/tmp/scratch`` so that you have enough space. - -- HPC compute node(s) with a standalone cluster: Create a cluster on any number of compute nodes - and then use all CPUs for your jobs. Refer to the [installation - instructions](#installing-a-spark-standalone-cluster-on-an-hpc). - -- AWS EMR cluster: The EMR scripts in the dsgrid repo at ``/emr`` will create a Spark cluster on - EC2 compute nodes with a cluster manager. The cluster manager allows multiple users to access a - single cluster and offers better job scheduling. Refer to the README.md in that directory. - - -Run Spark Applications ----------------------- -There are three ways of running Spark applications in Python. ``spark-submit`` and ``pyspark``, -provided by the Spark and pyspark installations, are recommended because they allow you to fully -customize the execution environment. More details follow -[below](#tuning-spark-configuration-settings). - -1. ``spark-submit`` - - This command will create a SparkSession, make that session available to the Python process, and - run your code. - -.. code-block:: console - - $ spark-submit [options] your_script.py [your_script_options] - -2. ``pyspark`` - - This command will create a SparkSession, make that session available to the Python process, and leave you - in the Python interpreter for an interactive session. - -.. code-block:: console - - $ pyspark [options] - -3. Inside Python - -.. code-block:: console - - $ python - >>> from pyspark.sql import SparkSession - >>> spark = SparkSession.builder.appName("your_app_name").getOrCreate() - -Run pyspark through IPython or Jupyter --------------------------------------- -You can configure ``pyspark`` to start ``IPython`` or ``Jupyter`` instead of the standard Python -interpreter by setting the environment variables ``PYSPARK_DRIVER_PYTHON`` and -``PYSPARK_DRIVER_PYTHON_OPTS``. - -IPython -------- -.. code-block:: console - - $ export PYSPARK_DRIVER_PYTHON=ipython - -Local mode: - -.. code-block:: console - - $ pyspark - -Cluster mode: - -.. code-block:: console - - $ pyspark --master=spark://$(hostname):7077 - -Now you are in IPython instead of the standard Python interpreter. - -Jupyter -------- -.. code-block:: console - - $ export PYSPARK_DRIVER_PYTHON=jupyter - $ export PYSPARK_DRIVER_PYTHON_OPTS="notebook --no-browser --port=8889 --ip=0.0.0.0" - $ export SPARK_HOME=$(python -c "import pyspark;print(pyspark.__path__[0])") - -Local mode: - -.. code-block:: console - - $ pyspark - -Cluster mode: - -.. code-block:: console - - $ pyspark --master=spark://$(hostname):7077 - -Pyspark will start a Jupyter notebook and you'll see the URL printed in the terminal. If you're on -a remote server, like in an HPC environment, you'll need to create an ssh tunnel in order to -connect in a browser. - -Once you connect in a brower, enter the following in a cell in order to connect to this cluster: - -.. code-block:: python - - from pyspark.sql import SparkSession - spark = SparkSession.builder.appName("your_app_name").getOrCreate() - -Spark UI --------- -The Spark master starts a web application at ``http://:8080``. Job information is -available at port 4040. You can monitor and debug all aspects of your jobs in this application. You -can also inspect all cluster configuration settings. - -If your Spark cluster is running on a remote system, like an HPC, you may need to open an ssh -tunnel to the master node. Here is how to do that on NREL's Kestrel cluster. - -On your laptop: -.. code-block:: console - - $ export COMPUTE_NODE= - $ ssh -L 4040:$COMPUTE_NODE:4040 -L 8080:$COMPUTE_NODE:8080 $USER@kestrel.hpc.nrel.gov - - -Installing a Spark Standalone Cluster -===================================== -The next sections describe how to install a standalone cluster on a local system and an HPC. - -Laptop ------- -.. note:: As stated earlier, the scripts mentioned in this section do not work in a native Windows - environment. You can still start a cluster in Windows; you just have to run the java commands - yourself. - -Download your desired version from https://spark.apache.org/downloads.html and extract it on -your system. - -.. note:: Choose the version that is set in ``dsgrid/setup.py`` for ``pyspark``. Major, minor, and - patch versions must match. - -.. code-block:: console - - $ cd # directions below assume $HOME - $ wget https://dlcdn.apache.org/spark/spark-3.3.1/spark-3.3.1-bin-hadoop3.tgz - $ tar -xzf spark-3.3.1-bin-hadoop3.tgz && rm spark-3.3.1-bin-hadoop3.tgz - - -The full instructions to create a cluster are at -http://spark.apache.org/docs/latest/spark-standalone.html. The rest of this section documents the -requirements for dsgrid. - -Set environment variables -~~~~~~~~~~~~~~~~~~~~~~~~~ -.. code-block:: console - - $ export SPARK_HOME=$HOME/spark-3.3.1-bin-hadoop3 - $ export PATH=$PATH:$SPARK_HOME/sbin - - -Note that after doing this your system will have two versions of ``pyspark``: - -- In your Python virtual environment where you installed dsgrid (because dsgrid installs pyspark) -- In ``$HOME/spark-3.3.1-bin/hadoop3/bin`` - -If you use a conda virtual environment, when that environment is activated, its ``pyspark`` will be -in your system path. Be sure not to add the spark bin directory to your path so that there are no -collisions. - -**Warning**: Setting ``SPARK_HOME`` will affect operation of your Python ``pyspark`` installation in -local mode. That may not be what you want if you make settings specific to the standalone cluster. - -Customize Spark configuration settings -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -.. code-block:: console - - $ cp $SPARK_HOME/conf/spark-defaults.conf.template $SPARK_HOME/conf/spark-defaults.conf - $ cp $SPARK_HOME/conf/spark-env.sh.template $SPARK_HOME/conf/spark-env.sh - -Set ``spark.driver.memory`` and ``spark.driver.maxResultSize`` in ``spark-defaults.conf`` to the -maximum data sizes that you expect to pull from Spark to Python, such as if you call -``df.toPandas()``. ``1g`` is probably reasonable. - -Set ``spark.sql.shuffle.partitions`` to 1-4x the number of cores in your system. Note that the -default value is 200, and you probably don't want that. - -Set ``spark.executor.cores`` and ``spark.executor.memory`` to numbers that allow creation of your -desired number of executors. Spark will try to create the most number of executors such that each -executor has those resources. For example, if your system has 16 cores and you assign 16g of memory -to the worker (see below), ``spark.executor.cores 3`` and ``spark.executor.memory 5g`` will result -in 3 executors. - -Start the Spark processes -~~~~~~~~~~~~~~~~~~~~~~~~~ -.. code-block:: console - - $SPARK_HOME/sbin/start-master.sh - -Start a worker with this command. Give the worker as much memory as you can afford. Executor memory -comes from this pool. You can also configure this in ``spark-env.sh``. - -.. code-block:: console - - $SPARK_HOME/sbin/start-worker.sh -m 16g spark://$(hostname):7077 - -If you add the ``sbin`` to your ``PATH`` environment variable, here is a one-liner: - -.. code-block:: console - - $ start-master.sh && start-worker.sh -m 24g spark://$(hostname):7077 - -Stop the Spark processes -~~~~~~~~~~~~~~~~~~~~~~~~ -Stop all of the processes when you complete your work. -.. code-block:: console - - $ stop-worker.sh && stop-master.sh - -.. _spark-on-hpc: - -HPC ---- -This section describes how you can run Spark jobs on any number of HPC compute nodes. -The scripts and examples described here rely on the SLURM scheduling system and have been tested -on NREL's Kestrel cluster. - -Install the Python package ``sparkctl`` to run the scripts described below. You can install it with -this command: - -.. code-block:: console - - $ pip install "sparkctl[pyspark]" - -The [sparkctl documentation](https://nrel.github.io/sparkctl/) has generic instructions to run -Spark in a variety of ways. The rest of this section calls out choices that you should make to run -Spark jobs with dsgrid. - -1. Choose compute node(s) with fast local storage. This example will allocate one node. Refer to this - `Kestrel documentation` _ - for more information for this type of compute node. - -.. code-block:: console - - $ salloc -t 01:00:00 -N1 --account=dsgrid --partition=nvme --mem=240G - -2. Configure Spark parameters based on the amount of memory and CPU in each compute node. - - This command must be run on a compute node. The script will check for the environment variable - ``SLURM_JOB_ID``, which is set by ``Slurm``. If you ssh'd into the compute node, it won't be set - and then you have to pass it as an argument. - -.. code-block:: console - - $ sparkctl configure --start - -Run ``sparkctl configure --help`` to see all options. - -Alternatively, or in conjunction with the above command, customize the Spark configuration files -in ``./conf`` as necessary per the HPC instructions. - -5. Ensure that the dsgrid application uses the Spark configuration that you just defined. - -.. code-block:: console - - $ export SPARK_CONF_DIR=$(pwd)/conf - $ export JAVA_HOME=/datasets/images/apache_spark/jdk-21.0.7 - -6. Follow the rest of the HPC instructions. - - -Tuning Spark Configuration Settings ------------------------------------ -In general you want to run Spark with as many executors as possible on each worker node. The -Amazon orchestration software along with the cluster manager *may* take care of that when -running on AWS (you will still have to adjust ``spark.sql.shuffle.partitions``). You will have to -perform more customizations when running a standalone cluster on your laptop or an HPC. - -There are multiple ways of setting parameters. These are listed in order of priority - later -methods will override the earlier methods when allowed. - -1. Global Spark configuration directory: This is ``$SPARK_HOME/conf`` or ``$SPARK_CONF_DIR``. - You can customize settings in ``spark-defaults.conf`` and ``spark-env.sh``. Make customizations - here if you will use the same settings in all jobs. - -2. Spark launch scripts: Use ``spark-submit`` to run scripts. Use ``pyspark`` to run interactively. - Both scripts offer the same startup options. You can choose to run in local mode or attach to - a cluster. You can override any setting from #1. Make changes here if you will use different - settings across jobs. Note that some settings must be made before the Spark JVM starts, like - ``spark.driver.memory``, and so this is your last chance to customize those values. - -3. SparkSession construction inside a Python process: You can customize things like executor - settings when you construct the ``SparkSession`` in Python. For example, this code block will - create a session where the job starts a single executor with a single core that uses all - available memory. - -.. code-block:: python - - from pyspark import SparkConf - from pyspark.sql import SparkSession - - conf = SparkConf().setAppName("my_app") - conf.set("spark.executor.cores", 1) - conf.set("spark.executor.memory", "16g") - conf.setMaster(cluster) - spark = SparkSession.builder.config(conf=conf).getOrCreate() - -4. Dynamic changes: You can make changes to a limited number of settings at runtime. You can't - change the number of executor cores because those have already been allocated. You can change - the number of shuffle partitions that Spark will use. You may want to change that value if the - sizes of the dataframes you're working on change dramatically. - -.. code-block:: python - - from pyspark.sql import SparkSession - - spark = SparkSession.getActiveSession() - spark.conf.set("spark.sql.shuffle.partitions", 500) - -Creating a SparkSession with dsgrid -=================================== -Ensure that the dsgrid software uses the cluster with optimized settings. If you start the dsgrid -Python process with the Spark scripts ``spark-submit`` or ``pyspark`` and set the ``--master`` -option, those scripts will create a SparkSession attached to the cluster and pass it to the Python -process. - -You can optionally set the ``SPARK_CLUSTER`` environment variable to the cluster URL and then -dsgrid will connect to it. - -.. code-block:: console - - $ export SPARK_CLUSTER=spark://$(hostname):7077 - -Using ``SPARK_CLUSTER`` is a bit simpler, but you cannot configure settings like -``spark.driver.memory``, which, as stated earlier, must be set before the JVM is created. - -spark-submit ------------- -Running dsgrid CLI commands through ``spark-submit`` requires cumbersome syntax because the tool -needs to - -1. Detect that the script is Python (which is why this example uses dsgrid-cli.py instead of - dsgrid). -2. Know the full path to the script (accomplished with the utility ``which``). - - Here's how to do that: - -.. code-block:: console - - $ spark-submit --master spark://$(hostname):7077 \ - $(which dsgrid-cli.py) \ - query project run \ - --registry-path=/scratch/${USER}/.dsgrid-registry \ - query.json - -.. note:: If you want to set a breakpoint in your code for debug purposes, you cannot use - spark-submit. - - -Spark Configuration Problems -============================ -Get used to monitoring Spark jobs in the Spark UI. The master is at -``http://:8080`` and jobs are at ``http://:4040``. If a job seems -stuck or slow, explore why. Then kill the job, make config changes, and retry. A misconfigured job -will take too long or never finish. - -This section explains some common problems. - -spark.sql.shuffle.partitions ----------------------------- -The most common performance issue we encounter when running complex queries is due to a non-ideal -setting for ``spark.sql.shuffle.partitions``. The default Spark value is 200. Some online sources -recommend setting it to 1-4x the total number of CPUs in your cluster. This -[video](https://www.youtube.com/watch?v=daXEp4HmS-E&t=4251s) by a Spark developer offers a -recommendation that has worked out better. - -Use this formula:: - - num_partitions = max_shuffle_write_size / target_partition_size - -You will have to run your job once to determine ``max_shuffle_write_size``. You can find it on the -Spark UI Stages tab in the Shuffle Write column. Your ``target_partition_size`` should be between -128 - 200 MB. - -The minimum partitions value should be the total number of cores in the cluster unless you want to -leave some cores available for other jobs that may be running simultaneously. - -Running out of space in local mode on the HPC ---------------------------------------------- -The ``/tmp`` directory on HPC filesystems is very small. If you run Spark local mode with default -settings, it will to use that directory for scratch space and then quickly fill it up and fail. -Set the environment variable ``SPARK_LOCAL_DIRS`` to an appropriate directory. - -.. code-block:: console - - $ export SPARK_LOCAL_DIRS=/tmp/scratch - -The scripts discussed above set this environment variable for standalone clusters on an HPC. - -Dynamic allocation ------------------- -This feature is disabled by default on standalone clusters. It is described -[here](https://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation). -This section does not necessarily line up with the Spark documentation and will change as we learn -more. - -We have observed two cases where enabling dynamic allocation on a standalone cluster significantly -improves performance: - -- When a Spark job produces a huge number of tasks. This can happen when checking dimension - associations as well as queries that map a dataset to project dimensions. -- When there is really only enough work for one executor but Spark distributes it to all executors - anyway. The inter-process communication adds tons of overhead. The executors also appear to do a - lot more work. With dynamic allocation Spark gradually adds executors and these problems don't - always occur. - -The second issue sometimes occurs when submitting a dataset to a project in the registry. -We recommend enabling dynamic allocation when doing this. If that is enabled then dsgrid code will -reconfigure the SparkSession to use a single executor with one core. - -Set these values in your ``spark-defaults.conf``. ``spark.shuffle.service.enabled`` must be set -before you start the workers. - -:: - spark.dynamicAllocation.enabled true - spark.dynamicAllocation.shuffleTracking.enabled true - spark.shuffle.service.enabled true - spark.shuffle.service.db.enabled = true - spark.worker.cleanup.enabled = true - -We have not observed any downside to having this feature enabled. - -Slow local storage ------------------- -Spark will write lots of temporary data to local storage during shuffle operations. If your joins -cause lots of shuffling, it is very important that your local storage be fast. If you direct Spark -to use the Lustre filesystem for local storage, you mileage will vary. If the system is idle, it -might work. If it is saturated, your job will likely fail. - -### Too many executors are involved in a job -Some of our Spark jobs, particularly those that create large, in-memory tables from dimension -records and mappings, perform much better when there is only one executor and only one CPU. If -you think this is happening then set these values in your ``spark-default.confg``:: - - spark.executor.cores 1 - # This value should be greater than half of the memory allocated to the Spark workers, which - # will ensure that Spark can only create one executor. - spark.executor.memory 15g - -One common symptom of this type of problem is that a job run in local mode works better than in a -standalone cluster. The likely reason is that the standalone cluster has many executors and local -mode only has one. - -A table has partitions that are too small or too large. -------------------------------------------------------- -Spark documentation recommends that Parquet partition files be in the range of 100-200 MiB, with -128 MiB being ideal. A very high compression ratio may change that. This affects how much data -each task reads from storage into memory. - -Check your partition sizes with this command: - -.. code-block:: console - - $ find -name "*.parquet" -exec ls -lh {} + - -Check the total size with this command: - -.. code-block:: console - - $ du -sh - -Divide the total size by the target partition size to get the desired number of partitions. - -If there are too few partitions currently: - -.. code-block:: python - - df.repartition(target_partitions).write.parquet("data.parquet") - -If there are too many partitions currently: - -.. code-block:: python - - df.coalesce(target_partitions).write.parquet("data.parquet") - -If you are partitioning by a column and find that there are many very small partition files, -repartition like this: - -.. code-block:: python - - df.repartition(column_name).write.partitionBy(column_name).parquet("data.parquet") - -.. _executors-spilling-to-disk: - -Executors are spilling to disk ------------------------------- -You are observing that the Spark job seems stalled. - -- Most of the work in a stage is complete, but one or two executors are still running. -- Only 1-2 CPUs out of the entire cluster show any activity. -- You see this log message over and over in the ``stderr`` log files - (e.g., ``spark_scratch/workers/app-20250115165825-0006/1/stderr``): - -.. code-block:: console - - 25/07/15 14:24:06 INFO UnsafeExternalSorter: Thread 60 spilling sort data of 4.6 GiB to disk (201 times so far) - -If this has occurred 201 times over more than an hour, then you might be better off canceling the -job and re-running with a different configuration. - -First, ensure that you have set ``spark.sql.shuffle.partitions`` to a reasonable value, as discussed -above. - -Second, your executor memory may be too small. Kestrel compute nodes have disproportionate CPUs with -respect to memory. By default, our configuration scripts will allocate 7 GB of memory per executor -and this is insufficient for some queries. The error message above indicates that the executor -was not able to perform a sort in memory, and so spilled to disk. This is very slow. - -Try to double or triple the executor memory. You can do this by setting the -``spark.executor.memory`` value in ``spark-defaults.conf``. You can set that value directly by -editing the file or indirectly by setting ``--executor-cores`` to 10 or 15 (default is 5), thereby -reducing the number of executors (by assigning each one more of the total available cores) and -giving each executor more memory. - -You can also acquire a bigmem node, which has 2 TB of memory. On these nodes, our Spark scripts will -allocate 70 GB of memory per executor. The debug partition on Kestrel usually has two bigmem nodes. - -Third, you may be experiencing data skew. This has happened frequently when performing dimension -mapping operations that explode data sizes by disaggregating or duplicating data. One or two -executors end up with significantly more data than the others, and get stuck. Refer to the next -section. - -Skew ----- -If your query will produce high data skew, such as can happen with a query that produces results -from large and small counties, you can use a salting technique to balance the data. For example, - -.. code-block:: python - - df.withColumn("salt_column", F.lit(F.rand() * (num_partitions - 1))) \ - .groupBy("county", "salt_column") \ - .agg(F.sum("county")) \ - .drop("salt_column") - -dsgrid will employ this technique for specific mapping types or when you enabled `handle_data_skew` -in a dataset's mapping plan. diff --git a/pyproject.toml b/pyproject.toml index 0caedc943..ce3398e4e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -115,11 +115,12 @@ doc = [ "numpydoc", "pandas-stubs", "ruff", - "sphinx~=7.2", - "sphinx-click~=5.0", - "sphinx-copybutton~=0.5.2", - "sphinx-tabs~=3.4", - "sphinx_argparse~=0.4.0", + "sphinx", + "sphinx-click", + "sphinx-copybutton", + "sphinx-tabs", + "sphinx-design", + "sphinx_argparse", "sphinxcontrib.programoutput", "autodoc_pydantic[erdantic]~=2.0", ]