[rocpd] Update CSV to SDK method of writing CSVs#111
Open
rocm-devops wants to merge 23 commits intoamd-stagingfrom
Open
[rocpd] Update CSV to SDK method of writing CSVs#111rocm-devops wants to merge 23 commits intoamd-stagingfrom
rocm-devops wants to merge 23 commits intoamd-stagingfrom
Conversation
- fix: missing 'sgpr_count' , 'arch_vgpr_count' and 'accum_vgpr_count' in kernel csv - add 'size' to scratch_memory csv - csv validation test added - TODO: check std::optional in libpyrocpd.cpp
* replace azure runners with internal * change to mi300a for debug * revert back to mi300 * move some of the load to mi300a * use mi300a for clang-tidy --------- Co-authored-by: Kandula, Venkateshwar reddy <Venkateshwarreddy.Kandula@amd.com>
* changing uuid abi * fix * review comments * fix CI fail * review comments * fix * adding static asserts * making constructor constexpr * fix CI fail * upadate UUID length to 16 bytes * fixing value64 Co-authored-by: Madsen, Jonathan <Jonathan.Madsen@amd.com> * Update CHANGELOG.md Co-authored-by: Madsen, Jonathan <Jonathan.Madsen@amd.com> --------- Co-authored-by: Madsen, Jonathan <Jonathan.Madsen@amd.com>
* Fix null handle - use .handle=0, not .handle=numeric_limits<>::max() * Update lib.common.hasher * Fix ROCPROFILER_CONTEXT_NONE * Use context operator== * Update CHANGELOG * Updated null handle for scratch memory and changed allocation test so that free ops account for null agent --------- Co-authored-by: Ian Trowbridge <Ian.Trowbridge@amd.com>
Increasing ROCr buffer to 4MBs.
* Adding callstack information to disassembly * changelog * Cleanup * Fix snapshots.json * Clang tidy fixes * Fix infinite recursion * Apply suggestions from code review Co-authored-by: Indic, Vladimir <Vladimir.Indic@amd.com> * Remove sibling transversal * Added docstrings * Apply suggestions from code review * Update source/include/rocprofiler-sdk/cxx/codeobj/code_printing.hpp * Review comments * Format + comments * Fmt * Add class name * Format * Fix static linkage * Making funcs inline --------- Co-authored-by: Giovanni <gbaraldi@amd.com> Co-authored-by: Indic, Vladimir <Vladimir.Indic@amd.com>
* Add sample data for avail and remove color code for non terminal output * review comments * review comments * add documentation * test fix
Co-authored-by: Bhardwaj, Gopesh <Gopesh.Bhardwaj@amd.com>
…how counters being profiled in metadata. (#325) * expose dimensional info in rocprofiler_counter_info_v1_t. * add counter_id in dim info. * address review comments * format. * address comments. * use array of pointers for dimensions_instaces. * format and comments. * address comments. * new line. * Update counter_defs.yaml * Update counter_defs.yaml * Update counter_defs.yaml * counter_defs. * format counter defs. * format counter defs. * format counter defs. * show only counters being profiled in metadata. * Format. * use config for counters and fix warnings. * add version for rocprofiler_counter_dimension_info_v1_t struct. * rename rocprofiler_counter_record_dimension_instance_v1_info_t. * account device id from pmc for counters metadata. * move dim structs to counters.h. * address comments to compare value. * fix tests. * Address comments. use pointer of arrays for ABI. * rebase. * fix build error. * use separate metadata::init() for rocprofv3. * also print not found counters. * precompute all the perf counters needed to be in metadata. * Misc. * format * Format. * rocprofiler::sdk::container::c_array * Address comments. * source/lib/output/metadata.cpp * lint. * add unit test for c_array. * add unit test and serialization support for c_array container. * Misc. * Clean files. * Format. * clang-tidy. * add more checks to c_array. * misc. typo * Addr comments. --------- Co-authored-by: Venkateshwar Reddy Kandula <vkandula@amd.com> Co-authored-by: Jonathan R. Madsen <Jonathan.Madsen@amd.com>
* Adding test and samples to decoder * Fix sample * Formatting * Fix multi test * Disable sample * Fix tests * Format * Version fix * Locking the decoder * Add atomic * Review comments * Format * Adding readme * merge conflict and adding PCS+ATT test * Review comments * Properly disable PCS test * Update tests/rocprofv3/advanced-thread-trace/CMakeLists.txt * Adding back env var test * Name fix * Preload sample * Addressing review comments * Update docs --------- Co-authored-by: Giovanni Baraldi <gbaraldi@amd.com>
- Fix context tracing domain bitset overflow - Previous behavior would enable all flags above ROCPROFILER_BUFFER_TRACING_MARKER_CORE_RANGE_API when this domain was enabled.
…n (#480) * Fix kernel dispatch for counter collection * Updated change log * Fix format * rename output csv file * Fix warnings * Address review comment * Address final review comment
Fix thread trace samples set tests properties
…se + query submodule + rocpd command-line tools (#488) * adding summary.py to generate tmp <category_region>_summary views * migrating CSV summary to SDK method of writing CSVs - Add domain_view to summary.py - omit the C++ code of writing CSV because it gets revered later anyway * Add summary subparser and write_sql_view_to_csv function * adding all <>_summary views generation to summary.py * add summary_per_rank feature * add --summary-per-rank * reconstruct generate_summary_view and create_domain_view -introduce by_rank * remove sqr and variance in summary views * use RocpdImportData instead of connection * two fixes on summary.py --modify the generate_summary_view function to return a tuple with view name and sql code add if_not_exits parameter to generete_summary_view * Refactor summary.py to allow output path and filename args, and apply time_window - clean up summary table column headers - only generate by-rank views if that param is specified * Add ProcessID to Hostname output and csv, so users can identify the system in the by-rank summaries * Summary.py, just add hostname to by-rank summaries, instead of creating mapping table * Summary - migrate csv writer to pandas, for more future flexibility * Adding a few simple tests for summary.py * Linting fixes * add region_categories to summary options - Automatically retrieve region categories from the database if argument is None * add backticks for view_names * fix tests after rebase * Made code review changes - fixed whitespace in CMakelists.txt - adding query.py module & subparser in __main__.py - refactor summary function to return query - used query.py to output csv - used query.py to also output summary to console - provided new command line options to select summary output to csv or console * Made fix to jinja template in query.py, as suggested by copilot * Consolidated output calls to query in export_view function based on feedback - refactored: helpers, query functions, create view functions - extended formats to include what query supports (md, html, pdf, json) - added json format to query, and changed orient=records - adding jinja2 and reportlab to requirements.txt * Add version_info for rocpd and roctx * Add rocpd commandline tool * Add executable permissions to source/bin/rocpd.py * Removed rocpd2query, and cleaned up --help examples --------- Co-authored-by: acanadas <acanadas@amd.com> Co-authored-by: Jin Tao <jintao12@amd.com> Co-authored-by: a-canadasruiz <Araceli.CanadasRuiz@amd.com> Co-authored-by: Jonathan R. Madsen <Jonathan.Madsen@amd.com>
* PCS test: cast agent name to str
* Removing SQTT buffer size limitation * Update source/lib/rocprofiler-sdk/thread_trace/core.cpp * Added testing for buffer size. Formatting. * Add test as unstable * Increase default buffer size * Apply suggestions from code review Co-authored-by: Indic, Vladimir <Vladimir.Indic@amd.com> * Fix typo from code review * Update tests/thread-trace/agent.cpp --------- Co-authored-by: Giovanni <gbaraldi@amd.com> Co-authored-by: Indic, Vladimir <Vladimir.Indic@amd.com>
* SWDEV-544115 Adding documentaiton for rocprofv3 advanced options * minor changes * updating rocpd documentation * updated changelog * adressed Feedback
* use new azure runners * update core-deb logic in continuous_integration.yml * more logic changes in continuous_integration.yml * changes to runs-on conditional in continuous_integration.yml * update runs-on conditional again on continuous_integration.yml * utulize matrix system list, add gpu variable and separate runner * Remove trailing comma from matrix * Delete .github/workflows/generate_afar.yml --------- Co-authored-by: Kandula, Venkateshwar reddy <Venkateshwarreddy.Kandula@amd.com> Co-authored-by: Elwazir, Ammar <Ammar.Elwazir@amd.com>
…d-comgr. (#555) * remove samples dependency on rocprofiler-sdk-amd-comgr. * add find package for amd_comgr. --------- Co-authored-by: Venkateshwar Reddy Kandula <vkandula@amd.com>
* Enable MI100 counters * Update counter_defs.yaml * Adding MI100 counters to tests * Added TCC counters --------- Co-authored-by: Sushma Vaddireddy <svaddire@amd.com>
- fix: missing 'sgpr_count' , 'arch_vgpr_count' and 'accum_vgpr_count' in kernel csv - add 'size' to scratch_memory csv - csv validation test added - TODO: check std::optional in libpyrocpd.cpp
rocm-devops
requested review from
ApoKalipse-V,
SrirakshaNag,
bgopesh,
bwelton,
jrmadsen and
vlaindic
as code owners
Author
|
This PR also includes changes to address #497 |
Author
|
@yhui_amdeng maybe this was my fault but if we are going to re-write the CSV output, it should be in Python. I mentioned I didn't feel like there needed to be any reason to drop down into C++ for writing CSV. |
Author
@jomadsen_amdeng Oh, is that what you intended? That was not understood. This PR was created to get rid of the CSV manager that existed before, and all the messy escape characters/quotes, etc, in the CSV headers. And to adopt the CSV style that generateCSV.cpp has. You commented that the escape characters/quotes were too error prone, so we wanted to simplify the rocpd CSV implementation like this first. Yes, we can certainly implement the CSV writes in python, and avoid dropping into C++ too. That'll take a bit longer. I'll get the ball rolling on that, in parallel. If you really don't want this, even for 7.0.1, then we can close this PR, but just keep in mind, the current rocpd csv.cpp implementation, will be quite error prone :( |
Contributor
|
This pull request has merge conflicts that need to be resolved. It cannot be imported to the ROCm/rocm-systems repo automatically. |
ammallya
pushed a commit
that referenced
this pull request
Oct 28, 2025
[ROCm/rocprofiler-sdk commit: 042c761]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
14 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR Details
Refactor CSV code to be more similar to SDK CSV code
Associated Jira Ticket Number/Link
https://ontrack-internal.amd.com/browse/SWDEV-543953
What type of PR is this? (check all applicable)
Technical details
Thru code review, @jomadsen_amdeng suggested the CSV generation code use similar functions/style as rocprofiler-SDK CSV code, so this refactor is to remove the cumbersome header format, and adopt the SDK way, which should be less error-prone and less of a maintenance burden.
Added/updated tests?
Some more in-depth CSV tests were added.
Updated CHANGELOG?
Added/Updated documentation?