[rocpd] Schema Updates

[systems-assistant]

PR Details

Associated Jira Ticket Number/Link

What type of PR is this? (check all applicable)

• Refactor
• Feature
• Bug Fix
• Optimization
• Documentation Update
• Continuous Integration

Technical details

Added/updated tests?

• Yes
• No, Does not apply to this PR.

Updated CHANGELOG?

• Yes
• No, Does not apply to this PR.

Added/Updated documentation?

• Yes
• No, Does not apply to this PR.

---

🔁 Imported from ROCm/rocprofiler-sdk#118
🧑‍💻 Originally authored by @rocm-devops

[bwelton]

This may have been on the original PR, but can you add some context about why this is being changed and what the effect of the change is?

[yhuiYH]

Suggested change

	for name in tbls:
	log(f"Inserting rows into {name} from {alias}.{name}")
	conn.execute(f'INSERT INTO "{name}" SELECT * FROM {alias}."{name}"')
	for name in tbls:
	log(f"Inserting rows into {name} from {alias}.{name}")
	rows = conn.execute(f'SELECT * FROM {alias}."{name}"').fetchall()
	if rows:
	col_count = len(rows[0])
	placeholders = ", ".join(["?"] * col_count)
	conn.executemany(
	f'INSERT INTO "{name}" VALUES ({placeholders})',
	rows
	)

Looking at merge.py, @bgopesh suggested I change this to a batch operation in my PR. Doing that in your implementation will halve the merge time! I'm looking at ways of incorporating your merge.py with the merge.py that the Silo team wrote in my PR. But this is a good change to pickup.

[yhuiYH]

This integrity check (I see you have it marked as optional here in the comments) will add a 66%-175% time hit to merge time. It may be better to skip it, or have it as a user-opt-in check (but what user will really want to turn this on?).
ie, merging 2 13MB DBs, merge time goes from 3.67 seconds to 1.33 seconds when we skip this check.
merging 16 13MB DBs, merge time goes from 37.03 seconds to 22.23 second when we skip this check.