feat: Add no-customization.sh for disk usage metrics#116
Conversation
This commit introduces `examples/secure-boot/no-customization.sh`, a new script designed primarily for collecting disk usage metrics during custom image builds. This script directly addresses the need for detailed disk analysis, a critical component in ensuring Dataproc images remain lightweight and optimized, as emphasized in the recent Dataproc 2.3 release which focused on reduced CVEs and smaller image footprints Dataproc on GCE image version 2.3 (optimized for FedRAMP _ Compliance).pdf]. During the development and review of the Dataproc 2.3 custom images, this `no-customization.sh` script, along with similar disk metric collection logic in `install_gpu_driver.sh`, was instrumental. These tools allowed for precise measurement of disk consumption at various stages of image creation. The data gathered directly informed decisions regarding package inclusions and default disk sizes, contributing significantly to the ~70% reduction in open-source software components and ~50% reduction in total CVEs observed in Dataproc 2.3 images Dataproc on GCE image version 2.3 (optimized for FedRAMP _ Compliance).pdf]. The metrics captured by this script (and the `install_gpu_driver.sh` exit handler) were fed back into `pre-init.sh` and other image generation orchestrators, allowing for iterative refinement and validation of the image size. This continuous feedback loop was crucial for achieving the lightweight and compliant image goals for Dataproc 2.3. The inclusion of this script in the repository makes this valuable diagnostic tool available for future image optimization efforts, especially relevant for new AI/ML images or subsequent releases. The `no-customization.sh` script leverages existing patterns for disk usage monitoring and cleanup within the custom images repository, including the use of `df`, `perl` for metric calculation, and the `dd` command for zeroing free space when `creating-image` metadata is present.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @cjac, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request introduces a new shell script, no-customization.sh, aimed at collecting and analyzing disk usage metrics during custom image builds. This tool is intended to provide crucial insights for optimizing image sizes, a process that was instrumental in achieving significant reductions in image footprint and CVEs for Dataproc 2.3 images. Its inclusion makes this diagnostic capability available for ongoing and future image optimization efforts.
Highlights
- New Script for Disk Metrics: Introduces
examples/secure-boot/no-customization.sh, a new shell script specifically designed to monitor and log disk space consumption during custom image builds. - Automated Disk Usage Analysis: The script incorporates an
exit_handlerthat processes the collecteddflogs upon completion, calculating and reporting key metrics such as starting, ending, maximum, and minimum disk usage, as well as the total increase. - Continuous Monitoring Mechanism: A
screensession is utilized to continuously recorddfoutput every 5 seconds, providing a detailed timeline of disk usage throughout the image build process. - Optional Disk Zeroing: Includes logic to conditionally zero out free disk space using the
ddcommand if acreating-imagemetadata attribute is present, which is a common practice for optimizing image size.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
There was a problem hiding this comment.
Code Review
This pull request introduces a new script for monitoring disk usage during image creation. The review focuses on a critical bug in the disk usage calculation logic within a Perl script and suggests a more robust and readable awk-based replacement. A potential race condition related to process synchronization is also pointed out.
| perl -e '($first, @samples) = grep { m:^/: } <STDIN>; | ||
| unshift(@samples,$first); $final=$samples[-1]; | ||
| ($starting)=(split(/\s+/,$first))[2] =~ /^(\d+)/; | ||
| ($ending)=(split(/\s+/,$final))[2] =~ /^(\d+)/; | ||
| @siz=( sort { $a => $b } | ||
| map { (split)[2] =~ /^(\d+)/ } @samples ); | ||
| $max=$siz[0]; $min=$siz[-1]; $inc=$max-$starting; | ||
| print( " samples-taken: ", scalar @siz, $/, | ||
| "starting-disk-used: $starting", $/, | ||
| " ending-disk-used: $ending", $/, | ||
| " maximum-disk-used: $max", $/, | ||
| " minimum-disk-used: $min", $/, | ||
| " increased-by: $inc", $/ )' < "/run/disk-usage.log" |
There was a problem hiding this comment.
The sort comparator { $a => $b } is incorrect for numeric sorting. The => operator (fat comma) is used for key/value pairs and does not perform the intended numeric comparison. This will result in incorrect maximum and minimum disk usage values being reported. For a descending numeric sort, you should use the numeric comparison operator <=>, like this: sort { $b <=> $a }.
Consider replacing the Perl script with a more robust, readable, and memory-efficient awk equivalent. awk is a standard tool well-suited for this kind of line-by-line processing.
awk '
/\/\// {
used = $3;
if (samples++ == 0) {
start = used;
min = used;
max = used;
}
if (used < min) { min = used; }
if (used > max) { max = used; }
last = used;
}
END {
if (samples > 0) {
inc = max - start;
print " samples-taken: " samples
print "starting-disk-used: " start
print " ending-disk-used: " last
print " maximum-disk-used: " max
print " minimum-disk-used: " min
print " increased-by: " inc
}
}' "/run/disk-usage.log"There was a problem hiding this comment.
Thanks. That was a typo of >= I think. I'll patch in a fix and amend the commit.
Regarding not using Perl: no thank you. My intention is to provide support for on Google platforms Perl at the same level as PHP and Python. Instead, let's work on ensuring the language has native support for all necessary interop interfaces. Join me on the p5p ml to discuss further.
| # Process disk usage logs from installation period | ||
| rm -f /run/keep-running-df | ||
| sync | ||
| sleep 5.01s |
There was a problem hiding this comment.
Using a fixed sleep to synchronize with the background df process is fragile and can lead to race conditions. If the system is under heavy load, the df command or other operations could be delayed, potentially causing this handler to proceed before the final disk usage is logged. A more robust approach would be to manage the background process via its Process ID (PID), allowing you to explicitly wait for it to terminate.
There was a problem hiding this comment.
You'll note on review of install_gpu_driver.sh that the df -h is interleaved with calls to sync. Can you tell me why I might have designed these disk usage checks in this manner?
|
this should go into #120 |
1 participant
This commit introduces
examples/secure-boot/no-customization.sh, a new script designed primarily for collecting disk usage metrics during custom image builds. This script directly addresses the need for detailed disk analysis, a critical component in ensuring Dataproc images remain lightweight and optimized, as emphasized in the recent Dataproc 2.3 release which focused on reduced CVEs and smaller image footprintsDuring the development and review of the Dataproc 2.3 custom images, this
no-customization.shscript, along with similar disk metric collection logic ininstall_gpu_driver.sh, was instrumental. These tools allowed for precise measurement of disk consumption at various stages of image creation. The data gathered directly informed decisions regarding package inclusions and default disk sizes, contributing significantly to the ~70% reduction in open-source software components and ~50% reduction in total CVEs observed in Dataproc 2.3 imagesThe metrics captured by this script (and the
install_gpu_driver.shexit handler) were fed back intopre-init.shand other image generation orchestrators, allowing for iterative refinement and validation of the image size. This continuous feedback loop was crucial for achieving the lightweight and compliant image goals for Dataproc 2.3. The inclusion of this script in the repository makes this valuable diagnostic tool available for future image optimization efforts, especially relevant for new AI/ML images or subsequent releases.The
no-customization.shscript leverages existing patterns for disk usage monitoring and cleanup within the custom images repository, including the use ofdf,perlfor metric calculation, and theddcommand for zeroing free space whencreating-imagemetadata is present.