From 4c611b238c30825b1181945e8d0408dad701e6a4 Mon Sep 17 00:00:00 2001 From: denisova-ok Date: Fri, 27 Jun 2025 14:41:57 +0300 Subject: [PATCH 1/3] add eng instructions --- src/common/analysis/box_plot_en.md | 59 +++++++++++++ .../analysis/dep_graph/wiki/README_en.md | 60 +++++++++++++ .../dep_graph/wiki/dep_graph_analysis_en.md | 15 ++++ .../analysis/dep_graph/wiki/dep_graph_en.md | 23 +++++ .../packages-charts/packages_charts_en.md | 70 +++++++++++++++ .../read_write_statistics/instruction_en.md | 19 ++++ .../analysis/statistics_analyzer/README_en.md | 87 +++++++++++++++++++ src/conf/README_en.md | 7 ++ src/experiment/README_en.md | 74 ++++++++++++++++ src/experiment/auto_conf/README.md | 2 +- src/experiment/auto_conf/README_en.md | 55 ++++++++++++ src/experiment_2/README_en.md | 31 +++++++ 12 files changed, 501 insertions(+), 1 deletion(-) create mode 100644 src/common/analysis/box_plot_en.md create mode 100644 src/common/analysis/dep_graph/wiki/README_en.md create mode 100644 src/common/analysis/dep_graph/wiki/dep_graph_analysis_en.md create mode 100644 src/common/analysis/dep_graph/wiki/dep_graph_en.md create mode 100644 src/common/analysis/packages-charts/packages_charts_en.md create mode 100644 src/common/analysis/read_write_statistics/instruction_en.md create mode 100644 src/common/analysis/statistics_analyzer/README_en.md create mode 100644 src/conf/README_en.md create mode 100644 src/experiment/README_en.md create mode 100644 src/experiment/auto_conf/README_en.md create mode 100644 src/experiment_2/README_en.md diff --git a/src/common/analysis/box_plot_en.md b/src/common/analysis/box_plot_en.md new file mode 100644 index 00000000..d249f824 --- /dev/null +++ b/src/common/analysis/box_plot_en.md @@ -0,0 +1,59 @@ +# box_plot.py +Python script for plotting graphs based on the results of the patch experiment (speeding_up) + +## Description +The script is designed to build graphs based on the results of the patch experiment (speeding_up). It reads data from text files containing the experiment results and plots the graphs. + +## How to run the program? +To configure the program's behavior, a command-line interface is implemented: +```bash +options: + -h, --help show this help message and exit + --file_path_no_pathes FILE_PATH_NO_PATHES + Enter the path to the file + --file_path_net_patch FILE_PATH_NET_PATCH + Enter the path to the file + --file_path_childrens_patch FILE_PATH_CHILDRENS_PATCH + Enter the path to the file + --file_path_all_patches FILE_PATH_ALL_PATCHES + Enter the path to the file + --plot_name PLOT_NAME + Enter title of plot + --save, --no-save Save the plot (default: True) + --show, --no-show Show the plot (default: True) + --standard_boxplot, --no-standard_boxplot + Use standard box plot (default: False) +``` + +### Arguments +- `file_path_no_pathes`, `file_path_net_patch`, `file_path_childrens_patch`, `file_path_all_patches`: arguments to specify paths to text files — results of the build experiment for `core-image-minimal` without patches, with the network load balancing patch, with the task priority patch, and with both patches respectively. By default, the script expects files `time1.txt`, `time2.txt`, `time3.txt`, `time4.txt` to be located in the subdirectory `speeding_up_results` relative to the script location. +- `plot_name`: the name used as the plot title and for saving the plot as a PNG file. Defaults to `Patch_analysis`. +- `save` and `show`: values to configure the program's behavior — whether to show or not show, save or not save the resulting plots. By default, the plot is both shown and saved. +- `standard_boxplot`: a flag to disable all customizations, leaving only the standard `boxplot` and plotting the values as red dots. + +## How to read the graphs? +- The y-axis shows the experiment execution time (in minutes). +- The x-axis shows which experiment was conducted (without patches, with the network load balancing patch, with the task priority patch, and with both patches). + +### Legend +- **Blue "boxes"** represent values between the 25th and 75th percentiles. +- **Blue horizontal lines ("whiskers")** represent values within 1.5 interquartile range from the quartiles. These lines end with blue caps. +- **Black dashed lines** represent the 1st and 99th percentiles. +- **Red dots** indicate values in the ranges [1st percentile, 25th percentile] and [75th percentile, 99th percentile]. +- Inside the blue box, the median value is labeled in black. +- The 1st and 99th percentiles are labeled in green and blue text respectively. + +## How to get the data? +It is necessary to run the `speeding_up_experiment` (src/common/scripts/speeding_up_experiment.sh) with different patch sets. It is recommended to specify the number of repetitions greater than 10 (in the current version we specify 30). + +The experiment saves the build results and elapsed time in the folder `src/buildstats_saves`. There is a file time.txt used for plotting the graphs. + +## Script output results + +### Result with the --standard_boxplot flag + +![Patch_analysis.png](./Patch_analysis.png) + +### Result without the --standard_boxplot flag + +![Patch_analysis_full.png](./Patch_analysis_full.png) diff --git a/src/common/analysis/dep_graph/wiki/README_en.md b/src/common/analysis/dep_graph/wiki/README_en.md new file mode 100644 index 00000000..b3ce4d10 --- /dev/null +++ b/src/common/analysis/dep_graph/wiki/README_en.md @@ -0,0 +1,60 @@ +## Using the program for sorting graph nodes and finding offsets + +First, it is necessary to run the parser and collect statistics (instructions for running the parser and collecting statistics are described here: https://github.com/moevm/os_profiling/blob/denisova_olga_parsing_ranking/statistics_analyzer/README.md) + +Also, before analyzing the dependency graph, it is necessary to create the file task-depends.dot (creating this file and visualizing the graph is described here: https://github.com/moevm/os_profiling/blob/denisova_olga_dep_graph/dep_graph/wiki/dep_graph.md) + +Then you can create a graph object and analyze it: + + G = nx.DiGraph(nx.nx_pydot.read_dot('./task-depends.dot')) + sorted_tasks = sort_start_time(parser.info) + sorted_nodes = [] + for node in G.nodes: # for each node find the corresponding index in sorted_tasks + index, start, end = match(node, sorted_tasks) + sorted_nodes.append((index, node, start, end)) + + sorted_nodes = sorted(sorted_nodes, key=lambda x: x[0]) + # write(sorted_nodes, 'tasks-order.txt') # tasks can be written to a file in the order of their start time + +In the example above, graph nodes are matched with objects from the statistics, and nodes are numbered in the order of their start time. + +For each node, you can find its "children" and find the offset between the end of the child's execution and the start of the parent's execution: + + results = [] + for node in G.nodes: + for child in G.neighbors(node): + for temp in sorted_nodes: + if temp[1] == node: + result1 = temp + if temp[1] == child: + result2 = temp + if result1[2] != -1 and result2[3] != -1: + offset = result1[2] - result2[3] + else: + offset = -1 + results.append((f'node: {node}, Started: {result1[2]}, child: {child}, Ended: {result2[3]}', offset)) + + results = sorted(results, key=lambda x: x[1], reverse=True) + + with open('task-order-sorted-offset.txt', 'w') as file: + file.writelines(f"{item[0]}, offset: {item[1]}\n" for item in results) + +In the example above, a file is created that contains information about each parent-child node pair and their offset. + +## Other ways to analyze the graph + +1) The method nx.bfs_layers allows composing BFS layers, which helps to look at the overall view of the graph's interconnections + + print(dict(enumerate(nx.bfs_layers(G, ['core-image-sato.do_build'])))) + +2) The method nx.is_tree helps check whether the graph is a tree + + print(nx.is_tree(G)) + +3) The method nx.is_connected allows finding out whether the graph is connected + + print(nx.is_connected(G)) + +4) The method nx.is_directed_acyclic_graph allows finding out whether the graph is acyclic + + print(nx.is_directed_acyclic_graph(G)) diff --git a/src/common/analysis/dep_graph/wiki/dep_graph_analysis_en.md b/src/common/analysis/dep_graph/wiki/dep_graph_analysis_en.md new file mode 100644 index 00000000..6067fb5a --- /dev/null +++ b/src/common/analysis/dep_graph/wiki/dep_graph_analysis_en.md @@ -0,0 +1,15 @@ +## Dependency Graph Analysis +### Checking for the Presence of Independent Nodes +Using the `nx.bfs_layers` method, "layers" of the dependency graph were found, that is, sets of nodes located at the same "distance" from a node chosen as the zero layer. The node `core-image-sato.do_rootfs` was selected as the zero layer node. Thus, if any node had no connections to the main set of nodes, it should not have appeared in any layer. Then a comparison was made between the total number of nodes in all layers and the total number of nodes in the graph; both turned out to be equal to 8782, which tells us that there are no independent nodes (or sets of nodes). + +### Finding the "Root" +Tasks are executed from the leaves to the "root", meaning at the start of the build, the leaf tasks run in parallel. By analyzing the `.dot` file, it was found that the last task to be executed (and the root of the graph) is the node `core-image-sato.do_build`, since no nodes depend on it. + +### Checking for a Tree Structure +It was determined that the dependency graph does not have a tree structure. This was clearly visible during further visualization of the layers. + +### Finding the Offset Between the End of the Child Node Execution and the Start of the Parent Node Execution +For each node, neighbor nodes were found (i.e., those nodes that are in direct dependency). Since the graph is directed, an edge from node A to node B exists if and only if node A depends on node B (in other words, node A is the parent of node B). For each such pair (parent node – child node), the time when the child node finished execution and when the parent node started execution was found (in this order, since the build goes from leaves to root). The offset (difference of the mentioned timestamps) was calculated, and a list of "parent-child" node pairs was created in order of decreasing offset. It turned out that the largest offsets belong to pairs where the parent node has many child nodes (i.e., the parent task depends on a large number of child tasks). + +### Visualization of Graph Layers +Using a script, smaller `.dot` files were obtained from the single `task-depends.dot` file created by BitBake according to BFS layers and visualized using Graphia. The link to the folder with screenshots: https://drive.google.com/drive/folders/1iDxxq7sxxZWLCpOhBl2Xb6UEQneB_4qW?usp=drive_link. The number in the file name corresponds to the layer number. The screenshot shows the nodes of the corresponding layer and their direct connections (for example, for the first layer, the screenshot shows nodes of the first layer as well as nodes from the 0th and 2nd layers, since the nodes of the 1st layer are connected to them). diff --git a/src/common/analysis/dep_graph/wiki/dep_graph_en.md b/src/common/analysis/dep_graph/wiki/dep_graph_en.md new file mode 100644 index 00000000..1a1b40c0 --- /dev/null +++ b/src/common/analysis/dep_graph/wiki/dep_graph_en.md @@ -0,0 +1,23 @@ +## Dependency Graph Construction + +To build the dependency graph, the file `task-depends.dot` is required, which can be obtained using the following command: `bitbake -g core-image-sato`. + +After obtaining the `task-depends.dot` file, we can build the dependency graph. The dependency graph was constructed in two ways: + +1) Using Python with the `plotly.graph_objects` library, an `.html` file was created containing the visualization of the dependency graph: +![image](https://github.com/moevm/os_profiling/assets/90854310/b7b2cced-3de4-4468-81e3-2fb6def2dc6f) + +In this file, it is possible to zoom in on specific parts of the graph; for example, let’s look at the node `core-image-sato.do_rootfs`: +![image](https://github.com/moevm/os_profiling/assets/90854310/7611a754-5283-4c77-8690-3b1cd9fe557e) + +2) The same dependency graph was also built using Graphia; in this case, the visualization of the graph is three-dimensional. The appearance of the graph: +![image](https://github.com/moevm/os_profiling/assets/90854310/8248dc71-0027-4b9b-9a1a-c95e4ef1dae7) +![image](https://github.com/moevm/os_profiling/assets/90854310/45265149-d92e-4017-a012-c34363296736) + +It is also possible to find a node by search and zoom in on a specific node. + +For `core-image-sato`, the resulting graph consisted of 8782 nodes and 33883 edges. + +## Questions Regarding the Dependency Graph + +The main question is whether the dependency graph has a root; in that case, it would be more convenient to perform traversal. diff --git a/src/common/analysis/packages-charts/packages_charts_en.md b/src/common/analysis/packages-charts/packages_charts_en.md new file mode 100644 index 00000000..d1c2b2d0 --- /dev/null +++ b/src/common/analysis/packages-charts/packages_charts_en.md @@ -0,0 +1,70 @@ +# Mapping Resource Usage to Packages + +## Available Data + +To map resource usage to specific packages, log files located in the `build/tmp/buildstats//` directory were analyzed. This directory contains subdirectories for each package involved in the build process. Each subdirectory includes log files such as `do_configure`, `do_install`, `do_compile`, etc. The log format is described in `yocto_buildstats.md`: +[yocto_buildstats.md](https://github.com/moevm/os_profiling/blob/denisova_olga_yocto_buildstats/wiki/yocto_buildstats.md) + +From these logs, we can extract the following characteristics: + +1. **Elapsed time** – total time taken +2. **utime** – user mode time +3. **stime** – system (kernel) mode time +4. **cutime** – user mode time of child processes +5. **cstime** – system mode time of child processes +6. **IO rchar** – bytes read from storage +7. **IO read_bytes** – estimated number of bytes read from storage (accurate for block-based file systems) +8. **IO wchar** – bytes written to disk +9. **IO write_bytes** – estimated bytes written to storage +10. **IO cancelled_write_bytes** – bytes written that were canceled due to page cache truncation +11. **IO syscr** – estimated number of read I/O operations +12. **IO syscw** – estimated number of write I/O operations +13. **rusage ru_utime** – user CPU time in seconds +14. **rusage ru_stime** – system CPU time in seconds +15. **rusage ru_maxrss** – peak resident set size in KB +16. **rusage ru_minflt** – number of minor page faults (handled without disk I/O) +17. **rusage ru_majflt** – number of major page faults (required disk I/O) +18. **rusage ru_inblock** – block input operations +19. **rusage ru_oublock** – block output operations +20. **rusage ru_nvcsw** – voluntary context switches +21. **rusage ru_nivcsw** – involuntary context switches +22. **Child rusage ru_utime** – same as (2), but for all child processes +23. **Child rusage ru_stime** – same as (3), for all child processes +24. **Child rusage ru_maxrss** – same as (15), for all child processes +25. **Child rusage ru_minflt** – same as (16), for all child processes +26. **Child rusage ru_majflt** – same as (17), for all child processes +27. **Child rusage ru_inblock** – same as (18), for all child processes +28. **Child rusage ru_oublock** – same as (19), for all child processes +29. **Child rusage ru_nvcsw** – same as (20), for all child processes +30. **Child rusage ru_nivcsw** – same as (21), for all child processes + +--- + +## Generating Resource Usage Charts + +A Python script was created to generate visual charts showing resource usage per package. The most insightful metrics are: + +- `utime`, `stime`, `cutime`, `cstime` +- `IO rchar`, `IO wchar` +- `rusage ru_utime`, `rusage ru_stime`, `rusage ru_maxrss` +- `Child rusage ru_utime`, `Child rusage ru_stime` + +Each metric is visualized as a separate graph and saved as a PNG image. + +Example chart: +![rusage ru_utime](https://github.com/moevm/os_profiling/assets/90854310/1d37013a-f817-43a9-876c-f306813b2d12) + +All generated charts are located in the `charts/` directory. +The code used to generate them is in the `make-charts.py` file. + +--- + +## Mapping Resources to Processes + +The directory `build/tmp/work` contains working subdirectories related to a specific architecture. These subdirectories include data about the packages. Each package directory contains a `temp/` folder with log files for each task (e.g., `log.do_.pid`) and scripts used by BitBake (`run.do_.pid`). These files are created and filled during the build. + +All packages from `build/tmp/work` match those in `build/tmp/buildstats`, with the exception of `gcc-source-13.2.0-13.2.0`. +The assumption is that using the PID in the filename, we can match specific resource metrics to the associated process. + +This directory structure is described in the official documentation: +[Yocto Project Reference Manual – Section 4.2.7.9](https://docs.yoctoproject.org/ref-manual/structure.html) diff --git a/src/common/analysis/read_write_statistics/instruction_en.md b/src/common/analysis/read_write_statistics/instruction_en.md new file mode 100644 index 00000000..16d5694d --- /dev/null +++ b/src/common/analysis/read_write_statistics/instruction_en.md @@ -0,0 +1,19 @@ +# Guide for Processing Read/Write Log Statistics + +1. To collect read/write statistics logs, run the build with the following command: + ```bash + strace -f -e trace=read,write,readv,writev -o yocto_trace_all.log bitbake core-image-minimal + ``` + +2. You also need a file containing a list of tasks with their corresponding PIDs. + This can be obtained by running the task ranking as described in the [ranking instructions](../statistics_analyzer/README.md). + After completion, the file `ranking_output.txt` will be generated. + +3. Next, process the logs using the `process_logs.py` script: + ```bash + python3 process_logs.py -l -t + ``` + +4. After execution, two files will be created in the `read_write_statistics/output/` directory: + - `process_statistics_rw.txt`: contains statistics for `read` and `write` operations + - `process_statistics_rwv.txt`: contains statistics for `readv` and `writev` operations diff --git a/src/common/analysis/statistics_analyzer/README_en.md b/src/common/analysis/statistics_analyzer/README_en.md new file mode 100644 index 00000000..ab11e7a7 --- /dev/null +++ b/src/common/analysis/statistics_analyzer/README_en.md @@ -0,0 +1,87 @@ +# Running the Parser + +Any program that uses internal statistics collection via the parser should be launched as follows: + +``` +python3 ./.py --poky -b +``` + +or + +``` +python3 ./.py --poky -t +``` + +--- + +## Using the Parser and Running Task Ranking + +First, retrieve information about available builds and get a timestamp-sorted list of builds: + +```python +args = create_parser_args() +timestamp = '' +timestamp_list = [] +poky_buildstats_path = os.path.join(args.poky_path, 'build/tmp/buildstats') +tree = list(os.walk(poky_buildstats_path)) +for item in tree: + if item[0] == poky_buildstats_path: + timestamp_list = item[1] +timestamp_list.sort(reverse=True) + +if args.timestamp is None and args.build_index is None: + print('No timestamp or build index specified') + return +elif args.timestamp is not None and args.build_index is not None: + print("Specify only timestamp or only build index") + return +if args.timestamp: + if args.timestamp in timestamp_list: + timestamp = args.timestamp + else: + print('No such timestamp') + return +else: + if len(timestamp_list) > args.build_index: + timestamp = timestamp_list[args.build_index] + else: + print('No such build index') + return +``` + +Next, create a `Parser` object and start collecting statistics: + +```python +parser = Parser(args.poky_path) +parser.get_data_from_buildstats(os.path.join(args.poky_path, 'build/tmp/buildstats', timestamp)) +``` + +--- + +After that, you can use the collected statistics for various purposes, such as: + +### Writing to Files + +```python +parser.write_data_about_all_packages() +for task_type in all_tasks: + parser.write_data_about_task(task_type) +``` + +--- + +### Ranking and Writing to File + +In this example, the top 10% longest-running tasks (default metric is `'Elapsed time'`) will be written to `ranking_output.txt`: + +```python +data = get_ranked_data_for_all_tasks(parser.info, parser.pid_info, border=0.1) +write_ranked_data(data, 'ranking_output.txt') +``` + +In this example, all tasks will be ranked by the `'Started'` metric and saved in ascending order: + +```python +data = get_ranked_data_for_all_tasks(parser.info, parser.pid_info, metric='Started', border=1, reverse=False) +write_ranked_data(data, 'ranking_output.txt') +``` diff --git a/src/conf/README_en.md b/src/conf/README_en.md new file mode 100644 index 00000000..0f24dcb0 --- /dev/null +++ b/src/conf/README_en.md @@ -0,0 +1,7 @@ +## Configs +**YOU MUST PASS THE CONFIG TO THE SCRIPT BEFORE EACH RUN** +- `default.conf` – the base config that will be used if you don’t explicitly set another config. +- `original.conf` – the config for the updated build (with all layers); it will be used by default unless the `--no-layers` flag is set. +- `experiment.conf` – the config used as a template for creating an experiment-specific config. + +> If you need to run a scenario not covered by the predefined options, you can provide a custom config path using the `--conf-file ` option. diff --git a/src/experiment/README_en.md b/src/experiment/README_en.md new file mode 100644 index 00000000..3c7896ee --- /dev/null +++ b/src/experiment/README_en.md @@ -0,0 +1,74 @@ +### Instruction — Step-by-Step Guide for Conducting the Experiment + +1. Set up SSH as described in the [SSH setup guide](/wiki/yocto_cache/ssh_connection.md). + +2. Fill in the [configuration file](src/setup_servers/auto_conf/[example]_experiment.conf) located at + `.../src/experiment/auto_conf/experiment.conf`: + + - a) `cache_ip`, `hash_ip` — IP addresses of your cache and hash servers (those configured in step 1). + - b) `cache_usr`, `hash_usr` — SSH usernames for your cache and hash servers (from step 1). + - c) `hash_port` — port to be used for the hash server. + - d) `cache_start_port` — starting port on the cache server; used for launching multiple cache instances. + - e) `cache_num_port` — number of ports to be used by cache servers; resulting in port usage from `{cache_start_port}` to `{cache_start_port + cache_num_port - 1}`. + - f) `step` — step increment for the experiment’s loop. + - g) `max_servers` — upper limit of cache server count in the test. + +3. During the experiment, output files are automatically generated in the repository root in the format `test_n_m`, where: + - `n` — number of cache servers used, + - `m` — experiment repetition number: + If `m=1`, the build was done **without** a hash server cache. + If `m=2`, the build was done **with** the hash server cache. + +--- + +### How the Experiment Works + +#### Hash Server +Everything related to hash server setup is located in `.../src/experiment/hash_server_setuper`, which also contains a `README.md`. + +#### Cache Server +To correctly conduct the experiment, clone the project repository on the machine intended to run as a cache server at this path: +`/home/user/Desktop/test` + +Then, inside the `src` directory, run: +```sh +python3 -m venv venv +source venv/bin/activate +cd ./experiment/cache_server_setuper/reqs && pip3 install -r requirements.txt +``` + +After setup, build the Yocto project using the `entrypoint.sh` script to populate the sstate-cache, or alternatively, manually replace the contents of `.../src/yocto-build/assembly/build/sstate-cache` with your own cache files. + +The main script for working with the cache server is: +`.../src/experiment/cache_server_setuper/manipulate_cache.sh` + +#### Running the Experiment +To start the experiment, navigate to the directory: +`.../src/experiment/` +and run: +```sh +./main.sh +``` + +--- + +### Features of the `./manipulate_cache.sh` Script + +1. Show usage/help: +```sh +./manipulate_cache.sh +``` + +2. Launch the full pipeline: +Creates and starts containers hosting `sstate-cache`. +```sh +./manipulate_cache.sh start +``` +Both `` and `` are optional. Default values: +- `port = 9000` +- `count_of_servers = 4` + +3. Stop and remove all cache containers: +```sh +./manipulate_cache.sh kill +``` diff --git a/src/experiment/auto_conf/README.md b/src/experiment/auto_conf/README.md index bd53426c..438def2d 100644 --- a/src/experiment/auto_conf/README.md +++ b/src/experiment/auto_conf/README.md @@ -15,7 +15,7 @@ cache_start_port=9000 cache_num_port=10 hash_port=8686 ``` -Выжно именование переменных. +Важно именование переменных. ### Интерфейс взаимодействия с файлом конфигурации Файл конфигурации оформлен под требования файла конфигурации python библиотеки `configparser`. diff --git a/src/experiment/auto_conf/README_en.md b/src/experiment/auto_conf/README_en.md new file mode 100644 index 00000000..3f194d48 --- /dev/null +++ b/src/experiment/auto_conf/README_en.md @@ -0,0 +1,55 @@ +### Auto Configuration File: `local.conf` + +A file named `auto_compose_local_conf.py` has been developed. In its current version, it is expected to be executed from the `build` directory. Whether this is correct or not will be clarified later, during the orchestration stage of experimental builds. + +For the script to function properly, it requires a configuration file `experiment.conf` to be placed in the same directory. + +Example contents of `experiment.conf`: +``` +[SSH] +cache_ip = 10.138.70.6 +cache_usr = user +hash_ip = 10.138.70.23 +hash_usr = user + +[SERVERS] +cache_start_port = 9000 +cache_num_port = 10 +hash_port = 8686 +``` +Variable names must match exactly. + +--- + +### Interface for Working with the Config File + +The configuration file is formatted according to the `configparser` format from the Python standard library. +It contains two sections: `SSH` and `SERVERS`. + +- The **SSH** section defines parameters for SSH connections — usernames and IPs of the cache and hash servers. +- The **SERVERS** section contains parameters for the experiment itself — starting port and the number of ports on the cache server, as well as the hash server port (IP addresses are taken from the SSH section and not repeated here). + +--- + +#### Working with the Config File in Python + +The file `config_parser.py` provides three functions: one for reading a single variable, one for reading the entire config, and one for writing a variable. + +1. `read_from_config(file_name, section_name, variable_name)` + Returns the value of the specified variable from the given file and section as a string. + +2. `print_config(file_name)` + Prints the full contents of the specified config file to the console. + +3. `write_to_config(file_name, section_name, variable_name, variable_value)` + Writes the specified value to the specified variable in the given section and config file. + +--- + +#### Working with the Config File in Bash + +In the file `read_config.sh`, a function `process_config` is implemented, which reads variables from a config file and applies them via `source`. + +Since `configparser`-style config files require section headers (`[SECTION]`) and typically include spaces around `=`, the function creates a temporary copy of the config, reformats it into a style suitable for `source`, sources it, and deletes the temporary file afterward. + +A demonstration script `example_using_read_config.sh` is also provided, which imports the function and makes use of the variables from the config file. diff --git a/src/experiment_2/README_en.md b/src/experiment_2/README_en.md new file mode 100644 index 00000000..0c09ee41 --- /dev/null +++ b/src/experiment_2/README_en.md @@ -0,0 +1,31 @@ +# Experiment with server filtering patch + +### Reproduction Instructions: +1. Start a hash server in any directory on your computer using the command: `bitbake-hashserv -b :8686` +2. Start a cache server on computer #2. I used a machine with IP `10.138.70.7`, user `user`, and port `9999`. +3. Now configure the experiment: + + a. Configuration in `auto_compose_local_conf.py` + ```py + target_working_server_port = 9999 # the port on the cache server that is open and used for serving cache + hash_ip_port = f'0.0.0.0:{8686}' # the port used by the local hash server from step 1 + cache_ip = '10.138.70.7' # IP of the machine serving the cache + cache_start_port = 8150 # start of the port range + cache_num_port = 50 # number of ports to use (experiment will run from 2 up to this number + 1) + ``` + + b. Configuration in `run.sh` + ```bash + max_servers=50 # maximum number of servers (minus one, effectively 50 + 1 will be tested) + step=1 # iteration step + ``` + You can also choose a specific poky version by setting a different commit hash there. + +4. Run the experiment using `./run.sh` + +### Workflow Logic +1. The file `auto_compose_local_conf.py` generates `local.conf` files for the experiment, placing them into the folder `./configs//local.conf` +2. If `poky` is not present in the current folder, it is cloned and checked out to a specific commit hash (`59db27de565fb33f9e4326e76ebd6fa3935557b9`) +3. The patch is copied and applied +4. The script runs two experiments — one representing the behavior **before** the patch and one **after**. Because of that, the build restart block is executed twice (with different output file names to avoid overwriting the results). The first run uses only the timing patch, while the second includes the filtering patch. +5. The build loop block does the following: activates the environment (`source oe-init-build-env`), copies the generated config from step 1, starts the build and saves logs, then deletes the build directory. From 8ab406ae5e277fe0fc804a85509f2fc96e3ef4bf Mon Sep 17 00:00:00 2001 From: Konstantin Evgenievich Kondratenko <90711883+KonstantinKondratenko@users.noreply.github.com> Date: Fri, 18 Jul 2025 12:37:21 +0300 Subject: [PATCH 2/3] fix links in box_plot en version --- src/common/analysis/box_plot.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/common/analysis/box_plot.md b/src/common/analysis/box_plot.md index b2e17490..e51a8cb4 100644 --- a/src/common/analysis/box_plot.md +++ b/src/common/analysis/box_plot.md @@ -55,9 +55,9 @@ options: ### Результат работы с флагом --standard_boxplot -![Patch_analysis.png](./Patch_analysis.png) +![Patch_analysis.png](./speeding_up_results/Patch_analysis_new_data_night.png) ### Результат работы без флага --standard_boxplot -![Patch_analysis_full.png](./Patch_analysis_full.png) \ No newline at end of file +![Patch_analysis_full.png](./speeding_up_results/Patch_analysis_full_new_data_night.png) From c5a812e8cebac949afa39bbd48aed6fe08999917 Mon Sep 17 00:00:00 2001 From: Konstantin Evgenievich Kondratenko <90711883+KonstantinKondratenko@users.noreply.github.com> Date: Fri, 18 Jul 2025 12:42:35 +0300 Subject: [PATCH 3/3] fix links in box_plot ru version --- src/common/analysis/box_plot_en.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/common/analysis/box_plot_en.md b/src/common/analysis/box_plot_en.md index d249f824..2dbcf3fc 100644 --- a/src/common/analysis/box_plot_en.md +++ b/src/common/analysis/box_plot_en.md @@ -52,8 +52,8 @@ The experiment saves the build results and elapsed time in the folder `src/build ### Result with the --standard_boxplot flag -![Patch_analysis.png](./Patch_analysis.png) +![Patch_analysis.png](./speeding_up_results/Patch_analysis_new_data_night.png) ### Result without the --standard_boxplot flag -![Patch_analysis_full.png](./Patch_analysis_full.png) +![Patch_analysis_full.png](./speeding_up_results/Patch_analysis_full_new_data_night.png)