wiki eng

src/common/analysis/box_plot.md

@@ -55,9 +55,9 @@ options:
 
 ### Результат работы с флагом --standard_boxplot
 
-![Patch_analysis.png](./Patch_analysis.png)
+![Patch_analysis.png](./speeding_up_results/Patch_analysis_new_data_night.png)
 
 
 ### Результат работы без флага --standard_boxplot
 
-![Patch_analysis_full.png](./Patch_analysis_full.png)
+![Patch_analysis_full.png](./speeding_up_results/Patch_analysis_full_new_data_night.png)

src/common/analysis/box_plot_en.md

@@ -0,0 +1,59 @@
+# box_plot.py
+Python script for plotting graphs based on the results of the patch experiment (speeding_up)
+
+## Description
+The script is designed to build graphs based on the results of the patch experiment (speeding_up). It reads data from text files containing the experiment results and plots the graphs.
+
+## How to run the program?
+To configure the program's behavior, a command-line interface is implemented:
+```bash
+options:
+  -h, --help            show this help message and exit
+  --file_path_no_pathes FILE_PATH_NO_PATHES
+                        Enter the path to the file
+  --file_path_net_patch FILE_PATH_NET_PATCH
+                        Enter the path to the file
+  --file_path_childrens_patch FILE_PATH_CHILDRENS_PATCH
+                        Enter the path to the file
+  --file_path_all_patches FILE_PATH_ALL_PATCHES
+                        Enter the path to the file
+  --plot_name PLOT_NAME
+                        Enter title of plot
+  --save, --no-save     Save the plot (default: True)
+  --show, --no-show     Show the plot (default: True)
+  --standard_boxplot, --no-standard_boxplot
+                      Use standard box plot (default: False)
+```
+
+### Arguments
+- `file_path_no_pathes`, `file_path_net_patch`, `file_path_childrens_patch`, `file_path_all_patches`: arguments to specify paths to text files — results of the build experiment for `core-image-minimal` without patches, with the network load balancing patch, with the task priority patch, and with both patches respectively. By default, the script expects files `time1.txt`, `time2.txt`, `time3.txt`, `time4.txt` to be located in the subdirectory `speeding_up_results` relative to the script location.
+- `plot_name`: the name used as the plot title and for saving the plot as a PNG file. Defaults to `Patch_analysis`.
+- `save` and `show`: values to configure the program's behavior — whether to show or not show, save or not save the resulting plots. By default, the plot is both shown and saved.
+- `standard_boxplot`: a flag to disable all customizations, leaving only the standard `boxplot` and plotting the values as red dots.
+
+## How to read the graphs?
+- The y-axis shows the experiment execution time (in minutes).
+- The x-axis shows which experiment was conducted (without patches, with the network load balancing patch, with the task priority patch, and with both patches).
+
+### Legend
+- **Blue "boxes"** represent values between the 25th and 75th percentiles.
+- **Blue horizontal lines ("whiskers")** represent values within 1.5 interquartile range from the quartiles. These lines end with blue caps.
+- **Black dashed lines** represent the 1st and 99th percentiles.
+- **Red dots** indicate values in the ranges [1st percentile, 25th percentile] and [75th percentile, 99th percentile].
+- Inside the blue box, the median value is labeled in black.
+- The 1st and 99th percentiles are labeled in green and blue text respectively.
+
+## How to get the data?
+It is necessary to run the `speeding_up_experiment` (src/common/scripts/speeding_up_experiment.sh) with different patch sets. It is recommended to specify the number of repetitions greater than 10 (in the current version we specify 30).
+
+The experiment saves the build results and elapsed time in the folder `src/buildstats_saves`. There is a file time.txt used for plotting the graphs.
+
+## Script output results
+
+### Result with the --standard_boxplot flag
+
+![Patch_analysis.png](./speeding_up_results/Patch_analysis_new_data_night.png)
+
+### Result without the --standard_boxplot flag
+
+![Patch_analysis_full.png](./speeding_up_results/Patch_analysis_full_new_data_night.png)

src/common/analysis/dep_graph/wiki/README_en.md

@@ -0,0 +1,60 @@
+## Using the program for sorting graph nodes and finding offsets
+
+First, it is necessary to run the parser and collect statistics (instructions for running the parser and collecting statistics are described here: https://github.com/moevm/os_profiling/blob/denisova_olga_parsing_ranking/statistics_analyzer/README.md)
+
+Also, before analyzing the dependency graph, it is necessary to create the file task-depends.dot (creating this file and visualizing the graph is described here: https://github.com/moevm/os_profiling/blob/denisova_olga_dep_graph/dep_graph/wiki/dep_graph.md)
+
+Then you can create a graph object and analyze it:
+
+    G = nx.DiGraph(nx.nx_pydot.read_dot('./task-depends.dot'))
+    sorted_tasks = sort_start_time(parser.info)
+    sorted_nodes = []
+    for node in G.nodes:  # for each node find the corresponding index in sorted_tasks
+        index, start, end = match(node, sorted_tasks)
+        sorted_nodes.append((index, node, start, end))
+
+    sorted_nodes = sorted(sorted_nodes, key=lambda x: x[0])
+    # write(sorted_nodes, 'tasks-order.txt')  # tasks can be written to a file in the order of their start time
+
+In the example above, graph nodes are matched with objects from the statistics, and nodes are numbered in the order of their start time.
+
+For each node, you can find its "children" and find the offset between the end of the child's execution and the start of the parent's execution:
+
+    results = []
+    for node in G.nodes:
+        for child in G.neighbors(node):
+            for temp in sorted_nodes:
+                if temp[1] == node:
+                    result1 = temp
+                if temp[1] == child:
+                    result2 = temp
+            if result1[2] != -1 and result2[3] != -1:
+                offset = result1[2] - result2[3]
+            else:
+                offset = -1
+            results.append((f'node: {node}, Started: {result1[2]}, child: {child}, Ended: {result2[3]}',  offset))
+
+    results = sorted(results, key=lambda x: x[1], reverse=True)
+
+    with open('task-order-sorted-offset.txt', 'w') as file:
+        file.writelines(f"{item[0]}, offset: {item[1]}\n" for item in results)
+
+In the example above, a file is created that contains information about each parent-child node pair and their offset.
+
+## Other ways to analyze the graph
+
+1) The method nx.bfs_layers allows composing BFS layers, which helps to look at the overall view of the graph's interconnections
+
+    print(dict(enumerate(nx.bfs_layers(G, ['core-image-sato.do_build']))))
+
+2) The method nx.is_tree helps check whether the graph is a tree
+
+    print(nx.is_tree(G))
+
+3) The method nx.is_connected allows finding out whether the graph is connected
+
+    print(nx.is_connected(G))
+
+4) The method nx.is_directed_acyclic_graph allows finding out whether the graph is acyclic
+
+    print(nx.is_directed_acyclic_graph(G))

src/common/analysis/dep_graph/wiki/dep_graph_analysis_en.md

@@ -0,0 +1,15 @@
+## Dependency Graph Analysis
+### Checking for the Presence of Independent Nodes
+Using the `nx.bfs_layers` method, "layers" of the dependency graph were found, that is, sets of nodes located at the same "distance" from a node chosen as the zero layer. The node `core-image-sato.do_rootfs` was selected as the zero layer node. Thus, if any node had no connections to the main set of nodes, it should not have appeared in any layer. Then a comparison was made between the total number of nodes in all layers and the total number of nodes in the graph; both turned out to be equal to 8782, which tells us that there are no independent nodes (or sets of nodes).
+
+### Finding the "Root"
+Tasks are executed from the leaves to the "root", meaning at the start of the build, the leaf tasks run in parallel. By analyzing the `.dot` file, it was found that the last task to be executed (and the root of the graph) is the node `core-image-sato.do_build`, since no nodes depend on it.
+
+### Checking for a Tree Structure
+It was determined that the dependency graph does not have a tree structure. This was clearly visible during further visualization of the layers.
+
+### Finding the Offset Between the End of the Child Node Execution and the Start of the Parent Node Execution
+For each node, neighbor nodes were found (i.e., those nodes that are in direct dependency). Since the graph is directed, an edge from node A to node B exists if and only if node A depends on node B (in other words, node A is the parent of node B). For each such pair (parent node – child node), the time when the child node finished execution and when the parent node started execution was found (in this order, since the build goes from leaves to root). The offset (difference of the mentioned timestamps) was calculated, and a list of "parent-child" node pairs was created in order of decreasing offset. It turned out that the largest offsets belong to pairs where the parent node has many child nodes (i.e., the parent task depends on a large number of child tasks).
+
+### Visualization of Graph Layers
+Using a script, smaller `.dot` files were obtained from the single `task-depends.dot` file created by BitBake according to BFS layers and visualized using Graphia. The link to the folder with screenshots: https://drive.google.com/drive/folders/1iDxxq7sxxZWLCpOhBl2Xb6UEQneB_4qW?usp=drive_link. The number in the file name corresponds to the layer number. The screenshot shows the nodes of the corresponding layer and their direct connections (for example, for the first layer, the screenshot shows nodes of the first layer as well as nodes from the 0th and 2nd layers, since the nodes of the 1st layer are connected to them).

src/common/analysis/dep_graph/wiki/dep_graph_en.md

@@ -0,0 +1,23 @@
+## Dependency Graph Construction
+
+To build the dependency graph, the file `task-depends.dot` is required, which can be obtained using the following command: `bitbake -g core-image-sato`.
+
+After obtaining the `task-depends.dot` file, we can build the dependency graph. The dependency graph was constructed in two ways:
+
+1) Using Python with the `plotly.graph_objects` library, an `.html` file was created containing the visualization of the dependency graph:  
+![image](https://github.com/moevm/os_profiling/assets/90854310/b7b2cced-3de4-4468-81e3-2fb6def2dc6f)
+
+In this file, it is possible to zoom in on specific parts of the graph; for example, let’s look at the node `core-image-sato.do_rootfs`:  
+![image](https://github.com/moevm/os_profiling/assets/90854310/7611a754-5283-4c77-8690-3b1cd9fe557e)
+
+2) The same dependency graph was also built using Graphia; in this case, the visualization of the graph is three-dimensional. The appearance of the graph:  
+![image](https://github.com/moevm/os_profiling/assets/90854310/8248dc71-0027-4b9b-9a1a-c95e4ef1dae7)  
+![image](https://github.com/moevm/os_profiling/assets/90854310/45265149-d92e-4017-a012-c34363296736)
+
+It is also possible to find a node by search and zoom in on a specific node.
+
+For `core-image-sato`, the resulting graph consisted of 8782 nodes and 33883 edges.
+
+## Questions Regarding the Dependency Graph
+
+The main question is whether the dependency graph has a root; in that case, it would be more convenient to perform traversal.

src/common/analysis/packages-charts/packages_charts_en.md

@@ -0,0 +1,70 @@
+# Mapping Resource Usage to Packages
+
+## Available Data
+
+To map resource usage to specific packages, log files located in the `build/tmp/buildstats/<timestamp>/` directory were analyzed. This directory contains subdirectories for each package involved in the build process. Each subdirectory includes log files such as `do_configure`, `do_install`, `do_compile`, etc. The log format is described in `yocto_buildstats.md`:  
+[yocto_buildstats.md](https://github.com/moevm/os_profiling/blob/denisova_olga_yocto_buildstats/wiki/yocto_buildstats.md)
+
+From these logs, we can extract the following characteristics:
+
+1. **Elapsed time** – total time taken  
+2. **utime** – user mode time  
+3. **stime** – system (kernel) mode time  
+4. **cutime** – user mode time of child processes  
+5. **cstime** – system mode time of child processes  
+6. **IO rchar** – bytes read from storage  
+7. **IO read_bytes** – estimated number of bytes read from storage (accurate for block-based file systems)  
+8. **IO wchar** – bytes written to disk  
+9. **IO write_bytes** – estimated bytes written to storage  
+10. **IO cancelled_write_bytes** – bytes written that were canceled due to page cache truncation  
+11. **IO syscr** – estimated number of read I/O operations  
+12. **IO syscw** – estimated number of write I/O operations  
+13. **rusage ru_utime** – user CPU time in seconds  
+14. **rusage ru_stime** – system CPU time in seconds  
+15. **rusage ru_maxrss** – peak resident set size in KB  
+16. **rusage ru_minflt** – number of minor page faults (handled without disk I/O)  
+17. **rusage ru_majflt** – number of major page faults (required disk I/O)  
+18. **rusage ru_inblock** – block input operations  
+19. **rusage ru_oublock** – block output operations  
+20. **rusage ru_nvcsw** – voluntary context switches  
+21. **rusage ru_nivcsw** – involuntary context switches  
+22. **Child rusage ru_utime** – same as (2), but for all child processes  
+23. **Child rusage ru_stime** – same as (3), for all child processes  
+24. **Child rusage ru_maxrss** – same as (15), for all child processes  
+25. **Child rusage ru_minflt** – same as (16), for all child processes  
+26. **Child rusage ru_majflt** – same as (17), for all child processes  
+27. **Child rusage ru_inblock** – same as (18), for all child processes  
+28. **Child rusage ru_oublock** – same as (19), for all child processes  
+29. **Child rusage ru_nvcsw** – same as (20), for all child processes  
+30. **Child rusage ru_nivcsw** – same as (21), for all child processes  
+
+---
+
+## Generating Resource Usage Charts
+
+A Python script was created to generate visual charts showing resource usage per package. The most insightful metrics are:
+
+- `utime`, `stime`, `cutime`, `cstime`  
+- `IO rchar`, `IO wchar`  
+- `rusage ru_utime`, `rusage ru_stime`, `rusage ru_maxrss`  
+- `Child rusage ru_utime`, `Child rusage ru_stime`
+
+Each metric is visualized as a separate graph and saved as a PNG image.
+
+Example chart:  
+![rusage ru_utime](https://github.com/moevm/os_profiling/assets/90854310/1d37013a-f817-43a9-876c-f306813b2d12)
+
+All generated charts are located in the `charts/` directory.  
+The code used to generate them is in the `make-charts.py` file.
+
+---
+
+## Mapping Resources to Processes
+
+The directory `build/tmp/work` contains working subdirectories related to a specific architecture. These subdirectories include data about the packages. Each package directory contains a `temp/` folder with log files for each task (e.g., `log.do_<task>.pid`) and scripts used by BitBake (`run.do_<task>.pid`). These files are created and filled during the build.
+
+All packages from `build/tmp/work` match those in `build/tmp/buildstats`, with the exception of `gcc-source-13.2.0-13.2.0`.  
+The assumption is that using the PID in the filename, we can match specific resource metrics to the associated process.
+
+This directory structure is described in the official documentation:  
+[Yocto Project Reference Manual – Section 4.2.7.9](https://docs.yoctoproject.org/ref-manual/structure.html)

src/common/analysis/read_write_statistics/instruction_en.md

@@ -0,0 +1,19 @@
+# Guide for Processing Read/Write Log Statistics
+
+1. To collect read/write statistics logs, run the build with the following command:
+    ```bash
+    strace -f -e trace=read,write,readv,writev -o yocto_trace_all.log bitbake core-image-minimal
+    ```
+
+2. You also need a file containing a list of tasks with their corresponding PIDs.  
+   This can be obtained by running the task ranking as described in the [ranking instructions](../statistics_analyzer/README.md).  
+   After completion, the file `ranking_output.txt` will be generated.
+
+3. Next, process the logs using the `process_logs.py` script:
+    ```bash
+    python3 process_logs.py -l <path to yocto_trace_all.log> -t <path to ranking_output.txt>
+    ```
+
+4. After execution, two files will be created in the `read_write_statistics/output/` directory:
+    - `process_statistics_rw.txt`: contains statistics for `read` and `write` operations  
+    - `process_statistics_rwv.txt`: contains statistics for `readv` and `writev` operations

src/common/analysis/statistics_analyzer/README_en.md

@@ -0,0 +1,87 @@
+# Running the Parser
+
+Any program that uses internal statistics collection via the parser should be launched as follows:
+
+```
+python3 ./<name>.py --poky <path to poky folder> -b <build index>
+```
+
+or
+
+```
+python3 ./<name>.py --poky <path to poky folder> -t <specific timestamp>
+```
+
+---
+
+## Using the Parser and Running Task Ranking
+
+First, retrieve information about available builds and get a timestamp-sorted list of builds:
+
+```python
+args = create_parser_args()
+timestamp = ''
+timestamp_list = []
+poky_buildstats_path = os.path.join(args.poky_path, 'build/tmp/buildstats')
+tree = list(os.walk(poky_buildstats_path))
+for item in tree:
+    if item[0] == poky_buildstats_path:
+        timestamp_list = item[1]
+timestamp_list.sort(reverse=True)
+
+if args.timestamp is None and args.build_index is None:
+    print('No timestamp or build index specified')
+    return
+elif args.timestamp is not None and args.build_index is not None:
+    print("Specify only timestamp or only build index")
+    return
+if args.timestamp:
+    if args.timestamp in timestamp_list:
+        timestamp = args.timestamp
+    else:
+        print('No such timestamp')
+        return
+else:
+    if len(timestamp_list) > args.build_index:
+        timestamp = timestamp_list[args.build_index]
+    else:
+        print('No such build index')
+        return
+```
+
+Next, create a `Parser` object and start collecting statistics:
+
+```python
+parser = Parser(args.poky_path)
+parser.get_data_from_buildstats(os.path.join(args.poky_path, 'build/tmp/buildstats', timestamp))
+```
+
+---
+
+After that, you can use the collected statistics for various purposes, such as:
+
+### Writing to Files
+
+```python
+parser.write_data_about_all_packages()
+for task_type in all_tasks:
+    parser.write_data_about_task(task_type)
+```
+
+---
+
+### Ranking and Writing to File
+
+In this example, the top 10% longest-running tasks (default metric is `'Elapsed time'`) will be written to `ranking_output.txt`:
+
+```python
+data = get_ranked_data_for_all_tasks(parser.info, parser.pid_info, border=0.1)
+write_ranked_data(data, 'ranking_output.txt')
+```
+
+In this example, all tasks will be ranked by the `'Started'` metric and saved in ascending order:
+
+```python
+data = get_ranked_data_for_all_tasks(parser.info, parser.pid_info, metric='Started', border=1, reverse=False)
+write_ranked_data(data, 'ranking_output.txt')
+```

src/conf/README_en.md

@@ -0,0 +1,7 @@
+## Configs
+**YOU MUST PASS THE CONFIG TO THE SCRIPT BEFORE EACH RUN**
+- `default.conf` – the base config that will be used if you don’t explicitly set another config.
+- `original.conf` – the config for the updated build (with all layers); it will be used by default unless the `--no-layers` flag is set.
+- `experiment.conf` – the config used as a template for creating an experiment-specific config.
+
+> If you need to run a scenario not covered by the predefined options, you can provide a custom config path using the `--conf-file <path>` option.