A secure and isolated malware analysis laboratory built using VMware Workstation, Windows 11 Pro, and FLARE-VM to perform static and dynamic malware analysis in a controlled environment.
- 📖 Project Overview
- 🎯 Objectives
- 🖥️ Lab Environment
- 🛠️ Tools Used
- ⚙️ Project Implementation
- Step 1: Creating the Virtual Machine
- Step 2: Disabling Automatic Windows Updates
- Step 3: Disabling Virus & Threat Protection
- Step 4: Disabling Microsoft Defender Real-Time Protection
- Step 5: Creating the Baseline Snapshot
- Step 6: Installing FLARE-VM
- Step 7: Creating the Post-Installation Snapshot
- Step 8: Malware Sample Collection
- Step 9: Configuring an Isolated Virtual Network
- 🔬 Static Malware Analysis
- ⚡ Dynamic Malware Analysis
- 📊 Dynamic Analysis Findings
- 🎯 Skills Gained
⚠️ Challenges Faced- 📚 Key Learnings
- 🏁 Conclusion
- 📁 Repository Structure
- ⭐ Acknowledgements
This project involved designing and building a secure, isolated malware analysis laboratory using VMware Workstation and Windows 11 Pro. The primary objective was to establish a controlled environment for safely analyzing malware samples through both static and dynamic malware analysis without compromising the host operating system.
The laboratory was configured by:
- Installing Windows 11 Pro inside VMware Workstation
- Configuring an isolated Host-only virtual network
- Disabling Windows security features required for malware analysis
- Installing the FLARE-VM malware analysis toolkit
- Creating VMware snapshots for system restoration
- Performing static and dynamic malware analysis using industry-standard tools
- Build a secure and isolated malware analysis laboratory.
- Install and configure Windows 11 Pro within VMware Workstation.
- Configure Windows security policies for malware analysis.
- Set up an isolated Host-only virtual network.
- Install and configure FLARE-VM.
- Create VMware snapshots for repeatable analysis.
- Perform static malware analysis.
- Perform dynamic malware analysis.
- Analyze malware behavior through process, registry, file system, and network monitoring.
- Identify and document Indicators of Compromise (IOCs).
| Component | Details |
|---|---|
| Host Operating System | Windows 11 |
| Virtualization Platform | VMware Workstation |
| Guest Operating System | Windows 11 Pro (64-bit) |
| Virtual Disk Size | 100 GB |
| Virtual Network | Host-only Network (VMnet2) |
| Analysis Toolkit | FLARE-VM |
| Analysis Type | Static and Dynamic Malware Analysis |
| Tool | Purpose |
|---|---|
| VMware Workstation | Created and managed the isolated virtual machine for malware analysis. |
| Windows 11 Pro | Guest operating system used for the malware analysis environment. |
| FLARE-VM | Malware analysis toolkit containing reverse engineering, forensic, and security analysis tools. |
| Windows PowerShell | Installed and configured FLARE-VM. |
| Local Group Policy Editor (gpedit.msc) | Configured Windows security and update policies. |
| Microsoft Defender Settings | Disabled security features to allow controlled malware execution within the virtual machine. |
| VMware Snapshot Manager | Created and managed snapshots for restoring the analysis environment. |
| VMware Virtual Network Editor | Configured an isolated Host-only network for safe malware execution. |
| HashMyFiles | Generated MD5, SHA-1, and SHA-256 hashes of malware samples. |
| VirusTotal | Verified malware hashes and reviewed antivirus detections and threat intelligence. |
| Strings | Extracted readable strings and Indicators of Compromise (IOCs) from malware executables. |
| ExifTool | Examined executable metadata such as timestamps, version information, and file properties. |
| Process Monitor (Procmon) | Monitored real-time file system, registry, process, and thread activity during malware execution. |
| Regshot | Compared Windows Registry snapshots before and after malware execution to identify registry changes. |
| Wireshark | Captured and analyzed network traffic generated during malware execution. |
| Task Manager | Identified the malware process and obtained its Process ID (PID) for Process Monitor filtering. |
A new virtual machine was created using VMware Workstation, and the Windows 11 Pro (64-bit) operating system was installed using the official Windows 11 ISO image. A 100 GB virtual hard disk was allocated to provide sufficient storage for malware samples, analysis tools, and snapshots.
This virtual machine serves as the isolated environment for performing both static and dynamic malware analysis while protecting the host operating system.
Figure 1: Windows 11 Pro installation inside VMware Workstation.
Figure 2: VMware virtual machine configuration used for the malware analysis laboratory.
To maintain a stable and reproducible malware analysis environment, Windows Automatic Updates were disabled using the Local Group Policy Editor. This prevents the operating system from installing updates that could alter the analysis environment or affect malware behavior during testing.
Navigation:
Computer Configuration → Administrative Templates → Windows Components → Windows Update → Manage End User Experience → Configure Automatic Updates
The Configure Automatic Updates policy was changed from Not Configured to Disabled.
Figure 3: Group Policy Editor showing the Configure Automatic Updates policy set to Disabled.
To enable controlled malware execution within the isolated virtual machine, Microsoft Defender Virus & Threat Protection settings were modified by disabling the available protection mechanisms. This prevents Windows Defender from automatically detecting or quarantining malware samples during analysis.
The following protection features were disabled:
- Real-Time Protection
- Cloud-Delivered Protection
- Automatic Sample Submission
- Tamper Protection (where applicable)
- Dev Drive Protection
Figure 4: Microsoft Defender Virus & Threat Protection settings configured for malware analysis.
To prevent Microsoft Defender Real-Time Protection from automatically re-enabling after a system restart, the corresponding Group Policy setting was configured using the Local Group Policy Editor. This ensures that malware samples can be analyzed without interruption from Windows Defender.
Navigation:
Computer Configuration → Administrative Templates → Windows Components → Microsoft Defender Antivirus → Real-Time Protection
The Turn off real-time protection policy was set to Enabled.
Figure 5: Local Group Policy Editor showing the Turn off real-time protection policy configured to Enabled.
After completing the initial virtual machine configuration, a Baseline snapshot was created using VMware Snapshot Manager. This snapshot serves as a clean restore point, allowing the virtual machine to be reverted to its original state before executing malware samples.
Creating a baseline snapshot ensures that the analysis environment can be quickly restored after malware execution, maintaining a clean, consistent, and repeatable testing environment.
Figure 6: VMware Snapshot Manager showing the Baseline snapshot created after the initial virtual machine configuration.
After preparing the virtual machine, Google Chrome was installed to download the required installation files, and the official FLARE-VM installation guide was followed. The installation was initiated using Windows PowerShell, which automatically downloaded and configured the FLARE-VM malware analysis toolkit.
The installation process took approximately two hours and required multiple automatic system restarts. Upon completion, FLARE-VM installed a comprehensive collection of malware analysis, reverse engineering, digital forensics, and incident response tools, providing a fully configured environment for malware research.
Figure 7: Windows PowerShell executing the FLARE-VM installation process.
After the successful installation of FLARE-VM, a second snapshot was created using VMware Snapshot Manager. This snapshot represents a fully configured malware analysis environment with all required tools installed and ready for use.
The post-installation snapshot enables the virtual machine to be quickly restored to a clean, fully configured state before analyzing new malware samples, ensuring a consistent and repeatable analysis workflow.
Figure 8: VMware Snapshot Manager showing the post-installation snapshot created after the successful installation of FLARE-VM.
Malware samples were collected from publicly available research repositories and trusted educational sources for analysis within the isolated virtual environment. Only samples intended for research and educational purposes were used throughout this project.
To gain exposure to different malware behaviors and analysis techniques, samples representing various malware families were selected and analyzed in the controlled laboratory environment.
Static malware analysis was performed without executing the malware sample, allowing its characteristics to be safely examined while identifying potential Indicators of Compromise (IOCs). This approach helps analysts understand the malware's structure, identify suspicious artifacts, and determine potential malicious behavior before runtime analysis.
The following static analysis techniques were performed:
Cryptographic hashes (MD5, SHA-1, and SHA-256) were generated using HashMyFiles to uniquely identify the malware sample and verify its integrity. These hashes can be used to compare the sample against threat intelligence databases and identify known malware.
Figure 9: HashMyFiles displaying the generated MD5, SHA-1, and SHA-256 hashes of the malware sample.
The generated file hash was submitted to VirusTotal to determine whether the malware sample had been previously identified. VirusTotal aggregates results from multiple antivirus engines and provides community threat intelligence, detection statistics, and additional information about the sample.
Figure 10: VirusTotal scan results showing malware detections and the analysis summary.
Strings analysis was performed to extract readable text embedded within the malware executable without executing it. This technique helps identify potential Indicators of Compromise (IOCs) such as file paths, URLs, registry keys, API calls, suspicious commands, and other embedded artifacts that may reveal the malware's functionality or intended behavior.
Figure 11: Strings analysis output displaying readable strings extracted from the malware executable.
ExifTool was used to examine the executable's metadata, including compilation timestamps, file properties, version information, and other embedded metadata. This information can provide valuable insights into the malware's origin, development, and potential behavior.
Figure 12: ExifTool output displaying the malware executable's metadata and file properties.
The static analysis techniques provided valuable insights into the malware's structure and characteristics without executing the sample. Information gathered during this phase helped identify potential Indicators of Compromise (IOCs) and established a foundation for the subsequent dynamic malware analysis, enabling safer and more informed behavioral investigation.
Before performing dynamic malware analysis, the virtual machine's network configuration was modified to ensure that malware execution remained isolated from external networks. A custom Host-only (VMnet2) virtual network was created using VMware Virtual Network Editor.
The Host-only network allows communication only between the host system and the virtual machine while preventing malware from accessing the Internet or communicating with external systems during analysis. DHCP was enabled to automatically assign IP addresses within the private network, providing a controlled and secure analysis environment.
This network configuration minimizes the risk of malware spreading or establishing communication with external Command-and-Control (C2) servers while still allowing controlled interaction between the host and the virtual machine.
Figure 13: VMware Virtual Network Editor showing the custom Host-only (VMnet2) network configuration used for malware analysis.
Dynamic malware analysis was performed by executing the malware sample within the isolated virtual machine while monitoring its behavior in real time. Unlike static analysis, this approach allowed the observation of the malware's runtime activities, system modifications, and potential Indicators of Compromise (IOCs).
The malware's interaction with the operating system was monitored using several industry-standard tools to analyze process activity, registry modifications, file system changes, network communications, and overall system behavior within a controlled and isolated environment.
Before executing the malware sample, a Pre-Analysis snapshot was created using VMware Snapshot Manager. This snapshot served as a restore point immediately prior to dynamic analysis, allowing the virtual machine to be reverted to a clean state after each malware execution.
Creating a dedicated pre-analysis snapshot ensured that multiple malware samples could be analyzed independently without retaining artifacts or system modifications from previous executions. This approach maintains a clean, consistent, and repeatable malware analysis environment.
Figure 14: VMware Snapshot Manager showing the Pre-Analysis snapshot created immediately before malware execution.
Before executing the malware sample, Regshot was used to capture the first registry snapshot (1st Shot). This snapshot recorded the initial state of the Windows Registry and served as a baseline for identifying changes introduced by the malware.
Figure 14: Regshot capturing the initial Windows Registry snapshot before malware execution.
The malware sample was executed with Administrator privileges inside the isolated virtual machine. Running the sample within a controlled environment ensured that its behavior could be safely observed without affecting the host operating system.
After execution, Task Manager was used to identify the running malware process. During this analysis, the malware was assigned Process ID (PID) 9544, which was used for targeted monitoring in Process Monitor.
Figure 15: Task Manager displaying the malware process with PID 9544.
Process Monitor (Procmon) was configured to filter events using PID 9544, displaying only the activities generated by the malware. This eliminated unrelated system events and simplified the analysis of file system, registry, process, and thread activity.
Figure 16: Process Monitor configured to monitor events generated by the malware process (PID 9544).
Wireshark was used to capture and analyze the network traffic generated during malware execution. This enabled the observation of DNS requests, connection attempts, and other network communications, helping determine whether the malware attempted to communicate with remote hosts or external Command-and-Control (C2) servers.
Figure 17: Wireshark capturing network traffic generated during malware execution.
After the malware completed execution, Regshot was used to capture the second registry snapshot (2nd Shot). The two snapshots were compared to identify registry keys and values that had been created, modified, or deleted during malware execution. The comparison results were exported as text reports for further analysis and documentation.
Figure 18: Regshot comparison summary highlighting registry changes before and after malware execution.
Figure 19: First section of the exported Regshot comparison report showing registry modifications introduced by the malware.
Figure 20: Second section of the exported Regshot comparison report showing additional registry changes.
The data collected from Process Monitor, Regshot, and Wireshark was reviewed to evaluate the malware's overall behavior. The analysis focused on:
- Process creation
- File system modifications
- Registry changes
- Persistence mechanisms
- Network activity
- Overall system behavior
The combined observations provided valuable insights into the malware's runtime behavior and enabled the identification of potential Indicators of Compromise (IOCs) while ensuring the analysis was conducted safely within the isolated malware analysis environment.
The dynamic analysis provided valuable insights into the malware's runtime behavior by correlating observations from Process Monitor, Regshot, and Wireshark. Monitoring the malware during execution enabled the identification of system modifications, runtime activities, and potential Indicators of Compromise (IOCs) that could not be observed through static analysis alone.
The malware initiated one or more processes that performed various system operations. Process Monitor captured file system, registry, process, and thread activity, providing detailed visibility into the malware's execution and runtime behavior.
The Regshot comparison report identified registry keys and values that were created, modified, or deleted during execution. These changes provided insight into the malware's interaction with the Windows Registry and revealed potential persistence or configuration mechanisms.
The malware created, modified, and accessed files and directories within the virtual machine. These file system changes provided useful forensic artifacts that can assist in identifying malware behavior and supporting further investigation.
Wireshark captured the network traffic generated during malware execution, including DNS queries, connection attempts, and other network communications. This analysis helped determine whether the malware attempted to establish external connections or communicate with remote systems.
By combining observations from Process Monitor, Regshot, and Wireshark, a comprehensive understanding of the malware's runtime behavior was obtained. The collected evidence highlighted process execution, registry modifications, file system activity, network communications, and other behavioral characteristics that contributed to identifying potential Indicators of Compromise (IOCs).
The findings obtained through dynamic analysis complemented the results of the static analysis, providing a more comprehensive understanding of the malware's functionality and behavior. Together, the two analysis approaches demonstrated the importance of conducting malware investigations within a secure, isolated, and repeatable analysis environment.
The following Indicators of Compromise (IOCs) were identified during the malware analysis:
| IOC Category | Observation |
|---|---|
| File Hashes | MD5, SHA-1, SHA-256 generated using HashMyFiles |
| Registry Changes | Registry keys created, modified, and deleted during execution |
| File System Activity | Files and directories accessed or modified by the malware |
| Process Activity | Malware process identified (PID 9544) |
| Network Activity | DNS queries, connection attempts, and network communications observed using Wireshark |
Throughout this project, I developed practical experience in:
- Virtual Machine Deployment
- Malware Analysis Lab Configuration
- VMware Snapshot Management
- Windows Security Configuration
- FLARE-VM Installation and Configuration
- Static Malware Analysis
- Dynamic Malware Analysis
- Hash Analysis
- Malware Identification
- Threat Intelligence using VirusTotal
- Strings Analysis
- Metadata Analysis using ExifTool
- Process Monitoring using Process Monitor
- Registry Analysis using Regshot
- Network Traffic Analysis using Wireshark
- Safe Malware Handling Practices
During the project, several challenges were encountered:
- Long FLARE-VM installation time
- Multiple automatic system restarts during installation
- Correctly configuring Windows security policies
- Maintaining an isolated and stable malware analysis environment
- Managing snapshots to ensure a clean and repeatable analysis workflow
This project provided hands-on experience in designing and configuring a secure malware analysis laboratory from the ground up. It strengthened my understanding of virtualization, Windows system administration, virtual network isolation, and the deployment of specialized malware analysis environments using FLARE-VM.
Through static analysis, I gained experience in generating cryptographic hashes, using VirusTotal for threat intelligence, extracting readable strings, and analyzing executable metadata to identify potential Indicators of Compromise (IOCs) without executing malware.
Dynamic analysis further enhanced my ability to monitor process activity, analyze registry modifications, capture network traffic, and observe malware behavior using Process Monitor, Regshot, and Wireshark.
Overall, this project reinforced the importance of maintaining VMware snapshots, documenting analysis findings, and conducting malware research within a secure, isolated, and repeatable environment while following safe malware handling practices.
This project involved designing and implementing a dedicated malware analysis laboratory using VMware Workstation, Windows 11 Pro, and FLARE-VM. By combining virtualization, virtual network isolation, VMware snapshots, and both static and dynamic analysis techniques, a secure and reusable environment was established for safely analyzing malicious software.
The project provided practical experience in investigating malware characteristics, runtime behavior, registry modifications, file system activity, and network communications while protecting the host operating system from potential compromise.
Overall, this project strengthened my practical skills in malware analysis, digital forensics, and cybersecurity research, providing a solid foundation for further learning in malware reverse engineering, incident response, and threat hunting.
Malware-Analysis-Lab/
│
├── README.md
├── LICENSE
│
└── images/
├── baseline-snapshot.png
├── defender-settings.png
├── flarevm-installation.png
├── hash-analysis.png
├── host-only-network-configuration.png
├── metadata-analysis.png
├── post-installation-snapshot.png
├── pre-analysis-snapshot.png
├── procmon-pid-filter.png
├── real-time-protection.png
├── regshot-comparison-report.png
├── regshot-comparison-report-part1.png
├── regshot-comparison-report-part2.png
├── regshot-initial-snapshot.png
├── strings-analysis.png
├── task-manager-malware-process.png
├── virustotal.png
├── vm-overview.png
├── windows-installation.png
├── windows-update-policy.png
└── wireshark-network-traffic.png
This project was completed for educational and research purposes to gain practical experience in malware analysis, digital forensics, and secure laboratory design using industry-standard tools and methodologies.