Dev to be merged with main

.gitignore

@@ -4,3 +4,5 @@ node_modules/
 **/model/*.csv
 **/model/*.json
 **/concepts/*.json
+import_solution_metadata/
+solution_metadata/

agile_operations/network_infrastructure/README.md

@@ -0,0 +1,181 @@
+# Network Infrastructure - Cybersecurity (TigerGraph Solution Kit)
+
+Cybersecurity is a crucial aspect of large organizations. Enterprises operate
+their own data centers and network infrastructure involving many different
+devices. Cyberattacks and other incidents can lead to issues such as data
+breaches, corrupted files, and loss of data - resulting in billions of
+dollars lost each year. One key capability for detecting and responding to
+these threats is a deep understanding of your organization's network
+infrastructure and how devices are connected.
+
+TigerGraph allows you to connect data from multiple sources and load data
+at terabyte scale. With TigerGraph visualizations, users can see the
+topology of their network infrastructure, understand how components relate,
+and run graph algorithms at scale to discover related incidents and events
+based on device topology in near real time.
+
+This solution kit provisions a complete graph environment - schema, sample
+data, and GSQL queries to help you:
+
+- Visualize router-firewall-switch-server topologies.
+- Identify single points of failure and the blast radius of device failures.
+- Trace incidents and alerts through network paths and time.
+- Find unsecured routes and devices with frequent alerts and incidents.
+
+---
+
+## Contents
+
+- [Overview](#overview)
+- [Components](#components)
+- [Prerequisites](#prerequisites)
+- [Setup Instructions](#setup-instructions)
+- [Query Explanations](#query-explanations)
+- [Mock Data](#mock-data)
+- [Insights Application](#insights-application)
+
+---
+
+## Overview
+
+Modern enterprises operate complex on-premise and hybrid data centers with
+routers, firewalls, switches, and servers forming intricate topologies.
+Cyber attacks, misconfigurations, and hardware failures can cascade across
+these networks, causing outages, data loss, and service disruptions.
+
+Graph databases like TigerGraph are well-suited for this type of analysis:
+they can represent topology, events, alerts, and incidents as a connected
+model and traverse it in real time. This solution kit combines:
+
+- **Network devices**  
+  A generic `Device` vertex plus specialized device types:
+  `Router`, `Firewall`, `Switch`, and `Server`.
+
+- **Events, alerts, and incidents**  
+  `Event`, `Alert`, and `Incident` vertices with associated classification
+  vertices (`Event_Type`, `Alert_Type`, `Incident_Type`) to capture what
+  happened and how it was categorized.
+
+- **Time hierarchy**  
+  A time dimension for temporal analysis and visualization:
+  `Time_Year`, `Time_Date_Month`, `Time_Date`, `Time_Date_Hour`,
+  `Time_Date_Minute`.
+
+- **Topology and connectivity**  
+  `Connect_To` edges between devices and impact/causal relationships such as:
+  `Impacts`, `Linked_With_Alert`, `Linked_With_Incident`, `From_Device`,
+  `To_Device`, plus relationships along the time hierarchy
+  (`Has_Minute`, `Has_Hour`, `Has_Date`, `Has_Month`, `Has_Year`).
+
+You can use the included queries as building blocks for operations,
+security analysis, and incident investigation - or extend the graph with
+your own device types, log sources, or analytics.
+
+> **Graph name:** This kit creates and uses a graph named `Network_Infrastructure`.
+
+---
+# Components
+
+This repository includes multiple components:
+
+- `data` - Sample data.
+- `load_jobs` - Scripts for data loading tasks.
+- `meta` - Solution Kit metadata - includes a TG Insights application.
+- `queries` - Collection of GSQL queries.
+- `schema` - Definition of database schema.
+- `readme.md` - This usage guide.
+- `setup.sh` - Automated setup script.
+
+
+## Prerequisites
+
+Before you run this solution kit, make sure you have:
+
+- **A running TigerGraph instance**  
+  - TigerGraph installed and running, or use the prebuilt kit on TG cloud.
+  - You must have permission to create graphs and run GSQL commands.
+
+- **GSQL client access**
+  - The `gsql` command-line tool available on the same machine/container where you cloned this repo.
+  - Ability to connect to your TigerGraph service
+
+- **Network access for sample data**
+  - Outbound internet access from the TigerGraph machine to read the sample 
+    CSV files from the public S3 bucket used in `loading_job/load_data.gsql`.  
+
+- **Shell environment**
+  - A Unix-like shell (Linux, macOS, or WSL) to run `setup.sh` and `queries/install_queries.sh`.
+  - Executable permissions for the scripts:
+    ```bash
+    chmod +x setup.sh
+    chmod +x queries/install_queries.sh
+    ```
+
+> **Graph name:** This kit creates and uses a graph named
+> `Network_Infrastructure`.
+
+# Setup Instructions
+The following instructions assume that you are running the following scripts
+with `gsql` command installed.
+
+If you don't yet have the `gsql` command available, see the TigerGraph documentation:
+
+- **Local GSQL shell on the server**  
+  [The GSQL Shell](https://docs.tigergraph.com/tigergraph-server/current/gsql-shell/)  
+  (explains how to run `gsql` directly on a TigerGraph server)
+
+- **Remote GSQL client (from your laptop or another machine)**  
+  [Using a Remote GSQL Client](https://docs.tigergraph.com/tigergraph-server/current/gsql-shell/using-a-remote-gsql-client)  
+  (explains how to download the GSQL client JAR, configure SSL, and create a `gsql` alias)
+Ensure that the script is executable with:
+```bash
+
+chmod +x setup.sh
+
+```
+Then, run the automated script using:
+```bash
+./setup.sh
+```
+
+
+The `setup.sh` script is designed to streamline the initial setup process by sequentially executing the following steps:
+
+1. **Schema Creation**: Initiates the schema creation process with the `schema/create_network_infrastructure_graph.gsql` script.
+2. **Data Loading**: Load data into the schema by running the data loading jobs with the scripts in the `loading_job` folder.
+3. **Query Installation**: Completes the setup by installing necessary queries through the `queries/install_queries.sh` script.
+
+## Query Explanations
+
+We have different queries to perform the following tasks:
+
+1. **Topology Visualization**: Explore the topology of the devices in the current dataset. There are multiple queries in this group that are suitable for different visualizations. The topology can be shown starting from router all the way down to the server, or it can be shown downstream, starting from other vertex like a router or switch.
+
+2. **Statistics**: Provide more statistics into the current network infrastructure. For instance, we can find top devices with highest amount of alerts or incidents.
+
+3. **Device Failure Impact Visualization**: Provide visualizations for a potential single point of failure. If one device along the path fails, we want to know if it's possible for other devices downstream to fail as well. The criteria for potential failed device is if there is no path from that device back to the working router.
+
+4. **Unsecure Server Visualization**: Provide visualizations of unsecured servers where it is possible to go from the router to the server without passing through the hardware firewall.
+
+5. **Incident Impact**: Find the impact of an incident based on the topology of the device within a specified radius.
+
+6. **Events Filtering**: Find the events based on some filters, like the device that is impacted by the event and the time range that the event occurs.
+
+7. **Find Potential Incidents that Cause an Event**: Find the potential source or cause of an event based on the connection of the different devices. The source or cause of the event has to be an Incident, although this can be changed.
+
+8. **Find Potential Events that Got Affected by an Incident**: Find the potential events that are caused by the input incident based on the connections of the different devices.
+
+## Mock Data
+
+The `data` folder is populated with sample data files. These files are crafted for testing and demonstration purposes.
+
+
+## Insights Application
+
+We have provided an insights application called "Network Infrastructure Insights". The applications provided the users with an intuitive view of the information in the graph.
+
+There are currently 4 pages in the Insights Application :
+- Network Infrastructure Topology Exploration
+- Network Infrastructure Topology Analysis
+- Event Visualizations
+- Event Causes Visualizations

agile_operations/network_infrastructure/queries/device_failure_impact_radius_visualization.gsql

@@ -1,6 +1,26 @@
 CREATE OR REPLACE QUERY device_failure_impact_radius_visualization (
     VERTEX<Device> device
 ) {
+    /*
+  Query Name:
+    device_failure_impact_radius_visualization
+
+  Purpose:
+    1. Find devices that do NOT have an alternative communication path.
+    2. Determine devices that will be impacted if the input device fails.
+    3. Display impacted devices and their interconnections.
+
+  Concept:
+    - If a device has an alternative path (redundant connectivity), it won't fail.
+    - Devices without alternative paths are marked as 'impacted'.
+
+  Inputs:
+    - device: The device whose failure we are analyzing.
+
+  Outputs:
+    - impacted_devices: Devices that will fail if the input device fails.
+    - @@edges_to_display: Edges between impacted devices for visual impact analysis.
+    */
 
     SetAccum<EDGE> @@edges_to_display;
     OrAccum<BOOL> @has_alternative_path;

agile_operations/network_infrastructure/queries/device_failure_impact_radius_visualization_with_subgraph_topology.gsql

@@ -1,6 +1,25 @@
 CREATE OR REPLACE QUERY device_failure_impact_radius_visualization_with_subgraph_topology (
     VERTEX<Device> device
 ) {
+    /*  
+  Query Name:
+    device_failure_impact_radius_visualization_with_subgraph_topology
+
+  Purpose:
+    1. Build the subgraph that contains the input device (based on connectivity).
+    2. Explore and detect devices that have alternative network paths (redundancy).
+    3. Identify devices that will be impacted (fail) if the given device fails.
+    4. Collect and return visualization data: subgraph topology and failure impact edges.
+
+  Input:
+    - device: The starting device whose failure impact we want to analyze.
+
+  Output:
+    - all_vertices_in_subgraph: Devices in the connectivity region of the input device.
+    - @@edges_to_display_in_subgraph: Edges representing overall subgraph structure.
+    - impacted_devices: Devices that do not have an alternative path (will fail).
+    - @@edges_to_display: Edges among impacted devices (failure impact radius).
+    */
     SetAccum<EDGE> @@edges_to_display;
     SetAccum<EDGE> @@edges_to_display_in_subgraph;
     OrAccum<BOOL> @has_alternative_path;
@@ -72,4 +91,4 @@ CREATE OR REPLACE QUERY device_failure_impact_radius_visualization_with_subgraph
 
 UPDATE DESCRIPTION OF QUERY device_failure_impact_radius_visualization_with_subgraph_topology "This query finds and visualizes the devices that will fail if the provided input device fails along with the subgraph that the input device was in. Use only for visualization purposes."
 
-UPDATE DESCRIPTION OF QUERY_PARAM device_failure_impact_radius_visualization_with_subgraph_topology.device "The input device (accepts devices of all types)."
+UPDATE DESCRIPTION OF QUERY_PARAM device_failure_impact_radius_visualization_with_subgraph_topology.device "The input device (accepts devices of all types)."

agile_operations/network_infrastructure/queries/downstream_device_topology_visualization.gsql

@@ -2,6 +2,29 @@ CREATE OR REPLACE QUERY downstream_device_topology_visualization (
     VERTEX<Device> device,
     UINT k_hop_switch_limit = 3
 ) {
+    /*
+  Query Name:
+    downstream_device_topology_visualization
+
+  Purpose:
+    Visualize the downstream topology from a given device, following the device hierarchy:
+      Router -> Firewall -> Switch -> Server
+    Also explores multiple downstream Switch layers (k-hop depth traversal).
+
+  Key Features:
+    . Identifies all downstream devices classified by device type.
+    . Follows device hierarchy dynamically based on input device type.
+    . Limits multi-hop Switch traversal using k_hop_switch_limit.
+    . Returns devices and connecting edges for visualization.
+
+  Inputs:
+    - device: Starting device.
+    - k_hop_switch_limit: Maximum depth for switch-to-switch iterations (default = 3).
+
+  Outputs:
+    - impacted_devices: All discovered downstream devices.
+    - @@edges_to_display: Edges to visualize the downstream network path.
+*/
 
     SetAccum<VERTEX> @@impacted_devices;
     SetAccum<EDGE> @@edges_to_display;

agile_operations/network_infrastructure/queries/explore_topology_from_all_router.gsql

@@ -1,4 +1,18 @@
 CREATE OR REPLACE QUERY explore_topology_from_all_router () {
+    /*
+  Query Name:
+    explore_topology_from_all_router
+
+  Purpose:
+    Visualize the entire network topology by:
+      - Retrieving all devices in the graph
+      - Exploring all connected edges using Connect_To
+      - Displaying both devices and their connections
+
+  Outputs:
+    - all_devices_with_connections: List of devices connected via Connect_To edges
+    - @@edges_to_display: All edges among connected devices for visualization
+*/
     SetAccum<EDGE> @@edges_to_display;
 
     all_devices = {Device.*};
@@ -13,4 +27,4 @@ CREATE OR REPLACE QUERY explore_topology_from_all_router () {
     PRINT @@edges_to_display AS edges_to_display;
 }
 
-UPDATE DESCRIPTION OF QUERY explore_topology_from_all_router "This query visualizes the network topology of all devices in the database. It shows the downstream connections from all routers."
+UPDATE DESCRIPTION OF QUERY explore_topology_from_all_router "This query visualizes the network topology of all devices in the database. It shows the downstream connections from all routers."

agile_operations/network_infrastructure/queries/explore_topology_from_multiple_routers.gsql

@@ -1,4 +1,27 @@
 CREATE OR REPLACE QUERY explore_topology_from_multiple_routers (SET<VERTEX<Router>> starter_router_set) {
+    /*
+  Query Name:
+    explore_topology_from_multiple_routers
+
+  Purpose:
+    Visualizes the downstream network topology starting from one or more routers.
+    Traverses devices in this structured order:
+      Router -> Firewall -> Switch -> Server
+
+  Key Features:
+     Uses BFS to exhaustively discover downstream switches.
+
+  Inputs:
+    starter_router_set - Set of router vertices to start the topology exploration.
+      If empty, all routers in the graph are selected automatically.
+
+  Outputs:
+    - all_visited_router_devices - Routers connected to input routers via Device_Has_Type
+    - all_visited_firewalls - First downstream stage (Firewalls)
+    - all_visited_switches - All connected Switches (multi-hop via BFS)
+    - all_visited_servers - Downstream Servers
+    - @@edges_to_display - All edges to display full topology
+*/
     OrAccum<BOOL> @visited;
     SetAccum<EDGE> @@edges_to_display;
 
@@ -85,4 +108,4 @@ CREATE OR REPLACE QUERY explore_topology_from_multiple_routers (SET<VERTEX<Route
 
 UPDATE DESCRIPTION OF QUERY explore_topology_from_multiple_routers "This query visualizes the network topology starting from the set of routers in the 'starter_router_set'."
 
-UPDATE DESCRIPTION OF QUERY_PARAM explore_topology_from_multiple_routers.starter_router_set "A set of vertices representing the starter router set to explore the network topology from."
+UPDATE DESCRIPTION OF QUERY_PARAM explore_topology_from_multiple_routers.starter_router_set "A set of vertices representing the starter router set to explore the network topology from."

agile_operations/network_infrastructure/queries/explore_topology_from_one_router.gsql

@@ -1,4 +1,28 @@
 CREATE OR REPLACE QUERY explore_topology_from_one_router (VERTEX<Router> starter_router) {
+    /*
+  Query Name:
+    explore_topology_from_one_router
+
+  Purpose:
+    Visualizes the downstream network topology starting from a single router.
+    Traverses devices in this structured order:
+      Router -> Firewall -> Switch -> Server
+
+  Key Features:
+    . Discovers all downstream Firewalls and Switches (including bypass paths)
+    . Uses BFS to find all connected Switches (multi-hop exploration)
+    . Captures all edges forming the full topology view
+
+  Input:
+    starter_router - A single Router vertex that acts as the exploration starting point.
+
+  Outputs:
+    - all_visited_router_devices - Devices directly connected to starter router via Device_Has_Type
+    - all_visited_firewalls - Firewalls downstream of the router
+    - all_visited_switches - All discovered Switches (including BFS expansion)
+    - all_visited_servers - Servers connected downstream of switches
+    - @@edges_to_display - Complete collection of edges forming the explored topology
+*/
     OrAccum<BOOL> @visited;
     SetAccum<EDGE> @@edges_to_display;
 
@@ -80,4 +104,4 @@ CREATE OR REPLACE QUERY explore_topology_from_one_router (VERTEX<Router> starter
 
 UPDATE DESCRIPTION OF QUERY explore_topology_from_one_router "This query visualizes the network topology starting from the router 'starter_router'."
 
-UPDATE DESCRIPTION OF QUERY_PARAM explore_topology_from_one_router.starter_router "The input router device to explore the network topology."
+UPDATE DESCRIPTION OF QUERY_PARAM explore_topology_from_one_router.starter_router "The input router device to explore the network topology."