lab49_advanced_shell_scripting.txt

RHCE RH254 HANDS-ON LAB: ADVANCED SHELL SCRIPTING
===============================================

LAB OBJECTIVE:
Master advanced shell scripting techniques including complex functions, error handling, signal processing, advanced parameter handling, and enterprise-grade script development

PREREQUISITES:
- RHEL 8/9 system with root access
- Understanding of basic shell scripting
- Knowledge of Linux system administration
- Familiarity with command-line tools

LAB SCENARIO:
Develop enterprise-grade shell scripts with advanced features including robust error handling, logging, configuration management, and automated system administration tasks.

EQUIPMENT NEEDED:
- RHEL system (192.168.1.20)
- Text editor (vim/nano)
- Various system utilities for script testing

LAB TASKS:

PART A: ADVANCED SCRIPT STRUCTURE AND FUNCTIONS
------------------------------------------------

1. Create advanced script template:
   # mkdir -p /opt/scripts/{templates,functions,configs,logs}
   # vim /opt/scripts/templates/advanced-script-template.sh
   
   #!/bin/bash
   #
   # Advanced Shell Script Template
   # Author: System Administrator
   # Version: 1.0
   # Description: Enterprise-grade script template with advanced features
   #
   
   # Script metadata
   readonly SCRIPT_NAME="$(basename "$0")"
   readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
   readonly SCRIPT_VERSION="1.0"
   readonly SCRIPT_AUTHOR="System Administrator"
   
   # Configuration
   CONFIG_FILE="${SCRIPT_DIR}/../configs/${SCRIPT_NAME%.sh}.conf"
   readonly LOG_FILE="/var/log/${SCRIPT_NAME%.sh}.log"
   readonly PID_FILE="/var/run/${SCRIPT_NAME%.sh}.pid"
   
   # Ensure required directories exist
   mkdir -p "$(dirname "$LOG_FILE")" "$(dirname "$PID_FILE")"
   
   # Global variables
   DEBUG=false
   VERBOSE=false
   DRY_RUN=false
   FORCE=false
   
   # Color codes for output
   readonly RED='\033[0;31m'
   readonly GREEN='\033[0;32m'
   readonly YELLOW='\033[1;33m'
   readonly BLUE='\033[0;34m'
   readonly NC='\033[0m' # No Color
   
   # Logging functions
   log_message() {
       local level="$1"
       local message="$2"
       local timestamp
       timestamp=$(date '+%Y-%m-%d %H:%M:%S')
   
       echo "[$timestamp] [$level] $message" >> "$LOG_FILE"
   
       case "$level" in
           "ERROR")
               echo -e "${RED}[ERROR]${NC} $message" >&2
               ;;
           "WARN")
               echo -e "${YELLOW}[WARN]${NC} $message" >&2
               ;;
           "INFO")
               [ "$VERBOSE" = true ] && echo -e "${GREEN}[INFO]${NC} $message"
               ;;
           "DEBUG")
               [ "$DEBUG" = true ] && echo -e "${BLUE}[DEBUG]${NC} $message"
               ;;
       esac
   }
   
   log_error() { log_message "ERROR" "$1"; }
   log_warn()  { log_message "WARN"  "$1"; }
   log_info()  { log_message "INFO"  "$1"; }
   log_debug() { log_message "DEBUG" "$1"; }
   
   # Error handling
   error_exit() {
       log_error "$1"
       cleanup
       exit "${2:-1}"
   }
   
   # Signal handling
   cleanup() {
       log_info "Cleaning up..."
       [ -f "$PID_FILE" ] && rm -f "$PID_FILE"
   }
   
   signal_handler() {
       local signal="$1"
       log_warn "Received signal: $signal"
       cleanup
       exit 130
   }
   
   # Set up signal traps
   trap 'signal_handler INT' INT
   trap 'signal_handler TERM' TERM
   trap 'cleanup' EXIT
   
   # Usage function
   usage() {
       cat << EOF
   Usage: $SCRIPT_NAME [OPTIONS] [ARGUMENTS]
   
   DESCRIPTION:
       Advanced shell script template with enterprise features
   
   OPTIONS:
       -h, --help          Show this help message
       -v, --verbose       Enable verbose output
       -d, --debug         Enable debug output
       -n, --dry-run       Show what would be done without executing
       -f, --force         Force execution without prompts
       -c, --config FILE   Use custom configuration file
       -V, --version       Show version information
   
   EXAMPLES:
       $SCRIPT_NAME --verbose
       $SCRIPT_NAME --config /path/to/config.conf
       $SCRIPT_NAME --dry-run --debug
   EOF
   }
   
   # Version information
   version_info() {
       cat << EOF
   $SCRIPT_NAME version $SCRIPT_VERSION
   Author: $SCRIPT_AUTHOR
   EOF
   }
   
   # Configuration loading
   load_config() {
       local config_file="${1:-$CONFIG_FILE}"
   
       if [ -f "$config_file" ]; then
           log_debug "Loading configuration from: $config_file"
           # shellcheck source=/dev/null
           source "$config_file"
       else
           log_warn "Configuration file not found: $config_file"
       fi
   }
   
   # Parameter validation
   validate_parameters() {
       return 0
   }
   
   # Main function
   main() {
       log_info "Starting $SCRIPT_NAME v$SCRIPT_VERSION"
   
       # Create PID file
       echo "$$" > "$PID_FILE" || error_exit "Unable to create PID file"
   
       # Load configuration
       load_config
   
       # Validate parameters
       validate_parameters || error_exit "Parameter validation failed"
   
       # Main script logic goes here
       log_info "Main script execution completed"
   }
   
   # Parameter parsing
   while [[ $# -gt 0 ]]; do
       case $1 in
           -h|--help)
               usage
               exit 0
               ;;
           -v|--verbose)
               VERBOSE=true
               shift
               ;;
           -d|--debug)
               DEBUG=true
               VERBOSE=true
               shift
               ;;
           -n|--dry-run)
               DRY_RUN=true
               shift
               ;;
           -f|--force)
               FORCE=true
               shift
               ;;
           -c|--config)
               CONFIG_FILE="$2"
               shift 2
               ;;
           -V|--version)
               version_info
               exit 0
               ;;
           -*)
               error_exit "Unknown option: $1"
               ;;
           *)
               break
               ;;
       esac
   done
   
   # Execute main function
   main "$@"

// Errors present which got fixed in script (in comparision to old script): 
// - readonly CONFIG_FILE was reassigned later → runtime error
// - Log and PID directories may not exist → redirection failure
// - PID file write can fail silently
// - Shellcheck-level robustness (safe sourcing, quoting)

2. Create advanced function library:
   # vim /opt/scripts/functions/advanced-functions.sh

   #!/bin/bash
   #
   # Advanced Shell Functions Library
   # Collection of reusable functions for enterprise scripts
   #
   
   # Safe defaults if main script did not define them
   : "${FORCE:=false}"
   : "${DRY_RUN:=false}"
   
   # Dummy log functions if not sourced from main template
   log_debug() { :; }
   log_info()  { :; }
   log_warn()  { echo "[WARN] $*" >&2; }
   log_error() { echo "[ERROR] $*" >&2; }
   
   # Check if running as root
   require_root() {
       if [[ $EUID -ne 0 ]]; then
           echo "This script must be run as root" >&2
           exit 1
       fi
   }
   
   # Check if command exists
   command_exists() {
       command -v "$1" >/dev/null 2>&1
   }
   
   # Prompt for confirmation
   confirm() {
       local prompt="${1:-Are you sure?}"
       local default="${2:-n}"
       local response
   
       if [ "$FORCE" = true ]; then
           return 0
       fi
   
       while true; do
           read -r -p "$prompt [y/N]: " response || return 1
           response=${response:-$default}
   
           case "$response" in
               [Yy]|[Yy][Ee][Ss])
                   return 0
                   ;;
               [Nn]|[Nn][Oo])
                   return 1
                   ;;
               *)
                   echo "Please answer yes or no."
                   ;;
           esac
       done
   }
   
   # Execute command with logging
   execute_command() {
       local cmd="$1"
       local description="${2:-Executing command}"
   
       log_debug "$description: $cmd"
   
       if [ "$DRY_RUN" = true ]; then
           log_info "[DRY RUN] Would execute: $cmd"
           return 0
       fi
   
       if eval "$cmd"; then
           log_info "$description: SUCCESS"
           return 0
       else
           local exit_code=$?
           log_error "$description: FAILED (exit code: $exit_code)"
           return "$exit_code"
       fi
   }
   
   # Retry function with exponential backoff
   retry_with_backoff() {
       local max_attempts="$1"
       local delay="$2"
       local command="$3"
       local attempt=1
   
       while [ "$attempt" -le "$max_attempts" ]; do
           log_debug "Attempt $attempt of $max_attempts: $command"
   
           if eval "$command"; then
               log_info "Command succeeded on attempt $attempt"
               return 0
           fi
   
           if [ "$attempt" -eq "$max_attempts" ]; then
               log_error "Command failed after $max_attempts attempts"
               return 1
           fi
   
           log_warn "Attempt $attempt failed, retrying in ${delay}s..."
           sleep "$delay"
           delay=$((delay * 2))
           attempt=$((attempt + 1))
       done
   }
   
   # Progress bar function
   progress_bar() {
       local current="$1"
       local total="$2"
       local width="${3:-50}"
       local prefix="${4:-Progress}"
   
       [ "$total" -eq 0 ] && return 1
   
       local percentage=$((current * 100 / total))
       local completed=$((current * width / total))
       local remaining=$((width - completed))
   
       printf "\r%s: [" "$prefix"
       printf "%*s" "$completed" | tr ' ' '='
       printf "%*s" "$remaining" | tr ' ' '-'
       printf "] %d%% (%d/%d)" "$percentage" "$current" "$total"
   
       if [ "$current" -eq "$total" ]; then
           echo
       fi
   }
   
   # File backup function
   backup_file() {
       local file="$1"
       local backup_dir="${2:-/tmp/backups}"
   
       if [ ! -f "$file" ]; then
           log_error "File not found: $file"
           return 1
       fi
   
       mkdir -p "$backup_dir" || return 1
   
       local backup_file
       backup_file="$backup_dir/$(basename "$file").$(date +%Y%m%d_%H%M%S).bak"
   
       if cp "$file" "$backup_file"; then
           log_info "File backed up: $file -> $backup_file"
           echo "$backup_file"
           return 0
       else
           log_error "Failed to backup file: $file"
           return 1
       fi
   }
   
   # Network connectivity check
   check_connectivity() {
       local host="${1:-8.8.8.8}"
       local timeout="${2:-5}"
   
       if ping -c 1 -W "$timeout" "$host" >/dev/null 2>&1; then
           log_debug "Network connectivity to $host: OK"
           return 0
       else
           log_warn "Network connectivity to $host: FAILED"
           return 1
       fi
   }
   
   # Service management functions
   ensure_service_running() {
       local service="$1"
   
       if systemctl is-active "$service" >/dev/null 2>&1; then
           log_debug "Service $service is already running"
           return 0
       fi
   
       log_info "Starting service: $service"
       if systemctl start "$service"; then
           log_info "Service $service started successfully"
           return 0
       else
           log_error "Failed to start service: $service"
           return 1
       fi
   }
   
   # Disk space check
   check_disk_space() {
       local path="${1:-/}"
       local threshold="${2:-90}"
   
       local usage
       usage=$(df -P "$path" | awk 'END {gsub(/%/, "", $5); print $5}')
   
       if [ "$usage" -gt "$threshold" ]; then
           log_warn "Disk usage for $path is ${usage}% (threshold: ${threshold}%)"
           return 1
       else
           log_debug "Disk usage for $path is ${usage}% (OK)"
           return 0
       fi
   }
   
   # Array manipulation functions
   array_contains() {
       local element="$1"
       shift
       local array=("$@")
   
       for item in "${array[@]}"; do
           if [ "$item" = "$element" ]; then
               return 0
           fi
       done
       return 1
   }
   
   # String manipulation functions
   trim_whitespace() {
       local string="$1"
       echo "$string" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//'
   }
   
   # Configuration file parsing
   parse_config_value() {
       local config_file="$1"
       local key="$2"
       local default_value="${3:-}"
   
       if [ -f "$config_file" ]; then
           local value
           value=$(grep -E "^${key}=" "$config_file" | cut -d'=' -f2- | tr -d '"' | tr -d "'")
           echo "${value:-$default_value}"
       else
           echo "$default_value"
       fi
   }

   // Errors present which got fixed in script (in comparision to old script): 
   // - Functions depend on log_* variables → fail if not sourced
   // - FORCE, DRY_RUN may be unset → [ "$VAR" = true ] errors
   // - df | tail -1 can break on localized output
   // - Arithmetic comparisons need numeric safety
   // - read -p must handle non-interactive shells
   // - Quoting & defensive checks

PART B: ERROR HANDLING AND DEBUGGING
-------------------------------------

1. Create error handling framework:
   # vim /opt/scripts/functions/error-handling.sh
   
   #!/bin/bash
   #
   # Advanced Error Handling Framework
   #
   
   # Safe defaults if not defined by caller
   : "${DEBUG:=false}"
   
   # Dummy logging & cleanup functions if not sourced
   log_error() { echo "[ERROR] $*" >&2; }
   log_warn()  { echo "[WARN] $*" >&2; }
   cleanup()   { :; }
   
   # Error codes
   readonly E_SUCCESS=0
   readonly E_GENERAL=1
   readonly E_MISUSE=2
   readonly E_NOEXEC=126
   readonly E_NOTFOUND=127
   readonly E_INVALID_ARG=128
   readonly E_FATAL=130
   
   # Error tracking
   ERROR_COUNT=0
   WARNING_COUNT=0
   
   # Stack trace function
   print_stack_trace() {
       local frame=0
       echo "Stack trace:" >&2
   
       while caller "$frame" >&2; do
           ((frame++))
       done
   }
   
   # Enhanced error exit
   die() {
       local exit_code="${1:-$E_GENERAL}"
       local message="${2:-An error occurred}"
   
       log_error "$message"
   
       if [ "$DEBUG" = true ]; then
           print_stack_trace
       fi
   
       cleanup
       exit "$exit_code"
   }
   
   # Error accumulator
   add_error() {
       local message="$1"
       ERROR_COUNT=$((ERROR_COUNT + 1))
       log_error "Error #$ERROR_COUNT: $message"
   }
   
   add_warning() {
       local message="$1"
       WARNING_COUNT=$((WARNING_COUNT + 1))
       log_warn "Warning #$WARNING_COUNT: $message"
   }
   
   # Check for accumulated errors
   check_errors() {
       if [ "$ERROR_COUNT" -gt 0 ]; then
           die "$E_GENERAL" "Script completed with $ERROR_COUNT error(s) and $WARNING_COUNT warning(s)"
       elif [ "$WARNING_COUNT" -gt 0 ]; then
           log_warn "Script completed with $WARNING_COUNT warning(s)"
       fi
   }
   
   # Assertion function
   assert() {
       local condition="$1"
       local message="${2:-Assertion failed}"
   
       if ! eval "$condition"; then
           die "$E_GENERAL" "ASSERTION FAILED: $message"
       fi
   }
   
   # Try-catch simulation
   try() {
       local command="$1"
       local error_handler="${2:-}"
   
       if ! eval "$command"; then
           local exit_code=$?
           if [ -n "$error_handler" ]; then
               eval "$error_handler $exit_code"
           else
               add_error "Command failed: $command (exit code: $exit_code)"
           fi
           return "$exit_code"
       fi
       return 0
   }

 //  Errors present which got fixed in script (in comparision to old script): 
 // - log_error, log_warn, cleanup, DEBUG may be undefined
 //- [ "$DEBUG" = true ] fails if DEBUG unset
 // - Arithmetic comparisons without quotes (shellcheck)
 // - Library should not break if sourced alone

2. Create debugging utilities:
   # vim /opt/scripts/functions/debug-utils.sh
   
   #!/bin/bash
   #
   # Debugging Utilities
   #
   
   # Safe defaults
   : "${DEBUG:=false}"
   
   # Dummy logger if not defined
   log_debug() { :; }
   
   # Debug trace
   debug_trace() {
       if [ "$DEBUG" = true ]; then
           echo "DEBUG: ${BASH_SOURCE[1]}:${BASH_LINENO[0]} ${FUNCNAME[1]}()" >&2
       fi
   }
   
   # Variable dumper
   dump_vars() {
       local prefix="${1:-}"
   
       if [ "$DEBUG" = true ]; then
           echo "=== Variable Dump ===" >&2
           if [ -n "$prefix" ]; then
               set | grep "^$prefix" >&2
           else
               set >&2
           fi
           echo "===================" >&2
       fi
   }
   
   # Function call tracer
   trace_calls() {
       if [ "$DEBUG" = true ]; then
           set -x
       fi
   }
   
   stop_trace() {
       set +x
   }
   
   # Performance timing
   start_timer() {
       TIMER_START=$(date +%s.%N)
   }
   
   end_timer() {
       local label="${1:-Operation}"
       local end_time
       local duration
   
       end_time=$(date +%s.%N)
   
       if command -v bc >/dev/null 2>&1 && [ -n "${TIMER_START:-}" ]; then
           duration=$(echo "$end_time - $TIMER_START" | bc)
           log_debug "$label took ${duration}s"
       fi
   }
   
   # Memory usage tracker
   check_memory_usage() {
       local process_name="${1:-$$}"
       local memory_kb
   
       memory_kb=$(ps -o rss= -p "$process_name" 2>/dev/null)
   
       if [ -n "$memory_kb" ]; then
           local memory_mb=$((memory_kb / 1024))
           log_debug "Memory usage: ${memory_mb}MB"
       fi
   }

// Errors fixed in script (in comparision to old script): 
// -DEBUG, log_debug, TIMER_START may be unset
// -bc dependency not validated
// -Arithmetic without guarding
// -Library must not crash when sourced alone

PART C: ADVANCED PARAMETER HANDLING
------------------------------------

1. Create parameter processing framework:
   # vim /opt/scripts/functions/parameter-handling.sh
   
   #!/bin/bash
   #
   # Advanced Parameter Handling Framework
   #
   
   # Safe fallback logger if main framework not sourced
   log_debug() { :; }
   
   # Safe fallback for config parsing if not sourced
   parse_config_value() { :; }
   
   # Parameter validation functions
   validate_integer() {
       local value="$1"
       local min="${2:-}"
       local max="${3:-}"
   
       if ! [[ "$value" =~ ^[0-9]+$ ]]; then
           return 1
       fi
   
       if [ -n "$min" ] && [ "$value" -lt "$min" ]; then
           return 1
       fi
   
       if [ -n "$max" ] && [ "$value" -gt "$max" ]; then
           return 1
       fi
   
       return 0
   }
   
   validate_email() {
       local email="$1"
       local regex="^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$"
   
       if [[ "$email" =~ $regex ]]; then
           return 0
       else
           return 1
       fi
   }
   
   validate_ip_address() {
       local ip="$1"
       local regex="^([0-9]{1,3}\.){3}[0-9]{1,3}$"
   
       if [[ "$ip" =~ $regex ]]; then
           IFS='.' read -ra octets <<< "$ip"
           for octet in "${octets[@]}"; do
               if [ "$octet" -gt 255 ]; then
                   return 1
               fi
           done
           return 0
       else
           return 1
       fi
   }
   
   validate_file_path() {
       local path="$1"
       local must_exist="${2:-false}"
   
       if [ "$must_exist" = true ] && [ ! -e "$path" ]; then
           return 1
       fi
   
       if [[ "$path" =~ ^[a-zA-Z0-9/_.-]+$ ]]; then
           return 0
       else
           return 1
       fi
   }
   
   # Advanced option parsing
   parse_long_options() {
       local args=("$@")
       local parsed_args=()
   
       for arg in "${args[@]}"; do
           case "$arg" in
               --*=*)
                   local option="${arg%%=*}"
                   local value="${arg#*=}"
                   parsed_args+=("$option" "$value")
                   ;;
               --*)
                   parsed_args+=("$arg")
                   ;;
               *)
                   parsed_args+=("$arg")
                   ;;
           esac
       done
   
       echo "${parsed_args[@]}"
   }
   
   # Configuration file parameter override
   override_with_config() {
       local config_file="$1"
       shift
       local variables=("$@")
   
       if [ -f "$config_file" ]; then
           for var in "${variables[@]}"; do
               local value
               value=$(parse_config_value "$config_file" "$var")
               if [ -n "$value" ]; then
                   declare -g "$var"="$value"
                   log_debug "Override from config: $var=$value"
               fi
           done
       fi
   }
   
   # Environment variable parameter override
   override_with_env() {
       local prefix="$1"
       shift
       local variables=("$@")
   
       for var in "${variables[@]}"; do
           local env_var="${prefix}_${var^^}"
           if [ -n "${!env_var:-}" ]; then
               declare -g "$var"="${!env_var}"
               log_debug "Override from environment: $var=${!env_var}"
           fi
       done
   }

   // Errors present which got fixed in script (in comparision to old script): 
   // -log_debug may be undefined
   // -parse_config_value may be undefined
   // -declare -g must not break older shells (safe usage)
   // -Unquoted command substitutions
   // -Defensive defaults so the library is safe when sourced alon

2. Create configuration management system:
   # vim /opt/scripts/functions/config-management.sh
   
   #!/bin/bash
   #
   # Configuration Management System
   #
   
   # Safe fallback loggers if main framework not sourced
   log_info()  { :; }
   log_error() { echo "[ERROR] $*" >&2; }
   
   # Safe fallback for config parsing
   parse_config_value() { :; }
   
   # Configuration file template generator
   generate_config_template() {
       local config_file="$1"
       local script_name="${2:-$(basename -- "$0" .sh)}"
   
       # Ensure target directory exists
       mkdir -p "$(dirname "$config_file")" || return 1
   
       cat > "$config_file" << EOF
   # Configuration file for $script_name
   # Generated on $(date)
   
   # General settings
   DEBUG=false
   VERBOSE=false
   LOG_LEVEL=INFO
   
   # Paths
   LOG_DIR=/var/log
   TEMP_DIR=/tmp
   BACKUP_DIR=/backup
   
   # Network settings
   TIMEOUT=30
   RETRY_COUNT=3
   
   # Email notifications
   ENABLE_EMAIL=false
   ADMIN_EMAIL=admin@example.com
   SMTP_SERVER=localhost
   
   # Custom settings (add your own below)
   # CUSTOM_SETTING=value
   EOF
   
       log_info "Configuration template created: $config_file"
   }
   
   # Configuration validation
   validate_config() {
       local config_file="$1"
       shift
       local required_vars=("$@")
   
       if [ ! -f "$config_file" ]; then
           log_error "Configuration file not found: $config_file"
           return 1
       fi
   
       local missing_vars=()
   
       for var in "${required_vars[@]}"; do
           local value
           value=$(parse_config_value "$config_file" "$var")
           if [ -z "$value" ]; then
               missing_vars+=("$var")
           fi
       done
   
       if [ "${#missing_vars[@]}" -gt 0 ]; then
           log_error "Missing required configuration variables: ${missing_vars[*]}"
           return 1
       fi
   
       return 0
   }
   
   # Dynamic configuration reloading
   reload_config() {
       local config_file="$1"
   
       if [ -f "$config_file" ]; then
           log_info "Reloading configuration from: $config_file"
           # shellcheck source=/dev/null
           source "$config_file"
           return 0
       else
           log_error "Cannot reload config, file not found: $config_file"
           return 1
       fi
   }

   // Errors which got fixed in script (in comparision to old script): 
   // - log_info, log_error may be undefined
   // - parse_config_value may be undefined
   // - source "$config_file" can break shellcheck / strict shells
   // - cat > "$config_file" fails if directory doesn’t exist
   // - Array handling needed defensive quoting

PART D: SYSTEM ADMINISTRATION SCRIPTS
--------------------------------------

1. Create system monitoring script:
   # vim /opt/scripts/system-monitor.sh
   
   #!/bin/bash
   #
   # Advanced System Monitoring Script
   #
   
   # Source function libraries
   source /opt/scripts/functions/advanced-functions.sh
   source /opt/scripts/functions/error-handling.sh
   source /opt/scripts/functions/debug-utils.sh
   source /opt/scripts/templates/advanced-script-template.sh 2>/dev/null || true
   
   # Safe defaults
   log_info()  { :; }
   log_debug() { :; }
   log_warn()  { echo "[WARN] $*" >&2; }
   log_error() { echo "[ERROR] $*" >&2; }
   load_config() { :; }
   
   # Script configuration
   readonly SCRIPT_NAME="system-monitor"
   readonly CONFIG_FILE="/opt/scripts/configs/system-monitor.conf"
   readonly LOG_FILE="/var/log/system-monitor.log"
   
   mkdir -p "$(dirname "$LOG_FILE")"
   
   # Default thresholds
   CPU_THRESHOLD=80
   MEMORY_THRESHOLD=85
   DISK_THRESHOLD=90
   LOAD_THRESHOLD=2.0
   
   # Monitoring functions
   check_cpu_usage() {
       local cpu_usage
       cpu_usage=$(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d'%' -f1 | cut -d'.' -f1)
   
       log_debug "CPU usage: ${cpu_usage}%"
   
       if [ "$cpu_usage" -gt "$CPU_THRESHOLD" ]; then
           add_warning "High CPU usage: ${cpu_usage}% (threshold: ${CPU_THRESHOLD}%)"
           log_info "Top CPU processes:"
           ps aux --sort=-%cpu | head -6 | tail -5
       fi
   
       echo "$cpu_usage"
   }
   
   check_memory_usage() {
       local memory_usage
       memory_usage=$(free | awk '/Mem:/ {printf("%.0f", $3/$2 * 100)}')
   
       log_debug "Memory usage: ${memory_usage}%"
   
       if [ "$memory_usage" -gt "$MEMORY_THRESHOLD" ]; then
           add_warning "High memory usage: ${memory_usage}% (threshold: ${MEMORY_THRESHOLD}%)"
           log_info "Top memory processes:"
           ps aux --sort=-%mem | head -6 | tail -5
       fi
   
       echo "$memory_usage"
   }
   
   check_disk_usage() {
       df -h | grep -E '^/dev/' | while read -r line; do
           local usage filesystem
           usage=$(echo "$line" | awk '{print $5}' | tr -d '%')
           filesystem=$(echo "$line" | awk '{print $6}')
   
           log_debug "Disk usage $filesystem: ${usage}%"
   
           if [ "$usage" -gt "$DISK_THRESHOLD" ]; then
               add_warning "High disk usage: $filesystem ${usage}% (threshold: ${DISK_THRESHOLD}%)"
               find "$filesystem" -type f -size +100M 2>/dev/null | head -5
           fi
       done
   }
   
   check_load_average() {
       local load_1min
       load_1min=$(uptime | awk -F'load average:' '{print $2}' | awk -F',' '{print $1}' | tr -d ' ')
   
       log_debug "Load average (1min): $load_1min"
   
       if command -v bc >/dev/null 2>&1; then
           if (( $(echo "$load_1min > $LOAD_THRESHOLD" | bc -l) )); then
               add_warning "High load average: $load_1min (threshold: $LOAD_THRESHOLD)"
           fi
       fi
   
       echo "$load_1min"
   }
   
   check_services() {
       local critical_services=("sshd" "systemd-logind" "NetworkManager")
   
       for service in "${critical_services[@]}"; do
           if systemctl is-active "$service" >/dev/null 2>&1; then
               log_debug "Service $service: RUNNING"
           else
               add_error "Critical service not running: $service"
           fi
       done
   }
   
   generate_report() {
       local report_file="/tmp/system-monitor-report-$(date +%Y%m%d_%H%M%S).txt"
   
       cat > "$report_file" << EOF
   SYSTEM MONITORING REPORT
   Generated: $(date)
   Hostname: $(hostname)
   Uptime: $(uptime)
   
   CPU Usage: $(check_cpu_usage)%
   Memory Usage: $(check_memory_usage)%
   Load Average: $(check_load_average)
   
   Disk Usage:
   $(df -h | grep -E '^/dev/')
   
   Network Interfaces:
   $(ip -o link show | awk -F': ' '{print $2}')
   
   Running Processes: $(ps aux | wc -l)
   
   Errors: $ERROR_COUNT
   Warnings: $WARNING_COUNT
   EOF
   
       log_info "System monitoring report generated: $report_file"
       echo "$report_file"
   }
   
   main() {
       log_info "Starting system monitoring"
       start_timer
   
       load_config
   
       check_cpu_usage >/dev/null
       check_memory_usage >/dev/null
       check_disk_usage
       check_load_average >/dev/null
       check_services
   
       local report_file
       report_file=$(generate_report)
   
       end_timer "System monitoring"
       check_errors
   
       log_info "System monitoring completed. Report: $report_file"
   }
   
   main "$@"

   // Errors which got fixed in script (in comparision to old script): 
   // - Log file directory may not exist
   // - load_config, log_*, add_error, add_warning, check_errors may be undefined
   // - bc dependency unchecked
   // - while read line without -r
   // - Subshell pipeline warning (acceptable, but made safe)
   // - Script should not crash if frameworks partially missing


2. Create user management script:
   # vim /opt/scripts/user-management.sh
   
   #!/bin/bash
   #
   # Advanced User Management Script
   #
   
   # Source function libraries
   source /opt/scripts/functions/advanced-functions.sh
   source /opt/scripts/functions/error-handling.sh
   source /opt/scripts/functions/parameter-handling.sh
   
   # Safe defaults
   log_info()  { :; }
   log_warn()  { echo "[WARN] $*" >&2; }
   log_error() { echo "[ERROR] $*" >&2; }
   
   require_root
   
   # Script configuration
   readonly SCRIPT_NAME="user-management"
   readonly CONFIG_FILE="/opt/scripts/configs/user-management.conf"
   readonly LOG_FILE="/var/log/user-management.log"
   
   mkdir -p "$(dirname "$LOG_FILE")"
   
   # Default settings
   DEFAULT_SHELL="/bin/bash"
   DEFAULT_HOME_DIR="/home"
   PASSWORD_LENGTH=12
   
   create_user() {
       local username="$1"
       local full_name="$2"
       local groups="$3"
       local shell="${4:-$DEFAULT_SHELL}"
   
       log_info "Creating user: $username"
   
       if ! [[ "$username" =~ ^[a-z][a-z0-9_-]*$ ]]; then
           add_error "Invalid username format: $username"
           return 1
       fi
   
       if id "$username" >/dev/null 2>&1; then
           add_warning "User already exists: $username"
           return 1
       fi
   
       execute_command "useradd -m -s '$shell' -c '$full_name' '$username'" \
           "Creating user $username" || return 1
   
       if [ -n "$groups" ]; then
           execute_command "usermod -a -G '$groups' '$username'" \
               "Adding user $username to groups: $groups"
       fi
   
       if command -v openssl >/dev/null 2>&1; then
           local password
           password=$(generate_password)
           echo "$username:$password" | chpasswd
           log_info "Generated password: $password"
       else
           add_warning "openssl not found; password not generated"
       fi
   
       log_info "User created successfully: $username"
   }
   
   delete_user() {
       local username="$1"
       local remove_home="${2:-true}"
   
       log_info "Deleting user: $username"
   
       id "$username" >/dev/null 2>&1 || {
           add_error "User does not exist: $username"
           return 1
       }
   
       confirm "Delete user $username?" || return 0
   
       local backup_file
       backup_file=$(backup_user_data "$username")
   
       local cmd="userdel"
       [ "$remove_home" = true ] && cmd="$cmd -r"
       cmd="$cmd '$username'"
   
       execute_command "$cmd" "Deleting user $username" || return 1
   
       [ -n "$backup_file" ] && log_info "User data backed up to: $backup_file"
   }
   
   modify_user() {
       local username="$1"
       local attribute="$2"
       local value="$3"
   
       id "$username" >/dev/null 2>&1 || {
           add_error "User does not exist: $username"
           return 1
       }
   
       case "$attribute" in
           shell)   execute_command "usermod -s '$value' '$username'" ;;
           groups)  execute_command "usermod -G '$value' '$username'" ;;
           home)    execute_command "usermod -d '$value' -m '$username'" ;;
           comment) execute_command "usermod -c '$value' '$username'" ;;
           *) add_error "Unknown attribute: $attribute"; return 1 ;;
       esac
   }
   
   generate_password() {
       local length="${1:-$PASSWORD_LENGTH}"
       openssl rand -base64 32 | tr -d "=+/" | cut -c1-"$length"
   }
   
   backup_user_data() {
       local username="$1"
       local backup_dir="/tmp/user-backups"
       local timestamp
       timestamp=$(date +%Y%m%d_%H%M%S)
   
       mkdir -p "$backup_dir"
   
       local backup_file="$backup_dir/${username}-backup-${timestamp}.tar.gz"
       if [ -d "/home/$username" ]; then
           tar -czf "$backup_file" -C /home "$username" 2>/dev/null
           echo "$backup_file"
       fi
   }
   
   list_users() {
       local filter="${1:-all}"
   
       case "$filter" in
           system)  awk -F: '$3 < 1000 {print $1}' /etc/passwd ;;
           regular) awk -F: '$3 >= 1000 && $3 != 65534 {print $1}' /etc/passwd ;;
           all)     awk -F: '{print $1}' /etc/passwd ;;
           *) add_error "Invalid filter: $filter"; return 1 ;;
       esac
   }
   
   main() {
       local action="$1"
       shift
   
       case "$action" in
           create) create_user "$@" ;;
           delete) delete_user "$@" ;;
           modify) modify_user "$@" ;;
           list)   list_users "$@" ;;
           *) echo "Usage: $0 {create|delete|modify|list}"; exit 1 ;;
       esac
   
       check_errors
   }
   
   main "$@"
   
   // Errors which got fixed in script (in comparision to old script): 
   // - Log directory may not exist
   // - Script must be run as root
   // - openssl dependency unchecked
   // - backup_user_data may return empty value
   // - Safe defaults for sourced frameworks

PART E: AUTOMATION FRAMEWORKS
-----------------------------

1. Create task automation framework:
   # vim /opt/scripts/task-automation.sh
   
   #!/bin/bash
   #
   # Task Automation Framework
   #
   
   # Ensure Bash (associative arrays + nameref)
   [ -n "$BASH_VERSION" ] || {
       echo "This script requires bash" >&2
       exit 1
   }
   
   # Source function libraries
   source /opt/scripts/functions/advanced-functions.sh 2>/dev/null || true
   source /opt/scripts/functions/error-handling.sh 2>/dev/null || true
   
   # Safe fallbacks if frameworks not loaded
   log_debug() { :; }
   log_info()  { :; }
   add_error() { echo "[ERROR] $*" >&2; }
   die()       { echo "[FATAL] $*" >&2; exit 1; }
   array_contains() {
       local element="$1"; shift
       for item in "$@"; do
           [ "$item" = "$element" ] && return 0
       done
       return 1
   }
   : "${E_GENERAL:=1}"
   
   # Task definitions
   declare -A TASKS
   declare -A TASK_DEPENDENCIES
   declare -A TASK_STATUS
   
   # Task registration
   register_task() {
       local task_name="$1"
       local task_function="$2"
       local dependencies="$3"
   
       TASKS["$task_name"]="$task_function"
       TASK_DEPENDENCIES["$task_name"]="$dependencies"
       TASK_STATUS["$task_name"]="pending"
   
       log_debug "Registered task: $task_name"
   }
   
   # Dependency resolution
   resolve_dependencies() {
       local task="$1"
       local resolved=()
       local unresolved=()
   
       resolve_task_deps "$task" resolved unresolved
       echo "${resolved[@]}"
   }
   
   resolve_task_deps() {
       local task="$1"
       local -n resolved_ref=$2
       local -n unresolved_ref=$3
   
       unresolved_ref+=("$task")
   
       local deps="${TASK_DEPENDENCIES[$task]}"
       if [ -n "$deps" ]; then
           for dep in $deps; do
               if ! array_contains "$dep" "${resolved_ref[@]}"; then
                   if array_contains "$dep" "${unresolved_ref[@]}"; then
                       die "$E_GENERAL" "Circular dependency detected: $task -> $dep"
                   fi
                   resolve_task_deps "$dep" resolved_ref unresolved_ref
               fi
           done
       fi
   
       resolved_ref+=("$task")
   
       # Remove from unresolved
       local temp_unresolved=()
       for item in "${unresolved_ref[@]}"; do
           if [ "$item" != "$task" ]; then
               temp_unresolved+=("$item")
           fi
       done
       unresolved_ref=("${temp_unresolved[@]}")
   }
   
   # Task execution
   execute_task() {
       local task_name="$1"
   
       if [ "${TASK_STATUS[$task_name]}" = "completed" ]; then
           log_debug "Task already completed: $task_name"
           return 0
       fi
   
       log_info "Executing task: $task_name"
       TASK_STATUS["$task_name"]="running"
   
       local task_function="${TASKS[$task_name]}"
   
       if [ -n "$task_function" ]; then
           if eval "$task_function"; then
               TASK_STATUS["$task_name"]="completed"
               log_info "Task completed successfully: $task_name"
               return 0
           else
               TASK_STATUS["$task_name"]="failed"
               add_error "Task failed: $task_name"
               return 1
           fi
       else
           add_error "Task function not found: $task_name"
           return 1
       fi
   }
   
   # Execute task with dependencies
   execute_task_with_deps() {
       local task="$1"
       local execution_order
       execution_order=($(resolve_dependencies "$task"))
   
       log_info "Execution order: ${execution_order[*]}"
   
       for task_name in "${execution_order[@]}"; do
           execute_task "$task_name" || return 1
       done
   }
   
   # Task status reporting
   report_task_status() {
       echo "Task Status Report:"
       echo "==================="
   
       for task in "${!TASKS[@]}"; do
           printf "%-20s: %s\n" "$task" "${TASK_STATUS[$task]}"
       done
   }
   
   // Errors which got fixed in script (in comparision to old script): 
   // - log_*, add_error, die, array_contains may be undefined
   // - E_GENERAL may be undefined
   // - declare -A requires Bash (guarded)
   // - local -n (nameref) requires Bash ≥ 4.3 (RHCE OK, but must be safe)
   // - Script must not crash if sourced alone


2. Create configuration deployment script:
   # vim /opt/scripts/config-deployment.sh
   
   #!/bin/bash
   #
   # Configuration Deployment Script
   #
   
   # Source function libraries
   source /opt/scripts/functions/advanced-functions.sh 2>/dev/null || true
   source /opt/scripts/functions/error-handling.sh 2>/dev/null || true
   source /opt/scripts/task-automation.sh 2>/dev/null || true
   
   # Safe fallbacks if frameworks not fully loaded
   log_info()  { :; }
   log_debug() { :; }
   log_warn()  { echo "[WARN] $*" >&2; }
   log_error() { echo "[ERROR] $*" >&2; }
   add_error() { echo "[ERROR] $*" >&2; }
   add_warning() { echo "[WARN] $*" >&2; }
   check_errors() { :; }
   
   command_exists() {
       command -v "$1" >/dev/null 2>&1
   }
   
   check_connectivity() {
       ping -c 1 -W 3 8.8.8.8 >/dev/null 2>&1
   }
   
   # Require root (config + services)
   [ "$EUID" -eq 0 ] || {
       log_error "This script must be run as root"
       exit 1
   }
   
   # Configuration deployment tasks
   task_backup_configs() {
       log_info "Backing up current configurations"
   
       local backup_dir="/tmp/config-backup-$(date +%Y%m%d_%H%M%S)"
       mkdir -p "$backup_dir"
   
       local config_files=(
           "/etc/httpd/conf/httpd.conf"
           "/etc/ssh/sshd_config"
           "/etc/hosts"
       )
   
       for config in "${config_files[@]}"; do
           if [ -f "$config" ]; then
               cp "$config" "$backup_dir/"
               log_debug "Backed up: $config"
           fi
       done
   
       log_info "Configuration backup completed: $backup_dir"
   }
   
   task_validate_configs() {
       log_info "Validating configuration files"
   
       if command_exists httpd; then
           execute_command "httpd -t" "Validating Apache configuration"
       fi
   
       execute_command "sshd -t" "Validating SSH configuration"
   
       log_info "Configuration validation completed"
   }
   
   task_deploy_configs() {
       log_info "Deploying new configurations"
   
       local source_dir="/opt/scripts/configs/templates"
       local configs=(
           "httpd.conf:/etc/httpd/conf/httpd.conf"
           "sshd_config:/etc/ssh/sshd_config"
       )
   
       for config_mapping in "${configs[@]}"; do
           local source_file="${config_mapping%%:*}"
           local target_file="${config_mapping##*:}"
   
           if [ -f "$source_dir/$source_file" ]; then
               execute_command \
                   "cp '$source_dir/$source_file' '$target_file'" \
                   "Deploying $source_file"
           fi
       done
   
       log_info "Configuration deployment completed"
   }
   
   task_restart_services() {
       log_info "Restarting services"
   
       local services=("httpd" "sshd")
   
       for service in "${services[@]}"; do
           if systemctl is-enabled "$service" >/dev/null 2>&1; then
               execute_command "systemctl restart '$service'" "Restarting $service"
           fi
       done
   
       log_info "Service restart completed"
   }
   
   task_verify_deployment() {
       log_info "Verifying deployment"
   
       local services=("httpd" "sshd")
   
       for service in "${services[@]}"; do
           if systemctl is-active "$service" >/dev/null 2>&1; then
               log_info "Service $service: RUNNING"
           else
               add_error "Service $service: NOT RUNNING"
           fi
       done
   
       if check_connectivity; then
           log_info "Network connectivity: OK"
       else
           add_warning "Network connectivity: FAILED"
       fi
   
       log_info "Deployment verification completed"
   }
   
   # Register tasks
   register_task "backup"   "task_backup_configs"    ""
   register_task "validate" "task_validate_configs"  "backup"
   register_task "deploy"   "task_deploy_configs"    "validate"
   register_task "restart"  "task_restart_services"  "deploy"
   register_task "verify"   "task_verify_deployment" "restart"
   
   # Main function
   main() {
       local target_task="${1:-verify}"
   
       log_info "Starting configuration deployment"
   
       if execute_task_with_deps "$target_task"; then
           log_info "Configuration deployment completed successfully"
       else
           log_error "Configuration deployment failed"
       fi
   
       report_task_status
       check_errors
   }
   
   # Execute main function
   main "$@"
   
   
   // Errors fixed in script (in comparision to old script): 
   // - Script assumes logging / error / task frameworks always exist
   // - Log file directory may not exist
   // - Script must be run as root (system config + services)
   // - command_exists, check_connectivity, execute_task_with_deps may be undefined
   // - Safe fallbacks required so script doesn’t crash if sourced standalone
   

PART F: TESTING AND VALIDATION
-------------------------------

1. Test advanced script template:
   # cp /opt/scripts/templates/advanced-script-template.sh /tmp/test-script.sh
   # chmod +x /tmp/test-script.sh
   # /tmp/test-script.sh --help
   # /tmp/test-script.sh --version
   # /tmp/test-script.sh --verbose --debug

2. Test function libraries:
   # source /opt/scripts/functions/advanced-functions.sh
   # command_exists "ls" && echo "Command exists test: PASSED"
   # source /opt/scripts/functions/parameter-handling.sh
   # validate_integer "123" 1 200 && echo "Integer validation test: PASSED"
   # check_connectivity "8.8.8.8" && echo "Connectivity test: PASSED"

3. Test error handling:
   # source /opt/scripts/functions/error-handling.sh
   # add_error "Test error message"
   # add_warning "Test warning message"
   # echo "Error count: $ERROR_COUNT, Warning count: $WARNING_COUNT"

4. Test system monitoring script:
   # chmod +x /opt/scripts/system-monitor.sh
   # /opt/scripts/system-monitor.sh
   # cat /var/log/system-monitor.log

   // If error comes while validating Test-4, then follow the below steps, or else move directly to Test-5
   //   Error: readonly SCRIPT_NAME / LOG_FILE errors because of Script-template variables conflict
      
   //   Solution: 
   // a.) Dont source the script template
   // source /opt/scripts/templates/advanced-script-template.sh // Dont do this step
      
   // b.) Update ths script - /opt/scripts/system-monitor.sh (Search around top of file). To define logging locally.
      
      log_message() {
          local level="$1"; shift
          printf '[%s] [%s] %s\n' "$(date '+%F %T')" "$level" "$*" >> "$LOG_FILE"
      }
      log_info()  { log_message INFO "$@"; }
      log_warn()  { log_message WARN "$@"; }
      log_error() { log_message ERROR "$@"; }
      log_debug() { :; }
      
   // Using own variable
      readonly SCRIPT_NAME="system-monitor"
      readonly CONFIG_FILE="/opt/scripts/configs/system-monitor.conf"
      readonly LOG_FILE="/var/log/system-monitor.log"
      
   // c.) Run the script
   // chmod +x /opt/scripts/system-monitor.sh
   // /opt/scripts/system-monitor.sh 

5. Test user management script:
   # chmod +x /opt/scripts/user-management.sh
   # /opt/scripts/user-management.sh list regular
   # /opt/scripts/user-management.sh create testuser "Test User" "users"

   // If error comes while validating Test-5, then follow the below steps, or else move directly to Test-6
   // Error: Logging executed as commands. Readonly variable conflicts
   
   // Solution:   
   // a.) Near the top of user-management.sh update:
      
      log_message() {
          local level="$1"; shift
          printf '[%s] [%s] %s\n' "$(date '+%F %T')" "$level" "$*" >> "$LOG_FILE"
      }
      log_info()  { log_message INFO "$@"; }
      log_warn()  { log_message WARN "$@"; }
      log_error() { log_message ERROR "$@"; }
      log_debug() { :; }
      
   // b.) Fix execute_command function inside advanced-function.sh
       
      execute_command() {
          local cmd="$1"
          local description="${2:-Executing command}"
          if [ "$DRY_RUN" = true ]; then
              log_info "[DRY RUN] Would execute: %s" "$cmd"
              return 0
          fi
          if eval "$cmd"; then
              log_info "%s: SUCCESS" "$description"
              return 0
          else
              local exit_code=$?
              log_error "%s: FAILED (exit code: %d)" "$description" "$exit_code"
              return "$exit_code"
          fi
      }
      
   // 3.) Test
      # /opt/scripts/user-management.sh create testuser_pass "Test User" "users"
      # id testuser_pass

6. Test task automation:
   # chmod +x /opt/scripts/config-deployment.sh
   # /opt/scripts/config-deployment.sh backup

7. Create test configuration:
   # mkdir -p /opt/scripts/configs
   # /opt/scripts/functions/config-management.sh generate_config_template /opt/scripts/configs/test.conf

TROUBLESHOOTING COMMANDS:
-------------------------
# bash -x /opt/scripts/script-name.sh  # Debug mode
# shellcheck /opt/scripts/script-name.sh  # Syntax checking
# tail -f /var/log/script-name.log  # Monitor logs
# ps aux | grep script-name  # Check running scripts

EXPECTED RESULTS:
-----------------
- Advanced script template with comprehensive features
- Robust error handling and debugging capabilities
- Advanced parameter processing and validation
- Reusable function libraries for common tasks
- System administration automation scripts
- Task automation framework with dependency resolution
- Configuration management and deployment automation

VALIDATION CHECKLIST:
---------------------
□ Advanced script template functional
□ Function libraries working correctly
□ Error handling and debugging operational
□ Parameter validation working
□ System monitoring script running
□ User management script functional
□ Task automation framework operational
□ Configuration deployment working
□ All scripts properly documented
□ Logging and debugging features active

CLEANUP:
--------
# rm -rf /opt/scripts
# rm /tmp/test-script.sh
# rm /var/log/system-monitor.log
# rm /var/log/user-management.log
# userdel -r testuser 2>/dev/null

MODIFIED LINES:
---------------------------------------------
30-235, 240- 485, 493-597, 602-681, 689-826, 831-930, 938-1100, 1106-1270, 1278-1424, 1430-1589, 1605, 846, 1620-1645, 1652-1688