Skip to content

README.md

Latest commit

 

History

History
433 lines (351 loc) · 17.9 KB

README.md

File metadata and controls

433 lines (351 loc) · 17.9 KB

Ops Scripts Toolkit

A practical shell and Python toolkit built from 15 years of real-world IT operations.

About

Tyrel Orde Fecha — System Administrator | Network Administrator | DataCenter Ops | DevOps | Cloud Engineer

Background

I started in on-prem and hybrid environments Cisco networks, bare-metal systems, and day-to-day data center operations—then expanded into AWS, automation, and containerized delivery.

This collection reflects that journey: practical tools used to run, troubleshoot, secure, and scale production environments. The current focus extends into Platform Engineering, FinOps, and AIOps while keeping operations simple, reliable, and measurable.

Core areas

  • Cloud / AWS operations
  • System administration and health checks
  • Networking and DNS diagnostics
  • Database operations (MySQL/RDS)
  • Storage and file-transfer operations (FTP, SMB/CIFS)

Find scripts quickly

Use the inventory tool to list and categorize everything:

./tool_inventory.py
./tool_inventory.py --json
./tool_inventory.py --category aws
./tool_inventory.py --category network
./tool_inventory.py --category database

Example files

  • uptime_incidents.example.json for uptime_sla_report.py
  • ldap_connect.env.example for ldap_connect_check.py
  • redis_health.env.example for redis_health_check.py
  • postgres_health.env.example for postgres_health_check.sh

Quick examples:

./uptime_sla_report.py --input uptime_incidents.example.json --target 99.9
source ldap_connect.env.example && ./ldap_connect_check.py --host "$LDAP_HOST" --port "$LDAP_PORT" --bind-dn "$LDAP_BIND_DN" --bind-password "$LDAP_BIND_PASSWORD" --base-dn "$LDAP_BASE_DN"

Security before publishing

  • Use environment variables or AWS profiles instead of hardcoded credentials
  • Keep private deployment details in private repos or private branches
  • Rotate any credentials that were ever committed

Naming standard

  • Filenames use snake_case for both .sh and .py scripts
  • Prefer verb-first names for action scripts (e.g., ec2_launch_instance.py)
  • Prefer noun-context names for checks/audits (e.g., dns_record_audit.sh)

Tool catalog

AWS / Cloud

  • check_instance_status.py
  • check_ses_status.py
  • ec2_launch_instance.py
  • ses_check_suppression.py
  • s3_list.py
  • s3_list_by_year.py
  • aws_billing_report.py
  • aws_billing_accounts.example.json
  • list_eks_ami_al2023.sh
  • ssm_connect.sh
  • ssm_tunnel.sh
  • kube_pod_triage.sh
  • uptime_sla_report.py

Network and DNS

  • check_domain.sh
  • dns_record_audit.sh
  • dns_resolver_chain_check.sh
  • dns_propagation_drift.sh
  • dns_ns_delegation_check.sh
  • dns_soa_serial_drift.sh
  • dnssec_validation_check.sh
  • dns_latency_heatmap.sh
  • anycast_resolver_fingerprint.sh
  • bgp_asn_context_lookup.sh
  • nat_egress_identity_drift.sh
  • dns_cache_behavior_probe.sh
  • dns_failover_switch_detector.sh
  • dns_authoritative_recursive_compare.sh
  • dns_split_horizon_detector.sh
  • dns_edns_capability_check.sh
  • dns_nxdomain_consistency_audit.sh
  • network_port_probe.sh
  • tcp_port_sweep_summary.sh
  • ssl_cert_expiry_check.sh
  • tcp_connection_watch.sh
  • tcp_connect_latency.sh
  • traceroute_path_summary.sh
  • snmp_device_check.sh
  • firewall_rules_audit.sh
  • path_mtu_probe.sh
  • interface_error_watch.sh
  • route_policy_audit.sh
  • packet_loss_jitter_probe.sh
  • http_timing_breakdown.sh
  • interface_bandwidth_sample.sh
  • interface_route_flap_watch.sh
  • packet_capture_helper.sh
  • tls_chain_analyzer.sh
  • dhcp_network_metadata.sh
  • neighbor_arp_ndp_audit.sh
  • multicast_diagnostics.sh
  • wireless_link_audit.sh
  • qos_dscp_probe.sh
  • pmtu_blackhole_detect.sh
  • ecn_capability_probe.sh
  • syn_backlog_pressure_check.sh
  • tcp_retransmission_hotspots.sh
  • wireguard_peer_audit.sh
  • wireguard_peer_add.sh
  • vpn_route_diagnose.sh
  • ping_server.sh
  • ip_address.sh
  • ipv6_default_route_audit.sh
  • ipv6_dns_aaaa_drift.sh
  • ipv6_reachability_matrix.sh
  • ipv6_path_drift_watch.sh
  • ipv6_ra_slaac_audit.sh
  • ipv6_reverse_ptr_check.sh
  • ipv6_listening_exposure_audit.sh
  • ipv6_firewall_policy_audit.sh
  • ipv6_firewall_risky_rules.sh
  • ipv6_ra_dhcpv6_consistency.sh
  • dual_stack_dns_preference_check.sh
  • dual_stack_connect_latency_compare.sh
  • dual_stack_happy_eyeballs_latency.sh
  • dns64_nat64_detection.sh
  • ipv6_neighbor_cache_pressure.sh
  • dual_stack_failover_probe.sh
  • resolver_v4_v6_consistency.sh
  • ipv6_temporary_address_audit.sh
  • anycast_path_divergence_watch.sh
  • ipv6_fragmentation_risk_audit.sh
  • dns_udp_doh_parity_check.sh
  • speed_test.py
  • what_is_my_ip.py
  • cisco_show_capture.sh
  • ldap_connect_check.py
  • ssh_hardening_audit.sh
  • auth_log_bruteforce_scan.sh

Database (MySQL/RDS)

  • rds_tools.sh
  • rds_profile.env.example
  • mysql_dump_rotate.sh
  • redis_health_check.py
  • postgres_health_check.sh

Storage / Transfer / Backup

  • backup_archive_retention.sh
  • backup_filesystem_prompt.sh
  • backup_filesystem_prompt_legacy_compression.sh
  • disk_usage_hotspots.sh
  • log_retention_cleanup.sh
  • cron_job_watchdog.sh
  • certbot_renewal_audit.sh
  • local_cert_expiry_scan.sh
  • ftp_mirror_lftp.sh
  • smb_mount_cifs.sh
  • file_integrity_baseline.py

System administration

  • admin_tools_menu.sh
  • linux_system_audit.sh
  • systemd_service_health.sh
  • http_health_check.sh
  • install_or_update_docker.sh
  • deploy_container_from_registry.sh
  • max_pods_calculator.sh
  • search.sh
  • password_generator.py
  • fail2ban_jail_audit.sh
  • fail2ban_unban_ip.sh
  • nginx_config_lint.sh
  • apache_vhost_audit.sh
  • sudoers_policy_audit.sh
  • os_patch_audit.sh
  • account_security_audit.sh
  • world_writable_scan.sh
  • security_baseline_run.sh

New additions in this cleanup pass

  • cisco_show_capture.sh: read-only Cisco show command capture via SSH
  • dns_record_audit.sh: DNS record audit for domain lists
  • network_port_probe.sh: quick TCP port status checker
  • mysql_dump_rotate.sh: compressed MySQL dumps with retention
  • ftp_mirror_lftp.sh: FTP/FTPS mirror helper with lftp
  • smb_mount_cifs.sh: SMB/CIFS mount helper using credential files
  • linux_system_audit.sh: host audit report generator
  • rds_tools.sh: rewritten as secure profile-based MySQL/RDS helper
  • aws_billing_report.py: rewritten to use role ARNs from JSON input (no hardcoded accounts/keys)

New additions in this expansion pass

  • ssl_cert_expiry_check.sh: checks cert expiry across multiple hosts
  • disk_usage_hotspots.sh: identifies largest directories/files quickly
  • systemd_service_health.sh: reports failed/inactive/restarting services
  • log_retention_cleanup.sh: age-based log cleanup with dry-run mode
  • tcp_connection_watch.sh: summarizes TCP states and top remote endpoints

New additions in this advanced pass

  • firewall_rules_audit.sh: audits ufw/nft/iptables rules plus listening ports
  • cron_job_watchdog.sh: validates cron command paths and warns on missing executables
  • snmp_device_check.sh: basic SNMP reachability and device metadata check
  • redis_health_check.py: Redis ping/info health with thresholds
  • postgres_health_check.sh: PostgreSQL connectivity and core health metrics
  • kube_pod_triage.sh: finds problematic pods/restarts and recent warning events

New additions in this network diagnostics pass

  • tcp_connect_latency.sh: repeated TCP connect latency checks with success/failure stats
  • path_mtu_probe.sh: binary-search MTU probing using DF ping payload tests
  • traceroute_path_summary.sh: traceroute/tracepath collection with compact hop summary
  • interface_error_watch.sh: interface error/drop counter sampling with delta alerts
  • dns_resolver_chain_check.sh: compares resolver answers/latency and flags mismatches
  • route_policy_audit.sh: audits ip rules/routes/default paths and route lookups
  • dns_propagation_drift.sh: detects DNS answer drift across public/internal resolvers
  • packet_loss_jitter_probe.sh: ping-based loss and jitter threshold probe per target
  • http_timing_breakdown.sh: breaks down DNS/connect/TLS/TTFB/total HTTP timings
  • tcp_port_sweep_summary.sh: scans port list/range and summarizes open/closed results
  • dns_ns_delegation_check.sh: compares recursive NS answers with trace delegation sets
  • interface_bandwidth_sample.sh: samples RX/TX throughput from interface byte counters
  • dns_soa_serial_drift.sh: compares SOA serials across resolvers to detect propagation lag
  • interface_route_flap_watch.sh: watches route-path changes over repeated samples
  • dnssec_validation_check.sh: checks AD/RRSIG/DNSKEY/DS DNSSEC signals per domain
  • packet_capture_helper.sh: bounded tcpdump helper with duration/packet limits and dry-run mode
  • dns_latency_heatmap.sh: resolver/domain latency table with min/avg/max sample stats
  • tls_chain_analyzer.sh: analyzes TLS chain, verify code, protocol/cipher, and expiry windows
  • anycast_resolver_fingerprint.sh: fingerprints resolver PoPs via CHAOS IDs and latency stats
  • dhcp_network_metadata.sh: collects interface/DHCP lease/default-route/resolver metadata
  • bgp_asn_context_lookup.sh: maps IPv4 targets to origin ASN/prefix/org context via DNS whois
  • neighbor_arp_ndp_audit.sh: audits neighbor tables for failed/incomplete entries and duplicate MAC patterns
  • nat_egress_identity_drift.sh: tracks public egress IP consistency across providers and rounds
  • multicast_diagnostics.sh: summarizes multicast addresses/routes/socket state with optional receive probe
  • dns_cache_behavior_probe.sh: repeats DNS queries to infer cache acceleration behavior
  • wireless_link_audit.sh: audits Wi‑Fi link state, signal quality, and driver metadata
  • dns_failover_switch_detector.sh: tracks resolver-order/set switches across sampling rounds
  • qos_dscp_probe.sh: compares loss/latency behavior across DSCP/TOS probe classes
  • dns_authoritative_recursive_compare.sh: compares recursive answers against authoritative NS union
  • pmtu_blackhole_detect.sh: detects likely PMTU blackhole patterns using DF payload bands
  • dns_split_horizon_detector.sh: compares internal vs external resolver answer unions
  • ecn_capability_probe.sh: tests endpoint behavior under non-ECN and ECN TOS classes
  • dns_edns_capability_check.sh: checks resolver EDNS/DO behavior across UDP buffer sizes
  • syn_backlog_pressure_check.sh: detects listening socket queue pressure and listen overflow counters
  • dns_nxdomain_consistency_audit.sh: validates NXDOMAIN consistency across resolver set
  • tcp_retransmission_hotspots.sh: surfaces retransmission pressure from kernel/socket indicators

New additions in this IPv6 diagnostics pass

  • ipv6_default_route_audit.sh: audits IPv6 global addressing, default route presence, and route lookups
  • ipv6_dns_aaaa_drift.sh: compares AAAA answers across IPv6 resolvers to detect drift/mismatch
  • ipv6_reachability_matrix.sh: runs IPv6 ICMP reachability checks with loss/latency summary table

New additions in this IPv6 diagnostics pass 2

  • ipv6_path_drift_watch.sh: tracks IPv6 route-path drift across repeated trace samples
  • ipv6_ra_slaac_audit.sh: audits RA/SLAAC sysctl posture and interface IPv6 address state
  • ipv6_reverse_ptr_check.sh: validates IPv6 PTR records with optional forward AAAA confirmation

New additions in this IPv6 security pass

  • ipv6_listening_exposure_audit.sh: flags wildcard IPv6 listeners outside an allowed-port baseline
  • ipv6_firewall_policy_audit.sh: audits IPv6 default firewall policy across nftables, ip6tables, and ufw
  • ipv6_firewall_risky_rules.sh: scans IPv6 firewall rules for broad allow/risky ACL patterns

New additions in this IPv6/dual-stack operations pass

  • ipv6_ra_dhcpv6_consistency.sh: checks RA/SLAAC posture against DHCPv6 client/lease signals
  • dual_stack_dns_preference_check.sh: compares A/AAAA availability and resolver ordering behavior
  • dual_stack_connect_latency_compare.sh: compares HTTP connect/TTFB/total timings between IPv4 and IPv6

New additions in this advanced dual-stack pass

  • dual_stack_happy_eyeballs_latency.sh: approximates Happy Eyeballs behavior by comparing per-round IPv4/IPv6 connect timing and success
  • dns64_nat64_detection.sh: detects likely DNS64 synthesis patterns (including 64:ff9b::/96 hits)
  • ipv6_neighbor_cache_pressure.sh: monitors degraded IPv6 neighbor-cache state ratio (FAILED/INCOMPLETE)

New additions in this dual-stack resilience pass

  • dual_stack_failover_probe.sh: tests real fallback behavior (IPv6 primary, IPv4 backup)
  • resolver_v4_v6_consistency.sh: compares answer consistency between IPv4 and IPv6 resolver endpoints
  • ipv6_temporary_address_audit.sh: audits privacy temporary IPv6 addressing posture per interface

New additions in this advanced network parity pass

  • anycast_path_divergence_watch.sh: detects path divergence to anycast targets across repeated traces
  • ipv6_fragmentation_risk_audit.sh: evaluates MTU-based IPv6 fragmentation risk for practical payload thresholds
  • dns_udp_doh_parity_check.sh: compares classic UDP DNS answers with DoH JSON answers for parity

New additions in this security/network pass

  • wireguard_peer_audit.sh: checks peer handshake freshness and transfer counters
  • vpn_route_diagnose.sh: validates interface routes and optional target ping
  • fail2ban_jail_audit.sh: jail and ban count visibility
  • fail2ban_unban_ip.sh: quick targeted/global IP unban helper
  • certbot_renewal_audit.sh: certbot inventory plus optional dry-run renewal
  • local_cert_expiry_scan.sh: scans local cert files for upcoming expiry

New additions in this enterprise pass

  • wireguard_peer_add.sh: safely appends a peer config and optional live apply
  • nginx_config_lint.sh: nginx syntax check plus included config inventory
  • apache_vhost_audit.sh: Apache syntax/vhost audit and enabled-site listing
  • ldap_connect_check.py: LDAP TCP, bind, and optional base DN search validation
  • uptime_sla_report.py: computes SLA uptime from incident windows in JSON

New additions in this security hardening pass

  • ssh_hardening_audit.sh: checks core sshd hardening settings
  • auth_log_bruteforce_scan.sh: surfaces repeated failed SSH login sources
  • sudoers_policy_audit.sh: finds broad sudo privilege and NOPASSWD patterns
  • file_integrity_baseline.py: baseline and verify file hashes for tamper detection
  • os_patch_audit.sh: patch/security update visibility across apt/dnf/yum
  • account_security_audit.sh: audits UID/account posture and password-policy settings
  • world_writable_scan.sh: detects world-writable files and non-sticky writable dirs
  • security_baseline_run.sh: runs security audits in one command and stores timestamped reports

security_baseline_run.sh modes:

  • default (--non-strict): fails only when checks return FAIL
  • --strict: fails when checks are FAIL or SKIPPED
  • --skip-check <name>: skip specific checks (repeatable), useful for environment-specific CI
  • --list-checks: print valid check names for use with --skip-check
  • --only-check <name>: run only selected checks (repeatable)
  • --only-group <name>: run by check group (network, deploy, storage; repeatable)
  • --list-groups: print valid group names for use with --only-group
  • --dry-run: preview run/skip decisions and generate a summary without executing checks
  • --output-json: write summary.json for CI/dashboard parsing
  • --json-pretty: pretty formatted JSON (default)
  • --json-compact: compact single-line JSON output
  • --json-file <path>: custom JSON output file path (relative to report dir or absolute)
  • --json-stdout: print JSON summary directly to stdout (for CI pipes)
  • --quiet: suppress informational logs (useful with --json-stdout in CI)

CI-friendly example (clean JSON to stdout):

./security_baseline_run.sh \
	--dry-run \
	--non-strict \
	--no-sudo \
	--only-group network \
	--output-json \
	--json-compact \
	--json-stdout \
	--quiet

CI artifact example (write JSON to a known path):

./security_baseline_run.sh \
	--dry-run \
	--non-strict \
	--no-sudo \
	--only-group network \
	--output-json \
	--json-compact \
	--json-file artifacts/security/summary.json \
	--quiet

Removed for public safety

  • Company-specific deployment scripts and internal one-off files were removed from this public-ready set.
  • Redundant scripts were consolidated (launch_ec2*, duplicate SES suppression checker, and backup alias entrypoint).

Public/private split recommendation

For your current setup, start with one repo and structure by folders:

  • shell/ for bash scripts
  • python/ for Python scripts
  • private/ (or separate private repo) for company-specific deployment scripts

When either shell or Python grows into a large standalone toolkit, split into dedicated repos.

CI automation

The repository includes a GitHub Actions workflow at .github/workflows/ci.yml that is manual-only (workflow_dispatch) and performs:

  • shell syntax checks (bash -n for all .sh files)
  • Python compile checks (python3 -m compileall -q .)
  • security_baseline_run.sh dry-run with JSON artifact output
  • artifact upload for artifacts/security/summary.json

This keeps daily script-copy/use workflows noise-free while still allowing an on-demand quality check before publishing major updates.

Release checklist

Before publishing updates, run this quick checklist:

  • verify shell syntax: find . -type f -name "*.sh" -print0 | xargs -0 -n1 bash -n
  • verify Python syntax: python3 -m compileall -q .
  • verify baseline runner JSON path: ./security_baseline_run.sh --dry-run --non-strict --no-sudo --only-group network --output-json --json-compact --json-file ./artifacts/security/summary.json --quiet
  • run a simple secret-pattern scan:
grep -RInE '(AKIA[0-9A-Z]{16}|aws_secret_access_key|BEGIN (RSA|EC|OPENSSH) PRIVATE KEY|password\s*=\s*"[^"]+")' . \
	--exclude-dir=.git

Local quality gate

Use pre-commit for local checks before each commit:

python3 -m pip install pre-commit
pre-commit install
pre-commit run --all-files

See CONTRIBUTING.md for script standards and contribution workflow.