Useful Things

Programming Infrastructure

• Python Programming
  • vulture: Find dead python code
  • pylama: Useful python code-audit tool
  • ty: A Python type checker in rust
  • uv: A rust replacement for pip
  • pyupgrade: Automatically upgrade python syntax to baseline against specific python versions
  • bandit: Find security problems in python code
  • complexipy: Rapidly find Python methods that are too complex
  • ruff: Lint python quickly
  • pss: Code-search tool written in python
• Python Debuggers / Profilers
  • pudb: Full-screen terminal Python debugger
  • pdoc: Documentation system for moderate-sized python projects
  • PySnooper: Debug python with decorators
  • snoop: Debug python with decorators; similar to PySnooper
  • pyinstrument: Call stack profiler for python libraries; detect why python is slow
  • py-spy: Sampling profiler for python code
• Python Compilers / Transpilers
  • codon: Compile Python into static binaries
  • PyO3: Write Python modules in rust, or embed Python in a rust binary
  • Cython: Accelerate Python with C
  • py2many: A python transpiler (i.e. convert Python to Rust, Go, C++, etc...)
  • py2exe: Compile Python to Windows binaries
  • pyinstaller: Compile to binary including required packages
  • Nuitka: Compile python code to a binary
  • edgepython: Compile a subset of python to a WASM binary to run "python" in browser / linux. Caution, no STDLIB included.
• YAML
  • yamllint: Simple yaml linter
  • yamlfmt: Google's yaml linter / formatter
  • yamlfix: Automatic yaml formatting fixer
• Markdown
  • rumdl: Markdown Linter
• VIM / IDE
  • ALE: Add IDE features to vim and neovim
  • Neovide: Enhanced neovim GUI with a focus on vim as an IDE
  • nvim-treesitter: Parser and auto-complettion for neovim
  • YouCompleteMe: Auto-completion for ALE
• Rust
  • rust-analyzer: Linter for rust projects
• Jujitsu: A trendy alternative to git for DCVS; developed at Google.

Languages / Toolkits

• OpenTofu: Open-source alternative to Terraform (which is now owned by IBM)
• Ansible
• Deno: Security-focused Rust JavaScript run-time; memory-safe alternative to Node.js

GitHub Actions

• act: Run GitHub actions locally on your laptop
• zizmor: Static YAML analysis for GitHub Actions YAML

Cloud Infrastructure

• checkov: Lint your Terraform, Dockerfiles and more
• ScoutSuite: Multi-cloud security auditing
• Terminal AWS: Manage your AWS infrastructure in the terminal

AI Toolkit

• rtk: CLI proxy that reduces LLM token consumption by 60-90% on common dev commands.

Python Language

Python: Code Infrastructure

• Libraries
  • click: Flexible CLI arguments library as an improvement over argparse
  • attrs: Python classes without boilerplate
  • traitlets: Python classes with strong typing and attribute observer operations.
  • loguru: Delgan's excellent Python logging framework
  • transitions: Object-oriented pythonic State Machine
  • dotenv: Manage / read .env files in Python
• Data Validation / Serialization
  • pydantic: Python data validation inside classes
  • marshmallow: Implement Python object serialization an de-serialization with type validation; this library is oriented around Python objects as dicts.
  • adaptix: Simplified Python data validation and conversion between sqlite, dict, dataclasses
• Python Web Development
  • flask: Simple python web-development framework
  • waitress: A production-grade WSGI web server (useful to serve flask apps in production)
  • dominate: Write HTML with python
  • scrapy: Web scraping framework
• typeguard: Python runtime type checker; needs a maintainer after v4.0.0
• gRPC: Fast, typed Python message passing library
• msgspec: Fast Python object serialization / deserialization with type checking; partially implemented in C. Data is supported in json, yaml, and MessagePack, and toml.
• msgpack: MessagePack for Python is an efficient binary serialization format. It lets you exchange data among multiple formats. See msgspec for more detail.
• trio: Python async and concurrency library

Python: General Usage

• pre-commit: Add hooks to git commit actions
• ultisnips
• cookiecutter: A disk and file templating framework
• pyautogui: Framework to exert control over GUI interfaces
• pywinauto: Framework to exert control over GUI interfaces
• jupyter-notebook: Dynamically analyze python code as a savable notebook
• GitPython: Manage git repositories with python
• rich: Library for terminal colorization
• ptftpd: Python TFTPd and PXE tool suite
• pyjwt: Python implementation of JWT (JSON Web Token)
• python-saml: Python SAML toolkit
• dnspython: Python interface to DNS queries
• splunk-sdk-python: Splunk SDK for Python
• hvac: Python interface to Hashicorp Vault
• pyshark: Python interface to tshark / Wireshark in the Terminal
• docker-py: Python API for the Docker ecosystem
• psutil: Cross platform process and system monitoring
• watchdog: Monitor file-system events in python
• invoke: Pythonic command execution and task management
• pysmb: Python SMB client
• pexpect: Pythonic Expect library
• pyparsing: Popular python parsing library via PEG parser
• TextFSM: Stateful templated python text parsing and value extraction
• ntc-templates: Templates to parse command output via TextFSM
• fuzzywuzzy: Fuzzy text matching
• rich: Command-line text formatting library
• python-prompt-toolkit: Build interactive Python command-line programs
• deepdiff: Diff python nested dicts
• dictdiffer: Diff python dicts
• PyGithub: GitHub v3 API python library
• atlassian-python-api: REST API for Atlassian products (i.e. Jira, Confluence)
• landslide: Markdown and Python-based slide deck library
• Database / SQL / ORM
  • peewee: Python ORM for sqllite, postgresql, mysql and cockroach DB
  • [SQLAlchemy][sqlalchemy]: The Database Toolkit for Python
  • records: Make SQL queries and ingest results as python objects
• JSON / YAML
  • orjson: Fast and correct json parsing library
  • pyyaml: Pythonic yaml handling
  • ruamel.yaml: Pythonic yaml handling
• Time Utilities (mostly useless after python stdlib introduced zoneinfo)
  • whenever: Time-handling replacement for Python datetime and some Arrow use-cases
  • Arrow: Flexible Python time-handling
• Cron / Durable Execution
  • absurd: Durable execution engine for reliable workflows / CICD; written by Armin Ronacher.
  • croniter: Parses cron schedules to iterate over datetime objects.
  • APScheduler: Python Task scheduling library
  • rocketry: Pythonic scheduling and cron-like recur using python decorators
• Graphing / Diagrams
  • plotly: Graphing and plotting library
  • matplotlib: Build graphs with Python
  • Seaborn: Simple statistical graphing package
  • diagrams: Diagrams as Python code
  • mermaid-py: Python interface to mermaid-js diagramming library
• Data Science / Computation
  • pandas: Pythonic data manipulation and statistical analysis library
  • polars: Python data manipulation and statistical analysis written in rust
  • prophet: Facebook's canned time-series forcasting library

Python: Network Automation

• fabric: Automate command execution on linux systems via SSH
• boto3: Automate AWS environments
• netmiko: Automate command execution on routers / switches via SSH
• nornir: Network inventory and execution abstractions
• exabgp: BGPd written in Python
• scrapli: Automate command execution on routers / switches via SSH
• suzieq: Canned network metrics collection
• stockpiler: Brett Lykins' config backup tool. Only supports Cisco and F5.
• meraki-cli: Meraki CLI tool for interacting with the dashboard
• py-junos-eznc: Junos automation library

Machine Learning

• keras
• scikit-learn
• langchain: Framework for building AI Agents
• textblob: Text processing and sentiment analysis
• MLAlgorithms: Clean Machine Learning examples

Rust Language

The best single-source of rust examples is Rust By Example.

General libraries

• rexpect: Rust expect library

CLI Argument parsing

• argh: Simplified Rust CLI arg parsing
• clap: Full-featured Rust CLI arg parsing; can be complex and often deprecates previously-used syntax

Error Handling

• snafu: Rust error handling
• anyhow: Rust Application error handling
• thiserror: Rust Application error handling

JavaScript / CSS / Web

• ECharts: Vue-based charting library
• Vue: Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

Go Language

• go-charts: Go charts library
• [plot][plot]: A repository for plotting and visualizing data
• delve: Go debugger; also see the delve youtube demo

Info

• what-happens-when: A detailed description of what happens when you type "google.com" into your browser and press Enter.

Tools

General Use utilities

• zellij: A rust terminal multiplexer
• difftastic: A text file differ that understands many programming languages. Diffs are intelligent based on the language in the file.
• taskwarrior: A task-list manager with a useful CLI
• just: just is like make, but without the Makefile syntax warts
• sd: Find and replace simplified
• ripgrep: Recursive grep through directories
• RustScan: A fast port-scanner in Rust (can replace nmap for some tasks)
• reveal.js: HTML and JS-based slide deck
• starship: Dynamic rust-powered shell prompts with meaningful info
• Win11Debloat: Powershell script to disable Microsoft trackers and other Win11 bloatware.
• warp: Windows / Linux / MacOS Shell with embedded AI agent capabilities.
• wasmtime: Run wasm binaries on linux
• gobackup: Server backup automation that uses anything from local storage to Amazon S3
• podman: Run Docker / OCI containers without root; replaces Docker for many workflows.
• stress-ng: Load-test CPU, Memory, Threading, Disk I/O
• rustdesk: An open-source remote desktop application designed for self-hosting, as an alternative to TeamViewer.
• System Monitoring
  • bottom: Flexible, Rust-based terminal monitoring tool. Favorite invocation - btm -bT
  • htop: Multi-core evolution of top
  • procs: List processes, rust replacement for ps
  • rustnet-monitor: Per-process network monitoring for your terminal: live TCP, UDP, and QUIC connections with deep packet inspection, sandboxed by default.
  • trippy: Traceroute / mtr written in Rust

Terminal utilites

• asciienema: A terminal session recorder built with Rust
• obs-studio: Record YouTube videos from your laptop

General Tools: Services

• VictoriaMetrics: Drop-in replacement for Prometheus. Highly scalable for large data sets
• perses: Forget Grafana, manage graph dashboards in git as code
• openobserve: Efficient log collection and management with up to 170x log storage compression ratios
• smallstep: CLI TLS Certificate Authority (CA)
• scylladb: Fast NoSQL DB

Network Tools: Services

• caddy: Go webserver / reverse proxy with auto-TLS certificate generation
• frp: A Golang fast reverse proxy to help you expose a local server behind a NAT or firewall to the internet.
• nsd: DNS server in C
• coredns: Golang DNS server that supports UDP, DoT, DoH, DoQUIC and more.
• unbound: Popular DNS resolver library
• httptap: Get a terminal waterfall diagram of website responses
• Netbox: Network source of truth; based on Django
• NIPAP: Open-source IP address management (IPAM)
• fail2ban: Monitor and black-hole repetitive attacks on your Linux server
• maltrail: Detect malicious network traffic
• secure_cartography: Use CDP / LLDP / SSH to compile accurate network diagrams
• influxdb: A time-series database
• FRrouting: The FRRouting Protocol Suite; open implementations of BGP, OSPF, EIGRP, PIM, etc... Use vtysh to manage the various routing daemons.

Network Tools: Diagnostics and Testing

• Ping / Traceroute
  • trippy: Traceroute / mtr written in Rust
  • mtr: Traceroute / mtr written in Rust
  • latency-monitor: One or two-way latency measurements
  • gping: Graph ping response times (written in Rust) in the terminal window
  • pingtracer: Progressive ping logger written for MS Windows; discovers all hops in the path and pings them
• Proxies
  • mitmproxy: An interactive TLS-capable intercepting HTTP proxy for penetration testers and software developers.
  • go-mitmproxy: A port of mitmproxy to Go
• Sniffers
  • sniffnet: Cross-platform network traffic capture with a nice GUI (Windows / MacOS / Linux)
• Monitoring / NMS
  • librenms: One stop shop for monitoring Cisco / Arista / etc...
  • Rust-Ping: Open source NMS written in Rust
  • network-monitor: Open source NMS written in Rust
  • nethogs: Detect which linux processes are consuming the most bandwidth
  • bandwhich: Detect which linux processes are consuming the most bandwidth (written in Rust)
  • sniffer: Detect which linux / Windows processes are consuming the most bandwidth (written in Go)
  • rrdtool: A time-series database, which intelligently compresses data
• Network Emulation
  • containerlab: Build arbitrary topologies of your favorite Cisco / Arista / Juniper / Nokia Operating System as a Docker container
  • vrnetlab: Tool to convert router images to a Docker container
• Network Testing
  • flent: A network performance test client, which requires [fping][fping] (client-side), netperf (server-side) or iperf2 (server-side)
  • iperf2: Classic network test tool
  • iperf3: Next-gen of iperf
  • crusader: Rust-based network testing and results graphing
  • goben: Measure TCP / UDP throughput between hosts
  • TRex: Cisco's flexible Network test tool
  • netperf: A network performance test server, written by Hewlett Packard. It's best to install the Debian package instead of trying to compile netperf

Embedded Operating Systems

• Zephyr: A popular RTOS for Embedded systems
• Embassy: A rust embedded operating system
• rtic: RTOS for ARM Cortex-M microcontrollers
• Tock: A secure RTOS for embedded system