Edge Computing & Internet Of Things 🛜

A qualitative compilation of production-ready edge computing projects with a focus on Edge-2-Cloud Data Engineering

Table of Contents

• Major IoT Clouds
• Major Edge OSS Projects
• Edge Messaging
  • Brokerless
  • Websockets
  • MQTT
  • Industrial
• Message Serialization
• Messaging & Data Streaming
  • Cloud Managed
  • On-Prem
• Stream Processing
  • Cloud Managed
  • On-Prem
• Databases
  • Cloud Managed
  • Time Series Database
  • Document Database (NoSQL)
  • Persistent Key Value DB
  • Relational Database
  • In Memory Database
• Batch Processing / Analytics
• Industrial IoT with Kubernetes
  • Kubernetes Distributions for Lightweight Edge
  • Kubernetes Networking
  • Multi-Cluster Networking
  • Service Mesh
  • Distributed Orchestration
  • Microservice Frameworks
  • Distributed Filesystem
  • FaaS

---

Projects in the curated list are filtered based on the following conditions:

• High Quality
• Production Ready
• Well Documented
• Well Maintained
• Cloud Ready
• Scalable

---

IoT on Hyperscalers

• AWS IoT - Unlock your IoT data and accelerate business growth
• Azure IoT Operations - Build interoperable IoT solutions that transform physical operations at scale.

Major Edge OSS Projects

• Eclipse Ditto - the open-source project of Eclipse IoT that provides a ready-to-use functionality to manage the state of Digital Twins
• Eclipse Hono - provides uniform (remote) service interfaces for connecting large numbers of IoT devices to a (cloud) back end.
• Akri - Akri is a Kubernetes Resource Interface that lets you easily expose heterogeneous leaf devices
• Akraino - LF Edge is an umbrella organization that establishes an open, interoperable framework for edge computing
• Shifu - Shifu provides users with a transparent framework for full-scene device hosting and integrated software development

Edge Messaging

MQTT

• HiveMQ - Enterprise MQTT Platform
• Nats.io - Connective Technology for Adaptive Edge & Distributed Systems
• Mosquitto - An Open Source MQTT v3.1/v3.1.1 Broker.
• VerneMQ - A distributed MQTT message broker.
• Waterstream - MQTT broker leveraging Apache Kafka as its own storage and distribution engine.
• NanoMQ - A light-weight and Blazing-fast MQTT Broker for IoT Edge platform.
• Narwhal - An extensible pub/sub messaging server for edge applications

Industrial

• Node CoAP - A client and server library for CoAP modeled after the http module.
• Rust CoAP - A Constrained Application Protocol (CoAP) library implemented in Rust.
• OPC-UA - Interoperability standard for the secure and reliable exchange of data
• ModBus - A data communications protocol for use with Modicon PLC's

Brokerless

• ZeroMQ Overview - An open-source universal messaging library
• NanoMSG - a simple high-performance implementation of several "scalability protocols"
• NNG - nanomsg-next-generation -- light-weight brokerless messaging

Websockets

• uws - Tiny WebSockets (access to the C++ library, µWebSockets, via Node.js)
• uWebsockets - is a lightweight and efficient MQTT broker designed to raise the bar for pub/sub performance.
• Tungstenite WS - Lightweight stream-based WebSocket implementation for Rust.
• Python Websockets - A library for building WebSocket servers and clients in Python
• Autobahn.ws - Open-source real-time framework for Web, Mobile & Internet of Things.
• socket.io - Bidirectional and low-latency communication for every platform

Message Serialization

• Apache Avro - Apache Avro™ is a data serialization system
• Apache Thrift - scalable cross-language services development
• CapNProto - Apache Avro™ is a data serialization system.
• JSON - is a lightweight data-interchange format.
• MessagePack - MessagePack is an efficient binary serialization format.
• CBOR - RFC 8949 Concise Binary Object Representation
• FlatBuffers - an efficient cross platform serialization library
• Protocol Buffer - a language-neutral, platform-neutral extensible mechanism for serializing structured data

Messaging and Data Streaming

Cloud Managed

• Azure Event Hub - a fully managed, real-time data ingestion service that’s simple, trusted, and scalable
• Azure Managed Kafka - Azure HDInsight, a customizable, enterprise-grade service for open-source analytics.
• Azure Queue Storage - a service for storing large numbers of messages
• Azure Event Grid - a highly scalable, serverless event broker that you can use to integrate applications using events.
• Confluent Cloud - Apache Kafka as a Service
• AWS MSK - Securely stream data with a fully managed, highly available Apache Kafka service
• AWS SQS - Fully managed message queuing for microservices, distributed systems, and serverless applications
• AWS Kinesis - Easily stream data at any scale

On-Prem

• Strimzi - Cloud Native Apache Kafka
• Apache Pulsar - a distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API.
• Apache Kafka - distributed publish-subscribe messaging system.
• Apache NiFi - Apache NiFi is an integrated data logistics platform for automating the movement of data between disparate systems.
• redpanda - A Kafka® replacement for mission critical systems; 10x faster. Written in C++.
• Fluentd - tool to collect events and logs.
• Kestrel - distributed message queue system.
• Apache ActiveMQ Artemis - The Next Generation Message Broker by ActiveMQ, AMQP 1.0 is a natively supported protocol.
• Apache Qpid Java Broker - A pure-Java AMQP message broker.
• Solace - Messaging Platform that supports AMQP 1.0
• RabbitMQ with AMQP 1.0 plugin.

Stream Processing

Cloud Managed

• AWS Kinesis Data Analytics - Gain actionable insights from streaming data with serverless, fully managed Apache Flink
• Azure Stream Analytics - Serverless real-time analytics, from the cloud to the edge
• KSQL DB - KSQL DB on Confluent Cloud

On-Prem

• Faust - stream processing and event processing in Python
• Siddhi - stream Processing and Complex Event Processing Engine
• KSQL - The database purpose-built for stream processing applications.
• Benthos - Declarative stream processing for mundane tasks and data engineering
• Go Streams - A lightweight stream processing library for Go
• Flogo - Elegant stream processing pipeline written entirely in Golang
• Apache Heron - A realtime, distributed, fault-tolerant stream processing engine
• Apache Beam - Implement batch and streaming data processing jobs that run on any execution engine.
• Apache Flink - Stateful Computations over Data Streams
• Apache Samza - A distributed stream processing framework

Batch Processing / Analytics

• Databricks - All your data, analytics and AI on one Lakehouse platform
• Apache Mahout - For Creating Scalable Performant Machine Learning Applications
• H2O - The #1 open-source machine learning platform for the enterprise
• Apache Drill - Schema-free SQL Query Engine for Hadoop, NoSQL and Cloud Storage.
• Apache Spark - Apache Spark™ is a unified analytics engine for large-scale data processing.

Databases

Cloud Managed

• Azure SQL - Build apps that scale with managed and intelligent SQL in the cloud
• Azure CosmosDB - Fast NoSQL / Graph / Relational database with open APIs for planet scale
• Azure PostgreSQL - Fully managed, intelligent, and scalable PostgreSQL
• Azure Data Explorer - Fast and highly scalable data exploration service.
• AWS Aurora - AWS Relational DB
• AWS DynamoDB - fast, flexible, and serverless NoSQL database for any scale
• AWS DocumentDB - Scale JSON workloads with ease using a MongoDB-compatible document database
• AWS Keyspaces - Run your Apache Cassandra workloads on a scalable, highly available, and managed wide column database service.
• AWS Neptune - Build applications that work with highly connected datasets using a fast, reliable graph database service.
• AWS Ledger - Provide transparent, immutable, cryptographically verifiable transaction logs with a fully managed ledger database service.
• AWS Timestream - Provide transparent, immutable, cryptographically verifiable transaction logs with a fully managed ledger database service.

On-Prem

Time Series Database

• InfluxDB - Scalable datastore for metrics, events, and real-time analytics
• TimescaleDB - An open-source time-series database optimized for fast ingest and complex queries
• VictoriaMetrics - fast, cost-effective monitoring solution and time series database
• Cortex - A horizontally scalable, highly available, multi-tenant, long term Prometheus.

Document Database (NoSQL)

• Crate Data - is an open source massively scalable data store. It requires zero administration.
• MongoDB - Document-oriented database system.
• RavenDB - A transactional, open-source Document Database.
• RethinkDB - document database that supports queries like table joins and group by.
• Couchbase - The Modern Database for Enterprise Applications
• Apache Cassandra - Manage massive amounts of data, fast, without losing sleep

Persistent KV DB

• TiKV - a highly scalable, low latency, and easy to use key-value database.
• BadgerDB - an embeddable, persistent and fast key-value (KV) database written in pure Go
• FoundationDB - scalable distributed database
• RocksDB - embeddable persistent key-value store for fast storage based on LevelDB.

Relational Database

• Citus - Distributed PostgreSQL
• Cockroach - Scalable, Geo-Replicated, Transactional Datastore.
• TiDB - TiDB is a distributed SQL database
• yugabyteDB - open source, high-performance, distributed SQL database compatible with PostgreSQL
• MariaDB - enhanced, drop-in replacement for MySQL
• Apache Calcite - the foundation for your next high-performance database
• Vitess - A database clustering system for horizontal scaling of MySQL

In Memory Database

• VoltDB - claims to be fastest in-memory database.
• BerkeleyDB - a software library that provides a high-performance embedded database for key/value data.
• LevelDB - a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.
• Memcached - high-performance, distributed memory object caching system

Industrial IoT with Kubernetes

Kubernetes Hybrid

• Azure Arc - Azure Arc simplifies governance and management by delivering a consistent multicloud and on-premises management platform.
• EKS Anywhere - Create and operate Kubernetes clusters on your own infrastructure

Kubernetes Distributions for Edge

• AKS Edge - AKS Edge Essentials is an on-premises Kubernetes implementation running containers at scale
• K3s - The certified Kubernetes distribution built for IoT & Edge computing
• microk8s - The best Kubernetes experience for developers, DevOps, cloud and edge

Kubernetes Networking

• Cilium - eBPF-based Networking, Observability, and Security
• Calico - most widely adopted solution for Kubernetes networking and security
• Flannel - Flannel is a simple and easy way to configure a layer 3 network fabric
• Antrea - Kubernetes networking based on Open vSwitch
• KubeEdge - An open platform to enable Edge computing
• Kube Router - networking solution built for operational simplicity and performance.

Multi-Cluster Networking

• Liqo - P2P, Seamless, Dynamic and Secure Resource Sharing
• Admiralty - The simplest way to deploy applications to multiple Kubernetes clusters.
• Submariner - direct networking between Pods and Services in different Kubernetes clusters
• Skupper - Skupper Multicloud communication for Kubernetes

Service Mesh

• Istio - the leading multi-cluster service mesh
• Linkerd - multi-cluster services with security, observability, and reliability
• Consul - Service Mesh for any runtime or cloud
• Traefik Mesh - Traefik Mesh is a straight-forward, easy to configure, and non-invasive service mesh
• Kuma - The universal Envoy service mesh for distributed service connectivity
• Istio - the leading service mesh

Distributed Orchestration

• Serf - decentralized solution for service discovery and orchestration
• etcd - a distributed, reliable key-value store for the most critical data of a distributed system

Microservice Frameworks

• Lightbend Akka - runtime for distributed, and fault tolerant event-driven applications on the JVM.
• Lightbend Lagom - opinionated microservices framework
• GoKit - A toolkit for microservices
• Goa - Goa provides a holistic approach for developing remote APIs and microservices in Go.
• Micro - Micro is a cloud platform for API development
• Dapr - Dapr is a portable, event-driven, runtime for building distributed applications across cloud and edge.

Distributed Filesystem

• Rook - Open-Source, Cloud-Native Storage for Kubernetes
• Minio - Object Storage for the Era of the Hybrid Cloud
• Longhorn - Cloud native distributed block storage for Kubernetes
• OpenEBS - OpenEBS builds on Kubernetes to enable Stateful applications to easily access Persistent Volumes

FaaS

• Serverless - zero-friction serverless development
• Apache Camel - easily integrate various systems consuming or producing data.
• Apache Openwhisk - Open Source Serverless Cloud Platform
• Fission - Open source, Kubernetes-native Serverless Framework
• KNative - Serverless Kubernetes functions
• Kubeless - The Kubernetes Native Serverless Framework
• Kyma - Extend your applications with Kubernetes
• Nuclio - High-Performance Serverless event and data processing platform