Update documentation.

Author: YeisonCardona

README.md

@@ -75,6 +75,20 @@ The framework offers a flexible configuration system, allowing users to customiz
 **Logging and Monitoring:**
 Comprehensive logging and monitoring capabilities are integrated into the framework, providing real-time insights into the network activity and performance metrics. This aids in troubleshooting and maintaining the health of the system.
 
+**File Streaming and Transfer:**
+ChaskiStreamer includes helpers to send files in chunks through the network using `push_file` and to accept incoming files when `allow_incoming_files` is enabled. This facilitates distributing large payloads without blocking the event loop.
+
+**Persistent Storage:**
+Nodes can store key/value pairs using an SQLite-backed `PersistentStorage`. Data can be requested or served to peers with the `ChaskiStorageRequest` message type.
+
+**Synchronous Interface:**
+`ChaskiStreamerSync` wraps the asynchronous streamer in a dedicated thread, offering a blocking API that integrates with traditional synchronous code bases.
+
+**Celery Integration:**
+The package ships with a custom Kombu transport (`chaski.utils.transport`) so Chaski can act as a Celery broker or backend.
+
+**Message Pool with TTL:**
+Each node keeps a bounded pool of recently processed messages. This avoids processing duplicates and provides a configurable time-to-live for cached entries.
 
 ## Chaski-Confluent components
 
@@ -83,21 +97,29 @@ Comprehensive logging and monitoring capabilities are integrated into the framew
 The Chaski_ Node is an essential component of the Chaski-Confluent system. It is responsible for initiating and managing
 network communication between distributed nodes. This class handles functions such as connection establishment,
 message passing, node discovery, and pairing based on shared subscriptions.
+Nodes keep track of their connections as "edges" where latency information and
+subscription data are stored. Each node can propagate received messages to its
+peers and caches recent messages in a bounded pool to avoid processing
 
 ### Chaski Streamer
 
 The Chaski-Streamer extends the functionality of Chaski-Node by introducing asynchronous message streaming capabilities.
 It sets up an internal message queue to manage incoming messages, allowing efficient and scalable message processing within a distributed environment.
 The ChaskiStreamer can enter an asynchronous context, enabling the user to stream messages using the `async with` statement.
-This allows for handling messages dynamically as they arrive, enhancing the responsiveness and flexibility of the system.
+This allows for handling messages dynamically as they arrive, enhancing the
+responsiveness and flexibility of the system. The streamer also supports
+chunked file transfer via `push_file` and can store temporary results in a
+`PersistentStorage` database. When synchronous behaviour is required,
+`ChaskiStreamerSync` exposes the same API from a background thread.
 
 ### Chaski Remote
 
 The Chaski-Remote class enhances the Chaski-Node functionality by enabling remote method invocation and interaction
 across distributed nodes. It equips nodes with the ability to communicate transparently, invoking methods and accessing
 attributes on remote objects as if they were local. This is achieved by utilizing the Proxy class, which wraps around
 the remote objects and provides a clean interface for method calls and attribute access.
-
+The remote node verifies module availability through a lightweight UDP check
+before the proxy is returned, ensuring that requested services are reachable.
 
 ## Asynchronous Communication Architecture
 
@@ -127,3 +149,12 @@ The CA in Chaski-Confluent can generate, sign, and distribute SSL certificates,
 security layer. This ensures that all communication between nodes is encrypted and authenticated,
 significantly reducing the risk of data breaches or unauthorized access.
 
+
+## Celery Transport and CLI Utilities
+
+Chaski includes a custom Kombu transport so it can be used as a Celery broker.
+The `ChaskiChannel` class relies on `ChaskiStreamerSync` to publish and consume
+tasks through the topic ``celery_tasks``.  Several command line scripts under
+`chaski/scripts/` make it easy to start a streamer root, a remote proxy or a
+certificate authority from the shell.
+

docs/source/index.rst

@@ -10,8 +10,8 @@ Documentation Overview
    :maxdepth: 2
    :name: mastertoc
 
-   notebooks/01-chaski_node.pct
-   notebooks/02-chaski_streamer.ptc
+   notebooks/01-chaski_node
+   notebooks/02-chaski_streamer
    notebooks/03-chaski_remote.ptc
    notebooks/04-certification_authority
    notebooks/05-celery

docs/source/notebooks/readme.ipynb

@@ -93,7 +93,22 @@
     "The framework offers a flexible configuration system, allowing users to customize various parameters such as timeouts, retry intervals, and buffer sizes. This adaptability helps in optimizing the performance according to specific requirements.\n",
     "\n",
     "**Logging and Monitoring:**\n",
-    "Comprehensive logging and monitoring capabilities are integrated into the framework, providing real-time insights into the network activity and performance metrics. This aids in troubleshooting and maintaining the health of the system.\n"
+    "Comprehensive logging and monitoring capabilities are integrated into the framework, providing real-time insights into the network activity and performance metrics. This aids in troubleshooting and maintaining the health of the system.\n",
+    "\n",
+    "**File Streaming and Transfer:**\n",
+    "ChaskiStreamer includes helpers to send files in chunks through the network using `push_file` and to accept incoming files when `allow_incoming_files` is enabled. This facilitates distributing large payloads without blocking the event loop.\n",
+    "\n",
+    "**Persistent Storage:**\n",
+    "Nodes can store key/value pairs using an SQLite-backed `PersistentStorage`. Data can be requested or served to peers with the `ChaskiStorageRequest` message type.\n",
+    "\n",
+    "**Synchronous Interface:**\n",
+    "`ChaskiStreamerSync` wraps the asynchronous streamer in a dedicated thread, offering a blocking API that integrates with traditional synchronous code bases.\n",
+    "\n",
+    "**Celery Integration:**\n",
+    "The package ships with a custom Kombu transport (`chaski.utils.transport`) so Chaski can act as a Celery broker or backend.\n",
+    "\n",
+    "**Message Pool with TTL:**\n",
+    "Each node keeps a bounded pool of recently processed messages. This avoids processing duplicates and provides a configurable time-to-live for cached entries."
    ]
   },
   {
@@ -108,20 +123,29 @@
     "The Chaski_ Node is an essential component of the Chaski-Confluent system. It is responsible for initiating and managing\n",
     "network communication between distributed nodes. This class handles functions such as connection establishment,\n",
     "message passing, node discovery, and pairing based on shared subscriptions.\n",
+    "Nodes keep track of their connections as \"edges\" where latency information and\n",
+    "subscription data are stored. Each node can propagate received messages to its\n",
+    "peers and caches recent messages in a bounded pool to avoid processing\n",
     "\n",
     "### Chaski Streamer\n",
     "\n",
     "The Chaski-Streamer extends the functionality of Chaski-Node by introducing asynchronous message streaming capabilities.\n",
     "It sets up an internal message queue to manage incoming messages, allowing efficient and scalable message processing within a distributed environment.\n",
     "The ChaskiStreamer can enter an asynchronous context, enabling the user to stream messages using the `async with` statement.\n",
-    "This allows for handling messages dynamically as they arrive, enhancing the responsiveness and flexibility of the system.\n",
+    "This allows for handling messages dynamically as they arrive, enhancing the\n",
+    "responsiveness and flexibility of the system. The streamer also supports\n",
+    "chunked file transfer via `push_file` and can store temporary results in a\n",
+    "`PersistentStorage` database. When synchronous behaviour is required,\n",
+    "`ChaskiStreamerSync` exposes the same API from a background thread.\n",
     "\n",
     "### Chaski Remote\n",
     "\n",
     "The Chaski-Remote class enhances the Chaski-Node functionality by enabling remote method invocation and interaction\n",
     "across distributed nodes. It equips nodes with the ability to communicate transparently, invoking methods and accessing\n",
     "attributes on remote objects as if they were local. This is achieved by utilizing the Proxy class, which wraps around\n",
-    "the remote objects and provides a clean interface for method calls and attribute access.\n"
+    "the remote objects and provides a clean interface for method calls and attribute access.\n",
+    "The remote node verifies module availability through a lightweight UDP check\n",
+    "before the proxy is returned, ensuring that requested services are reachable."
    ]
   },
   {
@@ -163,6 +187,20 @@
     "security layer. This ensures that all communication between nodes is encrypted and authenticated,\n",
     "significantly reducing the risk of data breaches or unauthorized access.\n"
    ]
+  },
+  {
+   "metadata": {},
+   "cell_type": "markdown",
+   "source": [
+    "## Celery Transport and CLI Utilities\n",
+    "\n",
+    "Chaski includes a custom Kombu transport so it can be used as a Celery broker.\n",
+    "The `ChaskiChannel` class relies on `ChaskiStreamerSync` to publish and consume\n",
+    "tasks through the topic ``celery_tasks``.  Several command line scripts under\n",
+    "`chaski/scripts/` make it easy to start a streamer root, a remote proxy or a\n",
+    "certificate authority from the shell.\n"
+   ],
+   "id": "f866782a2808b6e8"
   }
  ],
  "metadata": {

docs/source/notebooks/readme.rst

@@ -114,6 +114,27 @@ capabilities are integrated into the framework, providing real-time
 insights into the network activity and performance metrics. This aids in
 troubleshooting and maintaining the health of the system.
 
+**File Streaming and Transfer:** ChaskiStreamer includes helpers to send
+files in chunks through the network using ``push_file`` and to accept
+incoming files when ``allow_incoming_files`` is enabled. This
+facilitates distributing large payloads without blocking the event loop.
+
+**Persistent Storage:** Nodes can store key/value pairs using an
+SQLite-backed ``PersistentStorage``. Data can be requested or served to
+peers with the ``ChaskiStorageRequest`` message type.
+
+**Synchronous Interface:** ``ChaskiStreamerSync`` wraps the asynchronous
+streamer in a dedicated thread, offering a blocking API that integrates
+with traditional synchronous code bases.
+
+**Celery Integration:** The package ships with a custom Kombu transport
+(``chaski.utils.transport``) so Chaski can act as a Celery broker or
+backend.
+
+**Message Pool with TTL:** Each node keeps a bounded pool of recently
+processed messages. This avoids processing duplicates and provides a
+configurable time-to-live for cached entries.
+
 Chaski-Confluent components
 ---------------------------
 
@@ -124,7 +145,10 @@ The Chaski\_ Node is an essential component of the Chaski-Confluent
 system. It is responsible for initiating and managing network
 communication between distributed nodes. This class handles functions
 such as connection establishment, message passing, node discovery, and
-pairing based on shared subscriptions.
+pairing based on shared subscriptions. Nodes keep track of their
+connections as “edges” where latency information and subscription data
+are stored. Each node can propagate received messages to its peers and
+caches recent messages in a bounded pool to avoid processing
 
 Chaski Streamer
 ~~~~~~~~~~~~~~~
@@ -136,7 +160,11 @@ and scalable message processing within a distributed environment. The
 ChaskiStreamer can enter an asynchronous context, enabling the user to
 stream messages using the ``async with`` statement. This allows for
 handling messages dynamically as they arrive, enhancing the
-responsiveness and flexibility of the system.
+responsiveness and flexibility of the system. The streamer also supports
+chunked file transfer via ``push_file`` and can store temporary results
+in a ``PersistentStorage`` database. When synchronous behaviour is
+required, ``ChaskiStreamerSync`` exposes the same API from a background
+thread.
 
 Chaski Remote
 ~~~~~~~~~~~~~
@@ -147,7 +175,9 @@ nodes. It equips nodes with the ability to communicate transparently,
 invoking methods and accessing attributes on remote objects as if they
 were local. This is achieved by utilizing the Proxy class, which wraps
 around the remote objects and provides a clean interface for method
-calls and attribute access.
+calls and attribute access. The remote node verifies module availability
+through a lightweight UDP check before the proxy is returned, ensuring
+that requested services are reachable.
 
 Asynchronous Communication Architecture
 ---------------------------------------
@@ -183,3 +213,12 @@ The CA in Chaski-Confluent can generate, sign, and distribute SSL
 certificates, providing a robust security layer. This ensures that all
 communication between nodes is encrypted and authenticated,
 significantly reducing the risk of data breaches or unauthorized access.
+
+Celery Transport and CLI Utilities
+----------------------------------
+
+Chaski includes a custom Kombu transport so it can be used as a Celery
+broker. The ``ChaskiChannel`` class relies on ``ChaskiStreamerSync`` to
+publish and consume tasks through the topic ``celery_tasks``. Several
+command line scripts under ``chaski/scripts/`` make it easy to start a
+streamer root, a remote proxy or a certificate authority from the shell.