Contact Details
davidgs@redhat.com
Proposed Title
Building a desktop llm-d cluster with NVIDIA Jetson Orin Nano Dev Kits
Short Synopsis
I have 3 NVIDIA Jetson Orin Nano dev kits that I am combining with a gig-e switch and a Raspberry pi as the gateway. This will be a fully stand-alone llm-d cluster on which I can run LLMs
It won't be the most powerful cluster, but it will be a proof of concept for a desktop llm-d cluster.
I will also run some benchmark tests to see just how performance 9or not performant) this cluster is and how practical it might be for either testing or deployment.
Why This Post?
With the release of the NVIDIA Jetson Dev kits, and the growing importance of edge-based AI deployments, this could be an important step in showing how llm-d can enable greater flexibility in deploying AI solutions at the edge.
Target Audience
No response
Who Will Write It?
I plan to write this post myself
Author Details
I have worked in IoT/Edge for over 20 years and, as I have transitioned to AI (and llm-d) the importance of edge-based AI solutions has also started to gain mind-share. This post could be important in putting llm-d at the forefront of edge-based AI deployments.
Related Links
No response
Additional Context
No response
Contact Details
davidgs@redhat.com
Proposed Title
Building a desktop llm-d cluster with NVIDIA Jetson Orin Nano Dev Kits
Short Synopsis
I have 3 NVIDIA Jetson Orin Nano dev kits that I am combining with a gig-e switch and a Raspberry pi as the gateway. This will be a fully stand-alone llm-d cluster on which I can run LLMs
It won't be the most powerful cluster, but it will be a proof of concept for a desktop llm-d cluster.
I will also run some benchmark tests to see just how performance 9or not performant) this cluster is and how practical it might be for either testing or deployment.
Why This Post?
With the release of the NVIDIA Jetson Dev kits, and the growing importance of edge-based AI deployments, this could be an important step in showing how llm-d can enable greater flexibility in deploying AI solutions at the edge.
Target Audience
No response
Who Will Write It?
I plan to write this post myself
Author Details
I have worked in IoT/Edge for over 20 years and, as I have transitioned to AI (and llm-d) the importance of edge-based AI solutions has also started to gain mind-share. This post could be important in putting llm-d at the forefront of edge-based AI deployments.
Related Links
No response
Additional Context
No response