Description
spark-training-gotchas is an agent skill from the wshobson/agents collection that preflights and diagnoses ten known failure modes for machine learning training on NVIDIA DGX Spark. The DGX Spark uses a GB10 Grace Blackwell chip with an SM121 GPU, 128 GB of unified memory and an aarch64 CPU, and the skill names its ten recurring failure modes G1 to G10, across launch, memory, thermals, bandwidth and precision. Each is numbered so it can be checked by its number, and the numbering is used by checks that run these tests. The skill is meant to be read before a long run rather than after several hours of it. It applies when a training run fails to start with an import error or a segfault that does not point at the real cause, when a run runs out of memory while the GPU tools still show free memory, when throughput falls partway through a run that began well, before any multi-hour or multi-epoch job on GB10, when two Sparks are wired together and a parallelism strategy is needed, and when choosing between FP8 and NVFP4 precision for a run on the platform. The checks help the agent find which failure is likely, and the person running the job decides on the fix and confirms it on a short run. The skill is installed with the skills CLI from the wshobson/agents repository. Its skills.sh page showed about 2.3K installs, with Gen Agent Trust Hub and Socket passing and Snyk showing a warning at the time of the scan.…
Highlights
- Ten named failure modes, G1 to G10
- Preflight checks before a long run
- Launch, memory, thermal, bandwidth and precision issues
- Notes for two-Spark setups
- FP8 versus NVFP4 precision choice
- Installs with the skills CLI
Ratings & Reviews
No ratings yet
Be the first to rate spark-training-gotchas.
People also visit
spark-environment-setup
Set up a working ML training/inference environment on NVIDIA DGX Spark
debian-linux-triage
Expert-guided triage and resolution for Debian Linux system issues
flowstudio-power-automate-debug
Debug failing Power Automate cloud flows using the FlowStudio MCP server
spark-training-gotchas FAQ
What are the failure modes?
Ten recurring failure modes across launch, memory, thermals, bandwidth and precision, numbered G1 to G10.
When should I read it?
Before a long training run, rather than after several hours of it.
What if a run OOMs while the GPU tools show headroom?
The skill covers that case as one of the known failure modes.
Does it cover two machines?
Yes. It includes guidance for wiring two Sparks together before choosing a parallelism strategy.
How do I install it?
Run npx skills add https://github.com/wshobson/agents --skill spark-training-gotchas.
What security audits does it have?
skills.sh lists three audits: Gen Agent Trust Hub Pass, Socket Pass and Snyk Warn. A Warn or Fail is worth reading before you install.
What are the best spark-training-gotchas alternatives?
Popular alternatives to spark-training-gotchas include spark-environment-setup, spark-memory-thermal-ops and parallel-debugging. See all spark-training-gotchas alternatives