Self-hosted alternatives to spark-optimization

S

spark-optimization

Optimize Apache Spark jobs

spark-optimization is an agent skill from the wshobson/agents collection for optimizing Apache Spark jobs. It covers production patterns for partitioning data sensibly, caching data that is reused, reducing the cost of shuffles, which move data between machines and are often the slowest part of a job, and tuning memory and executor configuration. The core concepts begin with the Spark execution model, which explains how a job is split into stages and tasks, so that the agent can reason about where time is spent. The skill applies when optimizing slow Spark jobs, tuning memory and executor settings, implementing efficient partitioning, debugging Spark performance problems, scaling pipelines to large datasets, and reducing shuffle and data skew, where a few partitions hold most of the data and hold back the whole job. It is written for data engineers who run Spark on large datasets and need faster, cheaper jobs without guessing at the causes. The skill's guidance is most reliable when it is checked against the job's own metrics, such as the Spark UI's stage and task timings, before and after each change. The agent proposes changes and the data team decides which ones to adopt on production clusters. The skill is installed with the skills CLI from the wshobson/agents repository. Its skills.sh page showed about 10.5K installs and passing results from three security audits at the time of the scan.