Datamediumcommunity

spark-optimization

Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.

The full SKILL.md could not be loaded right now. You can still install it from the upstream repo.

Featured in these articles

Long-form deep dives where this skill is referenced or applied. Skim the article, then come back here to install.

ArticleAdvanced Patterns — Hooks, MCP Servers, Custom Tools, System PromptsOnce you've outgrown the defaults: hooks for deterministic side effects, MCP servers for org-specific data, custom tools, and system-prompt surgery.
ArticleSupabase — Postgres, Auth & Storage in OneReal Postgres, auth, file storage and pgvector behind one SDK. For an AI product, having your data and your vectors in the same database is a quiet superpower.
ArticleChunking — The Decision That Sets Your CeilingYou can't retrieve what you chunked badly. Chunking is the most under-rated lever in RAG — and the cheapest to get right.

spark-optimization

The 1,441 skills, demystified in one PDF.

Featured in these articles