- Home
- Workflow Orchestration Software
- dagster-slurm
On the recordRead 5 Oct 202614 facts
dagster-slurm
Plans and prices
Open-source software
Free
Dagster and Slurm required · Python 3.12 development conventions · Slurm cluster needed for cluster execution
Read 5 Oct 2026 · dagster-slurm.geoheil.com
On the record
- Cluster connectivity
- The product connects to Slurm clusters over SSH and supports key-based authentication, jump hosts, and configurable host-key checking.dagster-slurm.geoheil.com · read 5 Oct 2026
- Environment packaging
- Jobs use Pixi environments, and production mode expects a pre-deployed environment bundle on the cluster.dagster-slurm.geoheil.com · read 5 Oct 2026
- Execution modes
- The stable execution modes are local and Slurm, with one Slurm job per asset by default.dagster-slurm.geoheil.com · read 5 Oct 2026
- Experimental features
- Heterogeneous Slurm jobs are experimental; run-scoped allocation reuse is currently focused on Ray assets.dagster-slurm.geoheil.com · read 5 Oct 2026
- Intended users
- The docs describe laptop-first development and cluster use for compute-heavy Dagster assets, including model training and hyper-parameter sweeps.dagster-slurm.geoheil.com · read 5 Oct 2026
- Job submission
- It transfers the current checkout payload and Pixi-locked environment, submits jobs with Slurm, and returns job status and logs to Dagster.dagster-slurm.geoheil.com · read 5 Oct 2026
- Local development
- Assets can run locally without SSH or Slurm, and the project provides a Docker Compose Slurm stack for testing cluster workflows.dagster-slurm.geoheil.com · read 5 Oct 2026
- Observability
- Dagster can display Slurm state, logs, CPU efficiency, memory, elapsed time, and node-hours for asset runs.dagster-slurm.geoheil.com · read 5 Oct 2026
- Pipeline orchestration
- Dagster asset dependencies and lineage can connect data ingestion, Slurm compute, and downstream aggregation or publishing.dagster-slurm.geoheil.com · read 5 Oct 2026
- Purpose
- dagster-slurm lets Dagster assets run locally during development and on a Slurm cluster in production.dagster-slurm.geoheil.com · read 5 Oct 2026
- Ray integration
- Ray is a first-class launcher for distributed execution, GPU scheduling, and optional run-scoped allocation reuse.dagster-slurm.geoheil.com · read 5 Oct 2026
- Secrets handling
- Cluster connection details can be provided through runtime environment variables so credentials can be kept out of the repository.dagster-slurm.geoheil.com · read 5 Oct 2026
- Spark integration
- The documentation labels the Spark integration as work in progress.dagster-slurm.geoheil.com · read 5 Oct 2026
- Support
- The project directs users to GitHub Discussions for questions and how-to requests, GitHub Issues for bugs and feature requests, and Dagster Slack for broader ecosystem questions.dagster-slurm.geoheil.com · read 5 Oct 2026
