Kestra

KestraSince 2022-06

Kestra is an open-source, event-driven orchestration platform built to automate and coordinate complex data pipelines, microservices, and multi-agent AI systems. With a modern declarative YAML-first architecture, Kestra enables engineering teams to manage scheduled tasks, webhook triggers, distributed compute jobs, and LLM agent pipelines through code or a rich interactive UI. The platform provides over 600+ pre-built plugins spanning major cloud providers (AWS, GCP, Azure), databases (Postgres, Snowflake, BigQuery), and modern AI ecosystems (OpenAI, LangChain, Hugging Face, Vector DBs). Workflows can execute parallel compute tasks, branch conditionally, manage secrets securely, and handle automated retries with exponential backoff. Kestra eliminates the operational overhead of legacy orchestrators by running statelessly on top of modern container runtimes and Kubernetes, providing real-time workflow visualizers, sub-millisecond execution triggers, and enterprise-grade role-based access control.

Kestra’s architecture is built around an event-driven core powered by Apache Kafka or PostgreSQL for distributed queuing and high-throughput execution guarantees. Each workflow is version-controlled in Git as a declarative YAML specification, enabling full CI/CD integration and infrastructure-as-code automation. For AI engineering, Kestra serves as the deterministic execution backbone: triggering RAG indexing pipelines, coordinating distributed fine-tuning runs, provisioning transient GPU containers, and validating agent tool calls against production database replicas. The platform includes embedded Python, Node.js, and Bash script runners with isolated container sandboxes, comprehensive OpenTelemetry distributed tracing, and real-time execution dashboards.

Modal LabsSince 2022-11

Modal Labs is a high-performance serverless cloud platform that enables AI engineers and developers to run Python code in the cloud with instant access to thousands of CPUs, GPUs, and persistent network volumes. Founded by former Spotify CTO Erik Bernhardsson, Modal reimagines cloud computing with sub-second cold starts and zero infrastructure configuration. With Modal, you define your container image, dependencies, and GPU hardware directly inside standard Python code using simple decorators (e.g. `@app.function(gpu="H100")`). Modal handles container building, volume mounting, GPU scheduling, and automatic scaling down to zero in milliseconds, making it the premier choice for running generative AI models, ComfyUI video pipelines, and massive parallel batch jobs.

Modal operates a custom container runtime built in Rust that bypasses standard Docker daemon overhead, allowing container images to spawn in under 900 milliseconds. Its distributed filesystem mounts shared NetworkFileSystem (NFS) volumes across thousands of simultaneous workers with near-local NVMe read speeds. Modal supports NVIDIA T4, L4, A10G, A100 (40GB/80GB), and H100 SXM5 GPUs. Developers can attach web endpoints (`@app.web_endpoint`), schedule recurring cron tasks, execute distributed map-reduce jobs across tens of thousands of cores, and monitor live streaming logs via the interactive web console.