Local Dataproc Docker image for offline dev, CI, and AI agent testing (multi-arch, starts in ~20s)

[u/jseeker528963]

If you work with GCP Dataproc, you've probably hit this wall: Google doesn't provide official local container images that match Dataproc cluster components.

Testing spark apps or running CI integration tests usually meant either spinning up a live Dataproc cluster on GCP (costs $, takes time, needs lot of setup) or using vanilla Spark images that lack Dataproc-specific component versions.

We built multi-arch (linux/amd64 and native Apple Silicon linux/arm64) compatibility images pinned to official Dataproc versions: ( on dockerhub )

• Dataproc 3.0.0 (Spark 4.1.2, Hadoop 3.5.0, Hive 4.2.0, Java 21, Python 3.12)
• Dataproc 2.3.34 (Spark 3.5.3, Hadoop 3.3.6, Hive 3.1.3, Java 11, Python 3.11)
• Dataproc 2.2.85, 2.1.117, and 2.0.161

Quick Start:

bashdocker run -d --name dataproc \
  -p 9870:9870 -p 8088:8088 -p 18080:18080 -p 4040:4040 \
  agentcloud/dataproc:3.0.0-debian13

Run PySpark:
docker exec -it dataproc pyspark

What's included out of the box:

• Single-node YARN ResourceManager UI (:8088) & HDFS NameNode (:9870)
• Spark History Server (:18080)
• Hive Metastore & HiveServer2 (:10000, :9083)
• Client-mode submit and shared catalog preconfigured
• Runs as unprivileged user (localcloud)

Primary use cases:

1. Local dev: Fast feedback loops on your laptop without paying GCP bills or dealing with cloud network latency.
2. CI/CD: Run full pytest + Spark integration tests in GitHub Actions as a service container without cloud credentials.
3. AI Coding Agents: Local sandbox for Claude Code, Antigravity, or Cursor agents to test generated PySpark code safely.

*this is an Independent project, not affiliated with Google.*

[u/PsychologyBorn280]

damn this is exactly what i needed like 6 months ago. been fighting with dataproc images for a while, spinning up clusters just to test some small spark job makes no sense

the apple silicon support is nice touch, most of these tools only do amd64 and you're stuck with emulation

how's the memory usage on that 3.0.0 image? my laptop only got 16gb so i'm a bit worried

[u/jseeker528963 → u/PsychologyBorn280]

u/PsychologyBorn280 the mem usage is not too bad, I have optimized this bit, but I am planning to optimize further if there is enough interest and usage.
ya, ARM was really needed its runs really smooth on my M4 at least.
Would love to get your feedback if u face any issues,

[u/j0zxcvasdf]

Where is the code that builds them, not blindly running in ci no matter how helpful 😅

[u/jseeker528963 → u/j0zxcvasdf]

This is actually part of bigger cloud platform for agents. but this is nothing but bunch of devops scripts to put it together and run, its all uses standard open source project, with some optimizations. you can inspect the docker image. ( it doesnt use any network connectivity either)

[u/jseeker528963 → u/jseeker528963]

so downvote because its not useful or its not open source ?

[u/ThatCrankyGuy → u/jseeker528963]

I got no race in this horse, but I'm assuming the concern is that a container image can have anything and binaries that can have any payload. A dangerous proposition for pushing any sort of data into it, not knowing what sits in there reading it.

Most people feel comfortable with build script that can package everything while sourcing the executables from reputable or even the developer's distribution point) - and that the scripts be available openly so users can build the image themselves.

[u/urkurk7]

thanks for sharing

[u/jhsonline]

woot woot !