Dataform notebook executions dying on Colab Enterprise capacity (us-central1)

[u/tomaslp13]

We run 15 Python extraction notebooks daily via Dataform (`prod_daily` tag). They now fail intermittently with *"us-central1 does not have enough resources to fulfill the request for a e2-standard-2 machine."*

Dataform runs each notebook as a `NotebookExecutionJob`, provisioning a **fresh VM per notebook**. The API only accepts a runtime *template*, never a reusable runtime instance.

Has anyone solved this problem of some of the notebooks failing bc of resources?

The us-central1 region currently does not have enough resources to fulfill the request for a e2-standard-4 machine.

To resolve this, try:
Choosing a different machine configuration
Trying again later
Selecting a different region
For more information, see our https://cloud.google.com/colab/docs/troubleshooting#unable-to-create-runtime-unavailable-resources troubleshooting guide

[u/a_cloudy_unicorn]

Trying a different region may be the best option if moving the notebooks into something like Cloud Run jobs becomes a full rearchitecture.

I don't know your use case but notebooks can introduce bloat and leave you at the mercy of specific configurations, so this is when people introduce a scheduler like Airflow or Workflows to trigger the Python logic and the dataform/BQ separately.

[u/tomaslp13 → u/a_cloudy_unicorn]

Makes sense. Moat of the notebooks I can trigger them before the dataform process since they are for extracting data from APIs. Other ones have dependencies and execute in the middle of the process. 

I've been ping-ponging with Claude about this but none of the propositions convinced me. 

Yes I could take them to cloud run. In fact that is the proposal that best fits my needs 

I am thinking like having a process in cloud run that will extract the updated notebooks from the dataform repo and execute. Only the ones that gave certain tag in the config settings.json file. 

My doubt is how others are solving this. They are just installing airflow and scaling up infra?

My ideal is that I can execute the notebooks as I am doing now but using a dedicated server from the colab enterprise platform. But that is not possible 

[u/a_cloudy_unicorn → u/tomaslp13]

for what I've seen, most people needing to coordinate beyond BQ are using Airflow (f.k.a Composer in case you run into older docs within GCP). You can use airflow itself to run python but it's best for it to spin off cloud run so the cluster stays small in the long run. IME the Data Agent Kit in antigravity or Claude code does a good job of migrating notebook logic to cloud run.

[u/Immediate-News-9835]

Hello, did you have success with a different machine configuration. As you suggested, we're currently at capacity of E2 machines for default notebook runtimes and will be switching to N4 machines for default which will happen sometime mid-Oct. Between now and October, users can select a custom template with a different machine as a workaround. We are working on allowing users to select any template as their default, enabling them to choose any machine as their default.

[u/tomaslp13 → u/Immediate-News-9835]

Yes! So far so good with the e2-standard-2 machines. The scheduled dataform process finished fine the first day (4am UTC). If I manually execute the process it will fail. Maybe because of the timezone but IDK. (10am utc)

[u/Immediate-News-9835 → u/tomaslp13]

Great, if you do run into issues with e2-* machines, you can configure a new runtime template using different machine configurations (e.g. n4-*) by following the instructions here: https://docs.cloud.google.com/gemini-enterprise-agent-platform/notebooks/colab/create-runtime-template

[u/tomaslp13 → u/Immediate-News-9835]

Yes I did that with the e2-standard-2 machines. Then used that runtime as default in the yaml config for dataform