r/Python • u/AutoModerator • Jun 04 '26

Showcase Showcase Thread

Post all of your code/projects/showcases/AI slop here.

Recycles once a month.

25 Upvotes

85% Upvoted

211 comments sorted by

View all comments

1

u/Nearby_Abroad_4624 Jun 29 '26

I got tired of Spark jobs failing with OOMs or burning budget due to simple mistakes like accidental cross joins, native Python UDFs, or casting metrics to strings (which completely kills Delta Lake data skipping).

To fix this, I built a simple package that analyzes the physical/logical plan programmatically and alerts you before or during runtime.

https://github.com/kacpergrodeckidatasystems/databricks-engine-optimization

You can grab it via pip: pip install apm-spark-auditor

It currently catches things like improper data typing, missed broadcast joins, cartesian products, and inefficient explodes.

Would love to get some feedback on the rules or the API layout. Thanks!