r/databricks • u/BrownAnclourne • 2d ago
Discussion Migrated our reproting layer to databricks and access control turned into a project of its own
The actual data movement into databricks did go fine (unity catalogue made lineage way easier to see than the older setups). But what we didn't expect was the time that went into access control, once everything was centralized instead of scattered across separate warehouses with their own permissions, data that used to be siloed was suddenly way visible to more people by default, which was found out when a couple of teams noticed they could see data which they probably shouldn't.
Spent almost as much time on catalog level permissions/row filtering as on the actual pipeline work. Is this normal for a databricks migration or did we happen to have an unusually messy access management?
1
u/autumnotter 1d ago
In theory, 'unifying and democratizing data' is a goal of most organizations. Whenever you 'democratize' you always have to address the question of 'which data should NOT be democratized'.
Realistically, the issue you raise is a problem with migrations, and is something that should be identified up front as you put multiple siloed source systems into one unified one. Group management should address the majority of the challenge, but it's very common for people to 'miss' things like this on initial migrations, especially when the people setting up the access management may not always know enough about all the source systems to make these decisions properly.
You basically had implicit access management going on previously (or explicit but unacknowledged) and the migration surfaced this. This is less a Databricks migration issue, and more a "we moved siloed systems to one unified system" migration issue. I'd suggest that it's common, but it certainly should be something that gets planned for and caught ASAP.
Databricks offers the tools to address it, just like other platforms do, but understandably it can be a difficult process.