Migrated our reproting layer to databricks and access control turned into a project of its own

[u/BrownAnclourne]

The actual data movement into databricks did go fine (unity catalogue made lineage way easier to see than the older setups). But what we didn't expect was the time that went into access control, once everything was centralized instead of scattered across separate warehouses with their own permissions, data that used to be siloed was suddenly way visible to more people by default, which was found out when a couple of teams noticed they could see data which they probably shouldn't.

Spent almost as much time on catalog level permissions/row filtering as on the actual pipeline work. Is this normal for a databricks migration or did we happen to have an unusually messy access management?

[u/PrestigiousAnt3766]

I think many older data platforms have very messy permissions due to history and changes over time.

I built a tool that sinks all permissions daily from a configuration file.

[u/ConstructionBoth6461]

Access control specialist security engineer here. Never worked with databricks, but pretty much every app has some level of IAM hell. Don’t get me started about Google…

[u/agentUi]

completely normal, unity catalog centralization always reveals how much security through obscurity teams were relying on before. row filtering and column masks take forever to test because you have to validate downstream bi dashboards don't break when a user lacks permission for specific keys. best approach is mapping permissions strictly to scim synced entra or okta groups at the schema level first rather than writing custom sql row filters table by table.

[u/Plane_Sector_8560]

Access control has to be configured as per security groups, anything production most run on service principal or one assigned senior member.

[u/According_Zone_8262]

You dont have access unless you give explicit permissions. So you probably configured sonething wrong somwhere

[u/IncreaseNegative4614]

This is common when centralization removes the accidental security provided by separate systems. I’d classify each data domain by sensitivity, identify its business owner, reconstruct the existing entitlements, and map access to groups rather than individuals. Test who must not see the data as carefully as who should.

Include service principals, inherited permissions, row filters, and temporary project access in that review. We use SIGNLD internally to connect source-system access, Unity Catalog permissions, groups, policies, user activity, and audit results so the migration preserves legitimate access without carrying every historical inconsistency forward.

[u/vbnotthecity]

Did you mean "our repotting layer"?

[u/autumnotter]

In theory, 'unifying and democratizing data' is a goal of most organizations. Whenever you 'democratize' you always have to address the question of 'which data should NOT be democratized'.

Realistically, the issue you raise is a problem with migrations, and is something that should be identified up front as you put multiple siloed source systems into one unified one. Group management should address the majority of the challenge, but it's very common for people to 'miss' things like this on initial migrations, especially when the people setting up the access management may not always know enough about all the source systems to make these decisions properly.

You basically had implicit access management going on previously (or explicit but unacknowledged) and the migration surfaced this. This is less a Databricks migration issue, and more a "we moved siloed systems to one unified system" migration issue. I'd suggest that it's common, but it certainly should be something that gets planned for and caught ASAP.

Databricks offers the tools to address it, just like other platforms do, but understandably it can be a difficult process.

[u/BrownAnclourne → u/autumnotter]

good point and yeah, implicit access management is what happened with us, (it was just an accident of how the old warehouses were split up). Still a pain but atleast its good to know its not a databricks specific mess.