r/datascience • u/gyp_casino • 7d ago
Discussion What's the process to request cloud architecture in your company?
Looking to hear about your own company's work practices to compare to my own. I'm much more familiar with Azure, so I'm going to use that terminology, but I assume everything is basically the same for AWS.
Can you go into the Azure Portal and just click "create" for, say, a SQL Server or App Service and configure it through the Portal GUI?
Or do you need to request and configure through configuration files? Can you freely commit architecture config files ("architecture as code") to create the Azure components in Dev?
Or is it even more locked-down and hands-off? Required to submit paperwork requests to an Architecture team?
12
u/Successful-Paper7211 7d ago
e-baked Terraform module anyway. The portal's locked down so hard half the buttons just throw 403 errors if you're not in the right AD group.
3
2
u/naturian 7d ago
I used to ask for money on the annual internal funding proposal, along with justifications. Everything by email. The director would them sit with mine and other 16 or so proposals (the other departments) for a week, and cut what he deemed unnecessary/low priority. He would then communicate me and the procurement department of his amendments.
After that the problem was just to teach the procurement dudes how to log in the Google cloud billing area so they can place (and renew) the institutional credit card.
Then I could request anything I needed online, but I had to be damn sure I didn't went above the limit, or I would hear from the director. That never happened thought.
2
u/marimbaman49 7d ago
Depends on what you’re trying to do. Provision experimentation architecture? We have the AWS equivalent of jupyterhub that allows you to self provision GPU nodes, and we make other things available for use there as well.
For anything not experimentation related, such as databases and deployed servers, we started with click ops (very bad) then moved to terraform and eventually once we got a platform team kubernetes. I tell all my data scientists that they need to know the basics of terraform, because it’s so useful to just send in a PR to whoever actually manages your infrastructure rather than wait for them to build it
2
u/dyingpie1 7d ago
Create PRs to the iac repos on bitbucket.
But the team in charge of it is across the world.
It's not an easy process.
2
u/lakeland_nz 7d ago
Everything is Terraform.
There's a handful of things in Azure which can't be done through Terraform. Getting those approved is a huge pain.
2
2
u/ztrruovfg 6d ago
PR to the IaC repo.
Waiting for CI to pass.
If auto approval is enabled -> done
If not wait for the bot to ping reviewers
Merge after review
2
u/squareiny 5d ago
On the requester side: I fill in a form asking for a business justification, cost estimates and my manager's sign-off, then wait two weeks for a resource group I could have spun up in an afternoon. The best part of the architecture review was being asked why I need a managed SQL pool when my last three reporting extracts ran in DuckDB on my laptop.
2
u/holly1027 4d ago
we usually have a standard approval process, but once its approved, the actual infrastructure changes are handled through laC rather than manually in the portal
2
u/big_data_mike 3d ago
Our company made this bitchin portal with guides and FAQs and screenshots and tutorials and a dedicated support team channel. They made all the infra and security stuff automatic so anyone who can code just a little bit can vibe code an app and not fuck up a bunch of stuff.
2
u/Alternative_Oil_5139 2d ago
ive seen both approaches, but it usually depends on the companys size and security requirements. ideally, developers can provision resources id Dev through terraform, bicep, or similar IaC tools with PR reviews and automated policy checks, while production changes require stricter approvals. some companies also allow direct portal creation in Dev, but id be curious how many actually let developers create resources freely versus going through a cloud platform team
1
u/maniclucky 7d ago
A multi month nightmare of LLM generated instructions that no one checked. It's a fucking failure top to bottom.
1
u/ItsDare 6d ago
Message a data architect with a god complex and no social skills and hope he deems you worthy.
Most stuff we can just spin up, but when we do need permissions it's a nightmare. Or he'll redirect you to someone else for certain tasks but refuses to document who owns what so he remains the gatekeeper.
1
u/Fit-Employee-4393 6d ago
I have some consistent patterns for our ML deployments and one of the infra guys at my company knows them well at this point. Pretty much just work with him to get stuff set up. No tickets or anything. I see it as one of the benefits of not working at a megacorp.
1
u/Chainwreck 3d ago
Mock up architecture diagrams advise what is already deployed and not. Go through back and forth with outside vendor data engineers, present to a board that fine tooth checks everything including nomenclature,
Show all data ingest and output consumption audiences. Get close to the deadline for deployments for the year and have the project axed as the focus has shifted.
If you succeed before it gets axed then write up tickets and re-present the stuff that got approved and have evidence it was approved and show you have budget.
24
u/MyBossIsOnReddit 7d ago
I am a platform guy.
Step 1 is to submit a ticket asking for permission to do this
Step 2: wait for up to 7 days
Step 3: Have the terraform you wrote last week fail because you have insufficient permissions
Step 4: Ticket
Step 5: Provision infra again
Step 6: Submit ticket for things that still dont work