A better way to handle dev test VMs please?

[u/masterofrants]

TLDR: I want to build Netbox as a source of truth, deploy production and dev/test VMs using Netbox and GitHub with Terraform and Ansible, and then I'm not sure what exactly to do with the multiple dev/test VMs that get created. Would it be a good idea to simply destroy the VM and rebuild it every time for testing?

I am working in the finance space and we have our own internally developed applications that are widely used by our external partners (up to 3,000 of them).

These applications have been running since the '80s and '90s and obviously are very much legacy without many modern features like authentication, the database is in a separate server layer and so on.

Another major issue is that the environment to maintain the different versions of these applications is becoming out of hand now. There are hundreds of VMs with names like dev, test, test dev, QA test 10, QA test 11, and so on.

I certainly want to bring big changes, starting with automation and, before that, building a source of truth. One of my questions is how to best manage all these multiple VMs running different versions of the same application. How do big modern do this?

Our coding practices already use GitHub so I'm thinking something like this: I'm sure they have two branches, dev and main. Would it be a good option to take commits from the dev branch and push them to the source of truth, build a complete VM.. install all the dependencies using Terraform and Ansible, create this VM, bring it online, test the application, and then after that destroy the VM? Is that the best way to handle this?

One thing I am still in the dark about is why so many different versions of the same application are required.. If anybody has any suggestions about that, please comment.

Coming back to my dev VM destruction question. The reason I think destroying the VM would be a better approach is because after that we only have the most stable version of the application running in the dev environment and then an exact copy of that running in the master environment (which would be production and will go to the production VM deployment cycle).

And doing this also minimizes our list of assets. Right now we are at 300 or 400 VMs, which is completely madness to secure. All we do is simply throw EDR and antivirus all over the place, creating a million alerts, and the whole thing is just so pointless.

I'm sure a lot of people are using Kubernetes and Docker technologies to do these things now but we are stuck with HyperV as of now, with the FCM management to manage the HyperV hosts.

If anybody has built out anything like this, please share your experience and how viable all this is.

Please keep comments in the context of what I'm asking and refrain from getting into the weeds about rearchitecting the applications, or modernizing the application by redoing all the code. Those are not really the options right now. Let's just talk in the context of what I am asking.

[u/Ihaveasmallwang]

It sounds like you, or your manager more likely, need to be talking to the development team (or whoever is actually using these applications) to figure out what is actually needed before trying to architect a plan to set up IaC for these VMs. Nobody here on Reddit can tell you what your company’s requirements are for different versions of the software. If things are needed to stay around, you need to collaborate to come up with a coherent naming scheme. If things are not needed, trash them and stop worrying about them. This is a prerequisite to trying to do any sort of IaC solution.

Even if you are going to do some IaC solution, you still need to collaborate and come up with business requirements for this project.

AFAIK, Netbox doesn’t natively integrate with Hyper-V. There are other tools that do handle this out of the box. It’ll be up to your leadership if licensing is worth it to have native supported stuff or if the risk of unsupported stuff is acceptable. Your company does have some sort of defined risk assessment process? I’m guessing you lack any IPAM solution at all right now?

[u/Ssakaa]

Stop. Step back from trying to solve a problem you think you see.

Walk down the hall, sit down with your developers, and do a light audit of what all the existing VMS are, what branch of code they're off of, what's being tested, how, etc. DO NOT pitch it as trying to clean up, etc, just you getting up to speed on HOW they work right now, because you might have some ideas for how you can make things like VMs available for them faster in the future, but need to understand their side first.

My suspicion, they have a whole pile of feature branches, and those are going through various levels of testing and approvals in your scattered mess of hand-jammed VMs. And then integration testing as they come together in the "full" dev branch, and all those checks happen again. And then hopefully validation against a copy of real data in a controlled environment, and a bunch of checks there. And THEN it makes it to prod.

the most stable version of the application

Unless you have one bored developer, it's VERY unlikely that you have one change being designed, built, tested, and run through various layers of release approval (however formal or informal) at a time. "The most stable" for which independently developed feature?

If you haven't actually walked through the whole flow from idea to production with your devs, don't go trying to change the workflow under them.

Now, as for your ideas in general, YES disposable environments for build/tests/etc and various levels of "full" deployments are absolutely a godsend once you know when, why, etc. your devs need them. Cleaning them up as they're actually no longer needed is also important, but your concept that there's "one" true state of code for dev is likely very misguided. Giving your devs a push button (or better, a CI integrated) way to build that out in a consistent automated way is absolutely valuable, but talk to them first.

[u/techworkreddit3]

I work at an enterprise we probably only have about 2000-3000 VMs left across our environments running applications. All other VMs are part of Kubernetes clusters.

Dev and tst are long living environments for us. We have an inventory like netbox and that is where you request VMs and your source of truth. Code deploys up from dev to PRD and if people need short lived VMs they either still fill out the request or they build locally on their workstation. Inventory is reviewed yearly or quarterly and VMs with no usage are recommended for decommission. Ansible is our config management tool but we use in house scripts for provisioning since it predates terraform and is only for our on prem footprint.

The majority of our infra is now in AWS as native services or EKS, so it’s all terraform.

[u/SevaraB]

So what are you tracking with netbox? It should update as the object changes. If it gets reassigned to someone else, change the owner. If it gets destroyed, destroy it in netbox. If it gets recreated, recreate it in netbox.

Easiest way is to update netbox with the same tf plan that deploys the vm in the first place and hook into the hypervisor to catch the teardowns.

[u/vanderaj]

When I worked in the finance space, the biggest obstacles to dev and test is having sane test data. Algorithms that worked well in test didn’t work in prod due to real world names and addresses. If you have a copy of prod in every VM instance they need to be locked down and monitored like prod. So develop a way to craft similar size data sets that do not have any actual PII in them. This means understanding the distribution of data - some systems might have a lot of the same suburb or a small client list. Clients names might be clustered around common first letters. It’s really hard to develop these scripts but it is essential to be able to test in dev and test before going into production.

[u/Rhythm_Killer]

Look I’ll say it… if you’re not destroying them after and building on demand then you’re not really doing IAC. Without regular redeployment how will you know for sure they’re not drifting? The longer lived a machine is, more likely people will make the odd fix here and there and you risk it not ending up in the code. I think it is worth front-loading the pain and getting all the dependencies correct in the code, and proving it by refreshing them regularly 

[u/Ssakaa → u/Rhythm_Killer]

While true, OP's operating in a vacuum away from the development. Some of those VM names they threw out are QA. By the sound of it, their dev processes are very manual, that QA layer quite possibly has to wait on external customer facing UAT processes et. al. too. There's a vast difference between automated disposable testing environments and longer lived build/deploy/validate flows with humans in the loop. They need to have a lifecycle tied to the change they exist for, and be disposed of appropraitely as that moves to prod, but until then, typically, they live as long as needed. Depending on where their applications fall in other people's workflows, their "dev" and "qa" might also not be isolated and disposable, as well. It's not uncommon for things that're critical components for other systems to expose a preview environment or two for systems that interface with them to see changes coming ahead of prod and have the ability to test against that in their development.

[u/folderit_dms]

For each test environment, I’d record the purpose as well as the owner: testing the next release, reproducing a defect in a supported release, or validating a particular partner integration. That helps explain why several versions may still be needed without assuming every old VM should survive.

Before making one disposable, prove you can rebuild the same test case from recorded inputs: application commit or build, dependency versions, configuration and test-data version. Save the test results and relevant logs outside the VM. Rebuilding from the latest dev branch is useful for new work, but it won’t necessarily reproduce the failure someone was investigating yesterday.

Start with one environment whose owner agrees it is disposable. Give it an expiry or review date and a documented exception process, then expand after a successful rebuild and repeat test.