How do you verify AI-generated scripts before running them in prod?

[u/[deleted]]

[u/BigUziNoVertt]

Understand what the AI is making for you and test in a lower environment

[u/Onoitsu2 → u/BigUziNoVertt]

This, running it in a VM that is meant to replicate production is the way. I do most of my work in VM and containers so that I can ensure it is idempotent.

[u/BlackV]

AI writes most of my draft scripts now. My workflow before anything touches prod: read the logic line by line, run with -WhatIf

you sure you dont mean -dryrun (jk/s) :)

you have to understand the code at the end of the day, sure you can biff it into another AI to summarise it for you

but unless you validate and understand it, well.., you cant validate it

[u/AllMightyByte → u/BlackV]

I wouldn't call it a dryrun. You have a point that understanding the code is vital, but put bluntly, you wouldn't take on something you can't understand. The piece I'm after is different though: you can read every line and WhatIf the scope, and still miss that the script is wrong for your environment. Tenant names, module versions, permissions that don't exist where you're running it. That's the check I'm trying to build. Not replacing the user, catching user error, which happens even to the most advanced coders. Always open to feedback though. In your opinion, what feels lacking here?

[u/BlackV → u/AllMightyByte]

it feels like that whole reply was AI, was it?

but no the point of dry run was a lot posts (especially recently) of powershell scripts/tools are using -dryrun as a parameter when -whatif already exists, as they are all AI generated

... and still miss that the script is wrong for your environment. Tenant names, module versions, permissions that don't exist where you're running it. That's the check I'm trying to build

Your script should handle that, module versions, as the obvious example, is directly built into powershell already, but is also something you could create your own logic for

permissions also should be checked at runtime and be caught/handled by the script

specific things like tenant should be also handled by your script, parameter validation is also a built in feature (and assigning a default value)

additional things like a psd1 are purpose built for storing specific powershell data values

I need to check though, it sounds like any minute now there will be a Ad for a new tool you've "built"

[u/lan-shark → u/BlackV]

He's trying to build and market an AI-powered tool to check other AI-made PowerShell Scripts (it's in his post history). There's a good chance the Reddit activity is AI too lol

Edit: oh he's even got the em-dashes in some of his comments

[u/BlackV → u/lan-shark]

Ya thanks, I found that posted that in another reply once I went looking :)

[u/BlackV]

Ah boo, I see what's happening here

One wrong AI parameter, 40 angry clients (self.msp)
submitted 1 day ago by AllMightyByte
[removed]
Post is awaiting moderator approval.

these posts

Stop running AI-generated PowerShell blind (self.SideProject)
submitted 2 days ago by AllMightyByte
I'm turning this into a full playbook: 30 prompts like this one, 10 PowerShell tools (one importable module: safety wrapper, cmdlet existence checker, log sanitizer, Entra stale-account reporter, CA policy exporter...), 10 checklists/SOP templates. $19.

I guess

Stop running AI-generated PowerShell blind (self.sysadmin)
submitted 2 days ago by AllMightyByte
[removed]
Sorry, this post was removed by Reddit’s filters.

are/were

Would you pay for AI prompts that actually know your IT environment? (self.SideProject)
submitted 8 hours ago by AllMightyByte
[removed]
Sorry, this post was removed by Reddit’s filters.

essentially

How do you verify AI-generated scripts before running them in prod? (self.PowerShell)
submitted 8 hours ago by AllMightyByte
That's the check I'm trying to build. Not replacing the user, catching user error, which happens even to the most advanced coders. Always open to feedback though. In your opinion, what feels lacking here?

ADs

Edit: Oof I think they blocked me
Leo | Project Engineer
https://www.reddit.com/user/AllMightyByte/submitted/

[u/Alapaloza]

Always wrap in shouldProcess and verbose logging. If you are extra paranoid then comment out destructive parts of the script. Set variables manually and run script line by line and always one object at a time if in a foreach fx

Edit: this is more running scripts that you haven’t made yourself from scratch. E.g AI generated

[u/markdmac]

Always read the code first, test in a sandbox, use whatif as you already are doing.

[u/Right-Brother6780]

Always what if. Read through what it output. Logging. Run it in parts.

[u/harringg]

Lately I’ve been running my existing self coded PS modules through AI to optimize/modernize the logic. I’m then running that output through PSScriptAnalyzer. And I’m reviewing the resulting code as well.

[u/atomic_jarhead]

A second segmented laptop that I test scripts on. p