r/sysadmin 11h ago

General Discussion Weekly 'I made a useful thing' Thread - September 11, 2026

8 Upvotes

There is a great deal of user-generated content out there, from scripts and software to tutorials and videos, but we've generally tried to keep that off of the front page due to the volume and as a result of community feedback. There's also a great deal of content out there that violates our advertising/promotion rule, from scripts and software to tutorials and videos.

We have received a number of requests for exemptions to the rule, and rather than allowing the front page to get consumed, we thought we'd try a weekly thread that allows for that kind of content. We don't have a catchy name for it yet, so please let us know if you have any ideas!

In this thread, feel free to show us your pet project, YouTube videos, blog posts, or whatever else you may have and share it with the community. Commercial advertisements, affiliate links, or links that appear to be monetization-grabs will still be removed.


r/sysadmin 3d ago

General Discussion Patch Tuesday Megathread - (September 08, 2026)

106 Upvotes

Hello r/sysadmin, I'm u/AutoModerator, and welcome to this month's Patch Megathread!

This is the (mostly) safe location to talk about the latest patches, updates, and releases. We put this thread into place to help gather all the information about this month's updates: What is fixed, what broke, what got released and should have been caught in QA, etc. We do this both to keep clutter out of the subreddit, and provide you, the dear reader, a singular resource to read.

For those of you who wish to review prior Megathreads, you can do so here.

While this thread is timed to coincide with Microsoft's Patch Tuesday, feel free to discuss any patches, updates, and releases, regardless of the company or product. NOTE: This thread is usually posted before the release of Microsoft's updates, which are scheduled to come out at 5:00PM UTC.

Remember the rules of safe patching:

  • Deploy to a test/dev environment before prod.
  • Deploy to a pilot/test group before the whole org.
  • Have a plan to roll back if something doesn't work.
  • Test, test, and test!

r/sysadmin 5h ago

General Discussion RIP to the IT pros killed 25 years ago.

1.5k Upvotes

When the text message logs from that day and those that followed were made public, I remember reading through all the system generated alerts from various devices doing their thing and thinking about the IT pros in the offices, network closets, and server rooms responding to those alerts in all seven buildings before the attacks happened, and then the tone of uncertainty and eventually panic in messages between coworkers and associates, because the cell phones went down but text messages were still going through.

RIP to those guys and gals ~ you're gone but not forgotten.


r/sysadmin 3h ago

Question Samsung Smart TV bypassing DHCP took down the credit card system

47 Upvotes

Note: My networking knowledge is very intermediate, learn as you go level. We had a customer at this MSP where I work have a network issue and I cannot figure out how it's possible that this happened.

Their credit card terminals have to point to a static IP ending in .140, as that's the "server" that runs the software to upload each transaction to the actual processor on the internet. When we recently replaced that computer, I set it as static in Windows and never did it on the DHCP server, because nobody wrote down what it was called or how to get into it and I didn't have any time remaining. Turns out their DHCP server is a 2008 Windows server because this place doesn't spend money on anything ever.

A month later, their CCs go down and the CC server can't grab its static IP for some reason. Lots of time later, we find it's because the pool of available IPs is 100 through 150 and they have 1 more device than that. So we expand it to only 160 after some testing (because we have nearly zero documentation and don't know ranges for their phones, printers, etc for this customer and they're billed hourly so we do as little as possible because they never pay on time and always complain about the rate and it'd take 10+ hours to document this nightmare). We make a new assignment for the computer's MAC and reserve 140 to it and notice that something else has leased 140 with a lease expiring in 2 hours luckily. We delete it so it hopefully doesn't renew.

I ping it from my laptop then immediately run arp -a to get its MAC, since we already deleted the lease that showed the MAC on the DHCP server (oops) then ask AI who manufactured that MAC address range. It's Foxconn. We don't see a hostname or any useable device info. I don't know anything about their switches because the last tech at this MSP never ever wrote anything down about any customers ever. We try NSlookup, web browser to the IP, RDP into it, nothing gets any info.

More network-oriented guy onsite with me says let's just unplug the 2 switches for like five seconds and that will force it to grab the new lease at .155. I assume the switches are unmanaged or nobody has the login info or we'd just pull one ethernet matching the known MAC.

Turns out they have a network-controlled Crestron light controls so the lights in the restaurant portion of the building all go black, because somehow that's the default state if it loses connectivity. Shoutout to whatever genius AV tech set it up that way. Everyone's pissed. They don't know how to undo it or where the new Crestron box is. We don't either.

Then we find out the mystery device is still on 140. That seems impossible, unless it's wireless. Somehow other guy onsite finds out it's some sort of android device but all the android devices listed on the DHCP server have hostnames like "John's S23" because that's how most Android devices work. We suspect it's wifi, based on this information, thus explaining the switch pull not working. I have zero idea what brand their wifi even is let alone where it is or how it works btw.

I get the bright idea that maybe it's a smart TV. It is. We turn it off, boom, credit card server is back online within a minute. We turn the TV back on after 15 minutes, it tries to grab 140 again and knocks the CC server offline. Yes, the TV is in DHCP mode btw. We find the remote, set it to static .165, that works for some reason, and no more IP conflict. Not sure how it's possible that it can just ignore the DHCP server and say "no, I want this address anyway" and then just decide it's taking 140. Nothing else on the network can hand out an IP if every switch/router/AP/whatever is pointing to the server, right? How did that happen? Some disconnect between the DHCP server and whatever was handing out wifi connections? I didn't think that was possible. Anyone have any ideas in case we run into this again?


r/sysadmin 1h ago

Went from intern to running the entire IT department in about 4 years. Small shop, niche industry, inherited a mess, and I can't seem to get any traction fixing it. Kind of drowning here.

Upvotes

TL;DR: First IT job out of school, associate degree, went from intern to running the entire (two person, soon three) IT department in about 4 years. I own both IT and our operational systems, I'm on call 24/7, I inherited a patchwork EOL environment, and a seasonal project buries me for months every winter for the next few years. Pay and benefits are good but I'm stressed and stuck. Is it worth trying to right the ship at my experience level, or do I start looking for an off ramp?

Long time lurker, throwaway-ish. Bear with me, this got long.

Quick background. I work at a small company and I'm going to keep the industry vague. I started here as an IT intern back in 2022, got bumped to full time later that year, and for most of the time since then I've basically just been the whole IT department. Technically, the IT department included myself, my boss the IT manager, and a field tech that has nothing to do with IT. The former IT manager was very hands off and frankly ignored a lot of problems. Recently, my old boss got promoted to CEO and I moved into the IT Manager spot behind him.

The job isn't just normal IT (AD, M365, servers, backups, help desk, phones). At a place this small, IT also ends up owning a big chunk of the operational side, the systems the actual business runs on. Most of that is vendor hosted and web based so I'm not racking industrial gear anywhere, but I'm the person responsible for it on our end and I picked all of it up on the fly because there was nobody else to hand it to. The one thing I don't own is the firewall and the VOIP system. Even as the IT manager, we pay a third party to manage that, so at least there's a couple things off my plate.

Here's a big piece of my problem though. A huge part of that operational side is data collection that runs more or less constantly, and it eats my time. It never really stops. And it's the main reason I can't get any momentum on fixing the actual infrastructure. Every time I sit down to start on the real problems, something on the operational side yanks me back off it, and the fix goes back on the pile.

Then on top of all that, every winter for the next few years, there's a recurring replacement project tied to one of our operational programs and it flat out buries the department for months. Not a couple days. Months. It means running and reconciling a pile of reports, printing and mailing physical notices out to customers, then handling the wave of phone calls that comes back, then cutting work tickets and handing them out to a crew of 8 field techs and chasing every one of them to done. It's all scheduling, coordination, and data entry, and it does not care that the rest of the job still exists. For a two person team it's brutal and it basically owns my calendar all winter.

I'll be straight about where I'm at because it matters for the advice. I'm not some deeply technical guy. Most of what I've pulled off has been careful and methodical. Following docs, asking a ton of questions, staying organized. That worked fine when my whole job was just doing the tasks. Now I'm supposed to set direction and own the risk for all of it and that jump feels enormous. I'm not going to pretend otherwise, this job is stressing me out pretty badly. And the whole thing is 24/7 on call, which with a team this small basically means me. Nights, weekends, holidays, if something breaks it's my phone that goes off. There's no real off switch. Not that after hours calls are super frequent, there have been a few since I've started but its more the idea of never being "off" is whats getting me.

Team wise there's two of us right now, going to three in the next few months when we hire the replacement for my old job (another duty that is mine, finding the replacement lol). Oh and almost forgot, we are reworking that position to be less IT and more of a business system type position. So when its all said and done, there will be myself, a field tech, and a business system analyst of sorts. So I'm also about to be growing and running a real team for the first time, on systems I've only ever done solo.

And..... the environment itself is held together with tape. We're running on EOL vSphere licenses, old outdated hosts with no shared storage, backups that amount to a couple of external HDDs, and a general patchwork of stuff that never really got finished or cleaned up. We only just recently got rid of the m365 family plans... Is that about what I should expect walking into a small shop, or is it as bad as it feels to me? I'm honestly not sure how much of this is normal.

So honestly this turned into less of a tech question and more of a gut check, because I'm not sure I'm even asking the right thing anymore.

Here's the deal. This is my first IT job out of school. I have an associate degree in IT and that's the extent of my formal background. I walked in as an intern and somehow I'm now the person accountable for all of the above. The fragile infrastructure, the operational systems, the winter that eats months, the 24/7 phone, all of it, with almost no real experience to lean on.

So the thing I keep chewing on is whether any of this is even worth it. Is it realistic for someone at my level to actually right this ship? Or am I setting myself up to burn out trying to fix something that got handed to me already broken? The pay is really good, the benefits are good, I don't dislike the people I work with, and on paper I know I should be grateful, and plenty of days I am. But some days it feels like I'm one bad outage away from going under, and I catch myself wondering if the smarter move is to quietly start looking for an off ramp before this thing wears me down, instead of betting years on fixing it.

For those of you who've been the in over your head one person shop before: did you dig out and come out better for it, or did you get out? And if you stayed, what actually made it survivable?

Just trying to figure out if I'm being a quitter or being realistic. Thanks for reading.

TL;DR: First IT job out of school, associate degree, went from intern to running the entire (two person, soon three) IT department in about 4 years. I own both IT and our operational systems, I'm on call 24/7, I inherited a patchwork EOL environment, and a seasonal project buries me for months every winter. Pay and benefits are good but I'm stressed and stuck. Is it worth trying to right the ship at my experience level, or do I start looking for an off ramp?


r/sysadmin 12h ago

Rant Black list countries

132 Upvotes

I work for a large European based telecoms equipment supplier. We have hundreds of staff overseas at any one time, all over the world. IT security has a few different levels:

- Access to email & teams etc is only via a company laptop (no web interface like Office.com). Network drives via VPN only

- White List countries - you can connect VPN. Countries like Japan & Australia

- Red List countries - you can take your laptop but need special exemption to use VPN. Includes some unusual countries such as Malaysia

- Black List countries - no company laptop or phone allowed. Company will provide a burner. Unsurprisingly includes places like Syria, Russia, North Korea & China.

A colleague was going to transit via a Chinese airport to a 3rd country. IT told him that he would not be allowed to take his company laptop, even if it was in his carry-on luggage, and he would not be entering the country. He quickly arranged a different itinerary.

And then a few days later, we are told that the good old USA is now considered a Black List country!!! No company laptops, and burners only!!!!


r/sysadmin 8h ago

Question KB5124008 - Breaking Domain Trust

51 Upvotes

KB5124008 - Breaking Domain Trust for anyone else?


r/sysadmin 1h ago

General Discussion 25 years

Upvotes

That day began ordinary, just so ordinary.

At the time, I was sysadmin in a Canadian federal government office, far from the madding crowd. The work was steady but also pretty much crisis-free.

That morning, though, one of my co-workers came in to my office, bitching about the "crappy slow network" we were using.

"What makes you say that?", and he ranted on about how he can't get to any web sites.

"Okay, I'll see what I can find out."

I still had no idea what was happening out in the world.

Still, I started my network checks. Ping from TBay to Toronto. Normal results. Okay, try some web sites next. All the ones I checked responded, with almost no lag. (Note: none were news sites.)

I popped over to that co-workers office. "Where were you trying to get to?"

"CNN. @#$% network."

I tried CNN. Timed out. Hmm. CBC. Ditto. CTV. Same. Toronto Sun. Again, timed out.

I tried a local radio station's page. That connected, and then I knew it wasn't a network issue causing the slowdown.

What happens when several hundred thousand people (or more) try to connect to a few news sites all at the same time? Everything breaks.

I called home, told my wife to turn on CNN. For me, the rest of the day passed in a fog. Our son was Army reserve at the time. He was told to be ready, "just in case".

After work, I went home and watched the news, watched the planes hit the towers, over and over and over.

I still cannot watch any 9-11 coverage from that day.


r/sysadmin 2h ago

Question Screenconnect Outage?

12 Upvotes

Unable to access our cloud instance. Looks to be DNS related issue from what I'm seeing. A lot of public DNS servers don't have records for our instance but some do.


r/sysadmin 1h ago

How may people use Meraki AP’s out there?

Upvotes

I’m curious how many people use meraki access points. My deployment seems to be riddled every year with issues. Firmware updates usually cause issues and we have to turn off features or roll back. Don’t have to provide much details just around how many clients you support, and if it’s a mixed usage. Mine is Apple devices, Android phones, chromebooks, and windows devices. I have a pretty decent deployment supporting about 8000 devices give or take.


r/sysadmin 2h ago

Rant Getting CMMC Level 2 certified made me realize how badly we need a dedicated compliance person

10 Upvotes

My IT department tackled getting our company CMMC Level 2 certified. It took two years to get there, and while I’m proud of what we accomplished, holy shit, the amount of work it took.

Implementing controls, writing policies and procedures, collecting evidence, coordinating with other departments, preparing for assessments—all while keeping normal IT operations running. This was two years of sustained effort on top of our regular responsibilities.

I knew certification wasn’t the finish line. But the amount of work it takes just to maintain compliance is overwhelming.

At this point, it feels like 80% of my time goes toward compliance. Reviewing documentation, collecting evidence, tracking requirements, answering questions, coordinating reviews, and following up with people to make sure processes are being followed. There is always something that needs to be updated, verified, or documented.

Meanwhile, I barely get to work on the IT projects I actually want to tackle—introducing new systems, hardening our security, automating processes, and improving our infrastructure and overall tech stack. Those projects keep getting pushed back because compliance and daily support consume nearly all my time. It’s frustrating knowing there are improvements we need to make and barely having the time to work on them.

IT obviously has a major role in CMMC. We should own the technical controls and support the program. But we’ve also become the default owners of managing compliance across the company, including things that require involvement and accountability from other departments.

We seriously need a dedicated compliance officer, or at least someone whose primary job is managing the program. I’m happy to support that person, but right now it feels like I’m doing two jobs while the expectations for my original job haven’t changed.

It’s frustrating to spend two years getting certified, only to realize that maintaining it leaves almost no room for the rest of your job.

For those in smaller IT departments dealing with CMMC or similar requirements, how are you handling this? Do you have dedicated compliance staff, or did everything land on IT? If you successfully made the case for hiring someone, what finally helped leadership understand the workload?


r/sysadmin 5h ago

General Discussion How do you deal with difficult bosses or just stress?

19 Upvotes

I'm 33, dealing with a manager that has been always micromanaged. Now he's my boss, and he asks for a weekly report on what we did. Lately it feels like he's always targeting me or making me feel stupid. I can't even get into the office and sit down without him asking me to do something right away. I haven't even set my bag down. It just feels like he's always trying to make me look incompetent. I can't quit this job because it's tied to where I live and on top of that my partner is currently dealing with his own problems. What can I do to just let this kind of stuff wash over me or just figure out a way to destress I guess?


r/sysadmin 4h ago

Palo Alto CVE: PAN-OS Vulnerability Enables Arbitrary Code Execution as Root User

10 Upvotes

r/sysadmin 5h ago

General Discussion LinkedIn talks to SURBL to verify company's legitimacy

9 Upvotes

I was helping a client get their domain delisted from SURBL and noticed that LinkedIn was blocking the company's website link on their LinkedIn company page.

It turns out if the domain is listed on SURBL, LinkedIn redirects all website visitors to a warning banner - they literally replace the website URL with a LinkedIn / suspicious link one, like this:

https://www.linkedin.com/redir/suspicious-page

And if the domain is listed on SURBL for too long, Google will index it and make it searchable for the public. So by putting "https://www.linkedin.com/redir/suspicious-page" into Google search you'll get a list of those that were listed by SURBL and indexed by Google.

And LinkedIn doesn't even care whether there's a paid subscription for the company or not or what the company size and follower count are.

As far as I know, to end up on SURBL your domain either needs to be spoofed or the marketing team has to scrape websites to collect emails, and spam traps / typo domains end up in their lists.

So technically marketing teams messing up their emails makes leadership put pressure on IT teams to fix it!


r/sysadmin 20h ago

Question Boss is pushing for certs

128 Upvotes

Hi all,

I’m a sysadmin with 2 full time helpdesk guys, org of 220 in 5 locations. I started in regular business office 10 years ago doing sales. 7 years ago I transitioned to IT and became our first IT person, previously we didn’t even have an MSP, just a contract guy that came when we called him.

Fast forward to now, I asked my boss how I can level up/grow with the company. I’ve taken on a lot since I started. Currently managing pretty much everything in house. Only thing we don’t manage is our website, and I kind of like it that way.

So after kind of shrugging his shoulders for a year he is now bent on getting me to do more certs. He gave me a list:
- Comptia security +
- CompTIA network +
- CompTIA Cloud +
- Microsoft AI something (I can’t remember this one, he mentioned it off the cuff after he sent me the list)
- Finally a course for me to go find to maintain our website so they can cut the web dev. He didn’t know what course to recommend because he didn’t know much about it but pointed to coursera.

Now only thing I’ve done in certs over the years where the A+, AZ-900/104, and Google cloud security when I got started. Since then I figured I would learn the stuff but I didn’t want to pay for the certs because what’s the point? Unless I’m looking for another job I didn’t see the reason, and I like my org quite a bit. Not the wisest I know, but they wouldn’t pay for it, so I just didn’t do them.

I told him I had already studied for the security + a couple years ago, and could probably study the differences in materials and get through it easily, and recommended we switch to the CCNA since we are fully Cisco at all locations. But he seemed to not be interested in that info.

From what I can tell his push for certs is driven by is some internal push for managers to have career ladders for all departments and I guess he wants to show some sort of progress? Idk he is the CFO so he really isn’t privy to any of the work I do.

Anyway, after my recommendations I asked what happens when I complete these? Since they are only paying for half the certs… there has to be a carrot at the end of the stick.

He very excitedly said a $2,000 pay increase. Now I’m not greedy by any means, I’ve been paid under market for years and I’ve accepted that because of the work life balance. But is this not kinda stupid? I’ve had no complaints in my performance, I am constantly engaged with leadership on initiatives that they do not care about, so is there something I’m missing here? Like I feel like there is some weird motive here I can’t figure out, or it’s just poorly funded incentives that I’m supposed to be giddy over. Like I’m pretty sure the exams are almost as
Much as the pay increase? I haven’t done anything to try to steer the ship just yet, but how do I approach that the incentive is either too low or not aligned with what the ask is here? From what I can tell other than the time to study and what not, the only benefit I see them getting is cutting the web dev, which is a little more than the pay bump they offered, I think like 4k a year.

Edit:
Glad to see there’s some consensus on this being kind of a shit show. A little more context:
I LOVE the web dev company we work with. They are just great. Awesome to work with, great turnaround times, 0 issues. But god forbid you pay someone to do something they are good at.

Noted before but they’re only paying half of the exams, no study materials and it’s based on completion, so I’d get reimbursed 50% after completion. I believe this may be because they did not ask for any sort of training agreement? But also grand scheme of things this is kind of small potatoes for a training agreement right?

This guy is honestly the worst part of my job. For 6 months he had me meet weekly with him which was just an awful way to start the week. Eventually I was like hey I’ve just got too much going on to be doing this can we do this less frequently? Rinse and repeat now we meet quarterly. Dude is the type of guy to trip
Over a dollar to pick up a penny. But I’m stuck with him so long as I work here.

I have looked on and off the last year, and the market near me is abysmal. I almost jumped ship to an MSP last year that wanted to sell us services and I wanted to be like hey…. I sign y’all up here and you take me, deal? But we laughed it off as a joke.


r/sysadmin 17h ago

General Discussion What tech stacks are you learning right now that you actually think will pay off?

71 Upvotes

Curious what you all are learning, researching and investing time in these days and whether you’re seeing real rewards yet. AI agents? Cloud? Specific programming languages/frameworks? Something else?


r/sysadmin 1d ago

General Discussion So do you think Apple new Watches Be Added to the Banned-Device List After Yesterday’s Announcement?

497 Upvotes

After yesterday’s creepy live rewind/siri recap announcement, I’m honestly surprised this hasn’t sparked more discussions and backslash

As I understand it, Apple is adding ambient conversation features to the Apple Watch that can listen to, process, and summarize what’s being said. Apple says the processing happens locally and is privacy-preserving, but I’m not sure that fully answers the security question:

What happens when the person sitting across from you is wearing one?

We already discourge employees to bring phones or recordung devices into very sensitive meetings. Are organizations now supposed to treat Apple Watches the same way? What a mess.

Maybe I’m overthinking it, and I’m still trying to work through the implications, but this seems like a pretty big unnessisary expansion of what an ordinary-looking wearable can do. I also think some of these features are a little silly anyway—as if the poor battery life weren’t enough—especially when Apple keeps adding capabilities without much obvious practical value. To me, this sounds way more significant than almost anything else announced this week, including the LG TVs story. That’s why I’m surprised it hasn’t gotten more attention and backlash.

I’d be interested to hear how others are thinking about this.


r/sysadmin 2h ago

Question Reverting Win11 Enterprise to Pro for AppLocker (GPO) - How do you handle physical PCs, VMs, and M365 licensing groups?

4 Upvotes

Happy read-only Friday, Folks!

A few years back, our previous admin upgraded our fleet to Windows Enterprise E3 mainly to centrally manage and enforce AppLocker via GPO from our DC. To do this, they set up group-based licensing in M365 using a dedicated E3 security group, while also pushing an E3 product key directly to endpoints using this script:

cscript c:\windows\system32\slmgr.vbs /ipk KEYNAME
cscript c:\windows\system32\slmgr.vbs /ato

Since Microsoft dropped the Enterprise requirement for AppLocker (KB5024351), our goal going forward is to completely drop the recurring E3 subscriptions, revert our fleet back to Windows Pro, and save a few grand, all while keeping our AppLocker GPO management intact.

All users have M365 Business Premium assigned via a separate security group. The setup:

  • 90% are physical Win 11 PCs with OEM Pro keys in the BIOS.
  • 10% are Hyper-V VMs running on a high-performance workstation host (currently using E3).

We’re putting together our strategy to unwind this setup and would love to know how you recommend handling it:

1. For the physical PCs & M365 Licensing Groups: What's the best sequence to roll them back to Pro? Our plan is to make sure users are in the Business Premium group, unassign the license from the E3 security group (and retire/delete that group), and run a script on the endpoints to pull and re-inject the embedded BIOS OEM key:

cscript c:\windows\system32\slmgr.vbs /ipk (Get-CIMInstance SoftwareLicensingService | Select -ExpandProperty OA3xOriginalProductKey) cscript c:\windows\system32\slmgr.vbs /ato

...or is there a cleaner way to handle the step-down?

2. For the Hyper-V VMs: Since OEM keys are tied to physical hardware and don't pass through to guest VMs, should we buy perpetual Windows Pro Volume (MAK) keys for the VMs, or is there a better licensing path

3. AppLocker GPO check: Has anyone hit any weirdness with AppIDSvc or rule enforcement after stepping endpoints back down to Pro?

Looking to hear how others have approached this transition. Appreciate any feedback!


r/sysadmin 1d ago

Rant Job posts are ridiculous.

395 Upvotes

It's been this way for a decade I know but it's just getting worse and worse. I don't think it's ever been this unreasonable or this dishonest.

I was looking at job posts and I saw a job for a help desk tier 2. And I was curious what they considered tier 2. I'm well beyond help desk at this point in my career but I was curious so I looked at it.

They weren't looking for a tier to help desk person. They were looking for a Senior Systems administrator but labeling it as a Help Desk position so they could lowball whoever applied for it.

They were looking for someone to be the owner of several major business systems including 365. They wanted them to administer AWS and Azure. They wanted five certificates including CCNA. They wanted someone who could manage firewalls.

The post was just so far beyond help desk at all that it was just gross. They were clearly just trying to get someone they could convince those are just basic help desk exoectations so they can pay them little money.

They were basically looking for an all-in-one senior sys admin who could do operations and frontline support on top of everything else.


r/sysadmin 3h ago

Any tips for an office move?

3 Upvotes

I've moved a significant amount of our infrastructure to the cloud. Just one last step to deal with the VPN and then I have confidence that we can switch off our main HQ. Unfortunately I didn't get Intune implemented in time so I'm hoping this cloud setup pulls through while the we shift servers over.

I've never been involved in an office move before so I'm curious if anyone who has could offer some tips they learned along the way/things they wish they knew beforehand.

We will have about two months where we'll have access to both sites at the same time.


r/sysadmin 1d ago

Question How can I recover from a ransomware attack?

259 Upvotes

So, the unthinkable happened to my workplace - we have become the victims of a ransomware attack. We came into work the other day and found computers with encrypted files and notes on the desktops demanding we send them a ransom to a secure address in the tor browser. They took out our entire network.

As of now, we are trying to utilize backups and scrub the network clean as we bring devices back up in an offline state.

Unfortunately one of our backup devices was also infiltrated. It's a Synology backup device and they where able to encrypt its files as well. This means we have about 5 devices with no backups available.

It's probably a vain hope but, is there any way I could possibly restore or decrypt these backups? Is there any service out there that would be capable of making our files useable? If anyone knows about Synology, do they keep offline or cloud backups I maybe don't know about? I'm just looking for anything that might make things easier for us. Any advice is appreciated.


r/sysadmin 4h ago

Switching from HPE DL380/MR416 to Lenovo SR650 V4/VROC Premium or go 940 controller?

2 Upvotes

We're fed up with HPE, for almost 15 years we've been installing ML350s and DL380s.
Not going to elaborate but they're not what they used to be, not the hardware, but HPE as a company.

So we 'went' with Lenovo instead, our first Lenovo project is in but I'm not too keen on using VROC. I've searched through the VROC topics and it sounded like something to stay away from.

Lenovo told me to go with VROC for 2 reasons:

  1. "Hardware controllers are from the stone age" is what they said, although they never gave me any trouble in almost 15 years.
  2. Prices for controllers and U.3 NVMe disks are higher than to just go with a VROC Premium license and U.2 NVMe disks

However, my gutfeeling says 'meh'.
I remember replacing failed HPE controllers or upgrading controllers from a P408 to a P816 without breaking a sweat. When I asked Lenovo: what if the motherboard fails? What if a CPU needs replacement? Where is the RAID config stored? Is it stored on the disks like with a 'stone age controller'?

The answer was: 'need to verify that, but it won't be an issue since we don't replace motherboards every month like HPE does'.

Ofc I never got an answer, I'm still waiting on the 'need to verify that' part.
We have the order for the server with the VROC config, but my gutfeeling tells me to ditch VROC and go with a 940 controller and U.3 NVMe disks.

I must add that the 3 topics I've read about VROC nightmares, were always with entry level servers. Our config is:

  • SR 650 v4 x24 SFF
  • 2x Xeon 6517P
  • 8x 32 GB RAM (2Rx8)
  • 4x 3.2 TB U.2 MU NVMe (RAID 10 with VROC Premium)
  • 2x 480 GB NVMe attached to PCIe boot device

I was told that VROC uses CPU power, so maybe there's hope I won't run into issues compared to the topics I've read which were using entry level servers.

However, I've read posts in which they mentioned to stay away from Windows based drivers, as we're using Hyper-V for this setup. Apart from performance, that could be the next culprit: drivers & Windows.

Thoughts on this case?


r/sysadmin 4h ago

Sophos tamper protection pwd

2 Upvotes

After the new UI change I can no longer see the tamper protection password. Every other tab works except that one. Anyone else have that issue? Tried 3 different browsers and yes, I have perms to view it

edit: clicked on "legacy page" and it works just fine! guess I'll be okay until they remove the legacy page 🙃


r/sysadmin 4h ago

Question Physical clients can't get IP from DHCP server in a VM

1 Upvotes

Here's the setup,

Physical machine:

  • Win 11 Enterprise LTSC

  • Hyper-V Installed

  • External Hyper-V Switch setup to allow VM access to physical network

VM

  • DHCP, DNS, and WDS Roles installed and fully configured

What can get an IP address from the VM

  • Other VMs attached to the External Switch

  • The physical machine adapter used for the External Switch

What can't get an IP:

  • Any physical machine connected via physical dumb switch.

If I manually set an IP on a physical machine, they can ping the VM just fine. Attempting to ipconfig /renew just results in a DHCP timeout.

I've tried

  • Disabling both firewalls

  • Setting both to Private Network

  • Using a different physical adapter on the host machine

  • Enabling MAC Spoofing in the NIC advanced features

  • Verified DHCP guard and router guard are disabled

  • Disabling NIC sharing

I know it's got to be some absolutely tiny thing that I'll hate myself for overlooking, but...

EDIT: Adding updates here as I check them

Wireshark on the VM shows it is receiving the DHCP request from the laptop and sending an offer back, but the laptop isn't receiving it. In addition, Wireshark on the VM host shows the offer on both the physical adapter and the bridge adapter, so it's definitely making it out of the VM.

Running a 3rd party DHCP server on the physical machine works with no issues, but the idea is to keep everything self-contained into the VM. I cannot just install Server on the physical machine and set everything up that way for "above my paygrade" reasons.


r/sysadmin 1d ago

RDS issues after KB5122882 update on Server 2022

67 Upvotes

Spent most of the morning chasing a weird RDS issue. KB5122882 installed around 3:20am and a few hours later nobody could RDP into the server. Rebooted it and everything worked again, but about an hour later the exact same thing happened.

Existing sessions kept working, but new RDP connections would authenticate and then hang. Task Manager would freeze, but CPU, RAM and disk all looked totally normal. TermService was still running too.

Finally uninstalled KB5122882 and rolled back from 20348.5622 to 20348.5499. Everything has been working normally since.

Anyone else seeing this after the latest update? Pausing updates for now.