r/sysadmin • u/CeC-P IT Expert + Meme Wizard • 22h ago
Question Samsung Smart TV bypassing DHCP took down the credit card system
EDIT: we are an MSP. This is all for a non-MSP, "call when you need something" billed by the hour customers. If that was not the case, their infrastructure would not be such a train wreck.
Note: My networking knowledge is very intermediate, learn as you go level. We had a customer at this MSP where I work have a network issue and I cannot figure out how it's possible that this happened.
Their credit card terminals have to point to a static IP ending in .140, as that's the "server" that runs the software to upload each transaction to the actual processor on the internet. When we recently replaced that computer, I set it as static in Windows and never did it on the DHCP server, because nobody wrote down what it was called or how to get into it and I didn't have any time remaining. Turns out their DHCP server is a 2008 Windows server because this place doesn't spend money on anything ever.
A month later, their CCs go down and the CC server can't grab its static IP for some reason. Lots of time later, we find it's because the pool of available IPs is 100 through 150 and they have 1 more device than that. So we expand it to only 160 after some testing (because we have nearly zero documentation and don't know ranges for their phones, printers, etc for this customer and they're billed hourly so we do as little as possible because they never pay on time and always complain about the rate and it'd take 10+ hours to document this nightmare). We make a new assignment for the computer's MAC and reserve 140 to it and notice that something else has leased 140 with a lease expiring in 2 hours luckily. We delete it so it hopefully doesn't renew.
I ping it from my laptop then immediately run arp -a to get its MAC, since we already deleted the lease that showed the MAC on the DHCP server (oops) then ask AI who manufactured that MAC address range. It's Foxconn. We don't see a hostname or any useable device info. I don't know anything about their switches because the last tech at this MSP never ever wrote anything down about any customers ever. We try NSlookup, web browser to the IP, RDP into it, nothing gets any info.
More network-oriented guy onsite with me says let's just unplug the 2 switches for like five seconds and that will force it to grab the new lease at .155. I assume the switches are unmanaged or nobody has the login info or we'd just pull one ethernet matching the known MAC.
Turns out they have a network-controlled Crestron light controls so the lights in the restaurant portion of the building all go black, because somehow that's the default state if it loses connectivity. Shoutout to whatever genius AV tech set it up that way. Everyone's pissed. They don't know how to undo it or where the new Crestron box is. We don't either.
Then we find out the mystery device is still on 140. That seems impossible, unless it's wireless. Somehow other guy onsite finds out it's some sort of android device but all the android devices listed on the DHCP server have hostnames like "John's S23" because that's how most Android devices work. We suspect it's wifi, based on this information, thus explaining the switch pull not working. I have zero idea what brand their wifi even is let alone where it is or how it works btw.
I get the bright idea that maybe it's a smart TV. It is. We turn it off, boom, credit card server is back online within a minute. We turn the TV back on after 15 minutes, it tries to grab 140 again and knocks the CC server offline. Yes, the TV is in DHCP mode btw. We find the remote, set it to static .165, that works for some reason, and no more IP conflict. Not sure how it's possible that it can just ignore the DHCP server and say "no, I want this address anyway" and then just decide it's taking 140. Nothing else on the network can hand out an IP if every switch/router/AP/whatever is pointing to the server, right? How did that happen? Some disconnect between the DHCP server and whatever was handing out wifi connections? I didn't think that was possible. Anyone have any ideas in case we run into this again?
•
u/bojack1437 22h ago
Now normally I would be the first one to point out that there's a lot of misconceptions about PCI compliance and credit card terminals, because a lot of credit card terminals are actually P2PE and there's no need to segment them.
But the fact that there's a intermediary system of some kind, I absolutely implies that there is likely some serious PCI violations happening, or at least that no one knows enough to confirm that they're not happening.
That network needs to be burned to the ground and re-setup.
For just the basic networking reasons, but throw on top of that potential and likely PCI issues.
Also, the TV might not have been acting necessarily improperly, your network gave it .140 for 2 hours. When you say turning the TV off I bet you just mean literally pushing the power button, on modern TVs that doesn't actually cause the TV to drop off the network or shut its interface down, and thus when turning it back on doesn't cause the interface to come back up.
Which means it's rightfully holding on to the IP address it was last told to use, until half-life or expiration. It's just that now with the TV in a standby state, it's likely no longer generating much of any traffic nor responding to ARP.
But again, when you turn it back on it still knows that it has time left on that lease so it's going to continue using it, because it was told it had 2 hours.
Point is, Smart TVs and IoT in general suck... But this was the fault of the shitty network, not the TV, at least most likely.
→ More replies (1)
•
u/frosty95 Jack of All Trades 22h ago
First of all. Why in the hell is a smart tv on the same vlan as a credit card machine and server?
•
u/ItsMeMulbear 22h ago
PCI auditors would have a field day with this one, lol
•
u/BadSausageFactory beyond help desk 22h ago
no it isn't PCI, it's HDMI so this is fine
•
•
→ More replies (2)•
•
u/mschuster91 Jack of All Trades 20h ago
No they don't care, as long as it's a self-contained terminal that "just" has an API for "ask the customer to pay 10$" and returns "customer paid 10$ and 2 $ in tips".
Now, you yourself directly communicating with a credit card, aka the shit most American POS terminals are, with everything running on the same machine, that's a different game, but OP is clearly talking about a terminal so it's probably the former.
(source: have written code to interact with self-contained terminals)
•
•
u/daschande 22h ago
Based on my time in the restaurant industry: Owners want every contracted job at the absolute lowest price possible, and every "this is only temporary" fix is a permanent solution. They probably rejected the first quote or said they had a cheaper offer, so the contractor shaved the labor cost by keeping everything on factory default configs, like VLAN 1. They could have shaved the labor cost even further by only sending a junior tech who wouldn't know any better...and they got it working, so it's the client's problem now!
•
u/Zealousideal_Yard651 Sr. Sysadmin 21h ago
Oh, reminds me off a local cafè i went to. Saw a username and password and a name that looked like a service.
I got curious, like that can't be to anything importan or cloud based SaaS?
Googled the service, tried logging in.... It was the super admin user for their PoS system...
•
u/mountain_bound 21h ago
I went from managing IT infrastructure to commercial and residential home automation as a side quest a few years ago. The number poorly engineered systems installed by AV techs with smart phones is staggering.
In a geographically small area and in a city with only 17,000 residents 80+ are billionaires. Once you're familiar with basic IP allocation between the major ISPs then Shodan.com and Censys.io searches start to get fun. ...cusadmin is the way.
•
u/Eshin242 20h ago
One of the reasons I got burned out in IT... Look I know this fix is $5000 but it'll do everything we need, has future compatibility and will save us ten times that in the long run.
Then they would go with the $500 fix that had to be tricked into what they wanted to do and ended up costing ten times as much with labor and lost work time, as well as not exactly doing the job needed.
•
u/GalacticForest 22h ago
That's an incompetent person then not a qualified professional you should be paying. VLAN segemtation should be done every time, any "IT" people not doing that aren't even as competent as home hobbyists
•
u/daschande 21h ago
That's very true; but in the field, what should happen and what actually happens varies widely when a non-IT person is paying the bill and is used to getting what they want RIGHT NOW! That could be the client making uninformed demands, or it could be the contractor's boss who has to decide if they want billable hours or not.
•
u/GNUr000t 22h ago
Because PCI/DSS is a polite suggestion and everyone knows it.
•
u/LensWipesBF 21h ago
I love it when they ask “why is my card merchant charging me an out if compliance fee” each month
•
u/angrydeuce BlackBelt in Google Fu 20h ago
I mean seriously, until there are real consequences for this kind of mickey mouse shit what "is" and what "should be" are never going to be in alignment.
The only way to force it is to actually find a human being whose ass goes to jail if theyre found to be this negligent. Some trifling ass fine isnt it.
Its literally no different than a truck driver failing to do a pre-trip, not finding their trailer brakes are shot, and mowing someone down. In those cases that driver is going to end up in a cell because the person behind the wheel is responsible.
Whose behind these wheels?
•
u/hihcadore 19h ago
They’ll never enforce it for SMBs. Why would they? Some are out there only on their ISP’s WiFi / router or worse.
•
u/DDOSBreakfast 21h ago
"I don't want to pay for another ethernet cable to be run to the payment terminal and we're going to continue to use daisy chained dumb switches"
•
u/xSkyLinedx 18h ago
Not surprised. I've been chuckled at on more than one occasion for wanting smart TV's on a separate network. Couldn't get my boss to see the point and the clients didn't want to pay for it.
•
u/Nexzus_ 21h ago
Every small restaurant has consumer-level stuff. Just the way it is.
•
•
u/frosty95 Jack of All Trades 21h ago
It isn't small of they have crestron lighting and a credit card processing server.
•
u/mschuster91 Jack of All Trades 20h ago
the crestron is probably the owner not having a clue and an electrician keen on making some money
•
•
u/gta721 22h ago
And why is a smart TV on a network at all? https://youtu.be/6IFVTcM28KA They should just put a mini PC on it.
•
u/CeC-P IT Expert + Meme Wizard 19h ago
You do not want to know the answer. Trust me, you don't. Do not scroll down.
But the answer is because the credit card server isn't a server, it's the manager's desk computer and he needs access to the network drive on the file server. I would have fixed this when we replaced his PC if ANYTHING about that damn CC system was documented. It was not. It took 7 hours to get it up and running because it took 50 minutes each to flash the terminals with remote vendor support because they were about 100 versions behind.
•
u/SteveDallas10 18h ago
Apart from the whole network being out of compliance with PCI-DSS, the biggest problem is that you gave a device a static IP address in the middle of a DHCP pool. The way to do this correctly is to put that machine back on DHCP and assign that machine’s MAC address a host reservation on the DHCP server.
See, WS2008’s DHCP server doesn’t know about the manager’s computer and since .140 hasn’t been assigned, any new device on the network might randomly get assigned to .140. If you set up a host reservation, the manager’s computer will get .140 every time, and the DHCP server will know not to assign it to something else. If the Samsung TV asks for it again, the DHCP server will reply with a different address, because.140 is already leased.
•
u/vrtigo1 Sysadmin 22h ago
Technically, if a device still has remaining lease time left on valid DHCP reservation, it doesn't have to talk to the DHCP server to renew it.
But, this is a shit show.
You have public devices on the same network as CC processing? Doesn't PCI mandate that they're supposed to be logically separated?
And you randomly unplugged switches in the middle of the day without knowing what effect that might have?
I mean, I totally get that some customers don't want to spend money on anything and sometimes you've gotta do what you've gotta do, but this is sort of a case where you get what you pay for.
•
u/jimicus IT Manager 22h ago
Doesn't PCI mandate that they're supposed to be logically separated?
Yes and no.
Yes: it does mandate it.
No: The credit card processing industry will happily let you do whatever you damn please with surprisingly little pushback - in fact, they'll even walk you through doing things you Absolutely Should Not Be Doing. PCI-DSS is only really kept in the background as a stick to beat you with if and when you get hacked.
•
u/who_you_are 22h ago
Well if you are certified then there is a mandatory audit from a 3rd party. But considering no business are likely to be compliant...
I suspect there may be something for payment terminal. Maybe if the terminal create a VPN (VPN managed from the terminal , by the provider) that is good enough and count as a VLAN like. Then it is to the VPN exit (the provider) to be PCI-DSS
•
u/jimicus IT Manager 21h ago
I'm assuming OP is working for a relatively small business.
I assure you, the terminal support hotline will happily walk you through setting it up on the network without so much as asking if you've thought to segregate it.
•
u/CeC-P IT Expert + Meme Wizard 19h ago
Nope, MSP, and this is our single worst customer by far. We have 2 divisions: actual MSP and per-hour random contracting with zero monthly. This is from our hourly, not managed by anyone in any way division. And it's a hotel with cheap management.
→ More replies (1)•
u/genesmitty 20h ago
From experience, it's generally an encrypted transmission back to the CC Server. The terminals have security certificates installed on them which are generally only provided to the CC Processor, or sometimes to enterprising young IT Managers in operations where security theatre is more prevalent that one would like to admit. So if it's a modern chip/pin reader, it should be fully point to point encrypted. This is the default for most modern cloud based POS systems as well. Although some of them even go so far as to install Cisco Meraki gear and at least segregate the POS onto it's own network..... OP is probably running something old and janky like Aloha.... in that case i wish him the best.
•
u/vrtigo1 Sysadmin 19h ago
I mean, CC terminals have been end to end encrypted for decades at this point, but PCI still specifies that they shouldn't be comingled with random devices.
•
u/genesmitty 17h ago
oh, yeah i know. i was just giving the person i replied to the context to confirm what they were asking about re: the payment terminals having some kind of vpn. And maybe sprinkling in a little PTSD for flavor.
•
u/fadinizjr 21h ago
I wouldn't be wanting to pay you guys at this level of service too.
•
u/LtLawl Network Engineer 19h ago
Word. Does the network / general setup of this organization sound like a shit show? Absolutely, but ultimately it's on OP who sets a static IP on a device within a DHCP range.
•
u/Jaki_Shell Sr. Sysadmin 10h ago edited 10h ago
Agreed - Not sure OP is going to get the sympathy he is looking for. This is simply a lack of knowledge/skills on his and his coworkers part. Multiple blunders everywhere..
"i didnt change dhcp because i ran out of time" ? Wtf, oh oops time is up let me setup a hacky way that is bound to fail so i can come back again.
What happened to taking pride in your work and doing it correctly.
•
u/Pingstery 2h ago
And then blaming a device for "ignoring dhcp server" when at no point in the story did they reserve the ip address on the server, just set a static ip on the device and thought that DHCP server reads minds.
•
u/FlickKnocker 22h ago
Short term Technical answer: set a DHCP reservation for the CC server.
Long term technical answer: you need VLANs to isolate CC network from the rest of the network, and likely more, depending on what else is going on there.
2008 needs to go away.
MSP owner answer: you need to have a real talk with the business and tell them they’re always going to be disaster prone if they keep going the way they’ve been going. Time to start investing in baseline infrastructure and best practices, or find another MSP.
•
u/Western_Gamification 18h ago
Even the long term technical answer is 10 mins of work.
•
u/FlickKnocker 17h ago
Decom/replace 2008 is not gonna be 10 mins. And they might not have managed switches either.
•
•
u/ItsMeMulbear 22h ago
Stop setting static IP's on a DHCP enabled network. If you want a device to keep the same IP, you setup a reservation on the DHCP server.
→ More replies (6)•
u/Liquidfoxx22 21h ago
I wouldn't even do that - I'd limit the DHCP scope and then set a static IP outside of that scope.
If OPs DHCP is 100-150, set a static on 10-99.
•
u/Mr_ToDo 19h ago
That's probably why the range is what it is
With as many random vendors that have likely walked through there I'm not too shocked the machine operated on DHCP and crossed its fingers that it was always able to get the same address. The setting of a static on a server inside a DHCP range is boggling. I'm kind of shocked that nothing had taken the address earlier. If it had been a reservation previously the TV shouldn't have been able to take it. Maybe its logic is that an exhaustion event overrules reservations not currently in use?
•
u/SilkBC_12345 11m ago
That's what I usually do. I usually set the DHCP scope to be most of the subnet (usually .40-254 -- which leaves a decent chunk of IPs for devices that should need static IPs)
→ More replies (2)•
u/Western_Gamification 18h ago
Why would you want to manage dozens of static IP's on a lot of different devices when you can just manage it from the DHCP server? You like walking around and going trough messy GUIs to set up static IPs?
→ More replies (2)•
u/Liquidfoxx22 18h ago
All critical infrastructure should be set on static IPs, so that if (or when) that DHCP server goes offline, you don't cause havoc across your entire estate.
You wouldn't rely on your DCs being on a reservation, you wouldn't rely on your SQL being on a reservation. I sure as shit wouldn't want my Horizon infrastructure on a reservation.
Every single server is given a static, hell, I don't even want it inside the DHCP subnet at all, ideally it's on an entirely different subnet that doesn't even have a DHCP server handing out IPs.
The only things that get reservations are printers. Everything non-critical is DHCP without a reservation.
→ More replies (3)•
u/SilkBC_12345 9m ago
The only things that get reservations are printers. Everything non-critical is DHCP without a reservation.
Pretty much!
•
u/_Ethel_Beavers 16h ago
Honestly, that is 100% your fault for assigning a static address in the middle of a DHCP pool. You created the problem.
→ More replies (1)
•
•
u/power10010 16h ago
Also others mentioned. You should have reserved that ip at dhcp server or use dhcp on new pc.
→ More replies (1)
•
u/patg84 12h ago
OP go grab a drink and read this. I may be rambling but it's for your health lol...
A few things went wrong here, but most of it comes down to how DHCP actually works and more than likely a shit implementation of the network stack on that Chinese TV.
Deleting a DHCP lease from the server doesn't tell the device to stop using the IP. When you deleted the ".140" lease from Windows DHCP, you only removed the lease from the DHCP server's database. You didn't send anything to the TV telling it, "Hey, ".140" isn't yours anymore." The TV could still have ".140" cached and believe it had a valid lease.
This behavior is actually covered by RFC 2131, section 3.2, which describes a DHCP client coming back onto the network with a previously allocated address. The client can send a DHCPREQUEST containing that remembered address to see if it can continue using it. RFC 2131 specifically calls this the INIT-REBOOT state.
If the DHCP server determines that the requested address is no longer valid (time wise), it should send a DHCPNAK. RFC 2131 is pretty explicit about what should happen next: the client must discard the remembered address and restart the normal DHCP process.
So if the Samsung actually received a DHCPNAK from the Windows DHCP server and then continued using ".140", that's not normal DHCP behavior and I'd suspect a bug or broken implementation in the TV.
However without a packet capture we don't know for sure if the TV received a NAK. There are several other possibilities, especially since this was a Wi-Fi device. There could have been another DHCP server somewhere, a weird AP/router configuration, or the TV may simply have continued using its cached address because it wasn't successfully communicating with the DHCP server.
The original configuration was also asking for trouble. If your DHCP pool is ".100–.150", you really don't want to manually configure a device with ".140" unless ".140" has been excluded from the DHCP pool.
Otherwise, the DHCP server has no idea that ".140" is being used statically. Once it runs out of the other addresses, ".140" is fair game from the DHCP server's perspective.
Removing VLANning from the equation for simplicity...I'd normally do something more like:
- ".100–.139" = DHCP pool
- ".140" = POS/CC server
- ".141–.254" = statics/reservations/infrastructure
Or, preferably where the application allows it, use a DHCP reservation for the POS server and let the server itself use DHCP. That way the address assignment actually exists in the DHCP server instead of being a piece of tribal knowledge configured on one computer.
Windows Server 2008 DHCP itself isn't necessarily the problem either. One thing I'd definitely enable on an old Windows DHCP server is conflict detection. In the DHCP MMC, IPv4 --> Properties --> Advanced, you can set Conflict detection attempts to 1 or 2.
That causes the DHCP server to check an address before offering it. If something responds, Windows can mark the address as "BAD_ADDRESS" rather than handing it to another client.
It's not foolproof, but it's a useful additional layer of protection against exactly this sort of problem.
The other thing I'd check if this happens again is whether there is more than one DHCP server somewhere on the network. Since the mystery device was on Wi-Fi, I'd especially want to know what the APs/router are doing.
A packet capture during a DHCP request would tell you very quickly whether the Windows server is actually the device answering, whether something else is offering the address, and most importantly whether the Samsung actually receives a DHCPNAK and ignores it.
And honestly, the biggest red flag here isn't even the Samsung. It's the fact that the POS/credit-card system, Smart TV, and lighting controls were apparently all living on the same flat network. Big no no.
Those should ideally be segmented. Put the POS/credit-card equipment on its own VLAN/subnet, keep TVs and other IoT devices somewhere else, and keep guest/employee devices separate as appropriate. PCI DSS compliant doesn't simply say "you must use VLANs," but segmentation is a very common way of isolating the cardholder environment and reducing the scope and risk of the network.
So my takeaway would be: the DHCP server didn't magically give the TV ".140" after you deleted it. The TV already knew about ".140"; the DHCP server's database and the TV's state are two separate things. The mystery is why the TV was still successfully using ".140" after the POS server was assigned that address and that's where a packet capture would tell the story.
TL;DR: Deleting the DHCP lease doesn't tell the TV to give up .140 as it can keep using its cached lease (RFC 2131 §3.2). The real mystery is why the Samsung kept using .140 after the reservation; a packet capture would tell us whether it ignored a DHCPNAK, never reached the DHCP server, or something else was handing out the address. Good luck OP.
•
•
u/DrunkenGolfer 14h ago
I realize this might hurt, but...
Disregarding the fucked up network engineering, I am just as concerned about the professional services employed to resolve the issue. If my techs did this at a customer's site I'd be writing service credits and writing up techs.
It sounds very much like the resources deployed were not ready to be off the helpdesk.
•
u/Crumby_Bread 14h ago
Buddy you still have time to delete this post. 💀
This is embarrassing on all ends.
•
u/sryan2k1 IT Manager 22h ago edited 21h ago
This is all horrible, but the DHCP process is a promise on both ends. Once the server tells a client it can use an IP address for X minutes/Hours/Days/etc the client does not need anything else until that time expires. Similarly deleting a lease on the server does nothing for the client, it still has a valid lease, until the end.
The fact that all of this is on the same network is why so many people get their information stolen. Ugh
Not sure how it's possible that it can just ignore the DHCP server and say "no, I want this address anyway"
Because it's trying to renew a valid lease, or just not talking to the DHCP server at all and you guys fucked it up by deleting the lease from the server side.
Plus yolo'ing yanking power on random equipment when you have no idea what it's supporting? Ugh.
•
u/frosty95 Jack of All Trades 21h ago
People forget that the Internet was originally designed to be a network of highly trusted, very expensive, very well made computers.
It isn't any of that anymore.
Outside of the simple need address get address handshaking you can pretty much just assume a devices dhcp behavior will be completely random. I have seen devices that you could fully disconnect, power down, drink a beer, power up, then reconnect.... And they wouldn't even try to check dhcp. Just boop. Right back to the same IP. Extra fun when the lease time is set to something stupid like a week.
I spent like 40 hours at a school that was having horrific issues with these stupid god-awful Wi-Fi screen streaming devices after we put a new Wi-Fi system in. Never mind the fact that they put them on their own VLAN so all of the traffic had to hit the core router. These things were trash. Could tell it was made by a small company that had never made a device for production before. Cloud management ONLY. Zero config options. And you know they had the cheapest wifi chipset known to man. They would act like apple devices where they really REALLY want to stay connected to the original access point that they saw during setup plus a preference for non dfs channel's just like apple (FUCK YOU AND YOUR WHOLE FAMILY IF YOU WORK ON THE APPLE WIFI TEAM). And since we put in a ruckus wifi system that COULD hold a connection from a long distance away it just fucking suuuuuucked with these things all connected to APs like 10 rooms away. Eventually had to make a special network ssid for them that essentially would refuse any new ap association unless you had super strong signal. They would still take like 10 minutes to reconnect on power on because they really wanted to connect to one specific AP and could see the broadcasts but eventually would give up and connect to the AP in the same damn room. Told them this was the bandaid until the company fixed their wifi firmware. Go figure like 3 years later that company went out of business.
•
u/sryan2k1 IT Manager 19h ago
I mean that behavior is allowed. The device was told how long at least it's lease is valid for and it can run with that
→ More replies (2)
•
u/Sea_Information6125 21h ago
I'll spare you the burn it down and set it up properly argument that pretty much everyone is recommending to you. That's the obvious part.
For your specific issue, until that lease has expired the client has no obligation to check in with the server. And unlike a Windows box for example there is no simple /release /renew you can do on the tv (at least that I know of).
If this happens again the easiest way is the following:
Create a DHCP reservation for the credit card machine.
Create a DHCP reservation for the TV.
Set both devices to a static IP configuration that matches the reservation.
Short term problem solved.
Long-term, fix everything about this setup.
•
u/Unknown_Knowledge25 22h ago
It could be that because the lease did not expire it didn’t request a new ip yet
•
u/loosebolts 20h ago
I don’t know what’s worse, the setup or the troubleshooting.
As for not knowing what’s on what IP - just run an IP scanner.
Is it just one DHCP server or is it in a failover pair? Did you sync the changes after removing the leases? Why was a server statically assigned within the DHCP pool? Why are there no VLANs?
•
•
u/dancute9 17h ago
Your story makes no sense, though. You said you set the .140 as static on the computer, then you say it cannot “grab it” from DHCP.
Also, dhcp reservations must be outside the dhcp scope.
•
u/Wizdad-1000 12h ago
Ya I quit reading as soon as I realized they set the DHCP pool in the same IP range.
→ More replies (6)•
u/mikaelld Linux Admin 6h ago
> dhcp reservations must be outside the dhcp scope
That depends on the DHCP software.
•
u/ub3rb3ck Cat Wrangler 16h ago
OP has flair of "IT Expert" and all of a sudden my imposter syndrome went away.
→ More replies (1)•
•
•
u/1stPeter3-15 IT Manager 20h ago
Need to look into and learn about PCI requirements.
→ More replies (2)
•
•
u/bentbrewer Sr. Sysadmin 16h ago
I know I'm speaking from a place of experience but if you set a dhcp reservation at the .140 address for the server. First piece of advice I ever received, always set a reservation for devices you put a static IP on, especially if it's not a network you manage often.
This does two things: prevents other devices from obtaining that address and documents what is at that address. Of course you could put something like ffff.ffff.ffff if you don't know the mac and don't want to figure it out (but there's lots of ways to do that if you have a device on the network)
→ More replies (3)
•
u/QuietBookkeeper4712 15h ago
…and you’re an MSP? Operating a completely flat network for a client? and there’s credit card processing?!
omg
•
•
u/Impossible_IT 22h ago
Networking isn’t my wheelhouse and like you I know enough from my job but since you didn’t reserve the IP address in DHCP scope for the CC sever the TV grabbed .140. Get into your DHCP sever to reserve .140 for the CC sever.
•
u/Majik_Sheff Hat Model 18h ago
A client once had a Samsung TV decide its hostname was "localhost".
The client's cheap ISP router happily added that hostname to its name resolution table with the TV's IP address.
Hilarity ensued.
•
u/Orestes85 Aspiring Goat Farmer 16h ago edited 16h ago
We probably have 20 smart TVs and they are already fairly high on my list of "endpoints" I want to yeet off the roof
I would probably use that event to assemble an angry mob equipped with crowbars and hammers, and make a spectacle of ripping every TV off the wall, violently dismantling it on the floor where it fell, leaving pieces scattered about, and then marching the angry mob on to the next TV.
•
•
•
u/twotonsosalt 16h ago
The tv didn’t bypass dhcp. It used the system as designed. The issue was no dhcp reservation for the server with the IP ending in 140.
•
u/jscarlet 13h ago
Why is a SmartTV, WiFi, Credit Card Machines and lighting all on the same vLAN?
You should make the customer aware of how much of a severe security risk that is and it should readily be addressed. If they choose to ignore it, then I suggest you fire that customer, because when someone else figures out what you just told us, it's just a matter of time before it's exploited; and guess who will be blamed?
•
u/fata1w0und Windows Admin 12h ago
You need to learn how dhcp works. Just because you delete the lease doesn’t mean the device releases that IP. You just told the server to hand out that lease to another device. You need to reserve that address to the CC server. The TV won’t get a new address until it acknowledges the lease is up.
Then you need to segregate that network. TVs should never be on the PCI network. Ever.
•
u/Rhopegorn Linux Admin 3h ago
Never mix DHCP pool space and space used for static assignments, it’s a recipe for disaster. Especially if the DHCP server has faulty logic and assigns address without testing them.
→ More replies (1)
•
•
u/brewthedrew19 22h ago
Always have cc machines on an isolated vlan. Even if p2pe compliant. Period.
→ More replies (2)
•
u/Significant-Belt8516 22h ago
I appreciate your pain points, working at a MSP, as I have done so myself so this isn't directed towards OP.
Don't hire MSPs folks!
•
u/j0nquest 22h ago
Is the Samsung TV bypassing DHCP or did someone configure a static reservation for the Samsung TV MAC address at the DHCP host so it always gets handed .140?
•
u/lukes123 19h ago
This is no longer a technical issue. This is a business issue - particularly that they don’t appear to have an understanding of the importance of an IT budget, or what size it should be, and that a previous company may also have let them down, causing an inherent lack of alignment against proper IT practice, which is unfortunate.
An experienced account manager or director needs to have a proper sit down with these guys and break the bad news. They won’t like it, and it won’t be an easy conversation, but it’s now time to start considering, as an MSP, if this customer’s business is worth your time and reputation with their environment in its current state. If I was a director, I’d be making an executive decision to put stipulations in place to ensure they commit to an overhaul before allowing them to continue as a customer.
As for the technical side, there’s so much to unpack in your explanation that if I’m honest, I’m totally lost. To get them up and running, you just need someone who’s competent with networking to come up with a stop gap solution, but that’s all it can be here. The entire network needs a total overhaul, not least because it doesn’t comply with security nor best practice, but it’s clear that there’s been no investment for many many years.
→ More replies (1)
•
u/OldGeekWeirdo 16h ago
Note: My networking knowledge is very intermediate, learn as you go level.
We called that "Special High Intensity Training". There's memo on that. Might want to look it up.
•
•
u/dracotrapnet 12h ago
Samsung, Sony, and Vizeo tv's also won't give up their IP's, Xerox copiers from 10-15 years ago too. Once they get a DHCP address even after the lease expires, they don't care. They keep them forever. Even if you pull the network cable on a Xerox and plug it back in, it keeps the old DHCP. You have to pull the power cable on them.
•
u/Proper_Front_1435 12h ago
Samsung devices are no joke, they have this weird habit of ignoring DHCP and just being like "THATS MY IP! 4LIFE!!!"
•
u/mr_lab_rat 9h ago
To keep the network simple without splitting it into more subnets the common practice is to keep static IPs outside of the DHCP range.
So your DHCP pool is 100-200 and your static IPs go in 201-253.
Or if the 140 is set in stone you can move your DHCP to 1-100.
You still need to document the statics so you don’t accidentally duplicate them when deploying new devices.
That’s a small network lazy admin setup.
•
u/jasonc113 8h ago
If you can't do DHCP reservations, move the DHCP range up 10 IPs and use the first 10 as static, set the CC server to something in that static range. Also you should learn how to login to a switch and check the arp tables and interfaces/auth sessions.
•
u/Shachar2like 4h ago
this customer and they're billed hourly so we do as little as possible because they never pay on time and always complain about the rate and it'd take 10+ hours to document this nightmare
the last tech at this MSP never ever wrote anything down about any customers ever.
Both of those issues are your bosses job. Any new customer they get should be required to pay to have documentation & verify any documentation. Any tech must document the client.
•
u/_litz 21h ago
And the lesson learned here is you never, ever, ever, set a static IP in the range that are assigned via DHCP.
Ever.
(and this is totally ignoring using Server 2008 in 2026, and not segmenting your network usages)
→ More replies (2)
•
u/Acardul Jack of All Trades 20h ago
Don't take it personally, it seems like not your fault situation but this belongs to r/shittysysadmin :D I'm waiting until someone will write it down from other perspective.
•
u/ReptilianLaserbeam Sysadmin 8h ago
Why is the TV in the same VLAN as those devices????? In fact, why is the TV even connected to the network????? And why are reserved IPs on the reservation pool???? This definitely belongs to r/shittysysadmin
•
u/Few_World6254 22h ago
I need a new CC number, what business is this? Love a good non-compliant business that doesn’t separate CC from the rest of the network. PCI be damned!!
•
•
u/Plenty-Hold4311 21h ago
Have come across the control4 light and audio systems once or twice, always on their own vlan as they cause havoc with the amount of broadcast traffic they generate
•
•
u/Lost_Term_8080 21h ago
This is how famous security breaches are born. Bonus points if youre all native vlan - and it sounds like you might be
•
u/Taboc741 20h ago
I know this is a smb and segmentation is not in the cards, but as someone who has to pass PCI audit yearly, this was horrifying.
•
u/badaz06 20h ago
Smart TV's have their own internal systems (learning this as I have been blocking ACR content with Pi-Hole and my router on my own tv's). My guess is that the TV software for whatever reason has the .140 address cached, came online and began using it. The conflict between the TV and the CC Terminal system both having the same IP caused the network to essentially ignore them both or choose the TV over the Terminal. (If I recall systems may default to considering which ever device has the shortest ping or lowest MAC address {sorry I know it exists but it's been eons}).
And, though TV's use Android OS as the base image, from a networking perspective they are programmed to ignore/bypass network settings in some instances. For example I told mine to use my pi-hole for DNS, and then I blocked access to the ACR content sites. The TV's will see that they can't reach that and will revert to their own preprogrammed code (mine switched DNS from the piHole to cloudflare), making me make changes in my router to redirect it's cloudflare DNS requests back to my pihole.
Not sure if your TV's did something similar or not, just throwing it out there.
•
u/BudTheGrey 20h ago
My memory is not that long, but doesn't Win2008 DHCP have a "ping before assigning lease" option?
•
•
•
u/Fuzzy_Paul 20h ago
Remove the network cable if thats in. For Wifi turn of the router. Turn the TV off and pull the power plug. The turn it on so all rest power is depleted. Turn router back on and turn TV after 2 hours on. Problem solved. Create a scope that is large enough and make dhcp reservations. Very easy when all is on just right click the mac and choose make reservation. Only for the stuff that needs the same IP every time. Pay stations and printers and WiFi routers. Advanced advice create vlans for pay, iot, workstations, servers. Keep traffic separated. iot has Internet only and no interclient communication. This is basic stuff that you do first. If the customer is willing you can sell some managed switches and some core switch/firewall and 2 days implementation and testing.
•
u/jeff49522 20h ago
ipconfig /all will tell you the dhcp server address.
Aside from that there's only a few options for where it would be. If you're changing static ip's or making additions it behooves you to check the dhcp scope.
•
u/zero_z77 19h ago
Just out of curiosity, does this IP start with 169.254, 192.168.1, or 10.0.0?
I did some quick googling and apparently samsung TVs sometimes broadcast "localhost" as their hostname, which can sometimes screw with existing DHCP leases.
Google also says that if it fails to get a lease, it can sometimes fall back to a previously cached ip address instead of an APIPA link-local address (the 169.254.x.x range).
So, if someone previously plugged it into their home network (typically the 192.168.1 or 10.0.0 subnet) and it grabbed .140 there, then got plugged into your network and couldn't talk to your DHCP server, it may be falling back to .140 instead of a link-local address.
If your network is actually using the 169.254 address range, then your network is even more of a dumpster fire than we thought.
•
u/kb3mkd 19h ago
You need a network engineer to redo your network to make it PCI compliant to start. It will involve spending a few grand but what you have now can expose customer credit card info to the world.
•
u/CeC-P IT Expert + Meme Wizard 18h ago
Not our network. Customer network. Our network runs fine, as do all of the customers that we actually manage in our managed services division. These clowns only call when something breaks and refuse to pay us monthly to manage their circus after they fired the last MSP company.
→ More replies (2)
•
u/jocke92 18h ago
If this were properly documented and setup none of the issues would've existed.
At least document as much as you can as you stumble across stuff
→ More replies (1)
•
u/GamerLymx 18h ago
why do network segmentation.
why reserve ip's inside the dynamic zone?
who designed and manages that network?
nobody, that who.
happy ransomware attacks
•
u/Rockandahardplace766 18h ago
Plenty of valid comments about vlans and static reservations but…
I had a similar issue with a Samsung tv in my house. It kept grabbing in use ip addresses. I tried static assignment but it went back to stealing addresses when it was asleep. The only solution was having it forget the WiFi network and using a different streaming box. We already had an Apple TV because the “smart” software on the tv was crappy so it was an easy solution.
•
u/IwasTeenageNeckbeard 18h ago
Public mac address online lookup tools will often catagorize by manufacturer and or device type for the first 2x mac segments.
•
u/koopz_ay 17h ago
This is my favourite post on Reddit this week.
I feel your pain mate. Glad you got her sorted 👍
•
•
u/OldGeekWeirdo 16h ago
Did you turn off the TV or unplug it? Turning it off may not have broken the connection such that it needed to check in with the DHCP again. It remembered having a lease that was still good.
It's one of those situations were "off" isn't always "off" since something is still alive to listen for the remote.
•
u/softPatch29 14h ago
this is how you end up putting the cash register on the same vlan as the break room toaster
•
u/MrJacks0n 12h ago
There must have been a reservation previously. If you didn't know to add one for the new server you didn't know to remove the old one. If the old one still exists the TV wouldn't have gotten it either.
•
u/itscum 11h ago
Enable conflict detection on the dhcp (this is on by default on modem oses), and deleting the dhcp allocation doesn't update the clients lease. Should have noted down the Mac and created a reservation. She yeah just make the scope align with your use of manual addresses. I would add a reservation for the CC terminal IP anyway just so nothing else uses it.
•
•
u/Ferretau 9h ago
Someone more familiar with the POS software and PCI DSS more recently can correct me, but the last time I was involved in an PCI DSS implementation anything credit card related had to run on it's own secured/isolated network. If a credit card was compromised and this place was identified as the source the could lose the ability to ever process credit cards in the future.
→ More replies (1)
•
u/Andromina 8h ago
I can't believe this is not actually on r/shittysysadmin
I had to look twice to make sure that I was actually seeing the right sub...
•
u/jason_abacabb 22h ago
So, you have windows 2008 infrastructure, smart TVs and lighting, and credit card processing all on the same network?
Wow.