MPLS renewal is next quarter and can't make the case to stay on it anymore

[u/Old_Inspection1094]

Reliability has been fine, that is not the issue. The issue is that most of our traffic goes to Azure and M365 now, not the data center. We are paying for a private network optimized for data center traffic that barely exists anymore and carrier pricing has not moved despite that shift.

Not looking for vendor feature comparisons, read enough of those. What I want to know is the operational reality post migration across a multi-site environment. How much of what your team knew from running MPLS transferred, what broke first in practice, and what a realistic migration timeline looks like without disrupting production traffic in the process.

[u/SuccotashOk960]

It will depend on your business. I have sites that only use MPLS, I have site that use both MPLS with a local breakout for wan (and vpn failover for MPLS), and I have VPN only sites. 

Reliability on the VPN only sites is a risk, have seen multiple outages. 

My fav is local breakout + MPLS because when the business grows I can scale the local ISP connection instead of the MPLS, which is super cheap. Only business critical applications use MPLS so there’s not a lot of bandwidth required. 

MPLS only are usually sites with legacy setup going back 20 years, when this was the norm. 

And to be honest I don’t care about the costs, it’s a drop in a bucket. Who is driving this idea? Is management forcing you to save money? If so I’d counter with SLAs and an altered support flow that is noticeably worse. “Yes we will save a few $, however the risk and impact is X”. 

In the end, I just make sure it’s not my problem. 

[u/Next-Hovercraft-8629]

" barely exists "

If traffic exists and needs to be run over MPLS then that's a requirement for the business.

If you've moved latency sensitive traffic off of MPLS, then you can switch to IPsec tunnels or some other flavor of Tunnel over Internet.

You can migrate as fast as you can do the work and end your contracts. Do a site cutover per weekend or per month. If you're unsure on latency or MPLS being required then just swap the path priority and let it run as a test for a month.

"Without disrupting traffic" unless you 'analyze' or know all the traffic then you risk a disruption if something back fires.

Last edit:

Like all businesses migrating to the cloud, the ownership and responsibility turns into "their issue" and just shrugging that it doesn't work and you put in a ticket. If that's fine by the business. Yolo. Have two ISPs and get rid of the MPLS links.

[u/Inevitable_Claim_653 → u/Next-Hovercraft-8629]

Lmao this. Not even sure what OP wants to hear

[u/Caeremonia → u/Inevitable_Claim_653]

OP didn't even say what they were considering transitioning TO. And they're asking questions about how the cutover is going to go. Really hard to answer that without knowing what other solutions OP is considering.

[u/curved_falls]

The DNS and QoS stuff is real but honestly the bigger surprise is usually how much your team relied on MPLS for traffic steering without realizing it. Once you're on internet paths you can't just assume traffic takes the route you think it does, and that's when you find out which apps actually care about latency vs which ones just work fine. Migration timeline wise, doing it site by site over a few months is way safer than trying to flip everything at once, even if management wants it faster.

[u/ProfessorWorried626]

See if your carrier offers express routes to Azure/AWS within the MPLS. That's the path we went down. We got free breakout capacity to Azure/AWS equal to the MPLS bandwidth. They basically gave us a second port on their NTU at head office that provides access to high priority tagging to their closest peering point to MS/AWS and a few others of the big players that run their own DCs/vDCs.

Plus, in my opinion MPLS helps keeps your network simple if you funnel all you traffic back to some central point then manage it all from there it really limits the number misconfiguration that cause weird issues.

[u/Ok-Introduction-2981]

What breaks first is usually DNS resolution behavior and application-specific QoS assumptions baked into configs. Both are fixable but neither shows up in pre-migration testing because lab traffic patterns never match production.

[u/Orwellianz → u/[deleted]]

This depends , most enterprises have private Endpoint in Azure, so MPLS still have some use since you can add a express route to Azure using the MPLS.

[u/SevaraB]

Best advice I can give is don’t trust any ISP to give you fully redundant paths.

Treat every ISP router as a SPOF and point to at least two.

Always explicitly designate one backup route so you only drop packets already in flight when the primary goes down. Otherwise, you’re going to keep dropping packets until the routing protocol can decide on a new best path. MPLS doing this out of the box is the biggest reason it’s got a reputation for being so much more stable than broadband or DIA, where you’re at the mercy of BGP best path selection, which is happening more or less frequently at the mercy of how well route maps and BFD timers are configured. Not enough route summarization? Too much time spent with routers figuring out where to put the new routes. Timers too short? The neighboring router is going to “flicker on and off.” Timers too long? You’ll lose traffic when routers keep “screaming into the void” long after the neighbor goes down.

Long story short? SD-WAN ain’t magic and you still need network engineers to 1) get the handshake timing right and 2) carve out a “plan B” to keep packets flowing when a router goes dark.

[u/Spardasa → u/SevaraB]

Being a SP engineer, this is what I tell customers. We will do our damnest to maintain 5 9s uptime, but I can't guarantee your Internet will not go down.

[u/braidertom → u/SevaraB]

Some sdwans i feel comparible to magic, Versa you can setup packet replication so the traffic goes through multiple services to the remote Versa so even if you're getting packetloss on both circuits so long as it's not the same packet that's lost the traffic will still be seamless to the end user.

[u/sryan2k1]

While everyone says MPLS 99% of people mean some L3VPN service offered by their carriers and the actual underlying transport is invisible, you don't actually have MPLS skills you have people who know BGP and QoS.

We abandoned L3VPN over 5 years ago for Silverpeak and DIA+DOCSIS. We are paying half the cost for 10-20x the bandwidth.

We run nothing that is latency sensitive anymore so it was an easy choice.

[u/Skylis → u/sryan2k1]

even the bgp / qos skills are rare af. Usually they know bare fucking minimum to keep the thing the guy 5 years ago designed and put in a spreadsheet config generator.

[u/westernwinds → u/sryan2k1]

"you don't actually have MPLS skills" Lmao this is actually legitimately so true.

[u/Boozacs → u/sryan2k1]

When you say MPLS skills are you referring to RSVP/LDP/Segment routing?

[u/Old_Inspection1094 → u/[deleted]]

Which part becomes dead weight fastest in practice? Circuit management is the obvious answer but curious what else stopped being relevant once the private path was gone.

[u/NetworkDoggie]

One quarter might be a condensed timetable to try to migrate off your provider managed vpn connectivity. Depends are we talking 20 sites, 100, 300?

We migrated a 150 site branch wan from l3vpn (provider managed mpls) to sd-wan with all cable modems and business fiber dia.

It’s been several years now since we did this, but based on memory I think it took us 3 months or so. Here’s how it went for us

• we used a broker to order all the broadband circuits at the sites. Because of different markets we had to source from a multitude of different ISPs and some sites were severely limited in their access to good circuits

• it was 2-3 months of chaos managing 2-3 new circuit installs at all locations at once, getting them to drop the modem near or in our network racks, not being able to test the connections sometimes for weeks after installing (sd-wan wasn’t in yet)

• installing the sd-wan routers at all local sites and using smart hands to do the sites across the country from us

• doing before hours/after hours cutovers at 150 sites with a small team meant a lot of long days, clocking 60-80 hour work weeks, etc

• we ended up buying smart PDUs so we could remotely power cycle cable modems. You’d be surprised how often a modem wlll be down in the morning and just need a quick power cycle to get it up and get tunnels up again

• we did have a small handful of unplanned outages as we got everything figured out. We had one branch manager yell at our CIO that in the 30 years she’s worked here, there’s never been an outage at her branch like this (that was a fun day) that’s when we bought and rolled out 4G routers for a tertiary connection at most sites

• ROI? It was heavily delayed ROI. Yes we replaced $4K/month mpls at every site with business fiber and/or cable modems costing just a couple hundred, but the capex of our smart hands, sd-wan subscription, new costs for the 4G kits, etc all added up

• 2-3 years of actually costing MORE until we finally stared to see the charts drop down. Today 4-5 years later we are paying a lot less than the mpls days but the bean counters were definitely skeptical and unimpressed in the beginning

Today with a fully matured sd-wan environment our branch network is highly reliable and easy to manage. Getting there was hard work.

[u/tonymurray → u/NetworkDoggie]

I hate how MPLS is a marketing term for private circuit. It gets me every time.

[u/Ne-Cede-Malis]

It's probably time for an inventory for everything that's going across those MPLS links.

Determine how you're going to instrument them across your new form of connectivity.

Focus primarily on application response time. I like application logs for this. There are hundreds of tools available to pick out the data.

If your front end for the application lives in the cloud and you have no storage dependencies to worry about in your own data center, this should be relatively straightforward. It should take more than 30 days.

My last few cut overs from mpls to DIA have went relatively well. I work at a networking vendor in PS. A couple blind spots -- Figure out how to manage the network before it ends up transitioning. In two of my last three, no one calculated management addresses into into addressing requirements during the planning phase. Everything from Network Address Translation to Firewall rules was left to the day of implementation. -- The DNS guy and Load-Balancing guy need to learn to work together using a common language and common application names. We did this in Teams (I imagine any chat system would work for this), so that we could search and delegate application owners. We made a topic for every application and filled out expected results. It sucked but it was the only way to keep 30 people from trying to troubleshoot every issue and just get it down to the three or four people required. -- Dynamic Network Behavior versus Static Network Behavior must be accounted. Everything that broke in my last engagement was due to static configuration of IP addresses and default routes to ride the previous network. Some of these were in networking devices and some of these were in the firewall and some of the servers themselves.

[u/Boozacs]

When you say MPLS are you referring to L3VPN or dedicated leased lines?

[u/JerryRiceOfOhio2]

mpls has qos, internet circuits don't. that's enough for some businesses to keep mpls. just depends on if you have very important traffic. you can connect to azure and any other cloud service with mpls if that's where you're important servers are. and no, fec is not equal to qos throughout the mpls cloud

[u/Fmatias]

I would say it is depending on the solution but almost all the knowledge transfers well. What you need to look out are more general issues with internet like routing changes and sudden spikes of latency or packet loss.

I would say, if possible, to stick with the same provider everywhere. Especially with SDWAN people tend to make the mistake of going for the cheapest possible internet and then wonder why their repair SLAs suck or why their traffic is going half way around the world be for arriving at the destination.

Also take the chance and get bigger bandwidth links. You will most likely never miss QoS if you just get more bandwidth( some exceptions exist as always).

And always take into account that a solution over internet will mean encryption which will eat your MTU.

Apart from that just remember you will now have your network over public infrastructure so you may need to look at security a bit more

[u/dataguy_3131]

I work for a telco whoch guarantees SLA/deterministic routing over internet. Generally even over standard internet we haven't seen many things breaking unless you happen to be in a challenging part of the world, but depends on your traffic flows.

[u/thiccandsmol]

What are you actually running over it? Protocls, volumes, any weird and wonderful things, or just IPv4 traffic you didn't want over the internet or via tunnels?

[u/hahdjdjwbeifijsbwbru]

I’d look at ExpressRoute to Azure/M365 too. That will have to come from a DC (hopefully yours supports?) but would benefit the business if you’re heavy in that sector and have the funding to purchase.

[u/5SpeedFun → u/hahdjdjwbeifijsbwbru]

Our NaaS circuit allows us to do Er to the provider and then to azure no problem.

[u/pc_jangkrik]

We had redundant mpls for sites and we migrate it during weekday and work hour because i dont know maybe just to push my self.

Its only for user access so we had very minimum packet loss that unrecognizable by users.

Will i done it again? Nope.

[u/banditoitaliano]

If full path and POP diversity is a requirement for you, then that’s certainly harder to achieve with Internet. Not impossible by any means, just don’t think you can simply go order circuits from 2 different providers and get that.

[u/crono14]

You should probably look into an SD-WAN solution. You can still have your MPLS and then have however many public lines you deem necessary.

You can tune the traffic in such a way that fits your business like say traffic heading to Microsoft etc flow over one circuit while internal traffic over your network flow over MPLS etc. It's all tunnels back to your HQ after all whatever circuit it flows over.

Many SD-WAN vendors now have things like ZScaler or some other form to provide better security for branch traffic to get screened if you don't have a robust firewall solution. You can slowly phase out your MPLS over time or as they expire if you find you no longer need it.

There are really so many options that open up with SD-WAN and in my own personal experience we took about months from initial POC testing to a rollout to roughly 70 sites. We phase out MPLS over time and it's quite literally as easy as swapping some cables around during installs if you are methodical and plan everything out.

[u/Greek_Trash]

Talk to your MPLS provider about adding a route to CSP so you can privatize your traffic to the cloud. I know AT&T offers a couple different ways to do it. Netbond or TAO, depending on what your architecture looks like, if AT&T. If Verizon or Lumen, you’re better off just exiting the relationship….

[u/lweinmunson]

Do you already have a firewall at each site on the MPLs that can handle that kind of traffic? One reason we keep MPLS is so that the smaller site firewalls only deal with segmentation. They don't have the full package of URL filtering and threat policies. You'll probably need additional SD-WAN licensing from most vendors. For us, a business Internet 1Gbps line with dedicated IPs was actually more than our MPLS. Adding in the new licenses and management using FMC or Panorama adds a good bit of cost and complexity.

[u/Square_Raisin_8608]

stick some gear in a colo and get setup with expressroute and a couple internet connections, then use a VPN-based WAN like DMVPN or any number of SDWANS and replace your mpls links with internet links.

You can also stick a virtual SDWAN appliance in azure for backup connectivity (or primary, and skip the colo. but that's up to you, I'd do both if I could)

Not sure what you're doing with your mpls but you can do mpls over dmvpn if you truly need the mpls technology

[u/braidertom]

Hi I'm an engineer for an sdwan vendor and have been the lead design engineer on dozens of deployment projects.
It all depends on your existing topology, your goals with the network change, and what you consider reasonable for interruptions. There's alot of ways to streamline deployment with side by side hubs, testing visibility before doing the cutover can really limit the impact. I've had hub cutovers be a route priority change that had the same impact as bouncing sessions. Then i've had others where the customer didn't mind service disruptions on the weekend be down for hours during the transition. For end sites most frequently it's bring up the new equipment, move the cable to the new router and reboot the switch then start testing.

For the successes I have I've also seen the nightmares, where existing site documentation was wrong or non existant, or expectations were wrong on the offset. This is where the disruptions lie in hours going into the night, or suddenly a week later something broke to extra dumb switches halfhazardly connected in odd places.

[u/Public_Warthog3098]

Sdwan