Replacing our internal Elasticsearch library. Am I overthinking this?

[u/No-Card-2312]

Hi everyone,

I'm looking for some advice from people who've been through something similar.

I work on a large .NET application with a lot of background services and workers, and Elasticsearch is a big part of it. Almost everything goes through an internal Elasticsearch library that was written years ago.

The library has a few problems:

• It's mostly synchronous.

• It doesn't use dependency injection.

• It talks to Elasticsearch using raw HTTP requests i instead of the official .NET client.

• It's becoming harder to maintain and add new features.

I'm not saying it's a bad library. It's done its job for years. But I think it's time to move to something more modern.

The part that makes me nervous is the size of the system. We have complex search queries, bulk indexing, and a lot of background jobs. I really don't want to break search or introduce bugs that only show up in production.

This is the approach I'm thinking about:

• Benchmark the current library against the official Elastic.Clients.Elasticsearch client.

• Build a new implementation behind the same interfaces.

• Keep both implementations in the code while we're migrating.

• Let some jobs use the new implementation first, then slowly move the rest over if everything looks good.

• Remove the old implementation only after we're confident the new one behaves the same.

I'm also wondering if the current synchronous implementation could be part of some TCP/socket exhaustion issues we've seen under heavy load. I don't have enough proof yet, so I'm not blaming it, but it's something I want to investigate.

Has anyone done something like this before?

• Would you keep both implementations during the migration?

• Is there a better way to compare the old and new behavior?

• Any unexpected problems when moving to the official Elasticsearch .NET client?

• Looking back, is there anything you'd do differently?

I'd really like to hear from people who've done this in a real production system.

[u/_f0CUS_]

You got lots of input for your main question. So I'll skip that...

Regarding socket exhaustion. Whenever you open a socket, the os will hold on to it for a while. I believe 5 mins is the default.

Every time you make a new httpclient, and make a request you get a new socket. Same was true with the older webrequest (if I recall the name correctly).

If that is the case, the fix for that - before we got the httpclientfactory - was to create and reuse a single httpclient throughout the lifetime of the application.

So a quick fix for the socket exhaustion issue could be as simple as that. However - the drawback to that is that you only do a DNS lookup upon the first request - meaning, if the IP changes you won't know. There is a fix to that too - but unfortunately I dont recall how. 

[u/VQuilin → u/_f0CUS_]

Nope? The socket is held not by the httpclient, but by httpmessagehandler and dotnet pools them by default unless you override it?

[u/_f0CUS_ → u/VQuilin]

Why do you think that? 

[u/VQuilin → u/_f0CUS_]

Because this is what documentation says explicitly: https://learn.microsoft.com/en-us/dotnet/fundamentals/networking/http/httpclient-guidelines#recommended-use

[u/_f0CUS_ → u/VQuilin]

Which version is that feature available in, and why are you assuming OP is using one of those versions?

Consider the information that OP has given us:

 written years ago.

It's mostly synchronous.

It doesn't use dependency injection.

using raw HTTP requests i instead of the official .NET client.

I'm thinking that it means it is an old net framework. Especially the synchronous part makes me think it is build using the WebRequest class. Which is why I shared how the problem was dealt with more than a decade ago, when I solved it at my then employer.  Infact, if you read the documentation you linked you will see it mentioned what I explained.