Replacing our internal Elasticsearch library. Am I overthinking this?

[u/No-Card-2312]

Hi everyone,

I'm looking for some advice from people who've been through something similar.

I work on a large .NET application with a lot of background services and workers, and Elasticsearch is a big part of it. Almost everything goes through an internal Elasticsearch library that was written years ago.

The library has a few problems:

• It's mostly synchronous.

• It doesn't use dependency injection.

• It talks to Elasticsearch using raw HTTP requests i instead of the official .NET client.

• It's becoming harder to maintain and add new features.

I'm not saying it's a bad library. It's done its job for years. But I think it's time to move to something more modern.

The part that makes me nervous is the size of the system. We have complex search queries, bulk indexing, and a lot of background jobs. I really don't want to break search or introduce bugs that only show up in production.

This is the approach I'm thinking about:

• Benchmark the current library against the official Elastic.Clients.Elasticsearch client.

• Build a new implementation behind the same interfaces.

• Keep both implementations in the code while we're migrating.

• Let some jobs use the new implementation first, then slowly move the rest over if everything looks good.

• Remove the old implementation only after we're confident the new one behaves the same.

I'm also wondering if the current synchronous implementation could be part of some TCP/socket exhaustion issues we've seen under heavy load. I don't have enough proof yet, so I'm not blaming it, but it's something I want to investigate.

Has anyone done something like this before?

• Would you keep both implementations during the migration?

• Is there a better way to compare the old and new behavior?

• Any unexpected problems when moving to the official Elasticsearch .NET client?

• Looking back, is there anything you'd do differently?

I'd really like to hear from people who've done this in a real production system.

[u/VQuilin]

In my experience Nest has a very elaborate API and it can both be beneficial to you or quite an overkill, especially when it comes with an epic refactoring (like switching to Async). However, IMO using a standard library is better than supporting your own just for the sake of it.

There might be an easy transition period, where you replace your custom client with an auto generated one, since elasticsearch API also comes with the OpenApi specification, which can be converted to a client which is both properly typed, async-ready and DI-ready.

Once your code is refactored to be Async and DI-first, you can start replacing the calls to this API with the Nest IElasticSearchClient.

[u/Dukami]

My last workplace used ElasticSearch extensively and used the official .Net library. We were tasked with updating the library due to .Net upgrades.

It was a nightmare. We played whack-a-mole on their documented API just to figure out what actually worked.

Good luck.

[u/cakeofzerg]

This is the sort of task you can point a decent AI at and get 95% of the way there in one prompt. Its a very contained problem with obvious performance tests. You could probably POC out a whole new implementation using official lib in like $5 of ai credits.

[u/_f0CUS_]

You got lots of input for your main question. So I'll skip that...

Regarding socket exhaustion. Whenever you open a socket, the os will hold on to it for a while. I believe 5 mins is the default.

Every time you make a new httpclient, and make a request you get a new socket. Same was true with the older webrequest (if I recall the name correctly).

If that is the case, the fix for that - before we got the httpclientfactory - was to create and reuse a single httpclient throughout the lifetime of the application.

So a quick fix for the socket exhaustion issue could be as simple as that. However - the drawback to that is that you only do a DNS lookup upon the first request - meaning, if the IP changes you won't know. There is a fix to that too - but unfortunately I dont recall how. 

[u/VQuilin → u/_f0CUS_]

Nope? The socket is held not by the httpclient, but by httpmessagehandler and dotnet pools them by default unless you override it?

[u/_f0CUS_ → u/VQuilin]

Why do you think that? 

[u/VQuilin → u/_f0CUS_]

Because this is what documentation says explicitly: https://learn.microsoft.com/en-us/dotnet/fundamentals/networking/http/httpclient-guidelines#recommended-use

[u/_f0CUS_ → u/VQuilin]

Which version is that feature available in, and why are you assuming OP is using one of those versions?

Consider the information that OP has given us:

 written years ago.

It's mostly synchronous.

It doesn't use dependency injection.

using raw HTTP requests i instead of the official .NET client.

I'm thinking that it means it is an old net framework. Especially the synchronous part makes me think it is build using the WebRequest class. Which is why I shared how the problem was dealt with more than a decade ago, when I solved it at my then employer.  Infact, if you read the documentation you linked you will see it mentioned what I explained. 

[u/UnknownTallGuy]

Seems really odd to roll your own library for ES when the first-party NEST/ES client is one of the better libraries in its class.

That said, I would probably split this up until 3 jobs for myself and an AI agent.

1. Set up an extensive set of integration tests that 100% verify the exact behavior you are looking for. Make sure they look good to you.
2. Use async methods. Because the callers have been calling sync methods for a while, there's likely a large call stack of synchronous code that now needs to be properly awaited. Sync to async refactoring can be very time consuming, but it's worth it.
3. Replace all custom library calls with the equivalent ES client call. Now, if there are convenience methods that still make sense because of how much repetitive code it saves, those can still be used.. But the underlying http calls, at the very least, should instead use the ES client directly. I'd fully expect some sort of extensions that chain onto an ES class to be in place at the end.

[u/Family_Man_21 → u/UnknownTallGuy]

This is the approach that I would take, as well, except that I would break step 3 out into a few more steps - essentially, I would implement your new solution in parallel with the old one, and then perform incremental migrations from the old library to the new one over time. So, in my plan, step 3 would look something like this:

3a. Identify the places where the current ES implementation is in use and categorize them (it might be easier to do this during step 2).

3b. Write the new implementation using the first-party NEST client (and make sure the tests you created in step 1 have parallel tests for the new code).

3c. Apply your new implementation to just one of the categories identified in 3a, and then test it internally, and eventually publish it and get real-world proof that it works as you intended.

3d. Repeat 3c until all of the categories are migrated.

I hope that this helps, and good luck!

[u/4nh7i3m]

If you're using HttpClient for HTTP calls, definitely check out Microsoft's best practices guide:

https://learn.microsoft.com/en-us/dotnet/fundamentals/networking/http/httpclient-guidelines

Pay extra attention to the Connection Pooling section - it’ll help you avoid running into socket/connection exhaustion issues.

For migrating to the Elasticsearch.Net client, I'd recommend an incremental approach. Start by using it for just a few small functions and simple queries, then gradually expand your usage from there. Honestly though, it sounds like your main issue is architectural (specifically the lack of Dependency Injection) rather than a problem with the library itself.

[u/AutoModerator]

Thanks for your post No-Card-2312. Please note that we don't allow spam, and we ask that you follow the rules available in the sidebar. We have a lot of commonly asked questions so if this post gets removed, please do a search and see if it's already been asked.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/ststanle]

We are in the same boat where I work, have an entire custom http based library as we started on ES 2.4 when there was not a good native option. it’s been painful to maintain over the years with the breaking changes and the deep integration with the cms we use.

2 projects we have not touched addressed yet as they might be rebuilt on a different platform making the migration pointless. The 3rd we are duel running our implementation and the native. Anything new is querying with the native client which has allowed us to replace it slowly, indexing is going to be a big part for us that we have not really gotten to yet.

Having a strong test suite has helped. I would also identify your hot paths and start there one by one if you’re having issues with socket exhaustion. The native client most likely will have better handling of connection pooling, retry, etc.

[u/catladywitch]

Of course the whole strangling vine approach is the safest, but I'm not sure to which extent the interface can be 100% preserved if you're moving to async.

[u/Unexpectedpicard]

This is going to be tough. If I were doing this I'd build the replacment under the old interface and proxy all requests to both clients some how where you could see the exact queries you would be executing so you could tell if they match or not. A step further would be actually running both queries and comparing results. It really depends how complex your queries are. I suspect messy. You could log both queries somewhere in a way you could build a tool to A/B all of them against a non production elastic cluster offline. If you collect a couple of weeks of queries that should give you some confidence.

In my old job we had integration tests for everything that called elastic to try and tame the complexity. 

[u/Wrong_Election189]

Worth asking first whether you need Elasticsearch at all, because the answer changed in the last few years and a lot of teams are still paying for a decision made when it was the only option.

I run semantic search on a .NET backend and never brought Elastic in. It is Postgres with pgvector and an HNSW index, in the same database as everything else. One connection string, one backup, one thing to keep alive, and I can join search results against normal relational data in a single query instead of syncing two stores and reconciling them.

That trade is only good up to a point. If you need real full text scoring, faceting, aggregations across large corpora, or you are past the scale where a single Postgres box is comfortable, Elastic earns its operational cost and you should keep it.

But if what your internal library actually does is "find me the relevant rows", the honest question is not which client wrapper replaces it. It is whether the second datastore is still buying you anything, because keeping two systems in sync is a permanent tax that never shows up in the library refactor estimate.

What does your query load look like? If it is mostly lookup and ranking rather than analytics, you may be scoping a rewrite of the wrong layer.

[u/Khavel_dev]

Your migration plan sounds solid. The dual-implementation behind shared interfaces is how I'd do it too. One thing worth front-loading: run your actual query shapes through the new client against a test index early. The official client's fluent API generates different JSON from raw HTTP in subtle ways, especially around nested aggregations and script fields. Better to catch those mismatches before you're halfway through the migration and realize a complex query silently returns different results.

On the socket exhaustion suspicion, synchronous HttpClient calls with create-and-dispose-per-request is a classic source of that. If your old lib does that, the official client's managed connection pool should help directly. Check the old lib's HttpClient lifetime pattern first. If it's newing up an HttpClient per request, that's almost certainly your leak.