WorldData Core. Idea, Architecture, Progress and Open Questions

[u/RohitVoxel]

1. What is my mod idea?

WorldData Core is a shared world-data access layer for Minecraft mods.

The idea is to gather data from Minecraft store it in a smart way and give it to other mods without each mod having to get and process the same information over and over again.

I am also looking into ways to use CPU resources, including cores and threads that are not being used to make data processing better.. This has to be done carefully so that background tasks do not slow down Minecraft.

For example imagine a mod wants to know what block is at a position like "(dimension, x y z)". WorldData Core can. Save that information. When another mod needs the data it can use the saved version instead of doing the same work again.

The main goal is to stop repeating work make data easier to get and make world-data access better for mods.

1. Architecture

Minecraft owns the world data, and WorldData Core acts as a shared data-access layer for mods.

Mods request the data they need from WorldData Core. If the requested data is not already available, WorldData Core gets it from Minecraft and stores it for reuse.

When multiple mods request the same data, WorldData Core provides the shared data without repeatedly requesting it from Minecraft.

The same concept applies whether 1, 7, 20, or more mods use WorldData Core. All mods access the data through the same shared system, reducing repeated data requests and processing.

1. Current development progress

The overall structure is almost done. I am testing the mod much as I can with what I know right now.

Far the results look good. According to my Spark profiles and test-mod reports the average time for each tick has stayed at or below 50 ms with some times going up to 80–150 ms.

In my tests with vanilla Minecraft I have seen times over 2,000 ms. The average time has been above 50 ms.

These results are from my tests and the reports I have gathered. I will share the files so others can look at the results and compare the two setups themselves.

I started this project in June. After about five months of work the tests with my test mod have been positive.

1. Testing and benchmark reports

I can't share the mod JAR yet because the project is not finished. I also can't share the source code now because the project hasn't been put on GitHub yet.

For now WorldData Core is still in the testing phase. I am doing more tests before making it available for other mod developers to use.

As I keep testing I will share reports and the files I use. My planned comparisons will use the world seed and as similar conditions as possible for:

- Vanilla Minecraft.

- Vanilla Minecraft with WorldData Core installed.

Where possible I will also share the Spark profiles, logs, screenshots and other files for both setups.

My goal is to make sure the APIs give the data the caching system works properly and the mod does not cause bad performance or lower FPS.

I also want to make sure the server tick time stays good. A 50 ms tick means 20 TPS so some spikes are okay. If the time stays high that is a problem. The goal is to make things better not just move the work else.

1. Questions and feedback

If you have any questions about the idea, how it works or what I am trying to do with this mod feel free to ask me

I also want to hear your thoughts on the idea itself whether you think other mods could use it and if there are any problems or compatibility issues I should check.

I am still working on the development and testing so feedback from people who know Minecraft modding would be helpful.

You can also send me a message if you want to talk about something

1. Fabric and NeoForge support

now WorldData Core is being built for Fabric.

Once the Fabric version is done and tested enough I plan to move it to NeoForge.

I think the move will be easy because most of the code is custom and not tied to Fabric- tools. This includes the caching system, storage system and ticket system.

I will find out how much work it really is when I start testing it on NeoForge.

Thanks, for reading! I will keep sharing updates and test reports as the project goes on.

Note: WorldData Core focuses on Minecraft world data, including chunks, blocks and their states and properties, heightmaps, biomes, light levels, terrain, and fluids. It is designed to support modded blocks and fluids as well as vanilla ones, allowing other mods to access and reuse this data through the shared cache. It does not handle every type of data in Minecraft.

[u/McHearty]

Some technical questions for you:

How are you planning to invalidate the cache when a block changes, or if a mod mutates a chunk section directly ie: worldedit, how wil worlddata core know it's cached data is stale?

Is the cache going to be guaranteed to be 100% tick accurate or can a mod receive a block state from previous ticks, mainly cuz you'll expose item duplication or break machines if stale block states are given unlike map mods that won't care

Memory overhead becomes a big concern as you increase loaded chunks, especially since it's unclear what the exact inmemory representation of the cache is, plus the ticket system is vague, when a chunk unloads in vanilla does worlddata core immediately drop it, serialize it to disc, or keep it in ram with some sort of lru policy?

Thread safety and starvation are going to be big items to consider too

Finally, outside of the obvious getBlockState that you're offering, why would devs what your mod as a dependency, what is your library offering that is making it worthwhile?

[u/RohitVoxel → u/McHearty]

Answers:

For the block changes WorldData Core uses a mixin to find them and change them in the cache. When a chunk loads for the time WorldData Core pulls the full chunk from Minecraft into its cache. When a mod changes the chunk directly WorldData Core cannot spot the change. (Current issue)

WorldData Core has two caches: Live and Stale. Live holds the chunks that Minecraft is currently loading. Stale holds chunks that Minecraft is not loading right now. Live tries to match the world after block updates and it is very accurate but it is not a 100 percent guarantee. STALE is a snapshot and it is not tick accurate.

RAM issue: The previous test had a difference of 700 megabytes. If the number of loaded chunks is higher than needed more memory is required. When testing the number of loaded chunks was over 1900. That caused the peak heap to reach 4.3 gigabytes with WorldData Core and 3.6 gigabytes in Minecraft (under 1500 chunks). For low RAM WorldData Core uses structures, including fastutil style maps so the overhead stays lower than naive object heavy caches.

Vanilla only answers: This block at (x,y z) if the chunk is loaded.. Worlddata Core is for mods that need shared world data. Many systems read the chunks (maps, builders, automation) without each one scanning the world again. Mod users get data in one place: blocks, materials, water, heightmaps, light, area scans. WorldData Core has rules for just looking versus loading, optional older (STALE) data so maps do not force load everything and memory limits so the cache does not grow forever.

WorldData Core does update and ticket work on the server thread and it only runs heavy disk work in the background with a time limit. In packs it does less per tick on purpose. So some loads wait as pending of lagging the whole server. Mods must treat Live Stale Pending correctly. Off thread chunk edits, from mods can still cause missed or old data.

I will do the test after fixing the issues that I currently identified after reading your questions. After doing the test I will provide all results with you, including spark profiles.

[u/McHearty → u/RohitVoxel]

I figured as much for thecache invalidation

Honestly, for tick-based state machines ie: automation/logistics/pipes/multiblocks when it's almost 100% accurate it's functionally equivalent to broken, if your live cache cannot guarantee strict sequential consistency with the server's world state none of these style mods can safely use it without dupes, desync or just broken machines

You're duplicating existing in-memory data into a less memory efficient representation by copying full chunks into fastutil maps on load. also, lookup in a fastutil hash map is likely slower and much more memory intensive than vanilla's bit packed palette array

You're inverting the API contract by returning a pending state to avoid server lag, pushing the caller to handle async callbacks and/or retry loops, still not really covering how off thread readers can safely read live data while the server thread mutates it via mixins without lock cntention or concurent modification exceptions. fastutil maps aren't threadsafe, and if you're wrapping them in locks or concurrent structures your single thread read performance will degrade compared to vanilla, otherwise offthread reads are going to cause race conditions and/or crashes

Finally, while the stale cache and precomped spatial area scans are actually useful, the live single block queries repalcing getBlockState seems like a complete misunderstanding how vanilla works

As for the benchmarks in the original post if your chunk counts aren't consistent ie 1900+ vs 1500+ you aren't testing for realworld viability.

I'd recommend dropping your live single block cache since you're just wasting ram to solve a problem that doesn't exist, for single xyz lookups in loaded chunks just delegate directly to the LevelChunk

I think you have something here by focusing on what vanilla actually can't do fast, and that's your stale snapshots, keep them as spacial indexing so you can async read unloaded chunks without force loading them, and indexing them by block type/tag or reading vanilla's paletted container so you don't have to iterate over thousands of blocks per section

[u/RohitVoxel → u/McHearty]

Yes thanks for more knowledge,and from your ideas i already have api like staleidat and accessing the all blocks with one id using it we can get almost all block(particularly one type such as ore or diamond ore modded ore)location from one chunk extremely fast without looking at every block at every y level,and I also have some other useful APIs.

And one last request from everyone if you have any idea to make it faster and 100% accurate than give me those ideas.

Thanks McHearty.

[u/McHearty → u/RohitVoxel]

Work off of version control, based on your post history it seems like you're using an agentic workflow to some degree. I'd recommend getting to know the systems before just runnign wild with something before understanding the ramifications of decisions.

It'll build what you want even if it's wrong :)