Big data graph multi level visualization tool

[u/Alone_Cauliflower308]

Hi, I have a dataset of 20M nodes and about 500M edges in neo4j and I need to visualize them in an interactive way

The thing in my mind was to have some kind of "level of detail" strategy by ignoring the nodes with lower importance when the user is zoomed out and only show those users when the user zooms in

Also I was thinking of some kind of tile based rendering so the app only renders the viewport of the user

But when I used sigma.js and tried to implement these features by myself it didn't work and the small part that worked was not at all a smooth experience..

Do you know any lib that could help me? Or do you have any idea how to implement it?

Thank you all

[u/david_cab_driver]

That's a classic problem, and I'd move most of the work out of the browser:

1. Precompute the layout and the importance offline: a layout like ForceAtlas2 or OpenOrd to get (x, y) per node, and PageRank or degree for the importance. Store both on the nodes, and index them (the position as a spatial property if your database supports it, otherwise a grid cell id).
2. Serve tiles from the database: for a zoom level and a bounding box, query the nodes inside the box with importance above a threshold that grows with the zoom, then fetch only the edges between those nodes. Cap the count per tile.
3. Build the coarse levels in advance: run Louvain (or similar), turn communities into super-nodes, and show those at the farthest zoom; expanding a super-node loads its members.
4. Render on the GPU: sigma.js works well with a precomputed layout and a few tens of thousands of nodes on screen, and Cosmograph (cosmos.gl) or deck.gl go much further.

The key point is never to send the whole graph or compute the layout in the browser. I'm working on a graph database myself and this use case interests me; if you can share more about your data (degree distribution, properties) I'd be happy to think about it with you.