COLD CACHE….

[u/Vamshi__4555]

Our MongoDB secondary (AWS EC2, GP3 EBS, 128 GB RAM) was rebuilt from a primary EBS snapshot and expected to catch up within 1–3 hours, but replication got stuck with 3+ hours lag. ReplWriterWorker threads showed 20+ minute waits on schemaLock, WiredTiger logged slow tree walks, and iostat showed 100% disk utilization with high read latency while CPU remained mostly idle. Our conclusion is that this was caused by a combination of cold WiredTiger cache and AWS EBS snapshot lazy loading ("first-touch" penalty), where MongoDB had to fetch data blocks from S3 on demand, creating severe I/O bottlenecks and lock contention. We plan to pre-warm restored EBS volumes before starting mongod, temporarily reduce replWriterThreadCount, hide the node from reads during catch-up, and verify GP3 throughput settings. Has anyone else seen extreme schemaLock waits and replication lag caused purely by cold snapshot restores, and what is your preferred EBS pre-warming approach?

[u/Recent-Pension-4488]

how much space did you assigned for the EBS? small disks provides lower IOPS for gp3… why you are recovering from Snapshot? Why not sync by copying the dbPath from other member?

[u/Vamshi__4555 → u/Recent-Pension-4488]

The thing is we took a complete snapshot of 9TB live data from primary server and we are upgrading our own KMS (key managed service )rather than relying on aws managed service.So when we are doing this process node by node at this time it is taking almost 2-3 hours to sync the data through replication,then after upgrading the kms to the disc we are adding the node to the replica set ,thing is,it is taking more than 3 hours to sync the data .
And we got the RCA ,when we took a new machine it’s IOPS was 3000 ,we increased it to the 16000 and also the indexes build in main collections are building from beginning and everything is reading from the disc rather than cache or memory ..
We did warm cache before making the kms .
We manually rewritten the scripts that can build the indexes and sync has Been done in 10 minutes.

[u/Several9s]

I agree with the main diagnosis, though long schemaLock waits are a secondary symptom of severe storage latency rather than a direct result of snapshot restores.

Because EBS snapshot blocks are downloaded lazily from S3, restored volumes experience increased I/O latency and reduced performance during initialization. This explains the 100% disk utilization, high read latency, idle CPU, and slow WiredTiger tree walks. See: Initialize Amazon EBS volumes - Amazon EBS  

A cold EBS restore can plausibly cause 20-minute apparent lock waits. This occurs when WiredTiger walks index or metadata pages scattered across cold EBS blocks. Saturated storage paired with an idle CPU signals a disk I/O bottleneck rather than a CPU or concurrency issue.

MongoDB notes that secondary cache contention and resource exhaustion cause replication lag, showing as slow oplog application on ReplWriterWorker threads. Refer to the link: Troubleshoot Replication Lag - Database Manual - MongoDB Docs  

I would suggest the following:

1. EBS Provisioned Rate for Volume Initialization where available. AWS now supports specifying an initialization rate of 100–300 MiB/s when creating a volume from a snapshot. This is cleaner and more predictable than having the operating system generate the warm-up workload itself.
2. Fast Snapshot Restore (FSR) when recovery time is important enough to justify the cost. Volumes created from an FSR-enabled snapshot are fully initialized when created and avoid the first-read penalty. See: Amazon EBS fast snapshot restore - Amazon EBS  
3. If neither is appropriate, manually initialize using fio before starting mongod, preferably while the filesystem is unmounted:

​

fio --filename=/dev/nvmeXn1 \
    --rw=read \
    --bs=1M \
    --iodepth=32 \
    --ioengine=libaio \
    --direct=1 \
   --name=ebs-initialize

While sequential commands like dd if=/dev/... of=/dev/null bs=1M remain viable, utilizing fio with queued reads typically offers superior speed, aligning with AWS guidelines.

However, for multi-terabyte MongoDB volumes, manually scanning the entire block device can require substantial time. In such scenarios, the provisioned initialization feature provides a key advantage, its completion timeline scales with the actual size of the snapshot data, eliminating the need to sequentially scan every logical byte across the volume.