I just recently got this question from one of my readers and I just posted my response here for future reference:
I was impressed about your post about MongoDB because I have similar setup at my company and I was thinking maybe you could give me an advice.
We have production servers with mongodb 2.6 and replica set. /data /journal /log all separate EBS volumes. I wrote a script that taking snapshot of production secondary /data volume every night. The /data volume 600GB and it takes 8 hours to snapshot using aws snapshot tool. In the morning I restore that snapshot to QA environment mongodb and it takes 1 minute to create volume from snapshot and attach volume to qa instance. Now my boss saying that taking snapshot on running production mongodb drive might bring inconsistency and invalidity of data. I found on internet that db.fsynclock would solve the problem. But what is going to happen if apply fsynclock on secondary (replica set) for 8 hours no one knows.
We store all data (data+journal+logs) into the same EBS volume. That’s also what MongoDB documentation suggests: “To get a correct snapshot of a running mongod process, you must have journaling enabled and the journal must reside on the same logical volume as the other MongoDB data files.” (that doc is from 3.0 but it applies also to 2.x)
I suggest that you switch to having data+journal in the same EBS volume and after that you should be just fine with doing snapshots. The current GP2 SSD disks allows big volumes and great amount of IOPS so that you might be able to get away with having just one EBS volume instead of combining several volumes together with LVM. If you end up using LVM make sure that you use the LVM snapshot sequence which I described in my blog http://www.juhonkoti.net/2015/01/26/tips-and-caveats-for-running-mongodb-in-production
I also suggest that you do snapshots more often than just once per night. The EBS snapshot system stores only new modifications, so the more often you do snapshots, the faster each snapshot will be created. We do it once per hour.
Also after the EBS snapshot API call has been completed and the EBS snapshot process is started you can resume all your operations in the disk which was just snapshotted. In other words: The data is frozen at some atomic moment during the EBS snapshot API call. After that moment the snapshot will contain exactly that data what it was during that atomic moment. The snapshot progress just tells you when you can restore a new EBS volume from that snapshot and that your volume IO performance is degraded a bit because the snapshot is being copied to S3 behind the scenes.
If you want to use fsynclock (which btw should not be required if you use mongodb journal) then implement a following sequence and you are fine:
- fsynclock
- XFS freeze (xfs_freeze -t /mount/point)
- EBS snapshot
- XFS unfreeze
- fsyncUnlock (xfs_freeze -u /mount/point)
The entire process should not take more than a dozen or so seconds.