Can your DB lose your data?
If you have ever wondered if something called a data base can lose whatever you put in it then the short answer is yes. There are some good reasons this can happen as well as some more intricate cases where the reason is far less obvious. Lets start with the first ones.
Scenario 1 (hardware failure)
You have a database with a nighly backup and your disk drive fails midday, it’s easy to imagine that a few hours worth of data is permanently lost. Take another case: you are a customer of a major cloud provider and the whole server bursts into flames (the cause of the fire will not be revealed until 2022, but LiPo batteries and Apache Ignite remain the main suspects).
Solution: Replication across multiple data centers.
Major cloud providers offer a concept of an Availability Zone (AZ). AZs are separate data centers within a single region. They share service offerings, fast (and cheap) networking but are located in separate buildings. AWS claims that all AZs for a region are separated by many kilometers and no more 100km from each other. Azure says only that AZs are separated. In case of OVH SBG AZs were separate buildings located next to each other.
Scenario 2 (blackout)
But what if no physical damage is made and your server suffers only a power loss? ‘D’ in ACID (SQL side of things) stands for Durability and this is the quality we are to touch here. Yes, I know that ‘D’ in CRUD (NoSQL) stands for Delete but it’s only a coincidence as this type of problems is no different in both worlds. MongoDB lets you choose your desired level of durability. In case of a standalone (not replicated) instance writes are considered durable if they make it to the journal. Journal is a set of files created on a disk that contain pending changes to the dataset. They are applied every 60s to the main dataset. If a Mongo instance suffers a crash before the data makes it to the main set it will recover the next time it starts. There is a caveat though. Your storage may lie to you and still keep it in a volatile cache. Some disk arrays contain batteries to make sure the cache is aways flushed to disks, some of them are even smart enough to fallback to no caching when the battery fails keeping durability at the cost of performance but this may not always be the case.
Solution: Issue important writes a {j:true} flag. This will make sure all your data gets to the journal by waiting with your write acknowlegement. Use proper hardware.
Scenario 3 (network partition)
Another more DB specific (not only valid for MongoDB but for all databases) is the split brain scenatio a.k.a network partition (the ‘P’ in CAP). Suppose you have 3 replicas - master (or primary) and two slaves (or secondaries) and the data center (DC) containing your app and the master node loses network. Your app may still be able to make successful writes to the master node without even knowing that a new master has been elected outside the DC. When the network connection comes back your DB node reconnects to the cluster as a secondary and needs to rollback your writes to maintain consistency with other members. This means your data is lost.
Solution: All writes issued to a replica set need to reach majority of nodes before being acknowledged. This alone makes sure that even if n etwork partition occurs your data will be in the bigger group when a new primary will be elected. You can use write concern “majority” to achieve this in MongoDB.