Atlas MongoDB Cluster Setup

Data migration from Azure CosmosDB to Atlas Cluster

# mongodump from Azure Cosmos
STG_CONNECTION_STRING="mongodb://global-cosmos-mongo-stg:nIY5ZkhxozGEjuuuj0OnfSz4rQJ9yV4PauFtA4Foak5d9L9BqY6hHnTuG1QDIRhClnbb9HQazCISjdKBFdCrlA==@global-cosmos-mongo-stg.mongo.cosmos.azure.com:10255/?ssl=true&replicaSet=globaldb&retrywrites=false&maxIdleTimeMS=120000&appName=@global-cosmos-mongo-stg@"

mongodump --ssl --uri="$STG_CONNECTION_STRING" -o ~/Workspace/Mo2C/data/stage

# mongorestore into Atlas
mongorestore "mongodb+srv://global-mongo-stage:sGB7SyjBSU1zc50H@cluster0.bodh8.mongodb.net/iam?retryWrites=true&w=majority" ~/Workspace/Mo2C/data/stage

# mongodump from Atlas
mongodump --uri="mongodb+srv://global-mongo-stage:sGB7SyjBSU1zc50H@cluster0.bodh8.mongodb.net/iam" -o ~/Downloads

Problems

1. Global Writes Cluster requires M30+ tier Shard Compute, and it’s very expensive.
2. Destroy a Cluster taking very long time.
3. CosmosDB/Mongo has some indexes not compitable with Atlas Mongo. Key starting with $ or containing wildcard *
   • See details: Wildcard indexes
4. Atlas ConnectionString has variety DNS issues. Seeing from local and also inside K8s clusters, see below:

   This will require extra DevOps effort to solve the DNS across all evnironments and personal work stations. DNS resolver needs to have public DNS resolver like Google’s: 8.8.8.8 primary and 8.8.4.4 secondary.

   • K8s pod error: ```bash

     mongorestore “mongodb+srv://global-mongo-stage:sGB7SyjBSU1zc50H@cluster0.bodh8.mongodb.net/iam?retryWrites=true&w=majority” ~/tmp

   STG_CONNECTION_STRING=”mongodb://global-mongo-stage:sGB7SyjBSU1zc50H@cluster0-shard-00-00.bodh8.mongodb.net:27016,cluster0-shard-00-01.bodh8.mongodb.net:27016,cluster0-shard-00-02.bodh8.mongodb.net:27016,cluster0-shard-01-00.bodh8.mongodb.net:27016,cluster0-shard-01-01.bodh8.mongodb.net:27016,cluster0-shard-01-02.bodh8.mongodb.net:27016/iam?ssl=true&authSource=admin&retryWrites=true&w=majority” mongodump –ssl –uri=”$STG_CONNECTION_STRING” -o ~/tmp/db/

   2020-12-15T18:22:08.501+0000 error parsing command line options: error parsing uri: lookup _mongodb._tcp.cluster0.bodh8.mongodb.net on 10.0.0.10:53: no such host 2020-12-15T18:22:08.501+0000 try ‘mongorestore –help’ for more information

    - Solution: Update K8s coreDNS configuration:
   ```bash
    mongodb.net:53 {
    	errors
    	cache 30
    	forward . 8.8.8.8 8.8.4.4
    }
   

   • Local error:

      > mongorestore "mongodb+srv://global-mongo-stage:sGB7SyjBSU1zc50H@cluster0.bodh8.mongodb.net/iam?retryWrites=true&w=majority" ~/Workspace/Mo2C/data/stage/iam
      2020-12-15T18:03:26.432+0000	error parsing command line options: error parsing uri: lookup cluster0.bodh8.mongodb.net on [::1]:53: cannot unmarshal DNS message
      2020-12-15T18:03:26.432+0000	try 'mongorestore --help' for more information
     

   • Solution

   • Check networking

      telnet cluster0-shard-00-00.bodh8.mongodb.net 27017
      OR
      nc -w 3 -v cluster0-shard-00-00.bodh8.mongodb.net 27017
     

   FYI,

   • https://docs.mlab.com/troubleshooting-atlas-connection-issues/
   • https://stackoverflow.com/questions/61820151/minikube-kubernetes-pod-cant-connect-to-mongodb-atlas
   • https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/
   • https://coredns.io/2017/07/23/corefile-explained/
   • https://forums.docker.com/t/docker-dns-change-etc-resolv-conf-file/40617

5. Global Writing requires a compound index, which requires a field named as location field which our documents don’t have yet, and another secondary field.

6. There are restrictions over unique compound index against shard_keys

2020-12-17T20:06:32.668+0000	Failed: iam.identity: error creating indexes for iam.identity: createIndex error: (CannotCreateIndex) { atlas-wm7ojp-shard-0/atlas-wm7ojp-shard-00-00.bodh8.mongodb.net:27017,atlas-wm7ojp-shard-00-01.bodh8.mongodb.net:27017,atlas-wm7ojp-shard-00-02.bodh8.mongodb.net:27017: "cannot create unique index over { ownerId: 1, t: 1, email: 1, uname: 1, cid: 1 } with shard key pattern { location: 1.0, ownerName: 1.0 }", atlas-wm7ojp-shard-1/atlas-wm7ojp-shard-01-00.bodh8.mongodb.net:27017,atlas-wm7ojp-shard-01-01.bodh8.mongodb.net:27017,atlas-wm7ojp-shard-01-02.bodh8.mongodb.net:27017: "cannot create unique index over { ownerId: 1, t: 1, email: 1, uname: 1, cid: 1 } with shard key pattern { location: 1.0, ownerName: 1.0 }" }