Chef fails on s0 (mongos) and cannot login on c0-2

[bearproof]

With the default code, when I do a vagrant up, chef fails on s0 (mongos). Below is the stack trace.
Also, for the config servers, the admin users are not created and I cannot log in. For the mongod instances, everything works fine.

I'm still playing around with vagrant and chef, but I would like to contribute if I can. Just let me know what works and what not, and I can submit pull requests.

bearproof@bearproof-HP-EliteBook-8730w:~/work/altisource/mongodb-cluster$ vagrant provision s0
==> s0: Running triggers before provision...
==> s0: Cleaning cookbook directory
==> s0: Executing command "rm -rf cookbooks"...
==> s0: Command execution finished.
==> s0: Installing cookbook dependencies with berkshelf
==> s0: Executing command "berks vendor cookbooks"...
==> s0: Resolving cookbook dependencies...
==> s0: Fetching 'mongodb-cluster' from source at .
==> s0: Using chef-solo-search (0.5.1)
==> s0: Using build-essential (2.0.6)
==> s0: Using apt (2.6.0)
==> s0: Using mongodb-cluster (1.0.0) from source at .
==> s0: Using mongodb (0.16.1) from https://github.com/ceejh/chef-mongodb.git (at working)
==> s0: Using runit (1.5.10)
==> s0: Using yum (3.3.2)
==> s0: Using yum-epel (0.5.1)
==> s0: Using python (1.4.6)
==> s0: Vendoring apt (2.6.0) to cookbooks/apt
==> s0: Vendoring build-essential (2.0.6) to cookbooks/build-essential
==> s0: Vendoring chef-solo-search (0.5.1) to cookbooks/chef-solo-search
==> s0: Vendoring mongodb (0.16.1) to cookbooks/mongodb
==> s0: Vendoring mongodb-cluster (1.0.0) to cookbooks/mongodb-cluster
==> s0: Vendoring python (1.4.6) to cookbooks/python
==> s0: Vendoring runit (1.5.10) to cookbooks/runit
==> s0: Vendoring yum (3.3.2) to cookbooks/yum
==> s0: Vendoring yum-epel (0.5.1) to cookbooks/yum-epel
==> s0: Command execution finished.
==> s0: Installing Chef 11.16 Omnibus package...
==> s0: Downloading Chef 11.16 for ubuntu...
==> s0: downloading https://www.chef.io/chef/metadata?v=11.16&prerelease=false&nightlies=false&p=ubuntu&pv=12.04&m=x86_64
==> s0:   to file /tmp/install.sh.6883/metadata.txt
==> s0: trying wget...
==> s0: url https://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/12.04/x86_64/chef_11.16.4-1_amd64.deb
==> s0: md5 2ffff5b4d80e4dcffc917f8eb2003a31
==> s0: sha256  28b08975e7e33ac46c888616ec7fa232a0c624aeeda81e58a6047d2c6b62edfb
==> s0: downloaded metadata file looks valid...
==> s0: downloading https://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/12.04/x86_64/chef_11.16.4-1_amd64.deb
==> s0:   to file /tmp/install.sh.6883/chef_11.16.4-1_amd64.deb
==> s0: trying wget...
==> s0: Comparing checksum with sha256sum...
==> s0: Installing Chef 11.16
==> s0: installing with dpkg...
==> s0: (Reading database ... 
==> s0: 65470 files and directories currently installed.)
==> s0: Preparing to replace chef 11.16.4-1 (using .../chef_11.16.4-1_amd64.deb) ...
==> s0: Unpacking replacement chef ...
==> s0: Setting up chef (11.16.4-1) ...
==> s0: Thank you for installing Chef!
==> s0: Running provisioner: hostmanager...
==> s0: Running provisioner: chef_solo...
==> s0: Detected Chef (latest) is already installed
Generating chef JSON and uploading...
==> s0: Running chef-solo...
==> s0: stdin: is not a tty
==> s0: [2015-03-10T11:21:34+00:00] INFO: Forking chef instance to converge...
==> s0: [2015-03-10T11:21:34+00:00] WARN: 
==> s0: * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
==> s0: SSL validation of HTTPS requests is disabled. HTTPS connections are still
==> s0: encrypted, but chef is not able to detect forged replies or man in the middle
==> s0: attacks.
==> s0: 
==> s0: To fix this issue add an entry like this to your configuration file:
==> s0: 
==> s0: ```
==> s0:   # Verify all HTTPS connections (recommended)
==> s0:   ssl_verify_mode :verify_peer
==> s0: 
==> s0:   # OR, Verify only connections to chef-server
==> s0:   verify_api_cert true
==> s0: ```
==> s0: 
==> s0: To check your SSL configuration, or troubleshoot errors, you can use the
==> s0: `knife ssl check` command like so:
==> s0: 
==> s0: ```
==> s0:   knife ssl check -c /tmp/vagrant-chef/solo.rb
==> s0: ```
==> s0: 
==> s0: * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
==> s0: [2015-03-10T11:21:34+00:00] INFO: *** Chef 11.16.4 ***
==> s0: [2015-03-10T11:21:34+00:00] INFO: Chef-client pid: 7091
==> s0: [2015-03-10T11:21:38+00:00] INFO: Setting the run_list to ["recipe[chef-solo-search]", "recipe[mongodb::mongos]", "recipe[mongodb::user_management]"] from CLI options
==> s0: [2015-03-10T11:21:38+00:00] INFO: Run List is [recipe[chef-solo-search], recipe[mongodb::mongos], recipe[mongodb::user_management]]
==> s0: [2015-03-10T11:21:38+00:00] INFO: Run List expands to [chef-solo-search, mongodb::mongos, mongodb::user_management]
==> s0: [2015-03-10T11:21:38+00:00] INFO: Starting Chef Run for s0
==> s0: [2015-03-10T11:21:38+00:00] INFO: Running start handlers
==> s0: [2015-03-10T11:21:38+00:00] INFO: Start handlers complete.
==> s0: [2015-03-10T11:21:40+00:00] WARN: 10gen_repo is deprecated, use mongodb_org_repo
==> s0: [2015-03-10T11:21:43+00:00] WARN: Cloning resource attributes for template[mongodb.conf] from prior resource (CHEF-3694)
==> s0: [2015-03-10T11:21:43+00:00] WARN: Previous template[mongodb.conf]: /tmp/vagrant-chef/a9359d0e8f37fd0ce1e8da76ef4217a8/cookbooks/mongodb/recipes/install.rb:14:in `from_file'
==> s0: [2015-03-10T11:21:43+00:00] WARN: Current  template[mongodb.conf]: /tmp/vagrant-chef/a9359d0e8f37fd0ce1e8da76ef4217a8/cookbooks/mongodb/definitions/mongodb.rb:139:in `block in from_file'
==> s0: [2015-03-10T11:21:43+00:00] WARN: Cloning resource attributes for chef_gem[mongo] from prior resource (CHEF-3694)
==> s0: [2015-03-10T11:21:43+00:00] WARN: Previous chef_gem[mongo]: /tmp/vagrant-chef/a9359d0e8f37fd0ce1e8da76ef4217a8/cookbooks/mongodb/recipes/mongo_gem.rb:14:in `block in from_file'
==> s0: [2015-03-10T11:21:43+00:00] WARN: Current  chef_gem[mongo]: /tmp/vagrant-chef/a9359d0e8f37fd0ce1e8da76ef4217a8/cookbooks/mongodb/recipes/user_management.rb:1:in `from_file'
==> s0: [2015-03-10T11:21:45+00:00] INFO: service[mongos] started
==> s0: [2015-03-10T11:21:45+00:00] INFO: service[mongos] sending create action to ruby_block[config_sharding] (immediate)
==> s0: [2015-03-10T11:21:46+00:00] INFO: {"rs_default"=>["d1:27017", "d0:27017", "d2:27017"]}
==> s0: [2015-03-10T11:21:46+00:00] INFO: ["rs_default/d1:27017,d0:27017,d2:27017"]
==> s0: [2015-03-10T11:21:46+00:00] WARN: Could not connect to database: 'localhost:27017', reason Failed to connect to a master node at localhost:27017
==> s0: [2015-03-10T11:21:46+00:00] WARN: Could not connect to database: 'localhost:27017', reason Failed to connect to a master node at localhost:27017
==> s0: [2015-03-10T11:21:46+00:00] INFO: ruby_block[config_sharding] called
==> s0: [2015-03-10T11:21:46+00:00] INFO: [ruby_block[config_sharding]] sending add action to mongodb_user[admin] (delayed)
==> s0: [2015-03-10T11:21:46+00:00] WARN: Unable to connect to MongoDB instance, retrying in 1 second(s)...
==> s0: 
==> s0: 
==> s0: ================================================================================
==> s0: Error executing action `add` on resource 'mongodb_user[admin]'
==> s0: ================================================================================
==> s0: 
==> s0: 
==> s0: NoMethodError
==> s0: -------------
==> s0: undefined method `db' for nil:NilClass
==> s0: 
==> s0: 
==> s0: Cookbook Trace:
==> s0: ---------------
==> s0: /tmp/vagrant-chef/a9359d0e8f37fd0ce1e8da76ef4217a8/cookbooks/mongodb/providers/user.rb:10:in `add_user'
==> s0: /tmp/vagrant-chef/a9359d0e8f37fd0ce1e8da76ef4217a8/cookbooks/mongodb/providers/user.rb:127:in `block in class_from_file'
==> s0: 
==> s0: Resource Declaration:
==> s0: ---------------------
==> s0: # In /tmp/vagrant-chef/a9359d0e8f37fd0ce1e8da76ef4217a8/cookbooks/mongodb/recipes/user_management.rb
==> s0: 
==> s0:  14:   mongodb_user user['username'] do
==> s0:  15:     password user['password']
==> s0:  16:     roles user['roles']
==> s0:  17:     database user['database']
==> s0:  18:     connection node['mongodb']
==> s0:  19:     if node.recipe?('mongodb::mongos') || node.recipe?('mongodb::replicaset')
==> s0:  20:       # If it's a replicaset or mongos, don't make any users until the end
==> s0:  21:       action :nothing
==> s0:  22:       subscribes :add, 'ruby_block[config_replicaset]', :delayed
==> s0:  23:       subscribes :add, 'ruby_block[config_sharding]', :delayed
==> s0:  24:     end
==> s0:  25:   end
==> s0: 
==> s0: Compiled Resource:
==> s0: ------------------
==> s0: # Declared in /tmp/vagrant-chef/a9359d0e8f37fd0ce1e8da76ef4217a8/cookbooks/mongodb/recipes/user_management.rb:14:in `block in from_file'
==> s0: 
==> s0: mongodb_user("admin") do
==> s0:   action [:nothing]
==> s0:   retries 0
==> s0:   retry_delay 2
==> s0:   guard_interpreter :default
==> s0:   cookbook_name :mongodb
==> s0:   recipe_name "user_management"
==> s0:   password "admin"
==> s0:   roles ["userAdminAnyDatabase", "dbAdminAnyDatabase", "clusterAdmin"]
==> s0:   database "admin"
==> s0:   connection {"client_roles"=>[], "cluster_name"=>"vagrant", "shard_name"=>"default", "replica_arbiter_only"=>false, "replica_build_indexes"=>true, "replica_hidden"=>false, "replica_slave_delay"=>0, "replica_priority"=>1, "replica_tags"=>{}, "replica_votes"=>1, "auto_configure"=>{"replicaset"=>true, "sharding"=>true}, "configserver_url"=>nil, "root_group"=>"root", "user"=>"mongodb", "group"=>"mongodb", "init_dir"=>"/etc/init/", "init_script_template"=>"debian-mongodb.upstart.erb", "sysconfig_file"=>"/etc/default/mongodb", "sysconfig_file_template"=>"mongodb.sysconfig.erb", "dbconfig_file_template"=>"mongodb.conf.erb", "dbconfig_file"=>"mongodb.conf", "package_name"=>"mongodb-org", "package_version"=>nil, "default_init_name"=>"mongod", "instance_name"=>"mongos", "install_method"=>"mongodb-org", "is_replicaset"=>nil, "is_shard"=>nil, "is_configserver"=>nil, "reload_action"=>"restart", "apt_repo"=>"ubuntu-upstart", "template_cookbook"=>"mongodb", "key_file_content"=>"2pJs9ckLBfDZqyJvOqJKq8+syy9Vsd/lXkyN2wK2c66/6GEZKZvffFn+0O3d0fTX\n1rFEPxamTDg7KXwOF8H+v7aSn7k12sh2FpELX7uYKA2/BJ87elbjXngHX9b9HW7l\nDmXVyjgySpaUQu6UXJJD8iO+yaVZVv6BRDcyWMUNgx6VqUkHnyW/m8kHKXVYYsLW\nnQD/mJIVhRqnLo0Rc0wUl9FXjdFoz3MxRaG/PgOlENiqaiWTRKVAs9czS4CoVFQ0\nSGNemD7K9pasxh/LYj4HPLeXTlS1sZUS+djgX9ZI/cBzqlKbi3KMN15sjcRK7jpV\nq9+wM/IEtyUq5XeL2RC+pP7F9JKdgTHRvYjm3DHREtKc2LVeXzq87DmDjzyJhA4v\nQZ6iRM7gzX6h47PDoxgp6NTxPcUE+iGYPA7HeiBtUD5JLaqKCTYwF60l2Y/82cu/\nXS7edLnOKbody0o8scOM48jeiCKEUoaEvCUGtexd/XjqrJXWRlVlWzqkbrDrZpSu\nIpfpKGJ8T4A3REqoLbQw7MvVYsQcUOBseNbqpU3fRa9A/62UvM6J3hDLe5FlHRSj\nhN9hFNEbtVLx/6Am7lHanAQ6pRVuc2WhXGUH36VBYmgKmb+5mPm1k9MfmgU2e8E6\nLjW53OTC13Lwbse8mOU+8ZLeCbH7BStg8GUW90NjswtfFHYK4p0fBacX8peT8s2+\nqE70aYI5akTgBg1fOHhwm6UZPqmilD247jHwOtAhH1c4gpsSwceeX3l9n9d2Fx10\nh6+97xdClP7fgtBiluysCg2S7H+ZwVkOwP5XN+YhOIUWytCFp31F3YbnSCldfGQi\nAumuW1IjgZj66bABQWSSVxh0/c4+HceW007wWvaL4hRjaRboMqg+3wIQif/oku86\n56c8BBkczIJuNIVhKsBrt8/A6AWmpYsMpaEmI3FjXMZA99dUZnWh4SFSx2doBGQM\nNG44b0IaPk+wvaMzdyZljKZF20fV", "ruby_gems"=>{"mongo"=>nil, "bson_ext"=>nil}, "config"=>{"port"=>27017, "bind_ip"=>"0.0.0.0", "logpath"=>"/var/log/mongodb/mongodb.log", "logappend"=>true, "fork"=>false, "dbpath"=>nil, "nojournal"=>"", "rest"=>"", "smallfiles"=>"", "oplogSize"=>nil, "replSet"=>nil, "keyFile"=>"/etc/mongodb.key", "configdb"=>"c0:27019,c1:27019,c2:27019", "auth"=>true}, "mms_agent"=>{"api_key"=>nil, "package_url"=>"https://mms.mongodb.com/download/agent/%{agent_type}/mongodb-mms-%{agent_type}-agent", "monitoring"=>{"version"=>"2.2.0.70-1", "mmsApiKey"=>nil, "mmsBaseUrl"=>"https://mms.mongodb.com", "configCollectionsEnabled"=>true, "configDatabasesEnabled"=>true, "throttlePassesShardChunkCounts"=>10, "throttlePassesDbstats"=>20, "throttlePassesOplog"=>10, "disableProfileDataCollection"=>false, "disableGetLogsDataCollection"=>false, "disableLocksAndRecordStatsDataCollection"=>false, "enableMunin"=>true, "useSslForAllConnections"=>false, "sslRequireValidServerCertificates"=>false}, "backup"=>{"version"=>"2.1.0.106-1", "mmsApiKey"=>nil, "mothership"=>"api-backup.mongodb.com", "https"=>true, "sslRequireValidServerCertificates"=>false}, "user"=>"mongodb-mms-agent", "group"=>"mongodb-mms-agent"}, "sysconfig"=>{"DAEMON"=>"/usr/bin/$NAME", "DAEMON_USER"=>"mongodb", "DAEMON_OPTS"=>"--config mongodb.conf", "CONFIGFILE"=>"mongodb.conf", "ENABLE_MONGODB"=>"yes", "DAEMONUSER"=>"mongodb", "ENABLE_MONGOD"=>"yes", "ENABLE_MONGO"=>"yes"}, "ulimit"=>{"fsize"=>"unlimited", "cpu"=>"unlimited", "as"=>"unlimited", "nofile"=>64000, "rss"=>"unlimited", "nproc"=>32000}, "authentication"=>{"username"=>"admin", "password"=>"admin"}, "admin"=>{"username"=>"admin", "password"=>"admin", "roles"=>["userAdminAnyDatabase", "dbAdminAnyDatabase", "clusterAdmin"], "database"=>"admin"}, "users"=>[], "mongos_create_admin"=>true, "user_management"=>{"connection"=>{"retries"=>1, "delay"=>1}}, "mongod_create_user"=>{"retries"=>2, "delay"=>10}, "sharded_collections"=>{"foo.bar"=>"baz"}, "is_mongos"=>true}
==> s0:   username "admin"
==> s0: end
==> s0: 
==> s0: [2015-03-10T11:21:47+00:00] ERROR: Running exception handlers
==> s0: [2015-03-10T11:21:47+00:00] ERROR: Exception handlers complete
==> s0: [2015-03-10T11:21:47+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
==> s0: [2015-03-10T11:21:47+00:00] ERROR: mongodb_user[admin] (mongodb::user_management line 14) had an error: NoMethodError: undefined method `db' for nil:NilClass
==> s0: [2015-03-10T11:21:48+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

[cjhubert]

Are you using the master branch or the user-management branch? Sorry for any confusion. Since my PRs were never merged into the chef mongodb cookbook repo, this project has been at a standstill.

I haven't tried to vagrant up this in a while, let me see if it still works. Chef being chef and all.

[cjhubert]

Well, all the servers came up successfully for me in my user-management branch, the admin user was created, and the replicaset was initiated correctly... But for some reason the config servers are acting a little weird (telling me mongo isn't up). I'm looking into it, though I can't promise I have a lot of spare time at the moment.

In the meantime, this seems like a wonderful opportunity to say, have you checked out MongoDB MMS? I haven't used it myself, but it might get rid of the hassle of chef for you.

[bearproof]

I tried the user-management branch.

I see in the logs from the mongos instance :

2015-03-10T14:25:16.259+0000 [mongosMain] options: { config: "mongodb.conf", net: { bindIp: "0.0.0.0", port: 27017 }, security: { keyFile: "/etc/mongodb.key" }, sharding: { configDB: "c0:27019,c1:27019,c2:27019" }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongodb.log" } }
2015-03-10T14:25:16.298+0000 [mongosMain] warning: config servers c0:27019 and c1:27019 differ
2015-03-10T14:25:16.301+0000 [mongosMain] warning: config servers c0:27019 and c1:27019 differ
2015-03-10T14:25:16.304+0000 [mongosMain] warning: config servers c0:27019 and c1:27019 differ
2015-03-10T14:25:16.307+0000 [mongosMain] warning: config servers c0:27019 and c1:27019 differ
2015-03-10T14:25:16.307+0000 [mongosMain] ERROR: could not verify that config servers are in sync :: caused by :: config servers c0:27019 and c1:27019 differ: {} vs { chunks: "d41d8cd98f00b204e9800998ecf8427e", databases: "0195c739c938fdafd9632279694ea64e", shards: "d41d8cd98f00b204e9800998ecf8427e", version: "190c88080511ad7bb760abd6f7a28cb2" }
2015-03-10T14:25:16.307+0000 [mongosMain] configServer connection startup check failed

I didn't try yet MongoDB MMS, but I'll take a look into it. Also, I'll do some more digging on this side.

Thanks.

[cjhubert]

Interesting. Though I'm a little confused, were you able to bring up all machines in the branch or do you still get the above chef exception?

Right now on my mongos instance, I see this acting normal:

2015-03-10T15:28:13.819+0000 [LockPinger] cluster c0:27019,c1:27019,c2:27019 pinged successfully at Tue Mar 10 15:28:13 2015 by distributed lock pinger 'c0:27019,c1:27019,c2:27019/s0:27017:1425997635:1804289383', sleeping for 30000ms
2015-03-10T15:28:16.945+0000 [Balancer] distributed lock 'balancer/s0:27017:1425997635:1804289383' acquired, ts : 54ff0d90184828a71574f5e2