Separate out server acceptance tests

passenger, thin, unicorn, and webrick can be tested independently
This implies differences to how tests are run, so documentation was
updated
The advantage here is no chance of bleed-over between test cases,
whereas the previous way required imperfect automated cleanup between
test cases.
This also permits testing one thing at a time to save a little bit of
time. Unfortunately, it also means that testing all cases requires
considerably longer because a new VM must be provisioned for each case.

Author: Daniel Dreier

CONTRIBUTING.md

@@ -1,25 +1,47 @@
 Testing
 =======
 
-Tests on the puppet-puppet module can be run via `rake`.
+Puppet-puppet is tested via beaker and rspec-puppet. Rspec-puppet tests are run
+automatically for pull requests via travis-ci, or can be triggered manually
+by running `rake spec` after doing a `bundle install`.
 
-#### Beaker acceptance tests
-```
-BEAKER_destroy=no BEAKER_provision=onpass rake acceptance
+Beaker tests are run slightly differently for puppet-puppet than for other
+beaker-tested projects because the test suite should be run independently for
+each test case to prevent contamination between environments. For example,
+running the apache-passenger based puppet master test case will cause obvious
+conflicts if the nginx-unicorn puppet master is subsequently built on the same
+virtual machine. Rspec tags are used to accomplish this in the test cases.
+
+Running beaker tests for unicorn puppetmaster on the default nodeset (debian 7):
+```bash
+rspec . --tag servertype:unicorn --pattern "spec/acceptance/**/*_spec.rb"
 ```
 
-Run beaker acceptance tests on debian 7:
+Same as above, but only only destroying the VM if the build is successful, to
+help troubleshooting:
 ```
-BEAKER_set=debian-73-x64 BEAKER_destroy=onpass BEAKER_provision=no rake acceptance
+BEAKER_destroy=onpass BEAKER_provision=yes rspec . --tag servertype:unicorn --pattern "spec/acceptance/**/*_spec.rb"
 ```
-See spec/acceptance/nodesets for a list of possible node names; use the filename without .yml.
+
+Same as above, but on a different nodeset (see `spec/acceptance/nodesets` for
+options):
+```bash
+BEAKER_set=sles-11sp1-x64 BEAKER_destroy=onpass BEAKER_provision=yes rspec . --tag servertype:unicorn --pattern "spec/acceptance/**/*_spec.rb"
+```
+
+The presence of an OS/distro in the nodeset list does not imply support. The
+SLES example above is expected to fail most test cases but is included to lower
+the bar for future contributors who want to add support for additional distros.
+
+The rake command includes acceptance testing tasks, but these should not be
+used because they will run all of the acceptance tests on the same VM, which
+is expected to fail.
 
 If a beaker test fails, you can SSH into the environment if you use BEAKER_PROVISION=onpass.
 The path of the vagrantfile will be `.vagrant/beaker_vagrant_files/debian-73-x64.yml`
 if you followed the above instructions, and slightly different if you used a
 different nodeset. `cd` to that directory and `vagrant ssh` to access the VM.
 The tests that ran are in /tmp with randomly generated filenames.
-```
 
 #### rspec-puppet tests
 (note that these are run automatically by travis CI on pull requests)

spec/acceptance/agent_spec.rb

@@ -2,7 +2,7 @@
 
 describe 'agent', :unless => UNSUPPORTED_PLATFORMS.include?(fact('osfamily')) do
   context 'with default parameters' do
-    it 'should run with no errors' do
+    it 'should run with no errors', :agent do
       pp = <<-EOS
       class { 'puppet::agent': }
       EOS

spec/acceptance/passenger_server_spec.rb

@@ -0,0 +1,25 @@
+require 'spec_helper_acceptance'
+
+describe 'passenger server', :unless => UNSUPPORTED_PLATFORMS.include?(fact('osfamily')) do
+  context 'running on passenger' do
+    it 'should run with no errors', :servertype => 'passenger', :webserver => 'apache' do
+      pp = <<-EOS
+        class { "puppet::server":
+          servertype   => 'passenger',
+          ca           => true,
+          servername   => $::hostname,
+        }
+      EOS
+
+      # Run it twice and test for idempotency
+      apply_manifest(pp, :catch_failures => true)
+      expect(apply_manifest(pp, :catch_failures => true).exit_code).to be_zero
+    end
+
+    describe port(8140) do
+      it {
+        should be_listening
+      }
+    end
+  end
+end

spec/acceptance/server_spec.rb

@@ -0,0 +1,40 @@
+require 'spec_helper_acceptance'
+
+describe 'thin server', :unless => UNSUPPORTED_PLATFORMS.include?(fact('osfamily')) do
+
+  context 'running on thin' do
+    it 'should run with no errors', :servertype => 'thin', :webserver => 'nginx' do
+      pp = <<-EOS
+        class { "puppet::server":
+          servertype   => 'thin',
+          ca           => true,
+        }
+      EOS
+
+      # Run it twice and test for idempotency
+      apply_manifest(pp, :catch_failures => true)
+      expect(apply_manifest(pp, :catch_failures => true).exit_code).to be_zero
+    end
+
+    describe package('thin') do
+      it {
+        should be_installed.by('gem')
+      }
+    end
+
+    describe service('thin-puppetmaster') do
+      it {
+        should be_enabled
+      }
+      it {
+        should be_running
+      }
+    end
+
+    describe port(8140) do
+      it {
+        should be_listening
+      }
+    end
+  end
+end

spec/acceptance/thin_server_spec.rb

@@ -0,0 +1,45 @@
+require 'spec_helper_acceptance'
+
+describe 'server', :unless => UNSUPPORTED_PLATFORMS.include?(fact('osfamily')) do
+  context 'running on unicorn' do
+    it 'should run with no errors', :servertype => 'unicorn', :webserver => 'nginx' do
+      pp = <<-EOS
+        class { "puppet::server":
+          servertype   => 'unicorn',
+          ca           => true,
+        }
+      EOS
+
+      # Run it twice and test for idempotency
+      apply_manifest(pp, :catch_failures => true)
+      expect(apply_manifest(pp, :catch_failures => true).exit_code).to be_zero
+    end
+  describe command('puppet agent --test --server puppet') do
+    its(:exit_status) { should eq 0 }
+    its(:stderr) { should_not match /Forbidden request:/ }
+    its(:stderr) { should_not match /Error:/ }
+  end
+
+    describe package('nginx') do
+      it {
+        should be_installed
+      }
+    end
+
+    describe service('nginx') do
+      it {
+        should be_enabled
+      }
+      it {
+        should be_running
+      }
+    end
+
+    describe port(8140) do
+      it {
+        should be_listening
+      }
+    end
+  end
+
+end

spec/acceptance/unicorn_server_spec.rb

@@ -0,0 +1,34 @@
+require 'spec_helper_acceptance'
+
+describe 'server', :unless => UNSUPPORTED_PLATFORMS.include?(fact('osfamily')) do
+  context 'running on webrick/standalone' do
+    it 'should run with no errors', :server => 'webrick', :webserver => 'builtin' do
+      pp = <<-EOS
+        class { "puppet::server":
+          servertype   => 'standalone',
+          ca           => true,
+        }
+      EOS
+
+      # Run it twice and test for idempotency
+      apply_manifest(pp, :catch_failures => true)
+      expect(apply_manifest(pp, :catch_failures => true).exit_code).to be_zero
+    end
+
+    describe service('puppetmaster') do
+      it {
+        should be_enabled
+      }
+      it {
+        should be_running
+      }
+    end
+
+    describe port(8140) do
+      it {
+        should be_listening
+      }
+    end
+  end
+
+end

spec/acceptance/webrick_server_spec.rb

@@ -58,6 +58,9 @@
   EOS
   on host, "echo \"#{puppetfile}\" > /etc/puppet/Puppetfile"
   on host, "cd /etc/puppet; r10k puppetfile install"
+  on host, "mkdir -p /etc/puppet/environments/production/modules"
+  on host, "puppet config set --section master environmentpath '$confdir/environments'"
+  on host, "puppet config set --section master basemodulepath '$confdir/modules'"
 
   proj_root = File.expand_path(File.join(File.dirname(__FILE__), '..'))
   puppet_module_install(:source => proj_root, :module_name => 'puppet')