pending and skip do not work in tests that include the reactor context

[rdubya]

When trying to use skip or pending inside of examples, the examples do not function as expected. skip results in the call being ignored and the test appearing that it has passed and pending throws an exception. The following script should be able to be used to reproduce:

require 'async/rspec'

RSpec.describe 'test pending and skip' do
	include_context Async::RSpec::Reactor

	it 'should show as pending if skipped' do
		skip 'this should be skipped'
		expect(true).to be false
	end

	it 'should show as pending if pending is called' do
		pending 'this is pending'
		expect(true).to be false
	end
end

When I run it I get the following output:

test pending and skip
  should show as pending if skipped
  should show as pending if pending is called (FAILED - 1)

Failures:

  1) test pending and skip should show as pending if pending is called
     Failure/Error: pending 'this is pending'

     RuntimeError:
       `pending` may not be used outside of examples, such as in before(:context). Maybe you want `skip`?
     # /Users/rdubya/.rvm/gems/jruby-9.1.17.0/gems/rspec-core-3.8.0/lib/rspec/core/pending.rb:94:in `pending'
     # ./spec/test_spec.rb:107:in `block in (root)'
     # org/jruby/RubyBasicObject.java:1728:in `instance_exec'
     # /Users/rdubya/.rvm/gems/jruby-9.1.17.0/gems/rspec-core-3.8.0/lib/rspec/core/example.rb:254:in `block in run'
     # /Users/rdubya/.rvm/gems/jruby-9.1.17.0/gems/rspec-core-3.8.0/lib/rspec/core/example.rb:500:in `block in with_around_and_singleton_context_hooks'
     # /Users/rdubya/.rvm/gems/jruby-9.1.17.0/gems/rspec-core-3.8.0/lib/rspec/core/example.rb:457:in `block in with_around_example_hooks'
     # /Users/rdubya/.rvm/gems/jruby-9.1.17.0/gems/rspec-core-3.8.0/lib/rspec/core/hooks.rb:464:in `block in run'
     # /Users/rdubya/.rvm/gems/jruby-9.1.17.0/gems/rspec-core-3.8.0/lib/rspec/core/hooks.rb:604:in `block in run_around_example_hooks_for'
     # /Users/rdubya/.rvm/gems/jruby-9.1.17.0/gems/rspec-core-3.8.0/lib/rspec/core/example.rb:342:in `call'
     # /Users/rdubya/.rvm/gems/jruby-9.1.17.0/gems/async-rspec-1.12.0/lib/async/rspec/reactor.rb:55:in `block in run_example'
     # /Users/rdubya/.rvm/gems/jruby-9.1.17.0/gems/async-1.15.1/lib/async/task.rb:199:in `block in make_fiber'

Finished in 0.1073 seconds (files took 2.97 seconds to load)
2 examples, 1 failure

Failed examples:

rspec ./spec/test_spec.rb:106 # test pending and skip should show as pending if pending is called

This was ran in the context of a full test suite so there is potentially something else at play, but the basic issue seems to be that RSpec.current_example is nil so it can't be flagged like it should be. The current example is stored on the thread so I'm guessing it has something to do with how the example is being ran in a different thread, but haven't tracked down exactly where the break is.

[ioquatix]

Thanks for reporting this issue. Could you repro on MRI?

[ioquatix]

Please make a PR with your repro.

[rdubya]

Hi @ioquatix,
I can reproduce with mri using that same script. I'm pretty busy right now so at this point I'm just working around needing to use them. If I get a chance I'll create a PR but I don't know if/when that will be.

[emiltin]

We're not using async-rspec, but we're using Async and have support code that does much the same thing: run examples inside an async task.
We had the same problem. What I found was that it seems rspec depends on thread-local storage (which is actually fiber-local), including a reference to the current example. When you call skip() inside an async task, rspec therefore cannot set the skip flag correctly.

This is in line with what @rdubya mentions above.

A work-around that seems to work is to copy thead-local storage from current fiber when you enter around_each() and then insert it into the fiber of the async task:

  # called by rspec when each example is being run
  def self.around_each example
    thread_local_data = RSpec::Support.thread_local_data
    Async do |task|
      thread_local_data.each_pair { |key,value| RSpec::Support.thread_local_data[key] = value }
      example.run
    end
  end

Calling skip() inside an example now works, but I'm a bit worried whether it could have unintended side-effects. A better solutions is probably needed.

[ioquatix]

Yeah that makes sense. We will try to introduce a better way.