Tests run fine on sauce until we enable parallelization

[lanceblais]

Hello,

We're using RSpec and SauceLabs to run our Selenium tests.

We've followed all the information on the Wiki and we're able to run bundle exec rspec spec/path/to/e2e.

When we enable parallelization and run bundle exec parallel_rspec spec/path/to/e2e we get this error:

     Failure/Error: launch_cms_homepage()
     StandardError:
       No browser has been configured.

       It seems you're trying to run your tests in parallel, but haven't configured your specs/tests to use the Sauce integration.

       To fix this, add :sauce => true to your specs or make your tests subclasses of Sauce::TestCase or Sauce::RailsTestCase.

       For more details check the gem readme at https://github.com/saucelabs/sauce_ruby/blob/master/README.markdown
     # ./page_objects/shared/launch_browsers.rb:4:in `launch_cms_homepage'
     # ./spec/features/end_to_end/1/ma_agenda/ma_agenda_fixes_spec.rb:678:in `block (2 levels) in <top (required)>'

Note: launch_cms_homepage() is a method defined in one of the helpers we pull in.

Has this been seen before?

A related question, what strategy does the parallelization take for a folder of specs?

The e2e folder has e2e/1/test1.rb which has a bunch of describe blocks and e2e/1/test2.rb also has a bunch of specs. Is the parallelization at the describe block level, the file level, or other?

Thank you!

$ bundle exec gem list

*** LOCAL GEMS ***

bigdecimal (1.2.6)
bundler (1.9.6)
capybara (2.4.4)
childprocess (0.5.2)
cmdparse (3.0.1)
diff-lcs (1.2.4)
domain_name (0.5.24)
ffi (1.9.3)
highline (1.7.2)
http-cookie (1.0.2)
io-console (0.4.2)
jar_wrapper (0.1.7)
json (1.8.2)
mime-types (2.0)
mini_portile (0.6.0)
minitest (4.7.5)
multi_json (1.8.2)
net-http-persistent (2.9.4)
net-ssh (2.9.2)
net-ssh-gateway (1.2.0)
netrc (0.10.3)
nokogiri (1.6.3.1)
parallel (1.6.0)
parallel_tests (1.3.7)
power_assert (0.2.2)
psych (2.0.8)
rack (1.5.2)
rack-test (0.6.2)
rake (0.9.2.2)
rdoc (3.9.5)
require_all (1.3.2)
rest-client (1.8.0)
rspec (2.13.0)
rspec-core (2.13.0)
rspec-expectations (2.13.0)
rspec-mocks (2.13.0)
rspec-retry (0.4.0)
rubyzip (1.0.0)
sauce (3.5.6)
sauce_whisk (0.0.18)
selenium (0.2.11)
selenium-webdriver (2.45.0)
unf (0.1.4)
unf_ext (0.0.7.1)
websocket (1.0.7)
websocket-driver (0.3.0)
websocket-extensions (0.1.2)
xpath (2.0.0)
zip (2.0.2)

[bootstraponline]

Have you tried running in parallel by using the Rake task?

bundle exec rake sauce:spec

[bootstraponline]

Is the parallelization at the describe block level, the file level, or other?

parallelization is performed using parallel_tests which runs one process per file.

[lanceblais]

@bootstraponline we're using rspec with a spec_helper. We aren't using a Rakefile. Any help on why we'd use that?

[bootstraponline]

It would be interesting to know if it worked via Rakefile or if the error remained the same.

[lanceblais]

Ah I see. Ok, I assume the details are on the page about setting up the Rakefile. Our QA Engineer just always ran rspec :-) -- I'll get back to you

[bootstraponline]

I have an example repo.

• Create Rakefile
• Add

require 'bundler/setup'
require 'sauce'

• Run bundle exec rake sauce:spec

[DylanLacey]

The Sauce gem patches parallal_test's test division like so:

1. You have tests Alpha (a), Beta (b) and Delta (d)
2. You have Platforms 1 and 2
3. You ask to run 4 threads in parallel
4. parallel_tests will divide up Alpha, Beta and Delta across your threads; By default, it does this by file
5. The Sauce patch will take the files found by the parallel_tests gem, and create an array of the cross-product of the files and the configured browsers, eg:

# psuedocode
files = ['a1', 'a2', 'b1', 'b2', 'd1', 'd2']

1. The Sauce patch then gives these files back to parallel_tests, which divides them amongst the threads to run. (We use the default division strategy of files; Trying to use a different division strategy may cause fun and exciting errors.) Threads are then allocated files in a round-robin pattern, distributing the same test on different platforms across multiple threads:

thread_1 = ['a1',d1']
thread_2 =  ['a2','d2']
thread_3 =  ['b1']
thread_4 = ['b2']

By doing tests in parallel in this manner, we have a greater chance of evening out test time. The default Sauce gem running pattern, that of running each test sequentially against each browser, doesn't parallelise as well; A long running test will run on every browser in turn in the same thread, potentially taking significantly longer then every other thread. Additionally, in instances where there is more available concurrency then tests, there's no way to use higher concurrency.

Unfortunately, running in this test-distributing fashion requires us to do some funky stuff to load the required test config before a test run, hence relying on a rake task. The gem complains about not finding any browser because it knows it ran from the parallel_test integration, but not that it didn't run from the Sauce task, so it thinks something has gone wrong.

It'd be possible to fix this, I guess, by setting an environment variable when the rake task runs, telling child tasks that they're running from the Sauce specific integration; If we were to do that, however, I'd still want to issue a message saying that parallelism of that kind isn't always as efficient.

[bootstraponline]

parallel_tests uses processes to run in parallel not threads (even though it waits for them to finish via threads). In the ideal situation, it'd be per test and not per file via parallel_split_test. Manually breaking large test files down into smaller files for better performance isn't fun.

[lanceblais]

@bootstraponline How do I make the Rakefile only run tests within a certain folder? We've got 2 old ones and a new one that we run and running bundle exec rake sauce:spec is looking in the other 2 folders.

[lanceblais]

The tests get kicked off on Saucelabs by running bundle exec rake spec but as soon as I add the sauce part to the command it fails by trying to look in other directories.

So: same issue as above.

require 'bundler/setup'
require 'sauce'
require 'rspec/core/rake_task'

RSpec::Core::RakeTask.new(:spec) do |t|
  t.pattern = 'spec/features/end_to_end/**/*.rb'
end

[bootstraponline]

@bootstraponline How do I make the Rakefile only run tests within a certain folder?

@lanceblais The sauce:spec rake task accepts options.

rake sauce:spec[files,concurrency,test_options,parallel_options] # Run specs in parallel on Sauce Labs

Try passing the target folder as the files option to restrict execution.

[lanceblais]

@bootstraponline

I was only able to give it one file at a time (unless there's some way to do regex) by running: bundle exec rake sauce:spec test_files="spec/features/end_to_end/1/ma_agenda/ma_fixes.rb

When I run this command I get exactly the same output as I did when running with parrallel_rspec above.

So to answer your question: rake sauce:spec and parallel_rspec are now giving me the same behaviour of no browser has been configured..

Sorry for the delay, now what?

[bootstraponline]

I recommend creating a reduced test case that reproduces the issue and uploading it to github. The example I have on GitHub is working fine so I think there's something unique about your configuration.