Skip to content

Bug: proxy_component_test and other tests are unstable #57

@lurtz

Description

@lurtz

Description

The proxy_component_test fails when run concurrently with other tests:

vscode ➜ /workspaces/communication (main) $ bazel test --nocache_test_results //...
WARNING: /workspaces/communication/score/mw/com/impl/rust/BUILD:51:11: target '//score/mw/com/impl/rust:proxy_bridge_macros' is deprecated: Please change your dependency to //score/mw/com/impl/rust:bridge_macros
INFO: Analyzed 398 targets (0 packages loaded, 139 targets configured).
FAIL: //score/mw/com/impl/bindings/lola/test:proxy_component_test (see /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/test.log)
INFO: Found 263 targets and 135 test targets...
INFO: Elapsed time: 14.689s, Critical Path: 12.30s
INFO: 134 processes: 151 processwrapper-sandbox.
INFO: Build completed, 1 test FAILED, 134 total actions
//score/mw/com/message_passing:resmgr_factory_test                      SKIPPED
//score/mw/com/message_passing:resmgr_traits_test                       SKIPPED
//bazel/tools:invalid_schema_and_valid_json_test                         PASSED in 0.1s
...
//score/mw/com/message_passing:unit_test                                 PASSED in 0.4s
//score/mw/com/impl/bindings/lola/test:proxy_component_test              FAILED in 2.5s
  /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/test.log

Executed 133 out of 135 tests: 132 tests pass, 1 fails locally and 2 were skipped.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.

This failure does not always happen, but often enough to be annoying.

When the test itself is run in parallel with multiple instances, each one is failing:

vscode ➜ /workspaces/communication (main) $ bazel test //score/mw/com/impl/bindings/lola/test:proxy_component_test  --runs_per_test=12
INFO: Analyzed target //score/mw/com/impl/bindings/lola/test:proxy_component_test (0 packages loaded, 5 targets configured).
FAIL: //score/mw/com/impl/bindings/lola/test:proxy_component_test (run 10 of 12) (see /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/run_10_of_12/test.log)
FAIL: //score/mw/com/impl/bindings/lola/test:proxy_component_test (run 7 of 12) (see /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/run_7_of_12/test.log)
FAIL: //score/mw/com/impl/bindings/lola/test:proxy_component_test (run 4 of 12) (see /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/run_4_of_12/test.log)
FAIL: //score/mw/com/impl/bindings/lola/test:proxy_component_test (run 5 of 12) (see /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/run_5_of_12/test.log)
FAIL: //score/mw/com/impl/bindings/lola/test:proxy_component_test (run 2 of 12) (see /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/run_2_of_12/test.log)
FAIL: //score/mw/com/impl/bindings/lola/test:proxy_component_test (run 9 of 12) (see /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/run_9_of_12/test.log)
FAIL: //score/mw/com/impl/bindings/lola/test:proxy_component_test (run 8 of 12) (see /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/run_8_of_12/test.log)
FAIL: //score/mw/com/impl/bindings/lola/test:proxy_component_test (run 6 of 12) (see /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/run_6_of_12/test.log)
FAIL: //score/mw/com/impl/bindings/lola/test:proxy_component_test (run 11 of 12) (see /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/run_11_of_12/test.log)
FAIL: //score/mw/com/impl/bindings/lola/test:proxy_component_test (run 3 of 12) (see /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/run_3_of_12/test.log)
FAIL: //score/mw/com/impl/bindings/lola/test:proxy_component_test (run 12 of 12) (see /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/run_12_of_12/test.log)
FAIL: //score/mw/com/impl/bindings/lola/test:proxy_component_test (run 1 of 12) (see /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/run_1_of_12/test.log)

FAILED: //score/mw/com/impl/bindings/lola/test:proxy_component_test (Summary)
      /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/run_10_of_12/test.log
      /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/run_7_of_12/test.log
      /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/run_4_of_12/test.log
      /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/run_5_of_12/test.log
      /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/run_2_of_12/test.log
      /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/run_9_of_12/test.log
      /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/run_8_of_12/test.log
      /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/run_6_of_12/test.log
      /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/run_11_of_12/test.log
      /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/run_3_of_12/test.log
      /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/run_12_of_12/test.log
      /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/run_1_of_12/test.log
INFO: Found 1 test target...
Target //score/mw/com/impl/bindings/lola/test:proxy_component_test up-to-date:
  bazel-bin/score/mw/com/impl/bindings/lola/test/proxy_component_test
INFO: Elapsed time: 1.774s, Critical Path: 1.38s
INFO: 13 processes: 24 processwrapper-sandbox.
INFO: Build completed, 1 test FAILED, 13 total actions
//score/mw/com/impl/bindings/lola/test:proxy_component_test              FAILED in 12 out of 12 in 1.3s
  Stats over 12 runs: max = 1.3s, min = 0.7s, avg = 1.1s, dev = 0.2s
  /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/run_10_of_12/test.log
  /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/run_7_of_12/test.log
  /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/run_4_of_12/test.log
  /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/run_5_of_12/test.log
  /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/run_2_of_12/test.log
  /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/run_9_of_12/test.log
  /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/run_8_of_12/test.log
  /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/run_6_of_12/test.log
  /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/run_11_of_12/test.log
  /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/run_3_of_12/test.log
  /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/run_12_of_12/test.log
  /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/test/proxy_component_test/run_1_of_12/test.log

Executed 1 out of 1 test: 1 fails locally.

in #39 the solution was to not run any test, which uses lola (/dev/shm, /tmp/mw_com_lola) in parallel. I did that because no other test was using the service id 52719 as used by the proxy_component_test and had no other idea how to make it reliable.

Ideally the tests are isolated from each other, but that is not the case when they do not mock lola.

Error Occurrence Rate

Highly Intermittent

How to reproduce

bazel test --nocache_test_results //...

or

bazel test //...  --runs_per_test=10

The last command shows that more tests than proxy_component_test are affected.

//score/mw/com/impl/bindings/lola/service_discovery/client:service_discovery_client_test FAILED in 9 out of 10 in 3.7s
...
//score/mw/com/impl/bindings/lola/test:proxy_component_test              FAILED in 10 out of 10 in 1.7s
...
//score/mw/com/impl/bindings/lola/test:skeleton_component_test           FAILED in 9 out of 10 in 1.0s
...
//score/mw/com/impl/bindings/lola/test:skeleton_event_component_test     FAILED in 7 out of 10 in 1.1s
...
//score/mw/com/impl/plumbing:skeleton_service_element_binding_factory_test FAILED in 9 out of 10 in 1.3s

Supporting Information

No response

Classification

minor

Affected Version

0.5

Expected Closure Version

0.5

Category

  • Safety Related
  • Security Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions