Upgrade from 3.1.0 -> 4.0.0 causes ~10% of tests to intermittently fail

[kategengler]

Still investigating and endeavoring to reproduce. The upgrade on an app (full webpack Embroider) has a failing dependabot PR that I finally got a chance to look at -- ~10% of the tests intermittently fail with the upgrade when the suite is run as a whole but pass when run individually.

[NullVoxPopuli]

what is the output of pnpm why -r @ember/test-waiters?

[Techn1x]

Experiencing the same issue, also on latest embroider/webpack etc.

Running pnpm why -r @ember/test-waiters I could see that test-waiters v3 was being used by various addons still, so I added test-waiters v4 to my pnpm overrides to see if I could get around that........ but then I hit
@ember-data/request needs to depend on ember-auto-import in order to use @ember/test-waiters
which should be fixed by warp-drive-data/warp-drive#9614 and warp-drive-data/warp-drive#9619 but I'm stuck on ember-data 5.3.3 due to other problems.

Hope that helps

[kategengler]

I finally got back to this and had the same experience as @Techn1x

[NullVoxPopuli]

@kategengler problems resulting in needing an ember-data upgrade?

does the problem still exist with test-helpers 4.0.5? (or v5.1)

[kategengler]

When I add the override to force @ember/test-waiters v4, I get @ember-data/request needs to depend on ember-auto-import in order to use @ember/test-waiters, which it sounds like would require an ember-data upgrade to resolve. That's not an option for the project I am on, right now.

@ember/test-helpers is at 4.1.0, I haven't tried 5.1 yet.

[NullVoxPopuli]

is everyone using embroider?

[amk221]

Also experiencing tests failing on random things, for example
e.g.

await click('button');

console.log(find('.modal')); // randomly null 

await new Promise((resolve) => setTimeout(resolve, 10));

console.log(find('.modal')); // present

[Techn1x]

I got lucky and was able to update the projects I'm looking after to later versions of ember-data, from 5.3.3 -> 5.3.11, and I no longer have this issue. Big relief to get that update done, unblocked on a lot of things and the types are so much better.

Was able to do this without adding test-waiters 4 to pnpm overrides

is everyone using embroider?

yes

[NullVoxPopuli]

For everyone with this issue, could you comment back with two pieces of information?

pnpm why @ember/test-waiters

pnpx y-which @ember/test-waiters 

Or, a repro

Thanks!

[jrjohnson]

It's a full app reproduction, but you can see this in ilios/frontend#8418. and I'm guessing our mixed list of versions is part of the issue so I've been waiting on an ember-data release with this upgrade in it. In case it helps though, here is what I've got.

$ pnpm --filter frontend why @ember/test-waiters

Legend: production dependency, optional only, dev only

frontend@43.1.0 /Users/jrjohnson/Documents/code/ilios/frontend/packages/frontend (PRIVATE)

devDependencies:
@ember-data/adapter 5.3.11
├─┬ @ember-data/legacy-compat 5.3.11 peer
│ ├─┬ @ember-data/graph 5.3.11 peer
│ │ └─┬ @ember-data/store 5.3.11 peer
│ │   └─┬ @ember-data/request 5.3.11 peer
│ │     └── @ember/test-waiters 4.0.0
│ ├─┬ @ember-data/json-api 5.3.11 peer
│ │ ├─┬ @ember-data/graph 5.3.11 peer
│ │ │ └─┬ @ember-data/store 5.3.11 peer
│ │ │   └─┬ @ember-data/request 5.3.11 peer
│ │ │     └── @ember/test-waiters 4.0.0
│ │ └─┬ @ember-data/store 5.3.11 peer
│ │   └─┬ @ember-data/request 5.3.11 peer
│ │     └── @ember/test-waiters 4.0.0
│ ├─┬ @ember-data/request 5.3.11 peer
│ │ └── @ember/test-waiters 4.0.0
│ ├─┬ @ember-data/store 5.3.11 peer
│ │ └─┬ @ember-data/request 5.3.11 peer
│ │   └── @ember/test-waiters 4.0.0
│ └── @ember/test-waiters 4.0.0 peer
└─┬ @ember-data/store 5.3.11 peer
  └─┬ @ember-data/request 5.3.11 peer
    └── @ember/test-waiters 4.0.0
@ember-data/json-api 5.3.11
├─┬ @ember-data/graph 5.3.11 peer
│ └─┬ @ember-data/store 5.3.11 peer
│   └─┬ @ember-data/request 5.3.11 peer
│     └── @ember/test-waiters 4.0.0
└─┬ @ember-data/store 5.3.11 peer
  └─┬ @ember-data/request 5.3.11 peer
    └── @ember/test-waiters 4.0.0
@ember-data/legacy-compat 5.3.11
├─┬ @ember-data/graph 5.3.11 peer
│ └─┬ @ember-data/store 5.3.11 peer
│   └─┬ @ember-data/request 5.3.11 peer
│     └── @ember/test-waiters 4.0.0
├─┬ @ember-data/json-api 5.3.11 peer
│ ├─┬ @ember-data/graph 5.3.11 peer
│ │ └─┬ @ember-data/store 5.3.11 peer
│ │   └─┬ @ember-data/request 5.3.11 peer
│ │     └── @ember/test-waiters 4.0.0
│ └─┬ @ember-data/store 5.3.11 peer
│   └─┬ @ember-data/request 5.3.11 peer
│     └── @ember/test-waiters 4.0.0
├─┬ @ember-data/request 5.3.11 peer
│ └── @ember/test-waiters 4.0.0
├─┬ @ember-data/store 5.3.11 peer
│ └─┬ @ember-data/request 5.3.11 peer
│   └── @ember/test-waiters 4.0.0
└── @ember/test-waiters 4.0.0 peer
@ember-data/model 5.3.11
├─┬ @ember-data/graph 5.3.11 peer
│ └─┬ @ember-data/store 5.3.11 peer
│   └─┬ @ember-data/request 5.3.11 peer
│     └── @ember/test-waiters 4.0.0
├─┬ @ember-data/json-api 5.3.11 peer
│ ├─┬ @ember-data/graph 5.3.11 peer
│ │ └─┬ @ember-data/store 5.3.11 peer
│ │   └─┬ @ember-data/request 5.3.11 peer
│ │     └── @ember/test-waiters 4.0.0
│ └─┬ @ember-data/store 5.3.11 peer
│   └─┬ @ember-data/request 5.3.11 peer
│     └── @ember/test-waiters 4.0.0
├─┬ @ember-data/legacy-compat 5.3.11 peer
│ ├─┬ @ember-data/graph 5.3.11 peer
│ │ └─┬ @ember-data/store 5.3.11 peer
│ │   └─┬ @ember-data/request 5.3.11 peer
│ │     └── @ember/test-waiters 4.0.0
│ ├─┬ @ember-data/json-api 5.3.11 peer
│ │ ├─┬ @ember-data/graph 5.3.11 peer
│ │ │ └─┬ @ember-data/store 5.3.11 peer
│ │ │   └─┬ @ember-data/request 5.3.11 peer
│ │ │     └── @ember/test-waiters 4.0.0
│ │ └─┬ @ember-data/store 5.3.11 peer
│ │   └─┬ @ember-data/request 5.3.11 peer
│ │     └── @ember/test-waiters 4.0.0
│ ├─┬ @ember-data/request 5.3.11 peer
│ │ └── @ember/test-waiters 4.0.0
│ ├─┬ @ember-data/store 5.3.11 peer
│ │ └─┬ @ember-data/request 5.3.11 peer
│ │   └── @ember/test-waiters 4.0.0
│ └── @ember/test-waiters 4.0.0 peer
└─┬ @ember-data/store 5.3.11 peer
  └─┬ @ember-data/request 5.3.11 peer
    └── @ember/test-waiters 4.0.0
@ember-data/request 5.3.11
└── @ember/test-waiters 4.0.0
@ember-data/serializer 5.3.11
├─┬ @ember-data/legacy-compat 5.3.11 peer
│ ├─┬ @ember-data/graph 5.3.11 peer
│ │ └─┬ @ember-data/store 5.3.11 peer
│ │   └─┬ @ember-data/request 5.3.11 peer
│ │     └── @ember/test-waiters 4.0.0
│ ├─┬ @ember-data/json-api 5.3.11 peer
│ │ ├─┬ @ember-data/graph 5.3.11 peer
│ │ │ └─┬ @ember-data/store 5.3.11 peer
│ │ │   └─┬ @ember-data/request 5.3.11 peer
│ │ │     └── @ember/test-waiters 4.0.0
│ │ └─┬ @ember-data/store 5.3.11 peer
│ │   └─┬ @ember-data/request 5.3.11 peer
│ │     └── @ember/test-waiters 4.0.0
│ ├─┬ @ember-data/request 5.3.11 peer
│ │ └── @ember/test-waiters 4.0.0
│ ├─┬ @ember-data/store 5.3.11 peer
│ │ └─┬ @ember-data/request 5.3.11 peer
│ │   └── @ember/test-waiters 4.0.0
│ └── @ember/test-waiters 4.0.0 peer
└─┬ @ember-data/store 5.3.11 peer
  └─┬ @ember-data/request 5.3.11 peer
    └── @ember/test-waiters 4.0.0
@ember-data/store 5.3.11
└─┬ @ember-data/request 5.3.11 peer
  └── @ember/test-waiters 4.0.0
@ember/test-helpers 4.0.5
└── @ember/test-waiters 4.0.0
@embroider/router 2.1.8
└── @ember/test-waiters 3.1.0
ember-a11y-testing 7.1.0
├─┬ @ember/test-helpers 4.0.5 peer
│ └── @ember/test-waiters 4.0.0
└── @ember/test-waiters 3.1.0
ember-async-data 2.0.0
└── @ember/test-waiters 4.0.0
ember-cli-page-object 2.3.1
└─┬ @ember/test-helpers 4.0.5 peer
  └── @ember/test-waiters 4.0.0
ember-exam 9.1.0
└─┬ ember-qunit 9.0.1 peer
  └─┬ @ember/test-helpers 4.0.5 peer
    └── @ember/test-waiters 4.0.0
ember-mirage 0.3.3
└─┬ @ember/test-helpers 2.9.4
  └── @ember/test-waiters 3.1.0
ember-qunit 9.0.1
└─┬ @ember/test-helpers 4.0.5 peer
  └── @ember/test-waiters 4.0.0
ilios-common link:../ilios-common
├── @ember/test-waiters 4.0.0
├─┬ ember-async-data 2.0.0
│ └── @ember/test-waiters 4.0.0
├─┬ ember-cli-page-object 2.3.1
│ └─┬ @ember/test-helpers 4.0.5 peer
│   └── @ember/test-waiters 4.0.0
├─┬ ember-file-upload 9.3.0
│ ├─┬ @ember/test-helpers 4.0.5 peer
│ │ └── @ember/test-waiters 4.0.0
│ └── @ember/test-waiters 4.0.0
├─┬ ember-intl 7.1.4
│ └─┬ @ember/test-helpers 4.0.5 peer
│   └── @ember/test-waiters 4.0.0
├─┬ ember-simple-auth 7.1.3
│ ├─┬ @ember/test-helpers 4.0.5 peer
│ │ └── @ember/test-waiters 4.0.0
│ └── @ember/test-waiters 3.1.0
└─┬ ember-simple-charts 12.1.1
  └─┬ ember-async-data 2.0.0
    └── @ember/test-waiters 4.0.0

As it's installed in an add-on in this monorepo I switched to that package first packages/ilios-common and then ran

pnpx y-which @ember/test-waiters

  Node resolves @ember/test-waiters 
       to   ~/Documents/code/ilios/frontend/node_modules/.pnpm/@ember+test-waiters@4.0.0/node_modules/@ember/test-waiters/dist/index.js
       which is @ 4.0.0

     from   ilios-common @ 87.1.0 
       at   ~/Documents/code/ilios/frontend/packages/ilios-common

Edit: updated after removing and re-creating lock file.

[NullVoxPopuli]

test-helpers supports v4:

• https://github.com/emberjs/ember-test-helpers/blob/master/addon/package.json#L54

ember-async-data supports v4

• https://github.com/tracked-tools/ember-async-data/blob/main/ember-async-data/package.json#L54 (only in latest release)

your version of ember-data supports v4:

• https://github.com/emberjs/data/blob/v5.3.11/packages/request/package.json#L46

try deleting all occurances of test-waiters from your lockfile and re-install.
(or if you can delete your lockfile entirely, that would be easier)

[jrjohnson]

🤦, of course. I editing my above comment with the re-created lock file's version of why and pushed the branch back up.

[NullVoxPopuli]

This appears like it could be sus:

ember-a11y-testing 7.1.0
├─┬ @ember/test-helpers 4.0.5 peer
│ └── @ember/test-waiters 4.0.0
└── @ember/test-waiters 3.1.0

Opened a PR here: ember-a11y/ember-a11y-testing#589

in the mean time, you could attempt to use an override for test-waiters v4 -- it would require that all addons depending on test-waiters already support loading v2 addons (via auto-import).

that said,

test-waiters 3.x is supposed to be compatible with 4.x when both loaded at the same time.
So I have a hunch something is goofy.
that work was done here: https://github.com/emberjs/ember-test-waiters/releases/tag/v3.0.2