build: add pummel tests to ci runs #34289

Trott · 2020-07-10T05:06:48Z

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
commit message follows commit guidelines

gengjiawen · 2020-07-10T08:41:02Z

ASAN check failed with, since it works in normal test, we may need to skip it in asan, like in

node/test/parallel/test-crypto-dh-leak.js

Lines 7 to 8 in d5e1795

    
           if (process.config.variables.asan) 
        
             common.skip('ASAN messes with memory measurements');

Fail log:

AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:

  assert.ok(maxMem < 64 * 1024 * 1024)

    at process.<anonymous> (/home/runner/work/node/node/test/pummel/test-vm-memleak.js:61:10)
    at process.emit (events.js:326:22) {
  generatedMessage: true,
  code: 'ERR_ASSERTION',
  actual: false,
  expected: true,
  operator: '=='
}
Command: out/Release/node --max_old_space_size=32 /home/runner/work/node/node/test/pummel/test-vm-memleak.js

Trott · 2020-07-12T18:23:20Z

ASAN check failed with, since it works in normal test, we may need to skip it in asan, like in

Is this something you're seeing in CI or testing locally?

Same test failed on AIX. https://ci.nodejs.org/job/node-test-commit-aix/31507/nodes=aix71-ppc64/testReport/junit/(root)/test/pummel_test_vm_memleak/

gengjiawen · 2020-07-27T02:41:01Z

on macOS, not sure flaky

Also assert.js:103
  throw new AssertionError(obj);
  ^

AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:

false !== true

    at Timeout._onTimeout (/Users/runner/work/node/node/test/pummel/test-timers.js:55:10)
    at listOnTimeout (internal/timers.js:555:17)
    at processTimers (internal/timers.js:498:7) {
  generatedMessage: true,
  code: 'ERR_ASSERTION',
  actual: false,
  expected: true,
  operator: 'strictEqual'
}
Command: out/Release/node /Users/runner/work/node/node/test/pummel/test-timers.js

PR-URL: #34289 Reviewed-By: Richard Lau <rlau@redhat.com>

The pummel tests result in the Windows coverage runs in CI to exhaust memory, so we need to bump up the heap size. PR-URL: #34289 Reviewed-By: Richard Lau <rlau@redhat.com>

PR-URL: #34289 Reviewed-By: Richard Lau <rlau@redhat.com>

Trott · 2021-04-10T22:19:00Z

Landed in 6a1986d...2853b76

targos · 2021-04-20T15:11:26Z

These tests take quite a long time on the Raspberry Pis (and sometimes fail with a timeout). Could we disable pummel tests on node-test-binary-arm-12+ ?

Trott · 2021-04-20T23:00:41Z

These tests take quite a long time on the Raspberry Pis (and sometimes fail with a timeout). Could we disable pummel tests on node-test-binary-arm-12+ ?

Looking at Pi 2 results in https://ci.nodejs.org/job/node-test-binary-arm-12+/10400/, the longest running tests (and how long it took them to run in seconds) are:

pummel/test-crypto-dh-hash-modp18 (700.975)
pummel/test-policy-integrity (696.201)
pummel/test-crypto-dh-hash (451.29)
pummel/test-dh-regr (311.406)
pummel/test-next-tick-infinite-calls (260.476)
parallel/test-webcrypto-derivebits-pbkdf2 (170.465)
parallel/test-heapsnapshot-near-heap-limit (136.232)
sequential/test-net-bytes-per-incoming-chunk-overhead (82.495)
pummel/test-fs-watch-system-limit (75.903)
parallel/test-heapsnapshot-near-heap-limit-bounded (72.233)

I think it makes sense to:

move those above that aren't already in pummel into pummel,
move the pummel tests that only take a second or two to run to somewhere outside of pummel (although we'll need to check them for why they are in pummel in the first place--for example, if it's disk usage, that might warrant staying in pummel even if they only take a few seconds to run)
set up pummel tests to skip on Raspberry Pi.

I'll get to work on this tonight and hopefully have a PR together quickly.

Skipped the longest-running pummel tests on the Raspberry Pi devices in CI. Refs: nodejs#34289 (comment)

Move slower tests to pummel and skip on Raspberry Pi devices in CI. Refs: nodejs#34289 (comment)

Skipped the longest-running pummel tests on the Raspberry Pi devices in CI. Refs: nodejs#34289 (comment) PR-URL: nodejs#38394 Reviewed-By: Michaël Zasso <targos@protonmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Filip Skokan <panva.ip@gmail.com>

Move slower tests to pummel and skip on Raspberry Pi devices in CI. Refs: nodejs#34289 (comment) PR-URL: nodejs#38395 Reviewed-By: Michaël Zasso <targos@protonmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Filip Skokan <panva.ip@gmail.com>

Skipped the longest-running pummel tests on the Raspberry Pi devices in CI. Refs: #34289 (comment) PR-URL: #38394 Reviewed-By: Michaël Zasso <targos@protonmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Filip Skokan <panva.ip@gmail.com>

Move slower tests to pummel and skip on Raspberry Pi devices in CI. Refs: #34289 (comment) PR-URL: #38395 Reviewed-By: Michaël Zasso <targos@protonmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Filip Skokan <panva.ip@gmail.com>

The check for an 800ms window makesw assumptions about a setTimeout() not running late etc. Remove it. Refs: #34289 PR-URL: #38060 Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Juan José Arboleda <soyjuanarbol@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

The test is too slow to run on the Raspberry Pi bots. (It takes over 500 seconds to run on Pi 3 bots and over 900 seconds on Pi 2 bots.) Skip it on armv6 and armv7. Refs: #34289 PR-URL: #38076 Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

The test is too slow to run on the Raspberry Pi bots. (It takes over 900 seconds to run on Pi 3 bots and even more than that on Pi 2 bots.) Skip it on armv6 and armv7. Refs: #34289 PR-URL: #34289 Reviewed-By: Richard Lau <rlau@redhat.com>

ASAN increases memory usage, which in turn messes up the memory leak test. Skip the test on ASAN. PR-URL: #34289 Reviewed-By: Richard Lau <rlau@redhat.com>

The test is too slow to run on the Raspberry Pi bots. (It takes over 900 seconds to run on Pi 3 bots and even more than that on Pi 2 bots.) Skip it on armv6 and armv7. Refs: #34289 PR-URL: #34289 Reviewed-By: Richard Lau <rlau@redhat.com>

ASAN increases memory usage, which in turn messes up the memory leak test. Skip the test on ASAN. PR-URL: #34289 Reviewed-By: Richard Lau <rlau@redhat.com>

Skipped the longest-running pummel tests on the Raspberry Pi devices in CI. Refs: #34289 (comment) PR-URL: #38394 Reviewed-By: Michaël Zasso <targos@protonmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Filip Skokan <panva.ip@gmail.com>

nodejs-github-bot added build windows labels Jul 10, 2020