New make target check
authorMarkus Armbruster <armbru@pond.sub.org>
Sat, 14 Jul 2012 06:46:38 +0000 (08:46 +0200)
committerMarkus Armbruster <armbru@pond.sub.org>
Wed, 8 May 2013 04:55:11 +0000 (06:55 +0200)
commit49b2b13a908078b547b8472807e7702bf4a850db
treee5c9cd0ab471cbcfe888c37f48b06027616ddfc6
parent2fefdaed15a94f06a1000c564a13c65b4c5b0c6c
New make target check

Just a smoke test so far, extracted from src/scripts/nightly/.  This
makes the existing smoke test more easily accessible.  Noteworthy
differences:

* Instead of patching the code to make output more stable, postprocess
  the output to normalize it.

* Compare actual results to expected results instead of the previous
  test run's results.

* Much faster.  The old test harness used sleep liberally to "ensure"
  things always happen in the same order.

Known shortcomings:

* The smoke test hangs when the server fails to complete startup, or
  fails to terminate.

* Normalization of xdump hardcodes columns instead of getting them
  from xdump meta.

* Normalization of time values in xdump is an ugly hack.

* xdump meta column type isn't normalized.  Actual values can vary
  between systems, because the width of enumeration types is
  implementation-defined.  The smoke test works only when they're
  represented as int, which is the case on common systems.

* Currently expected to work only with thread package LWP and a
  random() that behaves exactly like the one on my development system,
  because:

  - Thread scheduling is reliably deterministic only with LWP

  - The PRN sequence produced by random() isn't portable

  - Shell builtin kill appears not to do the job in MinGW

  - The Windows server tries to run as service when -d isn't
    specified

Further work is needed to address these shortcomings.

Getting C programs behave exactly the same on all systems is hard.
We'll likely run into system-dependent differences that upset the
smoke test.  Floating-point computation seems particularly vulnerable.

Instead of updating src/scripts/nightly/ to use "make check", retire
it.  It hasn't been used in quite a while.  Investing more into our
homegrown auto-builder doesn't make sense, as canned auto-builders
such as Travis CI and Jenkins are readily available.

The shell scripts src/scripts/nightly/tests/?? become Empire batch
files tests/smoke/.  The shell scripts are actually shell boilerplate
around Empire batch files.  To make sure git recognizes the move, this
commit moves them unchanged.  tests/smoke-test strips the boilerplate
before it feeds the batch files to the client.  The next commit will
get rid fo that.
115 files changed:
Make.mk
src/scripts/nightly/.gitignore [deleted file]
src/scripts/nightly/common.sh [deleted file]
src/scripts/nightly/conf/ubuntu-lwp.i386.config [deleted file]
src/scripts/nightly/conf/ubuntu-pthread.i386.config [deleted file]
src/scripts/nightly/conf/win32.i386.config [deleted file]
src/scripts/nightly/cron/cron.ubuntu.i386 [deleted file]
src/scripts/nightly/cron/cron.win32.i386 [deleted file]
src/scripts/nightly/diff_to_prev.sh [deleted file]
src/scripts/nightly/nightlybuild.sh [deleted file]
src/scripts/nightly/patches/All/README [deleted file]
src/scripts/nightly/patches/All/emptime.c.patch [deleted file]
src/scripts/nightly/patches/All/journal.econfig [deleted file]
src/scripts/nightly/patches/All/pr.c.patch [deleted file]
src/scripts/nightly/patches/All/pre_update.econfig [deleted file]
src/scripts/nightly/patches/win32.i386/mingw.econfig [deleted file]
src/scripts/nightly/report.sh [deleted file]
src/scripts/nightly/tests/00/00 [deleted file]
src/scripts/nightly/tests/00/01 [deleted file]
src/scripts/nightly/tests/00/02 [deleted file]
src/scripts/nightly/tests/00/03 [deleted file]
src/scripts/nightly/tests/00/04 [deleted file]
src/scripts/nightly/tests/00/05 [deleted file]
src/scripts/nightly/tests/00/06 [deleted file]
src/scripts/nightly/tests/00/07 [deleted file]
src/scripts/nightly/tests/00/08 [deleted file]
src/scripts/nightly/tests/00/09 [deleted file]
src/scripts/nightly/tests/00/10 [deleted file]
src/scripts/nightly/tests/01/01 [deleted file]
src/scripts/nightly/tests/02/01 [deleted file]
src/scripts/nightly/tests/03/01 [deleted file]
src/scripts/nightly/tests/03/08 [deleted file]
src/scripts/nightly/tests/04/01 [deleted file]
src/scripts/nightly/tests/04/08 [deleted file]
src/scripts/nightly/tests/05/01 [deleted file]
src/scripts/nightly/tests/06/01 [deleted file]
src/scripts/nightly/tests/06/08 [deleted file]
src/scripts/nightly/tests/07/01 [deleted file]
src/scripts/nightly/tests/07/08 [deleted file]
src/scripts/nightly/tests/08/01 [deleted file]
src/scripts/nightly/tests/08/08 [deleted file]
src/scripts/nightly/tests/09/01 [deleted file]
src/scripts/nightly/tests/09/08 [deleted file]
src/scripts/nightly/tests/10/01 [deleted file]
src/scripts/nightly/tests/10/08 [deleted file]
src/scripts/nightly/tests/11/01 [deleted file]
src/scripts/nightly/tests/11/08 [deleted file]
src/scripts/nightly/tests/12/01 [deleted file]
src/scripts/nightly/tests/12/08 [deleted file]
src/scripts/nightly/tests/13/01 [deleted file]
src/scripts/nightly/tests/13/02 [deleted file]
src/scripts/nightly/tests/13/08 [deleted file]
src/scripts/nightly/tests/14/01 [deleted file]
src/scripts/nightly/tests/14/02 [deleted file]
src/scripts/nightly/tests/14/08 [deleted file]
src/scripts/nightly/tests/15/01 [deleted file]
src/scripts/nightly/tests/15/02 [deleted file]
src/scripts/nightly/tests/15/08 [deleted file]
src/scripts/nightly/tests/16/01 [deleted file]
src/scripts/nightly/tests/16/02 [deleted file]
src/scripts/nightly/tests/16/08 [deleted file]
src/scripts/nightly/tests/99/00 [deleted file]
src/scripts/nightly/tests/99/01 [deleted file]
src/scripts/nightly/tests/update [deleted file]
tests/normalize.pl [new file with mode: 0755]
tests/smoke-test [new file with mode: 0755]
tests/smoke/00/00 [new file with mode: 0644]
tests/smoke/00/01 [new file with mode: 0644]
tests/smoke/00/02 [new file with mode: 0644]
tests/smoke/00/03 [new file with mode: 0644]
tests/smoke/00/04 [new file with mode: 0644]
tests/smoke/00/05 [new file with mode: 0644]
tests/smoke/00/06 [new file with mode: 0644]
tests/smoke/00/07 [new file with mode: 0644]
tests/smoke/00/08 [new file with mode: 0644]
tests/smoke/00/09 [new file with mode: 0644]
tests/smoke/00/10 [new file with mode: 0644]
tests/smoke/01/01 [new file with mode: 0644]
tests/smoke/02/01 [new file with mode: 0644]
tests/smoke/03/01 [new file with mode: 0644]
tests/smoke/03/08 [new file with mode: 0644]
tests/smoke/04/01 [new file with mode: 0644]
tests/smoke/04/08 [new file with mode: 0644]
tests/smoke/05/01 [new file with mode: 0644]
tests/smoke/06/01 [new file with mode: 0644]
tests/smoke/06/08 [new file with mode: 0644]
tests/smoke/07/01 [new file with mode: 0644]
tests/smoke/07/08 [new file with mode: 0644]
tests/smoke/08/01 [new file with mode: 0644]
tests/smoke/08/08 [new file with mode: 0644]
tests/smoke/09/01 [new file with mode: 0644]
tests/smoke/09/08 [new file with mode: 0644]
tests/smoke/10/01 [new file with mode: 0644]
tests/smoke/10/08 [new file with mode: 0644]
tests/smoke/11/01 [new file with mode: 0644]
tests/smoke/11/08 [new file with mode: 0644]
tests/smoke/12/01 [new file with mode: 0644]
tests/smoke/12/08 [new file with mode: 0644]
tests/smoke/13/01 [new file with mode: 0644]
tests/smoke/13/02 [new file with mode: 0644]
tests/smoke/13/08 [new file with mode: 0644]
tests/smoke/14/01 [new file with mode: 0644]
tests/smoke/14/02 [new file with mode: 0644]
tests/smoke/14/08 [new file with mode: 0644]
tests/smoke/15/01 [new file with mode: 0644]
tests/smoke/15/02 [new file with mode: 0644]
tests/smoke/15/08 [new file with mode: 0644]
tests/smoke/16/01 [new file with mode: 0644]
tests/smoke/16/02 [new file with mode: 0644]
tests/smoke/16/08 [new file with mode: 0644]
tests/smoke/99/00 [new file with mode: 0644]
tests/smoke/99/01 [new file with mode: 0644]
tests/smoke/journal.log [new file with mode: 0644]
tests/smoke/server.log [new file with mode: 0644]
tests/smoke/smoke.out [new file with mode: 0644]