The other test output files have fixed names, and having just one with
a name that varies with the test name complicates things with no gain.
Signed-off-by: Markus Armbruster <armbru@pond.sub.org>
Warn the first time a server is started. Incorrect for info-test with
POSIX threads, so suppress the warning there.
Improve the warning message a bit while we're at it.
Signed-off-by: Markus Armbruster <armbru@pond.sub.org>
Normally, git's pre-commit hook protects us from them. However, when
expected test output contains trailing white space, we have to bypass
commit hooks. Unwanted space can then slip in if you don't pay
attention. I obviously didn't; clean up.
The previous commit should reduce the need for such hook suppression.
Signed-off-by: Markus Armbruster <armbru@pond.sub.org>
feed_input drops lines starting with a bar character '|', so they can
serve as comments. Syntax chosen because such lines shouldn't be
needed in tests. In particular, the server already ignores such lines
when it reads commands, because they get parsed as empty command with
a pipeline, and empty commands get ignored, regardless of
redirections.
Signed-off-by: Markus Armbruster <armbru@pond.sub.org>
Capturing the client's output tests both client and server, which is
nice. However, player input isn't visible in the resulting file,
which makes it more difficult to understand.
Route player output to journal (econfig key "keep_journal 2"), and
ignore client output.
Separate tests for the client would be useful.
Signed-off-by: Markus Armbruster <armbru@pond.sub.org>
random() may yield different pseudo-random number sequences for the
same seed on another system. For instance, at least some versions of
MinGW provide a random() in -liberty that differs from traditional BSD
(see commit c8231b12). Rather inconvenient for regression testing.
MT19937 Mersenne Twister is a proven, high-quality PRNG. Actual code
is reference code provided by the inventors[*]. Quick tests show
performance comparable to random().
Like random(), MT is not cryptographically secure: observing enough of
its output permits guessing its state, and thus its future output. I
don't think players can do that.
Drop the copy of BSD random() we added for Windows.
Like the previous commit, this changes the server's die rolls, and
makes fairland create a different random map for the same seed. Update
expected smoke test results accordingly.
[*] mt19937ar.sep.tgz downloaded from
http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/MT2002/emt19937ar.html
Just a smoke test so far, extracted from src/scripts/nightly/. This
makes the existing smoke test more easily accessible. Noteworthy
differences:
* Instead of patching the code to make output more stable, postprocess
the output to normalize it.
* Compare actual results to expected results instead of the previous
test run's results.
* Much faster. The old test harness used sleep liberally to "ensure"
things always happen in the same order.
Known shortcomings:
* The smoke test hangs when the server fails to complete startup, or
fails to terminate.
* Normalization of xdump hardcodes columns instead of getting them
from xdump meta.
* Normalization of time values in xdump is an ugly hack.
* xdump meta column type isn't normalized. Actual values can vary
between systems, because the width of enumeration types is
implementation-defined. The smoke test works only when they're
represented as int, which is the case on common systems.
* Currently expected to work only with thread package LWP and a
random() that behaves exactly like the one on my development system,
because:
- Thread scheduling is reliably deterministic only with LWP
- The PRN sequence produced by random() isn't portable
- Shell builtin kill appears not to do the job in MinGW
- The Windows server tries to run as service when -d isn't
specified
Further work is needed to address these shortcomings.
Getting C programs behave exactly the same on all systems is hard.
We'll likely run into system-dependent differences that upset the
smoke test. Floating-point computation seems particularly vulnerable.
Instead of updating src/scripts/nightly/ to use "make check", retire
it. It hasn't been used in quite a while. Investing more into our
homegrown auto-builder doesn't make sense, as canned auto-builders
such as Travis CI and Jenkins are readily available.
The shell scripts src/scripts/nightly/tests/?? become Empire batch
files tests/smoke/. The shell scripts are actually shell boilerplate
around Empire batch files. To make sure git recognizes the move, this
commit moves them unchanged. tests/smoke-test strips the boilerplate
before it feeds the batch files to the client. The next commit will
get rid fo that.
