Exercise version, show and xdump, except for xdump of game state.
The xdump part is mostly factored out of tests/smoke.
Signed-off-by: Markus Armbruster <armbru@pond.sub.org>
TREATIES has issues:
* Treaties can cover attack, assault, paradrop, board, lboard, fire,
build (s|p|l|n) and enlist, but not bomb, launch, torpedo and
enlistment centers.
* Usability is very poor. While a treaty is in effect, every player
action that violates a treaty condition triggers a prompt like this:
This action is in contravention of treaty #0 (with Curmudgeon)
Do you wish to go ahead anyway? [yn]
If you decline, the action is not executed. If you accept, it is.
In both cases, your decision is reported in the news.
You cannot get rid of these prompts until the treaty expires.
* Virtually nobody uses them.
* Virtually unused code is buggy code. There is at least one race
condition: multifire() reads the firing sector, ship or land unit
before the treaty prompt, and writes it back after, triggering a
generation oops. Any updates made by other threads while trechk()
waits for input are wiped out, triggering a seqno mismatch oops.
* The treaty prompts could confuse smart clients that aren't prepared
for them. WinACE isn't, but is reported to work anyway at least
common usage. Ron Koenderink (the WinACE maintainer) suspects there
could be a few situations where it will fail.
This feature is not earning its keep. Remove it. Drop command
treaty, consider treaty, offer treaty, xdump treaty, reject treaties.
Output of accept changed, obviously.
Signed-off-by: Markus Armbruster <armbru@pond.sub.org>
Adding or removing a command to/from a test has unfortunate effects:
* Before the previous commit: if the command consumes pseudorandom
numbers, all subsequent users of pseudorandom numbers get different
ones. This has always been a major headache.
* Since the previous commit: all subsequent users of pseudorandom
numbers get different ones whether the command consumes any or not.
That's even worse.
* If the command uses BTUs, subsequent prompts are changed. Not
nearly as bad as the above, but still annoying.
Create a new command __cmd to allow compensating for adding/removing
commands for tests. Throw in the ability to compensate treasury
changes for good measure. Three arguments: command count, BTU use,
money use.
Usage example: say you add a convert command to a test, and it uses 3
BTUs and $15. Then you compensate by adding "__cmd added 1 3 15"
right after it.
The command must not be available unless running_test_suite is on, of
course. Make it require the new player command capability TESTING,
and give that to all players when running_test_suite is on.
The command is intentionally not documented in info. Switch
running_test_suite off for info-test, to hide it (and any future
TESTING commands) from info-test.
Suppress the command counter increment for TESTING commands, so they
can be used without upsetting pseudorandom numbers
Signed-off-by: Markus Armbruster <armbru@pond.sub.org>
Just a smoke test so far, extracted from src/scripts/nightly/. This
makes the existing smoke test more easily accessible. Noteworthy
differences:
* Instead of patching the code to make output more stable, postprocess
the output to normalize it.
* Compare actual results to expected results instead of the previous
test run's results.
* Much faster. The old test harness used sleep liberally to "ensure"
things always happen in the same order.
Known shortcomings:
* The smoke test hangs when the server fails to complete startup, or
fails to terminate.
* Normalization of xdump hardcodes columns instead of getting them
from xdump meta.
* Normalization of time values in xdump is an ugly hack.
* xdump meta column type isn't normalized. Actual values can vary
between systems, because the width of enumeration types is
implementation-defined. The smoke test works only when they're
represented as int, which is the case on common systems.
* Currently expected to work only with thread package LWP and a
random() that behaves exactly like the one on my development system,
because:
- Thread scheduling is reliably deterministic only with LWP
- The PRN sequence produced by random() isn't portable
- Shell builtin kill appears not to do the job in MinGW
- The Windows server tries to run as service when -d isn't
specified
Further work is needed to address these shortcomings.
Getting C programs behave exactly the same on all systems is hard.
We'll likely run into system-dependent differences that upset the
smoke test. Floating-point computation seems particularly vulnerable.
Instead of updating src/scripts/nightly/ to use "make check", retire
it. It hasn't been used in quite a while. Investing more into our
homegrown auto-builder doesn't make sense, as canned auto-builders
such as Travis CI and Jenkins are readily available.
The shell scripts src/scripts/nightly/tests/?? become Empire batch
files tests/smoke/. The shell scripts are actually shell boilerplate
around Empire batch files. To make sure git recognizes the move, this
commit moves them unchanged. tests/smoke-test strips the boilerplate
before it feeds the batch files to the client. The next commit will
get rid fo that.
