Fix xdump nat column ip for connections from "long" IPv6 addresses
Broken in commit 3a7d7fa, which enlarged struct natstr member
nat_hostaddr[] from 32 to 46 characters, but neglected to update the
ca_len in nat_ca[]. Consequently, the address is truncated in xdump.
Can also break country * ?ip=... and such, but that's exotic.
Fix five year old show stopper bugs on big endian hosts
emp_server and empdump refuse to start on most big endian hosts,
because ef_verify_config() chokes on mdchr_ca[]:
Config meta uid 0 field type: value 0 is not in symbol table meta-type
Config meta uid 1 field type: value 0 is not in symbol table meta-type
Config meta uid 2 field type: value 0 is not in symbol table meta-type
Config meta uid 3 field type: value 0 is not in symbol table meta-type
Config meta uid 4 field type: value 0 is not in symbol table meta-type
Broken in commit 06a0036 (v4.3.12), which changed struct castr member
ca_type from packed_nsc_type (typedef'ed to char) to enum nsc_type,
but neglected to update the ca_type in mdchr_ca[].
On little endian hosts, the selector reads the least significant byte,
with sign extension. Happens to work, because the type values are all
sufficiently small integers.
On big endian hosts, the selector reads the most signiciant byte.
which is always zero (NSC_NOTYPE). Makes ef_verify_config() fail.
Except when sizeof(enum nsc_notype) == 1. Then selector type works
fine, and ef_verify_config() succeeds, but we run into the next
problem: the same commit also changed member ca_flags from nsc_flags
(typedef'ed to unsigned char) to int without updating the ca_type in
mdchr_ca[]. This breaks "only" xdump meta column flags.
v4.3.12 was released in April 2008. Either nobody has tried to run a
game on a big endian host since, or all who did gave up quietly,
without reporting the problem.
We clearly need to test on a wider range of machines.
Broken in commit 14ea670 (v4.3.8), which changed struct trdstr member
trd_type from char to short, but neglected to update the ca_type in
trade_ca[].
On little endian hosts, the selector reads the least significant byte,
with sign extension. Happens to work, because the type values are all
sufficiently small integers.
On big endian hosts, the selector reads the most signiciant byte,
which is always zero (EF_SECTOR). Messes up xdump trade badly.
Broken in commit 09248d0 (v4.3.8), which changed struct loststr member
lost_type from char to short, but neglected to update the ca_type in
lost_ca[].
On little endian hosts, the selector reads the least significant byte,
with sign extension. Happens to work, because the type values are all
sufficiently small integers.
On big endian hosts, the selector reads the most signiciant byte,
which is always zero (EF_SECTOR). Messes up xdump lost badly. Also
breaks lost * ?type=..., but that's exotic.
"Unusually long" topics are marked with a "!" in subject indexes.
This should use the line count of the formatted page, but that's too
much trouble, so commit 4c0b4c0 (v4.3.27) approximated it by "source
file has more than 9999 bytes". Change that to "source file has more
than 300 lines".
Declare subjects instead of picking them up automatically
Since subjects were added in Empire 2, we've always picked them up
from .SA requests. If you mistype a subject there, you get a "is a
NEW subject" warning, and incorrect subject pages. When building a
pristine tree, you get bogus "is a NEW subject" warnings for all
subjects. If you somehow delete the generated subjects.mk, but not
the generated subject files, the build breaks.
Declare subjects in Make variable subjects. Drop generated makefile
subject.mk.
Treat unknown topics in .SA arguments as errors. This replaces the
"$subj is a NEW subject" warning.
Treat subjects without member pages as errors. This replaces the "The
subject $subj has been removed" warning.
We used to do all the info indexing work in info.pl: find subjects,
create subjects.mk (to tell make the list of subjects), the subject
pages, and TOP.t. Worked, but touching an info page triggered a full
rebuild of all subject pages and TOP.t.
Commit 2f14064 (v4.3.0) tried to avoid that by splitting info.pl into
findsubj.pl, mksubj.pl, mktop.pl. findsubj.pl puts not just the
subjects into subjects.mk, but also make rules for the subject pages,
to guide their remaking. mksubj.pl creates a single subject page.
mktop.pl creates TOP.t.
Unfortunately, this doesn't work so well. Since subjects.mk doesn't
exist in a virgin tree, we use -include. Unwanted consequence:
findsubj.pl failure doesn't stop make. Moreover, the complex make
machinery breaks down when info sources get removed or subjects get
dropped.
Go back to the old method, except keep mktop.pl separate, as that part
works just fine, and use simpler make rules. mksubj.pl now creates
subjects.mk and all subject pages, like info.pl did.
This effectively reverts most of commit 2f14064. I'll address the
excessive rebuilding of subject pages in a different way shortly.
Remaking config.h and config.h.in updates the target only when its
contents actually changes. This is important, because after updating
config.h we need to recompile everything.
The make rules to do that are straight from the Autoconf manual. But
they don't work: the rules that connect config.h and config.h.in to
stamp-h and stamp-h.in don't have recipes. Since make doesn't have an
implicit rule either, it concludes that the target remains unchanged.
It still updates the prerequisites. The recipe for updating the stamp
files then change the the targetes behind make's back. Make misses
the change of config.h and/or config.h.in, and we get an incomplete
rebuild.
The rules need empty recipes instead. This Autoconf manual was fixed
accordingly in autoconf.git commit 6b42b38.
Mark obsolete pages with '+' in subject pages. Drop the separate
"Obsolete" subject page: move "info Innards" to subject "Server", and
"info update" to "Updates" (where it came from in commit a5764534,
v4.3.10).
Document GNU libc lossage in listen_addr doc string
Systems using GNU libc such as Linux are frequently configured in a
way getaddrinfo(NULL, ...) put the IPv4 wildcard "0.0.0.0" *before*
the IPv6 wildcard "::" in the result. Because of that, listen_addr ""
listens only on all IPv4 addresses. Workaround: listen_addr "::".
Fix wildcard bind to at least bind IPv4 or else IPv6 on OpenBSD
OpenBSD refuses to implement IPV6_V6ONLY, in violation of RFC 3493.
RFC 4038 frowningly recognizes this practice. The only way to bind
both IPv4 and IPv4 there is two separate sockets. Requires more
surgery than I can do now.
Since we can't have both IPv6 and IPv6 on OpenBSD with our single
socket, prefer IPv4, but if that doesn't work, do IPv6.
To prefer IPv6 instead, put 'listen_addr "::"' into econfig. Document
that in listen_addr's doc string.
Fix wildcard bind to bind both IPv6 and IPv4 on Windows & BSD
We rely on AF_INET6 wildcard bind() binding the AF_INET port, too,
i.e. IPV6_V6ONLY off. This should be the default according to RFC
3493 section 5.3, but isn't on Windows and BSD. RFC 4038 recognizes
this fact in section 4.2.
When IPV6_V6ONLY is on, an AF_INET6 wildcard bind only accepts
connections from IPv6 addresses. Thus, IPv4 doesn't work when
getaddrinfo() returns an AF_INET6 address first (which it should do
when the system has an IPv6 address configured).
Switch off IPV6_V6ONLY explicitly instead of relying on the default.
This makes IPv6 work on systems where IPV6_V6ONLY is on by default,
such as Windows and BSD.
Except for OpenBSD, which does not support switching it off. To be
addressed in the next commit.
Shouldn't fail. If it fails, but bind() works, the failure doesn't
matter. If bind() fails, we can just as well report that failure
instead of setsockopt()'s.
Change fairland island size probability distribution
Island size is randomly chosen from the interval [1..2*is+1], with
expected value is. Use two dice to roll the size instead of one.
This makes extreme sizes much less likely.
Make smoke test time out instead of hang when server misbehaves
The smoke test waits for the server completing startup by trying to
connect until it works. Hangs if the server doesn't complete startup
for some reason. Make it give up after 5s.
Likewise, The smoke test waits for the server to terminate by trying
kill -0 until it fails. Hangs if the server doesn't terminate. Make
it give up after 5s.
Fix extension of market bidding time when high bidder changes
Lots stay on the market until there's a bid and bidding time expires.
When the highest bidder changes, and less than five minutes of bidding
time are left, it gets extended by five minutes (since 4.0.7, actually
works since 4.0.9).
Normally, this ensures that the competition has at least five minutes
to react. Except when this is the first bid, bidding time may have
expired already. If it expired less than five minutes ago, the
competition still gets time to react, just less than it should. If it
expired earlier, the sale is executed immediately for units. For
commodities, the bidding time is set to expire in five minutes (since
4.2.0).
Instead of extending bidding time by five minutes, set it to expire in
five minutes, both for commodities and for units.
check_trade() converts the price to float, which can lose precision,
although only for ridiculously high prices. Has been broken since
4.0.0 introduced the market.
Avoid the conversion. Bulletins now show pre-tax price as $N instead
of $N.00.
Code dealing with money mixes int and long pretty haphazardly.
Harmless, because practical amounts of money fit into int on any
machine capable of running the server. Clean up anyway.
Fix melting of big piles of stuff by ridiculously heavy fallout
meltitems() computes #items * etus per update * fallout in type long.
Theoretical maximum is ITEM_MAX * etus * FALLOUT_MAX = 99980001 *
etus. Can overflow 32 bits for etus > 21. Has been broken since the
introduction of fallout in KSU.
Code dealing with reserves mixes int and long pretty haphazardly.
Harmless, because practical reserves fit easily on any machine capable
of running the server. Clean up anyway.
Code dealing with counting people mixes int and long pretty
haphazardly. Harmless, because practical populations fit into int
easily on any machine capable of running the server. Clean up anyway.
As long as symbol_by_value(), show_capab() and togg() support only
int, flags need to fit into int.
Not a problem in practice, because no machine capable of running
Empire has int narrower than 32 bits, and 32 bits suffice.
Some flags members are long instead of int: struct lchrstr member
l_flags, struct natstr member nat_flags, struct mchrstr member m_flags
are long. Waste of space on machines with long wider than int.
Change them to int.
Rearrange struct lchrstr and struct natstr to avoid holes.
Fix crash on edit s, p, u key 'U' with negative argument
ef_ensure_space() oopses on negative ID, but succeeds anyway. edit()
proceeds to ef_write(), which neglects to check for negative ID.
Since the ID isn't in the cache, it then passes a NULL old element to
callback prewrite(), which crashes.
Fix ef_ensure_space() to fail on negative ID. Commit 5173f8cd
(v4.3.0) made it oops, but neglected to make it fail.
Fix ef_write() to oops and fail on negative ID.
ef_write() still passes NULL old element to prewrite() when the ID
isn't in the cache. Doesn't actually happen, because we use
prewrite() callbacks only with fully cached tables. Fragile. Make
ef_open() fail when that assumption is violated.
Unify owner of units built by deities in foreign sectors
Newly built ships and land units are given to the player, planes and
nukes to the sector owner. Matters only for deities, because only
deities can build in foreign sectors. Stupid all the same.
This has always been inconsistent. Empire 1 gave ships and nukes to
the player, and planes to the sector owner. Chainsaw 3 added land
units, and gave them to the player. Empire 2 changed build to give
nukes to the sector owner.
Building doesn't work when the unit built is given to POGO, because
giving a unit to POGO destroys it. When build gives to the sector
owner, deities can't build in unowned sectors. When build gives to
the player, POGO can't build at all. That's more limiting, so change
build to always give to the sector owner.
We seed it with value of time(). It's the traditional way, but it
provides only a few bits of effective entropy when an attacker has a
rough idea when the program started.
Instead, seed with a kernel random number. If we can't get one, fall
back to a hash of gettimeofday() and getpid(). This should happen
only on old systems or Windows. Far worse than a kernel random
number, but far better than using time().
Note that fairland used to seed with time() + getpid() until commit 331aac2a (v4.2.20) dropped the getpid(), claiming it didn't improve
the randomness. Perhaps it didn't under Windows then, but it
certainly did elsewhere, so it was a regression.
random() may yield different pseudo-random number sequences for the
same seed on another system. For instance, at least some versions of
MinGW provide a random() in -liberty that differs from traditional BSD
(see commit c8231b12). Rather inconvenient for regression testing.
MT19937 Mersenne Twister is a proven, high-quality PRNG. Actual code
is reference code provided by the inventors[*]. Quick tests show
performance comparable to random().
Like random(), MT is not cryptographically secure: observing enough of
its output permits guessing its state, and thus its future output. I
don't think players can do that.
Drop the copy of BSD random() we added for Windows.
Like the previous commit, this changes the server's die rolls, and
makes fairland create a different random map for the same seed. Update
expected smoke test results accordingly.
[*] mt19937ar.sep.tgz downloaded from
http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/MT2002/emt19937ar.html
"random() % n" is sound only when n is a power of two. The error is
hardly relevant in Empire, because random() yields 31 bits, and our n
are always much smaller than 2^31. Fix it anyway.
Use smallest the 2^m >= n instead of n, and discard numbers exceeding
n.
Bonus: faster for me even in the worst case n = 2^m+1.
Like the recent change to damage(), this changes some of the server's
die rolls, only this time the effect is pretty pervasive. Worse,
fairland now creates a completely different random map for the same
seed. Update expected smoke test results accordingly.
damage() now uses random() % 32768 in chance() instead of random() %
100 inline, therefore can round differently for the same pseudo-random
number. Update expected smoke test results accordingly.
Aside: "random() % n" distributes evenly only when n is a power of
two. 100 isn't. However, because random() yields at least 31 bits,
and 100 is so much smaller than 2^31, the error is vanishingly small.
Clean up land unit retreat chance and fix its documentation
Change chance in percent lnd_retreat - lnd_effic - 1 to lnd_retreat -
lnd_effic. It's been that way since Empire 2, but I can't bring
myself to document the silly -1.
"info morale" wasn't updated when the retreat chance was changed in
Empire 2. Fix that.
The tech center doesn't have enough workers to use all materials in
some updates. How much get made depends on a die roll then. Tech
variations are inconvenient because they ripple through the rest of
the smoke test.
Make news item merging deterministic and safe for year 2038
News reporting merges news items into recent items with same contents.
For that purpuse, we keep a small cache of recent items. When a new
item can't be merged into an item in the cache, the oldest item gets
evicted to make space for the new one.
ncache() evicts the first item with the smallest timestamp (struct
nwsstr member nws_when). Timestamps are in seconds, therefore clashes
are common, and eviction depends on exact timing. Such indeterminism
can make the smoke test fail.
Moreover, ncache() assumes timestamps cannot exceed 0x7fffffff. If
they do, it always evicts the slot 0. They will in 2038.
Fix by evicting round robin. This always evicts the oldest item.
The airfield is a sector taken from player 8. How many updates it
takes to convert is highly variable. If it converts late, the
airfield may not be constructed in time. This is currently the case
for me.
Move the airfield to a more dependable sector.
For me, the smoke test now fails frequently, because of differences in
news. To be fixed next.
Just a smoke test so far, extracted from src/scripts/nightly/. This
makes the existing smoke test more easily accessible. Noteworthy
differences:
* Instead of patching the code to make output more stable, postprocess
the output to normalize it.
* Compare actual results to expected results instead of the previous
test run's results.
* Much faster. The old test harness used sleep liberally to "ensure"
things always happen in the same order.
Known shortcomings:
* The smoke test hangs when the server fails to complete startup, or
fails to terminate.
* Normalization of xdump hardcodes columns instead of getting them
from xdump meta.
* Normalization of time values in xdump is an ugly hack.
* xdump meta column type isn't normalized. Actual values can vary
between systems, because the width of enumeration types is
implementation-defined. The smoke test works only when they're
represented as int, which is the case on common systems.
* Currently expected to work only with thread package LWP and a
random() that behaves exactly like the one on my development system,
because:
- Thread scheduling is reliably deterministic only with LWP
- The PRN sequence produced by random() isn't portable
- Shell builtin kill appears not to do the job in MinGW
- The Windows server tries to run as service when -d isn't
specified
Further work is needed to address these shortcomings.
Getting C programs behave exactly the same on all systems is hard.
We'll likely run into system-dependent differences that upset the
smoke test. Floating-point computation seems particularly vulnerable.
Instead of updating src/scripts/nightly/ to use "make check", retire
it. It hasn't been used in quite a while. Investing more into our
homegrown auto-builder doesn't make sense, as canned auto-builders
such as Travis CI and Jenkins are readily available.
The shell scripts src/scripts/nightly/tests/?? become Empire batch
files tests/smoke/. The shell scripts are actually shell boilerplate
around Empire batch files. To make sure git recognizes the move, this
commit moves them unchanged. tests/smoke-test strips the boilerplate
before it feeds the batch files to the client. The next commit will
get rid fo that.
Really fix accepting connections from "long" IPv6 address
Commit ee01ac19 (v4.3.23) enlarged player member hostaddr from 32 to
46 characters, but missed natstr member nat_hostaddr. player_main()
copies hostaddr to nat_hostaddr. Can overrun the destination, but
fortunately just into nat_hostname.
Impact:
* Can makes praddr() print only a suffix of the address. Used by play
command, for player messages during login and logout, and for
logging.
* Can make player_main()'s test for "same address as last time" fail,
causing extra "Last connection" messages.
* Matching against econfig key privip is not affected.
Fix setsector not to disclose number of landmines to occupier
When the deity sets the number of mines with setsector, the sector
owner (if any) is told the resulting number of mines. Even for
occupied sectors, where mines belong to the old owner, and thus
shouldn't be disclosed. Oops.
Fix setsector not to tell the sector owner anything then.