769 lines
28 KiB
Text
769 lines
28 KiB
Text
Introduction
|
|
|
|
Empire is designed as a smart server with dumb clients. An Empire
|
|
client need to know nothing about the game. Even telnet would do. In
|
|
fact, emp_client is little more than a slightly specialized telnet.
|
|
|
|
In such a design, presentation is in the server, and it is designed
|
|
for human consumption. Ideally, presentation and logic are cleanly
|
|
separated, for easy selection between different presentations.
|
|
There's no such separation in the Empire server, and separating them
|
|
now would be a herculean effort.
|
|
|
|
Thus, smart clients have to work with output designed for humans.
|
|
That's not especially hard, just an awful lot of tedious work, and the
|
|
result gets easily broken by minor changes in output format, which
|
|
humans hardly notice.
|
|
|
|
Instead of making smart clients parse output of commands designed for
|
|
humans, one can add commands designed for machines. Such commands can
|
|
share the code implementing game rules with their counterparts for
|
|
humans. To do that cleanly means separating logic and presentation.
|
|
Implementing them with their own copy of the code is no good --- how
|
|
would you ensure that the two copies implement precisely the same game
|
|
rules?
|
|
|
|
Except for commands that actually don't do anything. There's a useful
|
|
class of such commands: commands to show game configuration and state
|
|
without altering it. The only game rules involved are those that
|
|
govern who gets to see what. Ensuring that those are obeyed is
|
|
tractable.
|
|
|
|
Empire has had one such command since the beginning: dump. Empire
|
|
4.0.6 added more: sdump, ldump, pdump and ndump. 4.0.7 added lost and
|
|
support for incremental dumps. These commands have served smart
|
|
clients well. However, they cover just the most important part of the
|
|
game state (sectors, ships, planes, land units, nukes), and no game
|
|
configuration. They are not quite complete even for what they attempt
|
|
to cover. Finally, their output is harder to parse than necessary.
|
|
|
|
The xdump command is designed to be the dump to end all dumps.
|
|
|
|
Like many good ideas, xdump has secondary uses. Dumping game state as
|
|
text can be one half of a game export/import facility. Useful to
|
|
migrate games to different machines or even, with some text mangling
|
|
perhaps, different server versions. We will see below that xdump
|
|
isn't quite sufficient for that, but it's a start.
|
|
|
|
Means to import game configuration let you customize your game without
|
|
recompiling the server. As we will see, configuration files have
|
|
different requirements, which xdump doesn't satisfy without some
|
|
extensions.
|
|
|
|
If game import code can edit everything, then a deity command capable
|
|
of editing everything is possible. Proof-of-concept code exists (not
|
|
in the tree yet).
|
|
|
|
|
|
Analysis of the data to dump
|
|
|
|
Game state consists of a fixed set of table files (sector, ship, ...),
|
|
telegram files, and a few miscellaneous files. Game configuration
|
|
consists of a fixed set of configuration tables and scalar parameters.
|
|
|
|
A table is an ordered set of records (table rows). All records have
|
|
the same fields (table columns), which are statically typed.
|
|
|
|
Fields may be integers, floating-point numbers, x- or y-coordinates,
|
|
symbols and symbol sets encoded as integers in a way specific to the
|
|
server version, and character arrays. Configuration table fields may
|
|
be pointers to zero-terminated strings, null pointers allowed. No
|
|
other pointers occur. Unions do not occur.
|
|
|
|
|
|
Requirements analysis
|
|
|
|
Requirements:
|
|
|
|
* Capable to dump all tables.
|
|
|
|
* Can do incremental dumps.
|
|
|
|
* Output is text.
|
|
|
|
* Output is reasonably compact.
|
|
|
|
* Output is trivial to parse. Triviality test: if it's easy in AWK, C
|
|
(no lex & yacc, just stdio), Lisp (just reader) and Perl (base
|
|
language, no modules), then it's trivial enough.
|
|
|
|
* Output identifies itself.
|
|
|
|
* Output is self-contained; symbol encoding is explicit.
|
|
|
|
* KISS: Keep it simple, stupid.
|
|
|
|
Non-requirements:
|
|
|
|
* Generality. We're not trying to design a general mechanism for
|
|
dumping C data.
|
|
|
|
* Completeness. We're not trying to dump stuff other than tables.
|
|
|
|
* Abstraction. We're not trying to hide how things are stored in the
|
|
server. When storage changes, xdump output will change as well, and
|
|
consumers need to be updated. This is not because abstraction
|
|
wouldn't be nice to have, just because we don't feel up to the task
|
|
of designing one.
|
|
|
|
|
|
Principles of Design
|
|
|
|
Traditional dumps have a dump function for every table. These
|
|
functions are simple, but exceedingly dull and repetitive.
|
|
|
|
The selector code works differently. Each table has a descriptor,
|
|
which among other things defines a dictionary of selector descriptors.
|
|
A selector descriptor describes a field (table column) visible to
|
|
players. This is what we call meta-data (data about data). The
|
|
selector code knows nothing about the individual tables, it just
|
|
interprets meta-data. That's smart, as it keeps the dull, repetitive
|
|
parts in more easily maintainable meta-data rather than code.
|
|
|
|
xdump follows the selector design, and uses the existing selector
|
|
meta-data. This requires extending the meta-data to configuration
|
|
tables, which weren't previously covered. It also requires some
|
|
generalization of selector descriptors, so that all fields can be
|
|
covered.
|
|
|
|
To sum up, meta-data consists of a table of tables, and for each table
|
|
a table of selectors (table of columns, so to speak). It is specific
|
|
to the server version and how it is compiled on the host.
|
|
|
|
To interpret a table xdump, you need its meta-data, because without it
|
|
you have no idea what the columns mean. As meta-data is just a bunch
|
|
of tables, xdump can dump it. But now you need meta-meta-data to make
|
|
sense of the meta-data. Fortunately, meta-meta-data is the same for
|
|
all xdumps, and therefore the recursion terminates with a single
|
|
meta-meta-table.
|
|
|
|
xdump dumps symbols and symbol sets as integers. To decode them, you
|
|
need to know what the symbol numbers and symbol set bits mean. For
|
|
this purpose, field meta-data includes the table ID of a symbol table.
|
|
A symbol table is a table of value-name pairs, with the value in the
|
|
leftmost column. You decode a symbol value by looking it up in the
|
|
symbol table. You decode a symbol set value by looking up its bits
|
|
(powers of two) in the symbol table.
|
|
|
|
Some integer fields are actually keys in other tables. For instance,
|
|
ship field type is a key in the table of ship types ship-chr, and
|
|
plane field ship is a key in the ship table. Key -1 is special: it's
|
|
a null key. Meta-data encodes these table reference just like for
|
|
symbols: the meta-data has the ID of the referenced table, and that
|
|
table has the key in the leftmost column. Obviously, that leftmost
|
|
column is a table key as well, referencing the table itself.
|
|
|
|
A table with its key in the leftmost column can be dumped partially.
|
|
Without such a key, you need to count records to find the record
|
|
index, and that works only if you can see a prefix of the complete
|
|
table.
|
|
|
|
The special table "ver" collects all scalar configuration parameters
|
|
in a single record. It does not occur in the table of tables.
|
|
|
|
|
|
Syntax of xdump command
|
|
|
|
See info xdump.
|
|
|
|
|
|
The xdump output Language
|
|
|
|
Because the output is to be parsed by machines, it needs to be
|
|
precisely specified. We use EBNF (ISO 14977) for syntax, except we
|
|
use '-' in meta-identifiers and omit the concatenation symbol ','.
|
|
|
|
table = header { record } footer ;
|
|
header = "XDUMP" space [ "meta" space ]
|
|
identifier space timestamp newline ;
|
|
identifier = id-chr { id-chr } ;
|
|
id-char = ? ASCII characters 33..126 except '"#()<>=' ? ;
|
|
timestamp = intnum ;
|
|
footer = "/" number newline ;
|
|
record = [ fields ] newline ;
|
|
fields = field { space field } ;
|
|
field = intnum | flonum | string ;
|
|
intnum = ? integer in printf %d format ? ;
|
|
flonum = ? floating-point in printf %g format ? ;
|
|
string = "nil"
|
|
| '"' { str-char } '"' ;
|
|
str-char = "\\" octal-digit octal-digit octal-digit
|
|
| ? ASCII characters 33..126 except '"' and '\\' ? ;
|
|
octal-digit = ? '0'..'7' ? ;
|
|
space = ' ' ;
|
|
newline = ? ASCII character 10 ? ;
|
|
|
|
Notes:
|
|
|
|
* The syntax for flonum is debatable. Precise conversion between
|
|
floating-point and decimal is hard, and C libraries are not required
|
|
to be precise. Using C99's %a format for flonum would avoid the
|
|
issue, but some programming environments may have trouble converting
|
|
that back to floating-point. We may change to %a anyway in the
|
|
future. Clients are advised to accept both.
|
|
|
|
* Strings syntax could perhaps profit from the remaining C escape
|
|
sequences. Except for '\"': adding that would complicate regular
|
|
expressions matching the string, and thus violate the `trivial to
|
|
parse' requirement
|
|
|
|
* Space is to be taken literally: a single space character. Not a
|
|
non-empty sequence of white-space.
|
|
|
|
Semantics:
|
|
|
|
* The table identifier in the header is one of the names in xdump table.
|
|
|
|
* The timestamp increases monotonically. It has a noticeable
|
|
granularity: game state may change between an xdump and the next
|
|
timestamp increase. If the table has a timestamp field, clients can
|
|
xdump incrementally by using a conditional ?timestamp>T, where T is
|
|
one less than the timestamp received with the last xdump of that
|
|
table.
|
|
|
|
Timestamp values are currently seconds since the epoch, but this
|
|
might change, and clients are advised not to rely on it.
|
|
|
|
* The number in the footer matches the number of records.
|
|
|
|
* Fields match their meta-data (see Meta-Data below).
|
|
|
|
* "nil" represents a null string (which is not the same as an empty
|
|
string). Otherwise, fields are to be interpreted just like C
|
|
literals.
|
|
|
|
|
|
Meta-Data
|
|
|
|
Table meta-data is in xdump table. Fields:
|
|
|
|
* uid: The table ID, key for xdump table. IDs depend on the server
|
|
version; clients should not hard-code them. This is the leftmost
|
|
field.
|
|
|
|
* name: The table name. Clients may identify tables by name.
|
|
|
|
Field meta-data for table T is in xdump meta T. The order of fields
|
|
in the xdump T matches the order of records in xdump meta T. Fields
|
|
of xdump meta T are:
|
|
|
|
* name: The field name. Matches the selector name. Clients may
|
|
identify fields by name. This is the leftmost field.
|
|
|
|
* type: The field's data type, a symbol. Clients should use this only
|
|
as key for the symbol table. Symbols are:
|
|
- "d", field uses intnum syntax
|
|
- "g", field uses flonum syntax
|
|
- "s", field uses string syntax
|
|
- "c", field uses string syntax
|
|
|
|
* flags: The field's flags, a symbol set. Flags are:
|
|
- "deity", field visible only to deities
|
|
- "extra", field not to be dumped
|
|
- "const", field cannot be changed
|
|
- "bits", field is a symbol set, field type must encode symbol "d",
|
|
field table must not be -1.
|
|
|
|
* len: If non-zero, then the record encodes an array with that many
|
|
elements. If field type encodes symbol "c", it is a character
|
|
array, which is dumped as a single string field. Else, the array is
|
|
dumped as len fields.
|
|
|
|
* table: Key for xdump table. Unless -1, it defines the table
|
|
referenced by the field value. Field type must encode symbol "d"
|
|
then.
|
|
|
|
Symbol table fields:
|
|
|
|
* value: The symbol's encoding as integer. If the symbol can be
|
|
element of a symbol set, this is a power of two.
|
|
|
|
* name: The symbol's name.
|
|
|
|
|
|
Notes on xdump Implementation
|
|
|
|
Overall impact on the server code is low.
|
|
|
|
To keeps xdump simple, storage of game state and game configuration
|
|
tables has been unified under the common empfile abstraction, making
|
|
nxtitem-iterators and selectors equally applicable to all tables.
|
|
|
|
xdump required a few extensions to meta-data, which may become useful
|
|
in other places as well:
|
|
|
|
* Selectors can now deal with arrays (revived struct castr member
|
|
ca_len). Not yet available on the Empire command line.
|
|
|
|
* Selector meta-data can now express that a selector value is a key
|
|
for another table (new struct castr member ca_table). The selector
|
|
code doesn't use that, yet.
|
|
|
|
* Selector flag NSC_EXTRA to flag redundant selectors, so that xdump
|
|
can ignore them.
|
|
|
|
Meta-data is in empfile[] (table meta-data), src/lib/global/nsc.c
|
|
(selector meta-data), src/lib/global/symbol.c (symbol tables). The
|
|
command is in src/lib/commands/xdump.c, unsurprisingly.
|
|
|
|
|
|
Hints on Using xdump in Clients
|
|
|
|
Let's explore how to dump a game. To make sense of a table, we need
|
|
its meta-data, and to make sense of that table, we need meta-meta
|
|
data. So we start with that:
|
|
|
|
[14:640] Command : xdump meta meta
|
|
XDUMP meta meta 1139555204
|
|
"name" 3 4 0 -1
|
|
"type" 5 4 0 32
|
|
"flags" 6 12 0 33
|
|
"len" 8 4 0 -1
|
|
"table" 9 4 0 -1
|
|
/5
|
|
|
|
To interpret this table, we have to know the field names and their
|
|
meanings. Clients hard-code them. They should be prepared to accept
|
|
and ignore additional fields, and to cope with changes in field order,
|
|
except they may rely on "name" coming first.
|
|
|
|
A word on hard-coding. Clients hard-code *names*. The numbers used
|
|
for table IDs and to encode symbols are none of the client's business.
|
|
|
|
The encoding doesn't normally change within a game. Except when the
|
|
game is migrated to a sufficiently different server. That's a
|
|
difficult and risky thing to do, especially as there are no tools to
|
|
help with migrating (yet). Clients may wish to provide for such
|
|
changes anyway, by decoupling the client's encoding from the server's,
|
|
and dumping fresh meta-data on login. Incremental meta-data dump
|
|
would be nice to have.
|
|
|
|
So we don't know how symbol type and symbol set flags are encoded. To
|
|
decode them, we need their symbol tables. However, we need flags and
|
|
type only for tables we don't know, and there's one more table we do
|
|
know, namely the table of tables. Let's dump that next, starting with
|
|
its meta-data:
|
|
|
|
[31:640] Command : xdump meta table
|
|
XDUMP meta table 1139556230
|
|
"uid" 9 0 0 26
|
|
"name" 3 4 0 -1
|
|
/2
|
|
|
|
Because xdump table is referenced from elsewhere (xdump meta meta
|
|
field table), the leftmost field must contain the key. Thus, the
|
|
leftmost field's meta-data field table must be the table ID of xdump
|
|
table itself. Let's try it:
|
|
|
|
[30:640] Command : xdump 26 *
|
|
XDUMP table 1139556210
|
|
0 "sect"
|
|
1 "ship"
|
|
[...]
|
|
9 "nat"
|
|
[...]
|
|
16 "sect-chr"
|
|
17 "ship-chr"
|
|
[...]
|
|
26 "table"
|
|
[...]
|
|
/45
|
|
|
|
It worked! Mind that the special table "ver" is not in the table of
|
|
tables.
|
|
|
|
Now dump the two symbol tables we postponed. Because xdump accepts
|
|
table IDs as well as names, we don't have to know their names:
|
|
|
|
[14:640] Command : xdump meta 32
|
|
XDUMP meta meta-type 1139555298
|
|
"value" 9 4 0 -1
|
|
"name" 3 0 0 -1
|
|
/2
|
|
|
|
[15:640] Command : xdump 32 *
|
|
XDUMP meta-type 1139555826
|
|
1 "d"
|
|
2 "g"
|
|
3 "s"
|
|
4 "d"
|
|
5 "d"
|
|
6 "d"
|
|
7 "d"
|
|
8 "d"
|
|
9 "d"
|
|
10 "d"
|
|
11 "d"
|
|
12 "d"
|
|
13 "g"
|
|
14 "c"
|
|
/14
|
|
|
|
[15:640] Command : xdump meta 33
|
|
XDUMP meta meta-flags 1139555303
|
|
"value" 9 4 0 -1
|
|
"name" 3 0 0 -1
|
|
/2
|
|
|
|
[24:640] Command : xdump 33 *
|
|
XDUMP meta-flags 1139555829
|
|
1 "deity"
|
|
2 "extra"
|
|
4 "const"
|
|
8 "bits"
|
|
/4
|
|
|
|
We now have complete meta-meta information:
|
|
|
|
name type flags len table
|
|
-----------------------------------------
|
|
name s (const) 0
|
|
type d (const) 0 meta-type
|
|
flags d (bits const) 0 meta-flags
|
|
len d (const) 0
|
|
table d (const) 0
|
|
|
|
Dumping the remaining tables is easy: just walk the table of tables.
|
|
Here's the first one:
|
|
|
|
[36:640] Command : xdump meta 0
|
|
XDUMP meta sect 1139556498
|
|
"owner" 6 0 0 9
|
|
"xloc" 10 0 0 -1
|
|
"yloc" 11 0 0 -1
|
|
"des" 4 0 0 16
|
|
[...]
|
|
/69
|
|
|
|
A whole load of tables referenced! Only one of them (not shown above)
|
|
is a symbol table.
|
|
|
|
owner references table nat. No surprise.
|
|
|
|
xloc and yloc together reference the sector table, but that's not
|
|
expressed in meta-data (yet).
|
|
|
|
Let's stop here before this gets too long and boring. Experiment
|
|
yourself! Check out example Perl code src/xdump.pl.
|
|
|
|
|
|
Analysis of xdump as Configuration File Format
|
|
|
|
xdump makes a lousy configuration format because it is unwieldy to
|
|
edit for humans. That's because configuration files have different
|
|
requirements than dumps:
|
|
|
|
* Can be edited by humans with common tools, including text editors
|
|
and spreadsheets.
|
|
|
|
Using text editors requires a nice fixed-width table layout.
|
|
Spreadsheet import requires trivial field separation. Tab character
|
|
field separator or fixed width columns should do. The syntax should
|
|
allow all that, but not require it.
|
|
|
|
Trouble spots:
|
|
|
|
- xdump's rigid horizontal and vertical spacing makes it impossible
|
|
to align things visually.
|
|
|
|
- xdump uses one line per record, which can lead to excessively long
|
|
lines.
|
|
|
|
- xdump's string syntax requires octal escape for space.
|
|
|
|
- No comment syntax.
|
|
|
|
* Each table is self-contained. You don't have to look into other
|
|
tables to make sense of it.
|
|
|
|
This conflicts with xdump's separation of data and meta-data. You
|
|
need the table's meta-data to identify fields, and the referenced
|
|
symbol tables to decode symbols.
|
|
|
|
* Easy to parse. Don't compromise legibility just to please some dumb
|
|
tool, though.
|
|
|
|
Since we're trying to apply xdump to the configuration file problem,
|
|
we get an additional requirement:
|
|
|
|
* Reasonably close to xdump. Translation between machine-readable and
|
|
human-readable should be straightforward, if meta-data is available.
|
|
|
|
This leads to a human-readable dialect of the xdump language.
|
|
|
|
|
|
Human-Readable xdump Language
|
|
|
|
Fundamental difference to basic, machine-readable xdump: the rigid
|
|
single space between fields is replaced by the rule known from
|
|
programming languages: white-space (space and tab) separates tokens
|
|
and is otherwise ignored. The space non-terminal is no longer needed.
|
|
|
|
Rationale: This allows visual alignment of columns and free mixing of
|
|
space and tab characters.
|
|
|
|
Comments start with "#" and extend to the end of the line. They are
|
|
equivalent to a newline.
|
|
|
|
Rationale: Follow econfig syntax.
|
|
|
|
Tables with a record uid in the leftmost field can be `split
|
|
vertically' into multiple parts. Each part must contain the same set
|
|
of records. The leftmost field must be repeated in each part. Other
|
|
fields may be repeated. Repeated fields must be the same in all
|
|
parts. Naturally, the parts together must provide the same fields as
|
|
a table that is not split.
|
|
|
|
Rationale: This is to let you avoid long lines. Line continuation
|
|
syntax would be simpler, but turns out to be illegible. Requiring
|
|
record uid is not technically necessary, as counting records works the
|
|
same whether a table is split or not. Except humans can't count.
|
|
Perhaps this should be a recommendation for use rather than part of
|
|
the language.
|
|
|
|
EBNF changes:
|
|
|
|
* Header and footer:
|
|
|
|
header = "config" identifier newline colhdr newline ;
|
|
colhdr = { identifier [ "(" ( intnum | identifier ) ")" ] } [ "..." ] ;
|
|
footer = "/config" newline ;
|
|
|
|
If colhdr ends with "...", the table is continued in another part,
|
|
which shall follow immediately.
|
|
|
|
Rationale:
|
|
|
|
- The xdump needs to identify itself as human-readable, hence change
|
|
from "XDUMP" to "config".
|
|
|
|
- The timestamp in the header is useless for the applications we
|
|
have in mind for human-readable xdumps. The number of records in
|
|
the footer is of marginal value at best, and a pain for humans to
|
|
update.
|
|
|
|
- The column header is due to the self-containedness requirement.
|
|
It contains just the essential bit of meta-data: the column names.
|
|
|
|
* Symbolic fields:
|
|
|
|
field = intnum | flonum | string | symbol | symset ;
|
|
|
|
Rationale:
|
|
|
|
- Syntax for symbols and sets of symbols is due to the
|
|
self-containedness requirement. Machine-readable xdump gets away
|
|
with just numbers, which have to be decoded using meta-data.
|
|
|
|
* Friendlier numbers and strings:
|
|
|
|
flonum = ? floating-point in scanf %g format ? ;
|
|
str-char = "\\" octal-digit octal-digit octal-digit
|
|
| ? ASCII characters 32..126 except '"' and '\\' ? ;
|
|
|
|
Rationale:
|
|
|
|
- Machine-readable floating-point syntax is too rigid. Accept
|
|
everything that scanf does. Could also change intnum to %i
|
|
format, which accepts octal and hexadecimal in C syntax, but that
|
|
seems not worth the documentation bother.
|
|
|
|
- Machine-readable syntax requires \040 instead of space in strings
|
|
to allow trivial splitting into fields. This is unacceptable here
|
|
due to the legibility requirement, hence the change to str-char.
|
|
|
|
* Parse nil as symbol:
|
|
|
|
string = '"' { str-char } '"' ;
|
|
|
|
Rationale: This is a technicality required to keep the parse
|
|
unambiguous.
|
|
|
|
* Symbols:
|
|
|
|
symbol = identifier ;
|
|
symset = "(" { symbol } ")" ;
|
|
|
|
The special symbol "nil" is to be interpreted as null string.
|
|
|
|
Rationale:
|
|
|
|
- The symbol set syntax is the simplest that could work. We need to
|
|
allow space between the symbols for legibility anyway, so why not
|
|
make it the delimiter. A stop token is required to find the end
|
|
of the field, and a start token is useful for distinguishing
|
|
between symbol and symset. Bracketing with some kind of
|
|
parenthesis is an obvious solution.
|
|
|
|
The resulting sub-language for records is a superset of
|
|
machine-readable sub-language for records.
|
|
|
|
See src/lib/global/*.config for examples.
|
|
|
|
|
|
Notes on Table Configuration Implementation
|
|
|
|
econfig key custom_tables lists table configuration files. At this
|
|
time, reading a custom table merges it with the built-in table, then
|
|
truncates the result after the last record read from the custom table.
|
|
|
|
Some of the tables are rather ugly in C, and cumbersome to edit. We
|
|
thus moved them to configuration files (src/lib/global/*.config). The
|
|
server reads them from builtindir before reading custom tables.
|
|
|
|
The code dealing with these files is in src/lib/common/conftab.c.
|
|
|
|
Actual work is done by src/lib/common/xundump.c, which accepts both
|
|
human-readable and machine-readable input. The parser is not precise;
|
|
it accepts human-readable syntax even within tables whose header marks
|
|
them machine-readable.
|
|
|
|
Symbolic index values in column headers are not implemented. They
|
|
occur in item selector pkg, which is an array indexed by values in
|
|
symbol table packing.
|
|
|
|
Configuration tables contain values that are not meant to be
|
|
customized. For instance, meta-data and symbol tables reflect the
|
|
encoding of C language constructs in the server. Selector flag
|
|
NSC_CONST marks them, so that the code can prohibit changes.
|
|
|
|
All tables are checked against meta-data on server startup by
|
|
ef_verify(). More elaborate checking would be nice, and probably
|
|
requires additional meta-data.
|
|
|
|
|
|
Appendix: Empire 3 C_SYNC --- A Cautionary Tale
|
|
|
|
Clients are just as important as the server, and it's too darn hard to
|
|
write a good client. In 1995, Ken Stevens decided to do something
|
|
about it.
|
|
|
|
Ken cast the problem as a data synchronization problem. Quote C_SYNC
|
|
RFC 5.1, section `Abstract':
|
|
|
|
This is a specification for a new method of synchronizing game data
|
|
in the Empire client with data in the server.
|
|
|
|
and section `Objectives':
|
|
|
|
This new mode of communication between the server and the client will
|
|
be called C_SYNC communication and will satisfy the following 6
|
|
criterea:
|
|
|
|
(1) Output format will be version independent. So if someone is
|
|
using an old EmpireToolkit, then it will still work with a newer
|
|
version of the server.
|
|
|
|
(2) Every C_SYNC message will be a self-contained packet. i.e. the
|
|
client will not need to depend on previous messages (header messages)
|
|
to determine the meaning of a C_SYNC message.
|
|
|
|
(3) A C_SYNC message will be able to represent any of the
|
|
player-accessible data that is contained in the server database (e.g.
|
|
enemy ships, nations).
|
|
|
|
(4) Bandwidth will be minimized (i.e. the format will be as
|
|
concise as possible) while remaining human-readable (i.e. no
|
|
binary messages). [Note that data compression may be added at a later
|
|
date, but if it is added, it will be added on a separate port to
|
|
maintain backwards compatability.]
|
|
|
|
(5) The client will be able to tell the server whether it wants
|
|
to receive C_SYNC messages and whether these messages can be sent
|
|
asynchroniously (via "toggle sync" and "toggle async" respectively).
|
|
|
|
(6) A portable ANSI C EmpireToolkit will be made available for
|
|
parsing C_SYNC messages and managing the data they contain.
|
|
|
|
C_SYNC worked by hooking into ef_write() & friends so it could
|
|
`synchronize' the client on game state changes.
|
|
|
|
Sounds jolly good, doesn't it?
|
|
|
|
Well, it was a failure, and Wolfpack ripped it out right away. Quote
|
|
the change log:
|
|
|
|
Changes to Empire 4.0.0 - Initial release
|
|
* Initial Wolfpack release - Long live the Wolfpack!!!!
|
|
[...]
|
|
* Removed C_SYNC. This is done for 2 reasons. 1) None of us like it or
|
|
wish to support it. 2) We envision a better scheme for doing similar
|
|
things will come along.
|
|
|
|
But *why* did it fail? Just because Steve McClure hated it? Nope.
|
|
C_SYNC failed for several different reasons, each of them bad, but
|
|
only the last one is truly fundamental.
|
|
|
|
a. Lack of a rigorous and complete definition. The RFC is long on
|
|
syntax, but short on semantics. For instance, the unit type was
|
|
encoded as a number. Unit characteristics happened to be dumped in
|
|
an order that matched these numbers, but that wasn't defined
|
|
anywhere.
|
|
|
|
b. Overly complicated syntax. Trouble with encoding of strings.
|
|
|
|
c. Buggy implementation. Malformed C_SYNC messages, duplicate
|
|
messages, missing messages, semantically incorrect messages, you
|
|
name it.
|
|
|
|
d. Change of crew before it was finished. Wolfpack took over and
|
|
understandable wasn't interested in this half-finished mess.
|
|
|
|
None of the above is a fundamental, inherent flaw of the idea. The
|
|
next one is more serious:
|
|
|
|
e. It failed to achieve objective (4), and therefore slowed down
|
|
clients too much to be of use in real-time combat. When you fired
|
|
from a bunch of ships, C_SYNC would push complete records for all
|
|
the ships and the target to you. Most of that data is redundant.
|
|
|
|
That's because C_SYNC didn't transmit state changes, it
|
|
resynchronized state, and the pieces of state it could transmit
|
|
were too large.
|
|
|
|
The network was slower then. But let's not be complacent. I/O is
|
|
slow. Always was, most likely ever will be.
|
|
|
|
Maybe sending the messages out of band (separate TCP stream) would
|
|
help. Maybe not.
|
|
|
|
And here comes the killer:
|
|
|
|
f. The data to sync is not readily available the server.
|
|
|
|
Yup. Think about it. The game state on the server is *not* the
|
|
same as on the client. The server grants the client a carefully
|
|
limited view on certain parts of server game state on certain
|
|
events.
|
|
|
|
To be complete, a machine-readable protocol must disclose as much
|
|
information as the human-readable output. Tracking server game
|
|
state changes cannot do that alone. For instance, lookout tells
|
|
you ship#, owner and location. That event does not trigger any
|
|
state change on the server!
|
|
|
|
To be correct, a machine-readable protocol must disclose no more
|
|
information than the human-readable output. When you observe a
|
|
server game state change, you can only guess what event triggered
|
|
it, and what it disclosed to which player. You're stuck with
|
|
conservative assumptions. That's the death knell for completeness.
|
|
Correct assumptions will be non-obvious, so correctness is
|
|
non-obvious, too, hence hard to achieve and maintain.
|
|
|
|
Bottom line: tracking server state change cannot support a complete
|
|
client protocol for hard theoretical reasons, and I believe it
|
|
cannot support a correct one for practical reasons.
|
|
|
|
Oddly enough, people criticized C_SYNC for all the flaws it had (and
|
|
some it hadn't), except for f.
|
|
|
|
What now? Throw up our hands in despair and give up? Nah. Ken tried
|
|
a shortcut, and it didn't work. That doesn't mean there's no way at
|
|
all. I believe the only way to get this done right is by tracking
|
|
*events*. Whenever something is printed to a player, be it live
|
|
connection or telegram, we need to transmit precisely the same
|
|
information in machine-readable form. Much more work.
|
|
|
|
xdump shares valuable ideas with C_SYNC, e.g. using selector
|
|
meta-data. It is, however, much more modest in scope. We're pretty
|
|
sure we can get it right and get it done in a reasonable time frame.
|