Home World Forum
Stars! AutoHost web forums

 Jump to Stars! AutoHost

 
 
Home » Stars! Clones, Extensions, Modding » Stars! Extensions » Detecting corrupt race files
Detecting corrupt race files Sun, 17 February 2019 23:58 Go to next message
ricks03 is currently offline ricks03

 
Officer Cadet 1st Year
Creator of TotalHost and Stars! utilities
Forum Administrator
Stars! AutoHost Administrator
Created TotalHost and Stars! utilities

Messages: 225
Registered: January 2012
Location: NC
 There's an old race bug that editing a race file, and making the plural of the name shorter, corrupts the file.

Anyone know of an easy programmatic way to test for that? starstat reports correctly for the file, so it's something more subtle.

Thanks!

https://www.irelandbybicycle.com
http://totalhost.sinister.net:999
https://github.com/ricks03/TotalHost

Report message to a moderator

Re: Detecting corrupt race files Mon, 18 February 2019 14:37 Go to previous messageGo to next message
raptor is currently offline raptor

 
Chief Warrant Officer 1
Creator of StarsWine
Created StarsWine

Messages: 138
Registered: June 2014
I toyed with this once but I don't remember how far I got. I can detail what little I did:

You can corrupt a race file by opening it, adding or removing more than 2 characters to the race name, and saving it again. At the time I had saved both copies of the files and was using a hex editor to compare the bytes that were different between the good/bad files. I remember that a couple bytes around the stored name were different, but I never got much past that.

My next step was to open/decrypt both files and deeply inspect the bit differences in the blocks to see if there was anything I could do to detect or repair, but I never finished.

That's all I remember.

Report message to a moderator

Re: Detecting corrupt race files Mon, 18 February 2019 15:01 Go to previous messageGo to next message
ricks03 is currently offline ricks03

 
Officer Cadet 1st Year
Creator of TotalHost and Stars! utilities
Forum Administrator
Stars! AutoHost Administrator
Created TotalHost and Stars! utilities

Messages: 225
Registered: January 2012
Location: NC
 Yeah, I struggle with untangling the decrypting blocks process. It's just not intuitive to me.

But I've got to do that anyway to code up something that calculated the race score, so I'll try reviewing what's different on the output (when I get that far).

https://www.irelandbybicycle.com
http://totalhost.sinister.net:999
https://github.com/ricks03/TotalHost

Report message to a moderator

Re: Detecting corrupt race files Mon, 18 February 2019 16:39 Go to previous messageGo to next message
raptor is currently offline raptor

 
Chief Warrant Officer 1
Creator of StarsWine
Created StarsWine

Messages: 138
Registered: June 2014
I narrowed the problem down to the FileFooterBlock (Block 0), which I think only contains the checksum of the file. Also I just found this:

https://starsautohost.org/sahforum2/index.php?t=msg&goto =20819&rid=1807

Which shows someone else already found that out and created a (now defunct) utility. I'm not sure what checksum algorithm is used yet, or what portion of the file data is checksummed. I'll have to experiment unless wumpus comes along to release his code sooner...

Report message to a moderator

Re: Detecting corrupt race files Tue, 19 February 2019 12:08 Go to previous messageGo to next message
craebild is currently offline craebild

 
Lieutenant

Messages: 568
Registered: December 2003
Location: Copenhagen, Denmark
 Wumpus' last post was in June 2008, so I don't think there is much chance he will be sending you the source code.

Med venlig hilsen / Best regards / Mit freundlichen Grüßen
Christian Ræbild / Christian Raebild

Report message to a moderator

Re: Detecting corrupt race files Tue, 19 February 2019 13:17 Go to previous messageGo to next message
ricks03 is currently offline ricks03

 
Officer Cadet 1st Year
Creator of TotalHost and Stars! utilities
Forum Administrator
Stars! AutoHost Administrator
Created TotalHost and Stars! utilities

Messages: 225
Registered: January 2012
Location: NC
 No harm in trying. I mean clearly _some_ people are still reading the forums. Smile

(I also tracked down a gmail address for him and emailed him.)

https://www.irelandbybicycle.com
http://totalhost.sinister.net:999
https://github.com/ricks03/TotalHost

Report message to a moderator

Re: Detecting corrupt race files Tue, 19 February 2019 18:32 Go to previous messageGo to next message
raptor is currently offline raptor

 
Chief Warrant Officer 1
Creator of StarsWine
Created StarsWine

Messages: 138
Registered: June 2014
I have to take a break and get back to normal life duties, but -

I have attempted to find the checksum algorithm for the race file but haven't yet succeeded. Using the starsapi code, I have run a variety of 16-bit CRC and other algorithms against several portions of the race file, both encrypted and decrypted, with/without the block header, with/without the file header, etc.

The only possible clue I've found is what is mentioned here:

https://starsautohost.org/sahforum2/index.php?t=msg&goto =41353&rid=1807&srch=terminator#msg_41353

But I tried several permutations with that, with no success. One nifty utility I found was this:

http://reveng.sourceforge.net/

But I haven't fully explored it use yet.

Report message to a moderator

Re: Detecting corrupt race files Wed, 20 February 2019 00:19 Go to previous messageGo to next message
ricks03 is currently offline ricks03

 
Officer Cadet 1st Year
Creator of TotalHost and Stars! utilities
Forum Administrator
Stars! AutoHost Administrator
Created TotalHost and Stars! utilities

Messages: 225
Registered: January 2012
Location: NC
 To confirm I understand you correctly. The problem is that Stars! doesn't recalculate the checksum correctly. We don't know how to calculate the checksum. That about sum it up?

https://www.irelandbybicycle.com
http://totalhost.sinister.net:999
https://github.com/ricks03/TotalHost

Report message to a moderator

Re: Detecting corrupt race files Wed, 20 February 2019 09:12 Go to previous messageGo to next message
raptor is currently offline raptor

 
Chief Warrant Officer 1
Creator of StarsWine
Created StarsWine

Messages: 138
Registered: June 2014
That's exactly it. I've verified all other parts of the race file are correct.

Report message to a moderator

Re: Detecting corrupt race files Wed, 20 February 2019 10:04 Go to previous messageGo to next message
raptor is currently offline raptor

 
Chief Warrant Officer 1
Creator of StarsWine
Created StarsWine

Messages: 138
Registered: June 2014
Narrowed it down a bit further:
- The FileFooterBlock (Block 0) contains the checksum unencrypted
- The checksum only changes if the decrypted PlayerBlock (Block 6) data changes

Report message to a moderator

Re: Detecting corrupt race files Wed, 20 February 2019 12:30 Go to previous messageGo to next message
ricks03 is currently offline ricks03

 
Officer Cadet 1st Year
Creator of TotalHost and Stars! utilities
Forum Administrator
Stars! AutoHost Administrator
Created TotalHost and Stars! utilities

Messages: 225
Registered: January 2012
Location: NC
 raptor wrote on Wed, 20 February 2019 09:12
That's exactly it. I've verified all other parts of the race file are correct.

That likely explains why my StarsPWD.pl utility can't reset passwords on race files, even though it works on other (.m, .hst) files. The utility simply swaps out the password for a blank one. If there's a checksum that the .exe uses, then swapping out the password also makes the file not match the checksum.

https://www.irelandbybicycle.com
http://totalhost.sinister.net:999
https://github.com/ricks03/TotalHost

Report message to a moderator

Re: Detecting corrupt race files Wed, 20 February 2019 12:37 Go to previous messageGo to next message
ricks03 is currently offline ricks03

 
Officer Cadet 1st Year
Creator of TotalHost and Stars! utilities
Forum Administrator
Stars! AutoHost Administrator
Created TotalHost and Stars! utilities

Messages: 225
Registered: January 2012
Location: NC
 Hmm. It also means the race information doesn't have a checksum (or at least it isn't used or validated) in the .m and .hst files. Because otherwise the same action (swapping out the password) would corrupt those files, and it doesn't

https://www.irelandbybicycle.com
http://totalhost.sinister.net:999
https://github.com/ricks03/TotalHost

Report message to a moderator

Re: Detecting corrupt race files Fri, 22 February 2019 10:59 Go to previous messageGo to next message
raptor is currently offline raptor

 
Chief Warrant Officer 1
Creator of StarsWine
Created StarsWine

Messages: 138
Registered: June 2014
I ran the game through 'reko' disassembler/decompiler. I'm not very good with assembly, but I found this decompiled chunk after a 'GetSaveFileName' call:
// 0517:4210: Register word16 fn0517_4210(Stack word16 wArg04, Register out ptr16 bxOut)
word16 fn0517_4210(word16 wArg04, ptr16 & bxOut)
{
	*bxOut = bx;
	word16 wLoc0A_15 = 0x00;
	word16 wLoc08_16;
	for (wLoc08_16 = 0x00; wLoc08_16 < 0x60; ++wLoc08_16)
	{
		word16 bx_36 = wLoc08_16 * 0x02;
		*bxOut = bx_36;
		wLoc0A_15 = wLoc0A_15 ^ Mem0[ds:bx_36 + wArg04:word16];
	}
	return wLoc0A_15;
}


That looks like a checksum. I'll look at it again later today or tomorrow.

Report message to a moderator

Re: Detecting corrupt race files Sun, 24 February 2019 17:42 Go to previous messageGo to next message
raptor is currently offline raptor

 
Chief Warrant Officer 1
Creator of StarsWine
Created StarsWine

Messages: 138
Registered: June 2014
I've implemented what I thought was as close to the decompiled checksum code that I could get. I can now calculate 1/2 the checksum about 50% of the time.. so something is still missing. I've even been brute forcing the know algorithm on different locations of the decrypted player block, but so far no good.

On another note, I got someone to disassemble the executable and got this nice picture of the race checksum assembly:

https://imgur.com/a/6P1VU7c

which seems to have more human readable information than I thought.

Report message to a moderator

Re: Detecting corrupt race files Sun, 24 February 2019 22:59 Go to previous messageGo to next message
ricks03 is currently offline ricks03

 
Officer Cadet 1st Year
Creator of TotalHost and Stars! utilities
Forum Administrator
Stars! AutoHost Administrator
Created TotalHost and Stars! utilities

Messages: 225
Registered: January 2012
Location: NC
 That's still amazing progress in just a couple of days!

https://www.irelandbybicycle.com
http://totalhost.sinister.net:999
https://github.com/ricks03/TotalHost

Report message to a moderator

Re: Detecting corrupt race files Wed, 27 February 2019 22:18 Go to previous messageGo to next message
raptor is currently offline raptor

 
Chief Warrant Officer 1
Creator of StarsWine
Created StarsWine

Messages: 138
Registered: June 2014
Well, I just found out that I can open all of my corrupted race files in 26jRC4 without issue. I've been using my old copy 27jRC3 most of the time. Was the checksum check disabled in the newer 26jRC4 version? It still records a checksum for a new race file...

D

Report message to a moderator

Re: Detecting corrupt race files Thu, 28 February 2019 22:33 Go to previous messageGo to next message
m.a@stars is currently offline m.a@stars

 
Commander

Messages: 2768
Registered: October 2004
Location: Third star to the left
 Maybe this info can help:

http://www.starsfaq.com/wiki.starsautohost.org/wiki/Patches_ and_versions.html

So many Stars, so few Missiles!

In space no one can hear you scheme! Deal

Report message to a moderator

Re: Detecting corrupt race files Tue, 14 July 2020 05:38 Go to previous messageGo to next message
russl5445 is currently offline russl5445

 
Crewman 3rd Class

Messages: 5
Registered: September 2008
Location: Tucson, AZ
 I've been banging my head against the checksum value in the Footer for a bit now...have you made any progress on it yourself?

Looks to me like:
- Checksum is calculated over the cleartext data (not the encrypted version)
- Footer stores the checksum in cleartext
- Agree, it appears to be XOR-related (a one-bit change in the cleartext causes a one-bit change in the Footer)

But I haven't been able to figure out exactly what the algorithm is. I've been calculating checksums on the blocks of the standard races (since that eliminates any possibility of differences in the file length or section header metadata, but the pattern isn't obvious to me yet.

Report message to a moderator

Re: Detecting corrupt race files Tue, 14 July 2020 09:14 Go to previous messageGo to next message
raptor is currently offline raptor

 
Chief Warrant Officer 1
Creator of StarsWine
Created StarsWine

Messages: 138
Registered: June 2014
The XOR algorithm I posted above is definitely the algorithm (the disassembly from several disassemblers show the same). However, it never seems to work right against a race file. I suspect there is some initialized/default data that is added to the in-game race data struct after a file is read in, and then that is checksummed. But I'm still unsure...

Report message to a moderator

Re: Detecting corrupt race files Tue, 14 July 2020 11:45 Go to previous messageGo to next message
russl5445 is currently offline russl5445

 
Crewman 3rd Class

Messages: 5
Registered: September 2008
Location: Tucson, AZ
 I suppose the next step is to change a race file, one property at a time, until I see a large change in the checksum. Seems tedious, but doable, I guess.

Report message to a moderator

Re: Detecting corrupt race files Tue, 14 July 2020 13:40 Go to previous messageGo to next message
russl5445 is currently offline russl5445

 
Crewman 3rd Class

Messages: 5
Registered: September 2008
Location: Tucson, AZ
 Hit a bit of inspiration, and so tried something out. Looks like the XOR algorithm might be working fine...so long as you're XORing the uncompressed (ASCII) race names, not the compressed form that's in the file. You can see this if you start with a standard Humanoid race, and then add "a" and later "l" to the name.

I haven't written up the code to verify this yet, but it looks likely to be the solution.

Did you have troubles with other file types as well? IIRC, the names in Design fields are compressed as well. Not sure about the fleet names.

Report message to a moderator

Re: Detecting corrupt race files Mon, 16 November 2020 02:51 Go to previous messageGo to next message
ricks03 is currently offline ricks03

 
Officer Cadet 1st Year
Creator of TotalHost and Stars! utilities
Forum Administrator
Stars! AutoHost Administrator
Created TotalHost and Stars! utilities

Messages: 225
Registered: January 2012
Location: NC
 I don't know about the race files, but it looks like
a) the file footer block is the turn year in the .m file
b) it's not encrypted

https://www.irelandbybicycle.com
http://totalhost.sinister.net:999
https://github.com/ricks03/TotalHost

Report message to a moderator

Re: Detecting corrupt race files Wed, 18 November 2020 23:58 Go to previous messageGo to next message
ricks03 is currently offline ricks03

 
Officer Cadet 1st Year
Creator of TotalHost and Stars! utilities
Forum Administrator
Stars! AutoHost Administrator
Created TotalHost and Stars! utilities

Messages: 225
Registered: January 2012
Location: NC
 russl5445 wrote on Tue, 14 July 2020 13:40
Hit a bit of inspiration, and so tried something out. Looks like the XOR algorithm might be working fine...so long as you're XORing the uncompressed (ASCII) race names, not the compressed form that's in the file. You can see this if you start with a standard Humanoid race, and then add "a" and later "l" to the name.

I haven't written up the code to verify this yet, but it looks likely to be the solution.

Did you have troubles with other file types as well? IIRC, the names in Design fields are compressed as well. Not sure about the fleet names.
OK, I'm turning my attention to your insight.

Can you give me a code example of what you're doing? Even a pseudo code example would help, as I don't quite grasp "XORing the uncompressed race names" The singular name? The plural name? Both? How are you XORing an ascii string?


https://www.irelandbybicycle.com
http://totalhost.sinister.net:999
https://github.com/ricks03/TotalHost

Report message to a moderator

Re: Detecting corrupt race files Sat, 15 May 2021 02:34 Go to previous messageGo to next message
ricks03 is currently offline ricks03

 
Officer Cadet 1st Year
Creator of TotalHost and Stars! utilities
Forum Administrator
Stars! AutoHost Administrator
Created TotalHost and Stars! utilities

Messages: 225
Registered: January 2012
Location: NC
 Testing with the decrypted data for Block 8, block 6, and Block 0

Some notes for the r file:

Block 8 in the race file changes bytes 12/13 when you save (the encryption seed). Saving a race again, even if you don't change anything, generates different Byte 12/13 in block 8.

Identical races create identical checksums in Block 0, even if the Block 8 byte 12/13 are different.

Conclusion: Block 8 doesn't matter for the checksum, it's only Block 6


Let's say the race name is "0".
Adding the same character ("0") at a time to the singular name, the two bytes in block 0 (the checksum) changes the value(s) in a VERY consistent pattern. So singular race name "0", "00", "000", "0000", etc to 16 characters

a b
a c
d c
d b
a b
a c
d c
d b
a b
a c
d c
etc

With a name of "0", add a 0 to the name ("00") makes the checksum:
BLOCK:0,Offset:138,Bytes:2 DATA DECRYPTED:248 92

If instead adding a 1 to the same name ("01") makes the checksum
BLOCK:0,Offset:138,Bytes:2 DATA DECRYPTED:248 93

If instead adding a 2 to the same name ("02") makes the checksum
BLOCK:0,Offset:138,Bytes:2 DATA DECRYPTED:248 94

So the second byte is increasing by the same amount as the value of the new character.

Adding a "Z" ("0Z")
BLOCK:0,Offset:138,Bytes:2 DATA DECRYPTED:248 54

adding a "!" ("0!")
BLOCK:0,Offset:138,Bytes:2 DATA DECRYPTED:248 77

Conclusion: the second byte is directly related to the value of the character added to the name, in the "0100000000000000" position in this case


A name of 1010101010101010 generates a value of
BLOCK:0,Offset:151,Bytes:2 DATA DECRYPTED:200 92

Where a name of 0101010101010101
BLOCK:0,Offset:151,Bytes:2 DATA DECRYPTED:200 93

The difference in the offset of 1s is the same difference as between "00" and "01".

https://www.irelandbybicycle.com
http://totalhost.sinister.net:999
https://github.com/ricks03/TotalHost

Report message to a moderator

Re: Detecting corrupt race files Sat, 15 May 2021 02:44 Go to previous messageGo to previous message
ricks03 is currently offline ricks03

 
Officer Cadet 1st Year
Creator of TotalHost and Stars! utilities
Forum Administrator
Stars! AutoHost Administrator
Created TotalHost and Stars! utilities

Messages: 225
Registered: January 2012
Location: NC
 One more note.
Adding the same character to both the singular and plural of the race name, and the checksum doesn't change.

https://www.irelandbybicycle.com
http://totalhost.sinister.net:999
https://github.com/ricks03/TotalHost

Report message to a moderator

Previous Topic: Utility to reset Stars! passwords
Next Topic: Utility to fix corrupt race files
Goto Forum:
  


Current Time: Tue Sep 10 02:37:52 EDT 2024
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.3.
Copyright ©2001-2011 FUDforum Bulletin Board Software