Home World Forum
Stars! AutoHost web forums

Jump to Stars! AutoHost


 
 
Home » Stars! Clones, Extensions, Modding » Stars! Extensions » Detecting corrupt race files
Detecting corrupt race files Sun, 17 February 2019 23:58 Go to next message
ricks03 is currently offline ricks03

 
Chief Warrant Officer 1

Messages: 146
Registered: January 2012
Location: NC
There's an old race bug that editing a race file, and making the plural of the name shorter, corrupts the file.

Anyone know of an easy programmatic way to test for that? starstat reports correctly for the file, so it's something more subtle.

Thanks!



https://www.irelandbybicycle.com
https://totalhost.sinister.net:999
https://github.com/ricks03/TotalHost

Report message to a moderator

Re: Detecting corrupt race files Mon, 18 February 2019 14:37 Go to previous messageGo to next message
raptor is currently offline raptor

 
Warrant Officer
Creator of StarsWine
Created StarsWine

Messages: 123
Registered: June 2014
I toyed with this once but I don't remember how far I got. I can detail what little I did:

You can corrupt a race file by opening it, adding or removing more than 2 characters to the race name, and saving it again. At the time I had saved both copies of the files and was using a hex editor to compare the bytes that were different between the good/bad files. I remember that a couple bytes around the stored name were different, but I never got much past that.

My next step was to open/decrypt both files and deeply inspect the bit differences in the blocks to see if there was anything I could do to detect or repair, but I never finished.

That's all I remember.

Report message to a moderator

Re: Detecting corrupt race files Mon, 18 February 2019 15:01 Go to previous messageGo to next message
ricks03 is currently offline ricks03

 
Chief Warrant Officer 1

Messages: 146
Registered: January 2012
Location: NC
Yeah, I struggle with untangling the decrypting blocks process. It's just not intuitive to me.

But I've got to do that anyway to code up something that calculated the race score, so I'll try reviewing what's different on the output (when I get that far).



https://www.irelandbybicycle.com
https://totalhost.sinister.net:999
https://github.com/ricks03/TotalHost

Report message to a moderator

Re: Detecting corrupt race files Mon, 18 February 2019 16:39 Go to previous messageGo to next message
raptor is currently offline raptor

 
Warrant Officer
Creator of StarsWine
Created StarsWine

Messages: 123
Registered: June 2014
I narrowed the problem down to the FileFooterBlock (Block 0), which I think only contains the checksum of the file. Also I just found this:

https://starsautohost.org/sahforum2/index.php?t=msg&goto =20819&rid=1807

Which shows someone else already found that out and created a (now defunct) utility. I'm not sure what checksum algorithm is used yet, or what portion of the file data is checksummed. I'll have to experiment unless wumpus comes along to release his code sooner...

Report message to a moderator

Re: Detecting corrupt race files Tue, 19 February 2019 12:08 Go to previous messageGo to next message
craebild is currently offline craebild

 
Lieutenant

Messages: 568
Registered: December 2003
Location: Copenhagen, Denmark
Wumpus' last post was in June 2008, so I don't think there is much chance he will be sending you the source code.



Med venlig hilsen / Best regards / Mit freundlichen Grüßen
Christian Ræbild / Christian Raebild

Report message to a moderator

Re: Detecting corrupt race files Tue, 19 February 2019 13:17 Go to previous messageGo to next message
ricks03 is currently offline ricks03

 
Chief Warrant Officer 1

Messages: 146
Registered: January 2012
Location: NC
No harm in trying. I mean clearly _some_ people are still reading the forums. Smile

(I also tracked down a gmail address for him and emailed him.)



https://www.irelandbybicycle.com
https://totalhost.sinister.net:999
https://github.com/ricks03/TotalHost

Report message to a moderator

Re: Detecting corrupt race files Tue, 19 February 2019 18:32 Go to previous messageGo to next message
raptor is currently offline raptor

 
Warrant Officer
Creator of StarsWine
Created StarsWine

Messages: 123
Registered: June 2014
I have to take a break and get back to normal life duties, but -

I have attempted to find the checksum algorithm for the race file but haven't yet succeeded. Using the starsapi code, I have run a variety of 16-bit CRC and other algorithms against several portions of the race file, both encrypted and decrypted, with/without the block header, with/without the file header, etc.

The only possible clue I've found is what is mentioned here:

https://starsautohost.org/sahforum2/index.php?t=msg&goto =41353&rid=1807&srch=terminator#msg_41353

But I tried several permutations with that, with no success. One nifty utility I found was this:

http://reveng.sourceforge.net/

But I haven't fully explored it use yet.

Report message to a moderator

Re: Detecting corrupt race files Wed, 20 February 2019 00:19 Go to previous messageGo to next message
ricks03 is currently offline ricks03

 
Chief Warrant Officer 1

Messages: 146
Registered: January 2012
Location: NC
To confirm I understand you correctly. The problem is that Stars! doesn't recalculate the checksum correctly. We don't know how to calculate the checksum. That about sum it up?


https://www.irelandbybicycle.com
https://totalhost.sinister.net:999
https://github.com/ricks03/TotalHost

Report message to a moderator

Re: Detecting corrupt race files Wed, 20 February 2019 09:12 Go to previous messageGo to next message
raptor is currently offline raptor

 
Warrant Officer
Creator of StarsWine
Created StarsWine

Messages: 123
Registered: June 2014
That's exactly it. I've verified all other parts of the race file are correct.

Report message to a moderator

Re: Detecting corrupt race files Wed, 20 February 2019 10:04 Go to previous messageGo to next message
raptor is currently offline raptor

 
Warrant Officer
Creator of StarsWine
Created StarsWine

Messages: 123
Registered: June 2014
Narrowed it down a bit further:
- The FileFooterBlock (Block 0) contains the checksum unencrypted
- The checksum only changes if the decrypted PlayerBlock (Block 6) data changes

Report message to a moderator

Re: Detecting corrupt race files Wed, 20 February 2019 12:30 Go to previous messageGo to next message
ricks03 is currently offline ricks03

 
Chief Warrant Officer 1

Messages: 146
Registered: January 2012
Location: NC
raptor wrote on Wed, 20 February 2019 09:12
That's exactly it. I've verified all other parts of the race file are correct.

That likely explains why my StarsPWD.pl utility can't reset passwords on race files, even though it works on other (.m, .hst) files. The utility simply swaps out the password for a blank one. If there's a checksum that the .exe uses, then swapping out the password also makes the file not match the checksum.



https://www.irelandbybicycle.com
https://totalhost.sinister.net:999
https://github.com/ricks03/TotalHost

Report message to a moderator

Re: Detecting corrupt race files Wed, 20 February 2019 12:37 Go to previous messageGo to next message
ricks03 is currently offline ricks03

 
Chief Warrant Officer 1

Messages: 146
Registered: January 2012
Location: NC
Hmm. It also means the race information doesn't have a checksum (or at least it isn't used or validated) in the .m and .hst files. Because otherwise the same action (swapping out the password) would corrupt those files, and it doesn't


https://www.irelandbybicycle.com
https://totalhost.sinister.net:999
https://github.com/ricks03/TotalHost

Report message to a moderator

Re: Detecting corrupt race files Fri, 22 February 2019 10:59 Go to previous messageGo to next message
raptor is currently offline raptor

 
Warrant Officer
Creator of StarsWine
Created StarsWine

Messages: 123
Registered: June 2014
I ran the game through 'reko' disassembler/decompiler. I'm not very good with assembly, but I found this decompiled chunk after a 'GetSaveFileName' call:
// 0517:4210: Register word16 fn0517_4210(Stack word16 wArg04, Register out ptr16 bxOut)
word16 fn0517_4210(word16 wArg04, ptr16 & bxOut)
{
	*bxOut = bx;
	word16 wLoc0A_15 = 0x00;
	word16 wLoc08_16;
	for (wLoc08_16 = 0x00; wLoc08_16 < 0x60; ++wLoc08_16)
	{
		word16 bx_36 = wLoc08_16 * 0x02;
		*bxOut = bx_36;
		wLoc0A_15 = wLoc0A_15 ^ Mem0[ds:bx_36 + wArg04:word16];
	}
	return wLoc0A_15;
}


That looks like a checksum. I'll look at it again later today or tomorrow.

Report message to a moderator

Re: Detecting corrupt race files Sun, 24 February 2019 17:42 Go to previous messageGo to next message
raptor is currently offline raptor

 
Warrant Officer
Creator of StarsWine
Created StarsWine

Messages: 123
Registered: June 2014
I've implemented what I thought was as close to the decompiled checksum code that I could get. I can now calculate 1/2 the checksum about 50% of the time.. so something is still missing. I've even been brute forcing the know algorithm on different locations of the decrypted player block, but so far no good.

On another note, I got someone to disassemble the executable and got this nice picture of the race checksum assembly:

https://imgur.com/a/6P1VU7c

which seems to have more human readable information than I thought.

Report message to a moderator

Re: Detecting corrupt race files Sun, 24 February 2019 22:59 Go to previous messageGo to next message
ricks03 is currently offline ricks03

 
Chief Warrant Officer 1

Messages: 146
Registered: January 2012
Location: NC
That's still amazing progress in just a couple of days!


https://www.irelandbybicycle.com
https://totalhost.sinister.net:999
https://github.com/ricks03/TotalHost

Report message to a moderator

Re: Detecting corrupt race files Wed, 27 February 2019 22:18 Go to previous messageGo to next message
raptor is currently offline raptor

 
Warrant Officer
Creator of StarsWine
Created StarsWine

Messages: 123
Registered: June 2014
Well, I just found out that I can open all of my corrupted race files in 26jRC4 without issue. I've been using my old copy 27jRC3 most of the time. Was the checksum check disabled in the newer 26jRC4 version? It still records a checksum for a new race file...

D

Report message to a moderator

Re: Detecting corrupt race files Thu, 28 February 2019 22:33 Go to previous messageGo to next message
m.a@stars is currently offline m.a@stars

 
Commander

Messages: 2760
Registered: October 2004
Location: Third star to the left
Maybe this info can help:

http://www.starsfaq.com/wiki.starsautohost.org/wiki/Patches_ and_versions.html



So many Stars, so few Missiles!

In space no one can hear you scheme! Deal

Report message to a moderator

Re: Detecting corrupt race files Tue, 14 July 2020 05:38 Go to previous messageGo to next message
russl5445 is currently offline russl5445

 
Crewman 3rd Class

Messages: 5
Registered: September 2008
Location: Tucson, AZ
I've been banging my head against the checksum value in the Footer for a bit now...have you made any progress on it yourself?

Looks to me like:
- Checksum is calculated over the cleartext data (not the encrypted version)
- Footer stores the checksum in cleartext
- Agree, it appears to be XOR-related (a one-bit change in the cleartext causes a one-bit change in the Footer)

But I haven't been able to figure out exactly what the algorithm is. I've been calculating checksums on the blocks of the standard races (since that eliminates any possibility of differences in the file length or section header metadata, but the pattern isn't obvious to me yet.

Report message to a moderator

Re: Detecting corrupt race files Tue, 14 July 2020 09:14 Go to previous messageGo to next message
raptor is currently offline raptor

 
Warrant Officer
Creator of StarsWine
Created StarsWine

Messages: 123
Registered: June 2014
The XOR algorithm I posted above is definitely the algorithm (the disassembly from several disassemblers show the same). However, it never seems to work right against a race file. I suspect there is some initialized/default data that is added to the in-game race data struct after a file is read in, and then that is checksummed. But I'm still unsure...

Report message to a moderator

Re: Detecting corrupt race files Tue, 14 July 2020 11:45 Go to previous messageGo to next message
russl5445 is currently offline russl5445

 
Crewman 3rd Class

Messages: 5
Registered: September 2008
Location: Tucson, AZ
I suppose the next step is to change a race file, one property at a time, until I see a large change in the checksum. Seems tedious, but doable, I guess.

Report message to a moderator

Re: Detecting corrupt race files Tue, 14 July 2020 13:40 Go to previous message
russl5445 is currently offline russl5445

 
Crewman 3rd Class

Messages: 5
Registered: September 2008
Location: Tucson, AZ
Hit a bit of inspiration, and so tried something out. Looks like the XOR algorithm might be working fine...so long as you're XORing the uncompressed (ASCII) race names, not the compressed form that's in the file. You can see this if you start with a standard Humanoid race, and then add "a" and later "l" to the name.

I haven't written up the code to verify this yet, but it looks likely to be the solution.

Did you have troubles with other file types as well? IIRC, the names in Design fields are compressed as well. Not sure about the fleet names.

Report message to a moderator

Previous Topic: Stars! AI
Goto Forum:
  


Current Time: Sat Oct 24 08:19:39 EDT 2020