Home World Forum
Stars! AutoHost web forums

Jump to Stars! AutoHost


 
 
Home » Stars! Clones, Extensions, Modding » Stars! Extensions » Detecting corrupt race files
Re: Detecting corrupt race files Sat, 15 May 2021 12:10 Go to previous messageGo to next message
ricks03 is currently offline ricks03

 
Officer Cadet 1st Year
Creator of TotalHost and Stars! utilities
Created TotalHost and Stars! utilities

Messages: 221
Registered: January 2012
Location: NC
Here's the raw checksum (decrypted) data for having a race name of "0" and each time adding another "0"
BLOCK:0,Offset:137,Bytes:2 DATA DECRYPTED:248 108 11111000 1101100
BLOCK:0,Offset:138,Bytes:2 DATA DECRYPTED:248 92 11111000 01011100
BLOCK:0,Offset:139,Bytes:2 DATA DECRYPTED:200 92 11001000 01011100
BLOCK:0,Offset:140,Bytes:2 DATA DECRYPTED:200 108 11001000 01101100
BLOCK:0,Offset:141,Bytes:2 DATA DECRYPTED:248 108 11111000 01101100
BLOCK:0,Offset:142,Bytes:2 DATA DECRYPTED:248 92 11111000 01011100
BLOCK:0,Offset:143,Bytes:2 DATA DECRYPTED:200 92 11001000 01011100
BLOCK:0,Offset:144,Bytes:2 DATA DECRYPTED:200 108 11001000 01101100
BLOCK:0,Offset:145,Bytes:2 DATA DECRYPTED:248 108 11111000 01101100
BLOCK:0,Offset:146,Bytes:2 DATA DECRYPTED:248 92 11111000 01011100
BLOCK:0,Offset:147,Bytes:2 DATA DECRYPTED:200 92 11001000 01011100
BLOCK:0,Offset:148,Bytes:2 DATA DECRYPTED:200 108 11001000 01101100
BLOCK:0,Offset:149,Bytes:2 DATA DECRYPTED:248 108 11111000 01101100
BLOCK:0,Offset:150,Bytes:2 DATA DECRYPTED:248 92 11111000 01011100
BLOCK:0,Offset:151,Bytes:2 DATA DECRYPTED:200 92 11001000 01011100

The same two bits are flipping in the first and second binary set. Even "0"s change the first checksum bye; odd ones change the second checksum byte



https://www.irelandbybicycle.com
http://totalhost.sinister.net:999
https://github.com/ricks03/TotalHost

Report message to a moderator

Re: Detecting corrupt race files Sat, 15 May 2021 12:29 Go to previous messageGo to next message
ricks03 is currently offline ricks03

 
Officer Cadet 1st Year
Creator of TotalHost and Stars! utilities
Created TotalHost and Stars! utilities

Messages: 221
Registered: January 2012
Location: NC
Here's the results for Leftover Resources, and Race Icon:

Leftover resources
BLOCK:0,Offset:137,Bytes:2 DATA DECRYPTED:248 109
BLOCK:0,Offset:137,Bytes:2 DATA DECRYPTED:248 108
BLOCK:0,Offset:137,Bytes:2 DATA DECRYPTED:248 110
BLOCK:0,Offset:137,Bytes:2 DATA DECRYPTED:248 111
BLOCK:0,Offset:137,Bytes:2 DATA DECRYPTED:248 104

Race icon
BLOCK:0,Offset:137,Bytes:2 DATA DECRYPTED:248 108
BLOCK:0,Offset:137,Bytes:2 DATA DECRYPTED:240 108
BLOCK:0,Offset:137,Bytes:2 DATA DECRYPTED:232 108
BLOCK:0,Offset:137,Bytes:2 DATA DECRYPTED:224 108
BLOCK:0,Offset:137,Bytes:2 DATA DECRYPTED:216 108
...
BLOCK:0,Offset:137,Bytes:2 DATA DECRYPTED:32 108
BLOCK:0,Offset:137,Bytes:2 DATA DECRYPTED:24 108
BLOCK:0,Offset:137,Bytes:2 DATA DECRYPTED:0 108


I'm using Name, Leftover Resources, and Race Icon because you can change them without the race point total changing.

Leftover Resources and Race Icon clearly flip a single bit, which in turn flips a single bit in one byte of the checksum. Race name adds a bit each time. Effectively adding a single bit each time, going even odd.

I suspect difficulty in translating the checksum comes from the race score being stored in the race, so every time you change most values, you actually change two values - the value you changed, and the total race point value.

So I suspect the checksum is something done on the even bytes of block 6, and something done on the odd bytes of block 6. CRC? XOR?

Hope that's useful.




https://www.irelandbybicycle.com
http://totalhost.sinister.net:999
https://github.com/ricks03/TotalHost

Report message to a moderator

Re: Detecting corrupt race files Sat, 15 May 2021 13:09 Go to previous messageGo to next message
ricks03 is currently offline ricks03

 
Officer Cadet 1st Year
Creator of TotalHost and Stars! utilities
Created TotalHost and Stars! utilities

Messages: 221
Registered: January 2012
Location: NC
and BAM!

$x1 = 0; $x2 = 0;
for (my $i = 0; $i < scalar @decryptedData; $i=$i+2) {
$x1 = $x1^$decryptedData[$i];
}
for (my $i = 1; $i < scalar @decryptedData; $i=$i+2) {
$x2 = $x2^$decryptedData[$i];
}
print "X1: $x1 X2: $x2\n";

produces the two bytes of the checksum. Which means I can now develop a tool that will fix a corrupted race file by recalculating the checksum and saving it. woot!!



https://www.irelandbybicycle.com
http://totalhost.sinister.net:999
https://github.com/ricks03/TotalHost

Report message to a moderator

Re: Detecting corrupt race files Sat, 15 May 2021 14:21 Go to previous messageGo to next message
ricks03 is currently offline ricks03

 
Officer Cadet 1st Year
Creator of TotalHost and Stars! utilities
Created TotalHost and Stars! utilities

Messages: 221
Registered: January 2012
Location: NC
Should also mean I can fix the Password reset utility to work on race files!


https://www.irelandbybicycle.com
http://totalhost.sinister.net:999
https://github.com/ricks03/TotalHost

Report message to a moderator

Re: Detecting corrupt race files Sat, 15 May 2021 15:55 Go to previous messageGo to next message
raptor is currently offline raptor

 
Chief Warrant Officer 1
Creator of StarsWine
Created StarsWine

Messages: 138
Registered: June 2014
This is the same algorithm as above. It will fail for some race files.

Report message to a moderator

Re: Detecting corrupt race files Sat, 15 May 2021 17:01 Go to previous messageGo to next message
ricks03 is currently offline ricks03

 
Officer Cadet 1st Year
Creator of TotalHost and Stars! utilities
Created TotalHost and Stars! utilities

Messages: 221
Registered: January 2012
Location: NC
Can you send me one it fails for? It's working on all my test race files.

Rick


[Updated on: Sat, 15 May 2021 17:02]




https://www.irelandbybicycle.com
http://totalhost.sinister.net:999
https://github.com/ricks03/TotalHost

Report message to a moderator

Re: Detecting corrupt race files Mon, 17 May 2021 11:57 Go to previous messageGo to next message
raptor is currently offline raptor

 
Chief Warrant Officer 1
Creator of StarsWine
Created StarsWine

Messages: 138
Registered: June 2014
Sent.

It fails on greater than 3/4 of the files as long as the race names have several characters in them. I think I found two more clues by dumping the running program memory and inspection the patterns in the structures:

1. The singular and plural race names must be word-aligned (16-bit or 2-byte)
2. Most times the ASCII equivalent of the race names are checksummed (hat-tip to russl5445)

For #2, sometimes the stars-encoded text is used instead but I'm not sure why yet. I suspect it has to do with a size comparison, and some sort of algorithm fallback.


[Updated on: Mon, 17 May 2021 11:57]

Report message to a moderator

Re: Detecting corrupt race files Mon, 17 May 2021 12:03 Go to previous messageGo to next message
ricks03 is currently offline ricks03

 
Officer Cadet 1st Year
Creator of TotalHost and Stars! utilities
Created TotalHost and Stars! utilities

Messages: 221
Registered: January 2012
Location: NC
(corrected)
The checksum formula not using the names at all:
for (my $i = 0; $i <= scalar @decryptedData -($singularNameLength+$pluralNameLength+2); $i=$i+2) {
$checkSum1 = $checkSum1^$decryptedData[$i];
}
for (my $i = 1; $i < scalar @decryptedData -($singularNameLength+$pluralNameLength+2); $i=$i+2) {
$checkSum2 = $checkSum2^$decryptedData[$i];
}

I think works in all cases that the two names are identical to each other. Which is still broken, but a datapoint Smile


[Updated on: Mon, 17 May 2021 12:45]




https://www.irelandbybicycle.com
http://totalhost.sinister.net:999
https://github.com/ricks03/TotalHost

Report message to a moderator

Re: Detecting corrupt race files Mon, 17 May 2021 12:37 Go to previous messageGo to next message
raptor is currently offline raptor

 
Chief Warrant Officer 1
Creator of StarsWine
Created StarsWine

Messages: 138
Registered: June 2014
Yeah, that's the original algorithm just with two loops and it can work every once in a while. The issue isn't with the algorithm, but with the input data.

Report message to a moderator

Re: Detecting corrupt race files Mon, 17 May 2021 12:50 Go to previous messageGo to next message
ricks03 is currently offline ricks03

 
Officer Cadet 1st Year
Creator of TotalHost and Stars! utilities
Created TotalHost and Stars! utilities

Messages: 221
Registered: January 2012
Location: NC
Corrected the above. My point is that formula works for all cases where the two names match. Not including the name generates the correct checksum if the singular and plural names are the same. It doesn't work if the're different, but the same length (although if they're different by one character that's the next character (AA AB) the checksum will also vary by a single bit.

So the net result of the XOR (or whatever) of the name(s) doesn't change the value. That means the formula isn't correct, but it _does_ tell you that the net result of the name calculation of singular and plural washes out somehow in that case. Which might provide insight? Esp. in the context of XOR, where XORing a number against itself returns 0, and returning a number against 0 returns itself.


[Updated on: Mon, 17 May 2021 12:51]




https://www.irelandbybicycle.com
http://totalhost.sinister.net:999
https://github.com/ricks03/TotalHost

Report message to a moderator

Re: Detecting corrupt race files Mon, 17 May 2021 12:57 Go to previous messageGo to next message
raptor is currently offline raptor

 
Chief Warrant Officer 1
Creator of StarsWine
Created StarsWine

Messages: 138
Registered: June 2014
Yes, I understand that. What I'm saying is that the formula *is* correct, without a doubt, as it's from from the disassembly (see above for the single-loop version). The input data is what is incorrect and needs to be formatted properly somehow: block 6 is somehow overlaid into a slightly different structure that is then checksummed.

Next clue: both singular and plural names seem to each be part of a 32 byte array

Report message to a moderator

Re: Detecting corrupt race files Mon, 17 May 2021 13:08 Go to previous messageGo to next message
ricks03 is currently offline ricks03

 
Officer Cadet 1st Year
Creator of TotalHost and Stars! utilities
Created TotalHost and Stars! utilities

Messages: 221
Registered: January 2012
Location: NC
I'm off to test a merge of that theory.
The first byte of the checksum is the even bytes, where the characters are converted to their ASCII value. It's working manually out to 4 characters.



https://www.irelandbybicycle.com
http://totalhost.sinister.net:999
https://github.com/ricks03/TotalHost

Report message to a moderator

Re: Detecting corrupt race files Mon, 17 May 2021 14:02 Go to previous messageGo to next message
ricks03 is currently offline ricks03

 
Officer Cadet 1st Year
Creator of TotalHost and Stars! utilities
Created TotalHost and Stars! utilities

Messages: 221
Registered: January 2012
Location: NC
Right, and the different formatting is of just the singular/plural name data.


https://www.irelandbybicycle.com
http://totalhost.sinister.net:999
https://github.com/ricks03/TotalHost

Report message to a moderator

Re: Detecting corrupt race files Mon, 17 May 2021 14:20 Go to previous messageGo to next message
ricks03 is currently offline ricks03

 
Officer Cadet 1st Year
Creator of TotalHost and Stars! utilities
Created TotalHost and Stars! utilities

Messages: 221
Registered: January 2012
Location: NC
So the formula appears to be If the Race Name is ABCDE / EFGHI

<other data that doesn't include the race name > 0 0 A E B F C G D H E I
where the values of the individual letters are converted to ASCII, with XOR'd even bytes for the first checksum value, and XOR'd odd bytes for the second checksum value.

That makes perfect sense for why ABCDE / ABCDE doesn't change the Checksum value, because you're XOR'ing A^A, C^C, E^E and 0^0, B^B,D^D, which don't change the value.

It's still not perfect, as it only works at the moment for an odd # of characters that is the same in both singular and plural, but it works for any combination of any characters for an odd number of characters. Just have to figure out how to fill in the blanks.

Edit: Literally, fill in the array of ascii values for singular and plural with 0s, so each is 15 characters long.


[Updated on: Mon, 17 May 2021 14:32]




https://www.irelandbybicycle.com
http://totalhost.sinister.net:999
https://github.com/ricks03/TotalHost

Report message to a moderator

Re: Detecting corrupt race files Mon, 17 May 2021 15:32 Go to previous messageGo to next message
ricks03 is currently offline ricks03

 
Officer Cadet 1st Year
Creator of TotalHost and Stars! utilities
Created TotalHost and Stars! utilities

Messages: 221
Registered: January 2012
Location: NC
This code appears to work for all cases:
singular name being random characters of random length
plural name being random characters of random length

sub raceCheckSum {  # calculate a race checksum
  # The race checksum is calculated from the array of decrypted data of Block 6
  #  without the singular/plural race name data
  # The singular and plural race names are recalculated as ord arrays, 
  #  and each padded to 15 characters
  # Then the data arrray has added to it:
  # + 0 0    (to replace the race name size fields I suspect) 
  # + the 1st two ord from the singular name, and the 1st two ord from the plural
  # + the 2nd two ord from singular, and 2nd two ord from plural
  # ...
  # 
  # the 1st checksum byte is the XOR of the even data bytes
  # the 2nd checksum byte is the XOR of the odd data bytes
  
  my ($decryptedData, $singularRaceName, $pluralRaceName) = @_;
  my @decryptedData = @{ $decryptedData };
  my ($checkSum1, $checkSum2);
  my $datalength = scalar @decryptedData - (1 + length($singularRaceName) + 1 + length($pluralRaceName) + 1);
  my @dData = @decryptedData[0..$datalength];
  # get the ascii values of the Race names - singular
  my @singularRaceNameOrd = unpack("C*", $singularRaceName); #array of ascii values
  unshift (@singularRaceNameOrd, 0); # add a starting 0
  for (my $i = scalar @singularRaceNameOrd; $i <= 15; $i++) {
   push (@singularRaceNameOrd, 0); # Pad out the array
  }
  my @pluralRaceNameOrd = unpack("C*", $pluralRaceName); #array of ascii values
  unshift (@pluralRaceNameOrd, 0); # add a starting 0
  for (my $i = scalar @pluralRaceNameOrd; $i <= 15; $i++) {
    push (@pluralRaceNameOrd, 0); # Pad out the array
  }
  for (my $i=0; $i <= 15 ; $i=$i+2) { # add ords to the array in pairs (as we do odd/even)
    push (@dData, $singularRaceNameOrd[$i]);
    push (@dData, $singularRaceNameOrd[$i+1]);
    push (@dData, $pluralRaceNameOrd[$i]);
    push (@dData, $pluralRaceNameOrd[$i+1]);
  }
  for (my $i = 0; $i < scalar @dData; $i=$i+2) {
    $checkSum1 = $checkSum1^int($dData[$i]);  # Force numification
  }
  # Checksum 2: Odd bytes
  for (my $i = 1; $i < scalar @dData; $i=$i+2) {
    $checkSum2 = $checkSum2^int($dData[$i]); # Force numification
  }
  print "Calculated Race Checksum: $checkSum1  \t$checkSum2\n";
  return $checkSum1, $checkSum2;
} 

[edited to clean it up a bit]

That makes it work by the original algorothm. It correctly calculated the checksum for the file you sent me.

An alternate way would be XORing the decryptedData before the name, and then just XORing the value after that with the ord values from the names, but this is (in theory) more programmatically consistent with the formula you decoded -- apply the formula to the (modified) data.



[Updated on: Mon, 17 May 2021 16:49]




https://www.irelandbybicycle.com
http://totalhost.sinister.net:999
https://github.com/ricks03/TotalHost

Report message to a moderator

Re: Detecting corrupt race files Mon, 17 May 2021 18:32 Go to previous messageGo to next message
ricks03 is currently offline ricks03

 
Officer Cadet 1st Year
Creator of TotalHost and Stars! utilities
Created TotalHost and Stars! utilities

Messages: 221
Registered: January 2012
Location: NC
Darned close, but still not quite right.

Edit: found it; I had taken a shortcut, and sometimes calculated the name length by the string, which is clearly not a good idea due to the encodeOneByte characters.

sub raceCheckSum {  # calculate a race checksum
  # The race checksum is calculated from the array of decrypted data of Block 6
  #  without the singular/plural race name data
  # The singular and plural race names are recalculated as ord arrays, 
  #  and each padded to 15 characters
  # Then the data arrray has added to it:
  # + 0 0    (to replace the race name size fields I suspect) 
  # + the 1st two ord from the singular name, and the 1st two ord from the plural
  # + the 2nd two ord from singular, and 2nd two ord from plural
  # ...
  # 
  # the 1st checksum byte is the XOR of the even data bytes
  # the 2nd checksum byte is the XOR of the odd data bytes
  
  my ($decryptedData, $singularRaceName, $pluralRaceName, $singularNameLength, $pluralNameLength) = @_;
  my @decryptedData = @{ $decryptedData };
  my ($checkSum1, $checkSum2);
  my $datalength = scalar @decryptedData - (1 + $singularNameLength + 1 + $pluralNameLength + 1);
  my @dData = @decryptedData[0..$datalength];
  # get the ascii values of the Race names - singular
  my @singularRaceNameOrd = unpack("C*", $singularRaceName); #array of ascii values
  unshift (@singularRaceNameOrd, 0); # add a starting 0
  for (my $i = scalar @singularRaceNameOrd; $i <= 15; $i++) {
   push (@singularRaceNameOrd, 0); # Pad out the array
  }
  my @pluralRaceNameOrd = unpack("C*", $pluralRaceName); #array of ascii values
  unshift (@pluralRaceNameOrd, 0); # add a starting 0
  for (my $i = scalar @pluralRaceNameOrd; $i <= 15; $i++) {
    push (@pluralRaceNameOrd, 0); # Pad out the array
  }
  for (my $i=0; $i <= 15 ; $i=$i+2) { # add ords to the array in pairs (as we do odd/even)
    push (@dData, $singularRaceNameOrd[$i]);
    push (@dData, $singularRaceNameOrd[$i+1]);
    push (@dData, $pluralRaceNameOrd[$i]);
    push (@dData, $pluralRaceNameOrd[$i+1]);
  }
  for (my $i = 0; $i < scalar @dData; $i=$i+2) {
    $checkSum1 = $checkSum1^int($dData[$i]);  # Force numification
  }
  # Checksum 2: Odd bytes
  for (my $i = 1; $i < scalar @dData; $i=$i+2) {
    $checkSum2 = $checkSum2^int($dData[$i]); # Force numification
  }
  return $checkSum1, $checkSum2;
}



[Updated on: Mon, 17 May 2021 19:08]




https://www.irelandbybicycle.com
http://totalhost.sinister.net:999
https://github.com/ricks03/TotalHost

Report message to a moderator

Re: Detecting corrupt race files Tue, 18 May 2021 00:41 Go to previous messageGo to next message
raptor is currently offline raptor

 
Chief Warrant Officer 1
Creator of StarsWine
Created StarsWine

Messages: 138
Registered: June 2014
I think it is solved now. Here is what I had to do:

1. Create a byte array of 192 bytes (96 16-bit words)
2. Copy in all of Block 6 up until the start of the singular race name (don't include it)
3. Decode the singular name and insert it into the array at position 128
4. Decode the plural name and insert it into the array at position 160
5. Run the XOR algo on the array

All my test files are checksummed properly. Replacing the checksum on corrupt files created with the above steps allows the race file to be opened in Stars! again. This is working for both 2.6jrc4 and 2.7jrc3.

I will commit code to the starsapi project shortly. Please verify.

Report message to a moderator

Re: Detecting corrupt race files Tue, 18 May 2021 01:22 Go to previous messageGo to next message
ricks03 is currently offline ricks03

 
Officer Cadet 1st Year
Creator of TotalHost and Stars! utilities
Created TotalHost and Stars! utilities

Messages: 221
Registered: January 2012
Location: NC
I thought is was solved 5 hours ago Smile
My above code works in every test case I have, and adds the data to the array differently I believe. Yours seems larger, as I process the name as the 15 characters ord'd . 192 bytes? What else are you putting in there for the 16 characters x 2?


StarsRace.pl in the Totalhost code is updated to that model, and working both to detect corrupt and non-corrupt race files, and fix the corrupt ones.

What you're saying for moving the data around makes more sense programmatically. So I suspect we're doing something different for the same results. But I'm looking at the bytes for singular/plural in pairs, not in series.

Ahh, I"ll bet you're adding the name after the split, where I'm doing it before.



[Updated on: Tue, 18 May 2021 04:14]




https://www.irelandbybicycle.com
http://totalhost.sinister.net:999
https://github.com/ricks03/TotalHost

Report message to a moderator

Re: Detecting corrupt race files Tue, 18 May 2021 10:07 Go to previous messageGo to next message
raptor is currently offline raptor

 
Chief Warrant Officer 1
Creator of StarsWine
Created StarsWine

Messages: 138
Registered: June 2014
Correctness and binary compatibility is my goal. The actual in-game data structure for a race file's Block 6 is 192 bytes. It is then checksummed first, then compressed/truncated and saved into the race file. When a race file is loaded, Block 6 is then expanded/overlaid into the same 192 byte structure, then verified against the checksum.

Report message to a moderator

Re: Detecting corrupt race files Tue, 18 May 2021 18:00 Go to previous messageGo to next message
raptor is currently offline raptor

 
Chief Warrant Officer 1
Creator of StarsWine
Created StarsWine

Messages: 138
Registered: June 2014
Example utility in the Java starsapi to automatically fix race files is now found here:

https://github.com/stars-4x/starsapi/blob/master/src/main/ja va/org/starsautohost/starsapi/tools/CorruptedRaceFileFixer.j ava

Report message to a moderator

Re: Detecting corrupt race files Mon, 28 June 2021 15:44 Go to previous message
ricks03 is currently offline ricks03

 
Officer Cadet 1st Year
Creator of TotalHost and Stars! utilities
Created TotalHost and Stars! utilities

Messages: 221
Registered: January 2012
Location: NC
In addition to StarsRace.pl I've gone ahead and created StarsRace.EXE so a local install of Perl isn't required.

It will output the race data, and detect/correct for a corrupted race file.



https://www.irelandbybicycle.com
http://totalhost.sinister.net:999
https://github.com/ricks03/TotalHost

Report message to a moderator

Previous Topic: Utility to reset Stars! passwords
Next Topic: Utility to fix corrupt race files
Goto Forum:
  


Current Time: Wed Feb 01 07:36:41 EST 2023