Home World Forum
Stars! AutoHost web forums

Jump to Stars! AutoHost


 
 
Home » Stars! Clones, Extensions, Modding » Stars! Extensions » xy modification after game has started
Re: xy modification after game has started Sun, 06 July 2014 01:55 Go to previous messageGo to next message
raptor is currently offline raptor

 
Chief Warrant Officer 1
Creator of StarsWine
Created StarsWine

Messages: 138
Registered: June 2014
You're right! I have correct the code.

I also started on a full API for all Block types. Now I really need to take a break...

Report message to a moderator

Re: xy modification after game has started Sun, 06 July 2014 03:38 Go to previous messageGo to next message
m.a@stars is currently offline m.a@stars

 
Commander

Messages: 2765
Registered: October 2004
Location: Third star to the left
Outstanding! Cheers

Do you think hardcoding every blocktype as its own class is the best way to go? Or just the easiest way to start? Sherlock



So many Stars, so few Missiles!

In space no one can hear you scheme! Deal

Report message to a moderator

Re: xy modification after game has started Sun, 06 July 2014 09:07 Go to previous messageGo to next message
raptor is currently offline raptor

 
Chief Warrant Officer 1
Creator of StarsWine
Created StarsWine

Messages: 138
Registered: June 2014
I'm open to other ideas and suggestions, but my current idea was that you could do something like the following (not real code):

1. blocklist = Decryptor.readFile("game.xy")
2. Modify a block in the blocklist
3. (eventually) Encryptor.writeFile(blocklist, "game.xy")

Or:

1. Create new types of Blocks as java objects (FileHeaderBlock, etc.), setting their properties through Java.
2. build up a block list
3. Encryptor.writeFile(blocklist, "game.xy")

The whole idea is to be able to edit the data using standard java objects, then reading/writing to a file using those objects. Internally, we could easily have other data types as members of the blocks themselves, then allow API access to them.

I don't know the best way to go about it.. I just started coding and this is what happened. Above all, I want to make the code semi-easy to follow as that will lower the barrier to entry for contributions. Smile

Report message to a moderator

Re: xy modification after game has started Sun, 06 July 2014 15:14 Go to previous messageGo to next message
m.a@stars is currently offline m.a@stars

 
Commander

Messages: 2765
Registered: October 2004
Location: Third star to the left
raptor wrote on Sun, 06 July 2014 15:07
The whole idea is to be able to edit the data using standard java objects, then reading/writing to a file using those objects. Internally, we could easily have other data types as members of the blocks themselves, then allow API access to them.

I don't know the best way to go about it.. I just started coding and this is what happened. Above all, I want to make the code semi-easy to follow as that will lower the barrier to entry for contributions. Smile

Sounds good. Hardcoding could do for starters, but I'm guessing some parameterization will be needed for better flexibility. At the very least, avoid having to hardcode so many things if there's a way to "softcode" the main details (type, size, sub-blocks, kinds of payload, etc) for each blocktype. Work at computer



So many Stars, so few Missiles!

In space no one can hear you scheme! Deal

Report message to a moderator

Re: xy modification after game has started Sun, 06 July 2014 16:53 Go to previous messageGo to next message
raptor is currently offline raptor

 
Chief Warrant Officer 1
Creator of StarsWine
Created StarsWine

Messages: 138
Registered: June 2014
Maybe I'm misunderstanding what you mean by 'hard-code'. Right now, I have hard-coded nothing except the deserialization code in things like FileHeaderBlock (decode() method). If this is what you refer to, then yes, I believe that we could do better - I was thinking that we could implement our own serialization class that could encode/decode data points in some sort of regular pattern, and apply the rules to each of the block types. Then it would just be a matter of laying out the class structure appropriately. I'm not that experienced in the area of serialization, but I'm a quick study (or at least a stubborn study).

I am curious, though, what coding abilities do you have (and could you help out)? If you've worked with some of the other developers, maybe you could answer some of my questions... like, why do I have to swap bytes so often?

Ideally, I'd love to get my hands on as much source code from any of the other projects as possible. Although, from what I gather, I'm the first to post source code? which seems crazy...

Report message to a moderator

Re: xy modification after game has started Sun, 06 July 2014 20:31 Go to previous messageGo to next message
m.a@stars is currently offline m.a@stars

 
Commander

Messages: 2765
Registered: October 2004
Location: Third star to the left
raptor wrote on Sun, 06 July 2014 22:53
Maybe I'm misunderstanding what you mean by 'hard-code'. Right now, I have hard-coded nothing except the deserialization code in things like FileHeaderBlock (decode() method). If this is what you refer to, then yes, I believe that we could do better - I was thinking that we could implement our own serialization class that could encode/decode data points in some sort of regular pattern, and apply the rules to each of the block types. Then it would just be a matter of laying out the class structure appropriately.

You're probably right. Must be my romantic side wanting to have everything handled by just one super-flexible magic class that didn't have anything hardcoded at all but instead relied on pre-filled data tables and whatnot. Hit over head


Quote:
I am curious, though, what coding abilities do you have (and could you help out)?

From Assembler and C to Java, including Basic, SQL and shell scripting, plus a knack for "black-box" testing, though I seem to be developing some kinda laziness lately which pushes me more towards the design side of things and away from the thrills of pure coding. Rolling Eyes Whip


Quote:
If you've worked with some of the other developers, maybe you could answer some of my questions... like, why do I have to swap bytes so often?

I'd bet there must be some integer math going on which handles bytes in pairs or even quartets, but the disassembly didn't quite piece together. I wanted to look into that this weekend but the hours somehow seemed to fly past in a blink. Wall Bash


Quote:
I'd love to get my hands on as much source code from any of the other projects as possible. Although, from what I gather, I'm the first to post source code? which seems crazy...

It is, but the general consensus is far from settled on the issue, since most "source code" would concern the (en)decryption routines, and too little has been done on the far more interesting fronts of bug-stomping, cheat-killing, MM-automation and scenario-creation. Sherlock



So many Stars, so few Missiles!

In space no one can hear you scheme! Deal

Report message to a moderator

Re: xy modification after game has started Sun, 06 July 2014 22:56 Go to previous messageGo to next message
raptor is currently offline raptor

 
Chief Warrant Officer 1
Creator of StarsWine
Created StarsWine

Messages: 138
Registered: June 2014
Quote:
...the general consensus is far from settled on the issue, since most "source code" would concern the (en)decryption routines, and too little has been done on the far more interesting fronts of bug-stomping, cheat-killing, MM-automation and scenario-creation. Sherlock


OK. I'm not really the type of person to get into debate much, but if the community has anxiety over cheating then maybe prioritization of bug/cheat-detection would be good.

As for the byte swapping - Maybe I'll write a decryptor in c++ and see if the bytes swapping issues are the same - it could be due to the endianness of the Java virtual machine, but I'm not sure. I did notice that the internal primes table had them swapped, so it could be how Windows 16-bit binaries were compiled back in the day. I'm just speculating, though.

Report message to a moderator

Re: xy modification after game has started Mon, 07 July 2014 11:25 Go to previous messageGo to next message
raptor is currently offline raptor

 
Chief Warrant Officer 1
Creator of StarsWine
Created StarsWine

Messages: 138
Registered: June 2014
I think I just figured out the byte swap problem (which now seems obvious): binary data was written on a 16bit little-endian platform (Windows 3.1). This means the byte stream should probably be read as such.

A side-effect of this is that any description of a stars block that has one-byte-or-smaller pieces may be incorrect. For instance, alignment of the FileHeaderBlock here:

http://wiki.starsautohost.org/wiki/FileHeaderBlock

probably has bytes 16 and 17 switched. This also clarifies instances where there is weirdly packed data, such as in places like the block header shown here:

http://wiki.starsautohost.org/wiki/Technical_Information#Gen eral_File_Structure

I'm thinking that the file byte stream should be read in swapped to begin with - but this may change paradigms a little. It will definitely reduce deserialization code.

Report message to a moderator

Re: xy modification after game has started Mon, 07 July 2014 12:48 Go to previous messageGo to next message
XAPBob is currently offline XAPBob

 
Lt. Commander

Messages: 957
Registered: August 2012
Deal with 16 little endiannin the import, then the reat is much easier.

Ok, a bit or a rewrite, but me=uch easiernomgoing.

Report message to a moderator

Re: xy modification after game has started Mon, 07 July 2014 14:29 Go to previous messageGo to next message
m.a@stars is currently offline m.a@stars

 
Commander

Messages: 2765
Registered: October 2004
Location: Third star to the left
raptor wrote on Mon, 07 July 2014 04:56
if the community has anxiety over cheating then maybe prioritization of bug/cheat-detection would be good.

That anxiety has kept toolmakers in the dark while cheaters had a clear (and hidden) advantage. Stars! hasn't evolved, and we've seen where stagnation leads when the player base shrinks. Wall Bash

Now the playing field is leveled and a new era begins! wOOt 2 Cheaters will also fear being automatically discovered/nerfed, which is an improvement. Yes


Quote:
I did notice that the internal primes table had them swapped, so it could be how Windows 16-bit binaries were compiled back in the day.

Good guess. That's how ints (two bytes) are commonly handled in the Wintel world. Whip


[Updated on: Mon, 07 July 2014 14:39]




So many Stars, so few Missiles!

In space no one can hear you scheme! Deal

Report message to a moderator

Re: xy modification after game has started Mon, 07 July 2014 14:37 Go to previous messageGo to next message
m.a@stars is currently offline m.a@stars

 
Commander

Messages: 2765
Registered: October 2004
Location: Third star to the left
raptor wrote on Mon, 07 July 2014 17:25
A side-effect of this is that any description of a stars block that has one-byte-or-smaller pieces may be incorrect. For instance, alignment of the FileHeaderBlock here:

http://wiki.starsautohost.org/wiki/FileHeaderBlock

probably has bytes 16 and 17 switched. This also clarifies instances where there is weirdly packed data, such as in places like the block header shown here:

http://wiki.starsautohost.org/wiki/Technical_Information#Gen eral_File_Structure

Who knows with the Jeffs and their binary shenanigans. Confused


Quote:
I'm thinking that the file byte stream should be read in swapped to begin with - but this may change paradigms a little. It will definitely reduce deserialization code.

"Swapped" for Java, perhaps, but "natural" for Wintel C/C++. Twisted Evil



So many Stars, so few Missiles!

In space no one can hear you scheme! Deal

Report message to a moderator

Re: xy modification after game has started Sat, 19 July 2014 02:18 Go to previous messageGo to next message
raptor is currently offline raptor

 
Chief Warrant Officer 1
Creator of StarsWine
Created StarsWine

Messages: 138
Registered: June 2014
I think I have settled on an optimal way of reading the byte stream. The data I'm after usually comes in 1, 2, or 4 byte chunks (before bit-twiddling to get higher packed data points). Here are the rules I follow:

1. If a single byte, read the byte
2. If two bytes, read them in swapped (little-endian)
3. If four bytes, read them in swapped

You can see how I implement it here:

https://github.com/raptor/stars/blob/master/python/decryptor .py#L253

So basically, instead of bonkers structures that are parsed directly from the byte stream, without swapping, that look like this:

XXXXXXXX YYYYYYXX NNYYYYYY NNNNNNNN

(This is a 4 byte planet chunk from the XY file)

X = X coordinate offset
Y = Y coordinate
N = Planet Name ID

The byte stream now looks like this:

NNNNNNNN NNYYYYYY YYYYYYXX XXXXXXXX

Everything lines up! This makes the decoding code much simpler and easier to follow.

On another note - I find I'm enjoying writing python,so I'm not sure I'll continue with the Java. Also, my time is very limited and porting code like this from decompiled C# is slow-going. Honestly, it's slightly demoralizing that there is source code out there that has most of my work done already, but it was never released (XyliGUN's tools in particular interest me, but they were not written in a language that's easily decompilable).

I encourage anyone who wants to add to my work by either forking the project in github or even sending me other code to merge. Smile

Report message to a moderator

Re: xy modification after game has started Sat, 19 July 2014 12:20 Go to previous messageGo to next message
m.a@stars is currently offline m.a@stars

 
Commander

Messages: 2765
Registered: October 2004
Location: Third star to the left
Nice! Cool

And yes, 32 bit ints are usually written in D C B A order on a Win16 platform. Teleport

I guess XyliGUN didn't use C# for his very promising StarsPlayerTool? Sherlock

Take heart, never before have all the interesting and sorely-needed tools that can revive Stars! been so close to finally being created! Cheers



So many Stars, so few Missiles!

In space no one can hear you scheme! Deal

Report message to a moderator

Re: xy modification after game has started Sat, 19 July 2014 15:20 Go to previous messageGo to next message
raptor is currently offline raptor

 
Chief Warrant Officer 1
Creator of StarsWine
Created StarsWine

Messages: 138
Registered: June 2014
Oh good - I'm glad you verified the 32bit integer structure for me... I couldn't seem to find info on it but it seemed to work out each time.

Sadly no, All of XyliGUN's tools were written in c/c++ or delphi/pascal (not sure).

Report message to a moderator

Re: xy modification after game has started Sat, 19 July 2014 19:57 Go to previous messageGo to next message
m.a@stars is currently offline m.a@stars

 
Commander

Messages: 2765
Registered: October 2004
Location: Third star to the left
Nao problemo. I got used to little-endianness back in the old days. Hit Computer

I gather no really good decompilers exist for C/C++, still one of the most used languages and one of the few that can translate virtually anything found in binary code? Confused



So many Stars, so few Missiles!

In space no one can hear you scheme! Deal

Report message to a moderator

Re: xy modification after game has started Sat, 19 July 2014 22:44 Go to previous messageGo to next message
raptor is currently offline raptor

 
Chief Warrant Officer 1
Creator of StarsWine
Created StarsWine

Messages: 138
Registered: June 2014
Not really... I mean, they exist, but they can never really generate code that is all that useful. I had tried two decompilers (RecStudio and boomerang) and they mostly generate unintelligible code. See here for the boomerang one (RecStudio was much worse):

http://www.megafileupload.com/en/file/551487/StarsPlayerTool -c.html

Languages that compile to an intermediary byte-code that then runs on a VM is perfect for decompilation (like Java or C#), but not ever c/c++/pascal/etc. that compile to machine code. Sad

Report message to a moderator

Re: xy modification after game has started Sun, 20 July 2014 02:42 Go to previous messageGo to next message
m.a@stars is currently offline m.a@stars

 
Commander

Messages: 2765
Registered: October 2004
Location: Third star to the left
raptor wrote on Sun, 20 July 2014 04:44
See here for the boomerang one (RecStudio was much worse)

Heh. Looks like so much pure assembler code. Lots of refactoring required! Sherlock

I used to hear good things about IDA years ago. If boomerang is the state of the art nowadays then indeed we have a problem. Shocked

Note: the exe says it's MS Visual C++. It also includes planet names, lots of messages and possibly also debug symbols, none of which appear in the decompilation. Hit Computer


[Updated on: Sun, 20 July 2014 18:14]




So many Stars, so few Missiles!

In space no one can hear you scheme! Deal

Report message to a moderator

Re: xy modification after game has started Mon, 21 July 2014 00:22 Go to previous messageGo to next message
raptor is currently offline raptor

 
Chief Warrant Officer 1
Creator of StarsWine
Created StarsWine

Messages: 138
Registered: June 2014
Which one are you looking at? I decompiled StarsPlayerTool.exe, but pulling out raw strings does indeed show names for Blocks 6,7,8,12,13,16,17,20,26,28,30. Most of this data is already in the decompiled sources for PaulCr's tools, of which I'd like to exhaust first before I attempt learning about disassembly.

Report message to a moderator

Re: xy modification after game has started Mon, 21 July 2014 12:11 Go to previous messageGo to next message
m.a@stars is currently offline m.a@stars

 
Commander

Messages: 2765
Registered: October 2004
Location: Third star to the left
raptor wrote on Mon, 21 July 2014 06:22
Which one are you looking at? I decompiled StarsPlayerTool.exe, but pulling out raw strings does indeed show names for Blocks 6,7,8,12,13,16,17,20,26,28,30.

2013.09.01. IDA-free finds all strings and many func names, and IDA6demo flags most RTL calls too. You can forget about Boomerang's fake C src. IDA's true ASM is much clearer. Pirate

Quote:
Most of this data is already in the decompiled sources for PaulCr's tools, of which I'd like to exhaust first before I attempt learning about disassembly.

By all means, milk PaulCr's code for all its worth. I can handle the (dis)assembly if needs be, though I reckon the exe itself will be more useful as benchmark than what can be guessed about structs and algorithms from the ASM. Work at computer

Are there any more tools that might harbor other useful information? Sherlock



So many Stars, so few Missiles!

In space no one can hear you scheme! Deal

Report message to a moderator

Re: xy modification after game has started Fri, 15 November 2019 12:27 Go to previous message
ricks03 is currently offline ricks03

 
Officer Cadet 1st Year
Creator of TotalHost and Stars! utilities
Created TotalHost and Stars! utilities

Messages: 221
Registered: January 2012
Location: NC
The NSA posted Ghidra, their open-course decompiling tool.

https://www.nsa.gov/resources/everyone/ghidra/




https://www.irelandbybicycle.com
http://totalhost.sinister.net:999
https://github.com/ricks03/TotalHost

Report message to a moderator

Previous Topic: NSA Helping to Decompile Stars!?
Next Topic: Utility to clean .m files
Goto Forum:
  


Current Time: Wed Feb 01 08:27:24 EST 2023