Home World Forum
Stars! AutoHost web forums

Jump to Stars! AutoHost


 
 
Home » Stars! Clones, Extensions, Modding » Stars! Extensions » "message pane" scraper
"message pane" scraper Fri, 26 July 2013 13:12 Go to next message
XyliGUN is currently offline XyliGUN

 
Ensign
Stars! V.I.P


Messages: 325
Registered: July 2004
Location: Russia, St.Petersburg

Several peoples asked me by mail is it possible to dump messages from m file.
Instead of answer to each of you let me just start a new topic here on forum.

First of all there are 3 "types" of messages in Stars:
1. server messages, which are part of the m file (like fleet X has been dismantled for XkT of minerals at the starbase orbiting Planet Y.)
2. client messages (like, you just found planet X) - these are not stored in m file, but generated by client based on data from m and h files.
3. in-game messages between players.

1 - can be extracted easily.
2 - needs to be generated based on comparision of data from m & h files
3 - can be extracted, but there are some minor issues related to non-english letters (like russian or german).

Second question is the form, in which you would like to have these messages?
Messages in Stars!.exe are stored like this "\\F has been dismantled for \\vkT of minerals at the starbase orbiting \\p."
m file just contain a message id and message args, which can be planet id, design id, fleet id or number of pop/minerals etc.
All of this can exported to csv, xml, txt, but which one would you like, depends of how are you going to use it.
Any hints on this?

Also there are several options regarding security and password:
1. it can be used by player itself in that case tool can be public, but it should verify password
2. it can be used by SAH only and in that case it will ignore password protection, since SAH cannot know players passwords

Comment, suggestions are welcome! Smile



"Progress isn't made by early risers. It's made by lazy men trying to find easier ways to do something."
Robert A. Heinlein, Time Enough For Love

Report message to a moderator

Re: "message pane" scraper Sat, 27 July 2013 05:09 Go to previous messageGo to next message
m.a@stars is currently offline m.a@stars

 
Commander

Messages: 2765
Registered: October 2004
Location: Third star to the left
1) Would be *very* useful! Teleport
3) Would be quite useful too, even if some letters were "missing" from the logs. Rolling Eyes

Make mine csv (tab-separated txt, just like Stars! own Reports). It should be easy to combine with the known list of exe messages. Sherlock



So many Stars, so few Missiles!

In space no one can hear you scheme! Deal

Report message to a moderator

Re: "message pane" scraper Sat, 27 July 2013 07:21 Go to previous messageGo to next message
XAPBob is currently offline XAPBob

 
Lt. Commander

Messages: 957
Registered: August 2012
Fleet ids and olanet ids are easy to extract from the f and map files - so could even be left as is

I'd go for line per message (1) with variables as $P[XX] and $F[XX] ( can you see where im going?

Report message to a moderator

Re: "message pane" scraper Tue, 30 July 2013 13:44 Go to previous messageGo to next message
XyliGUN is currently offline XyliGUN

 
Ensign
Stars! V.I.P


Messages: 325
Registered: July 2004
Location: Russia, St.Petersburg

Looks like you a going to do some sort of the automation?
Well, the list of the all possible server/client messages generated by the Stars! can be found here.
What we can try to do is to export csv with the following information: message ID and all of the args values.
Then you can go ahead and do what you want with that data. Will it work for you?



"Progress isn't made by early risers. It's made by lazy men trying to find easier ways to do something."
Robert A. Heinlein, Time Enough For Love

Report message to a moderator

Re: "message pane" scraper Tue, 30 July 2013 16:28 Go to previous messageGo to next message
Bystander is currently offline Bystander

 
Chief Warrant Officer 1
Duel club Champion 2007
Duel Club Champion 2007

Helped track down one or more Stars bugs

Messages: 141
Registered: June 2003
Location: Tampa, Florida, USA

Interesting about the variety of possible messages. Some seem to be from earlier versions of Stars!

"014 04 There was a massive blood bath at \n involving \i fleets of \i races. There were no survivors."

That would be a big change from current game. Would need simultaneous damage.


"\o attempted to transfer \V \m to \o. Only \v00 Colonists were successfully received. The rest perished in Deep Space!"

I vaguely remember a game when I was just starting out in Stars! I had accidentally carried over the unload orders when I added a waypoint in space between planet drops. Some overflow situation happened and it told me some strange number like 63 octillion colonists had perished in deep space. Nowadays, I think it just cancels the drop order.


"Your starbase at \p failed to build a new ship of type \i because someone lost the plans!"
Pretty sure Stars! just removes all the impacted ships from all planetary queues when a design is deleted.




Report message to a moderator

Re: "message pane" scraper Tue, 30 July 2013 18:05 Go to previous messageGo to next message
m.a@stars is currently offline m.a@stars

 
Commander

Messages: 2765
Registered: October 2004
Location: Third star to the left
XyliGUN wrote on Tue, 30 July 2013 19:44
Looks like you a going to do some sort of the automation?
Well, the list of the all possible server/client messages generated by the Stars! can be found here.
What we can try to do is to export csv with the following information: message ID and all of the args values.
Then you can go ahead and do what you want with that data. Will it work for you?

Sounds great! Very Happy



So many Stars, so few Missiles!

In space no one can hear you scheme! Deal

Report message to a moderator

Re: "message pane" scraper Tue, 30 July 2013 18:14 Go to previous messageGo to next message
XAPBob is currently offline XAPBob

 
Lt. Commander

Messages: 957
Registered: August 2012
XyliGUN wrote on Tue, 30 July 2013 18:44
Looks like you a going to do some sort of the automation?
Well, the list of the all possible server/client messages generated by the Stars! can be found here.
What we can try to do is to export csv with the following information: message ID and all of the args values.
Then you can go ahead and do what you want with that data. Will it work for you?

Yes, that would be truly lovely Smile
Obviously convenient for me to it at a local level butbif you feel that that risks the decryption code then importing it into the SAH codebase would work too (optional file attachment on email?)

Prefer it compiled as a linux app as well, but that reqlly might just be me...


[Updated on: Wed, 31 July 2013 02:14]

Report message to a moderator

Re: "message pane" scraper Wed, 31 July 2013 13:19 Go to previous messageGo to next message
XAPBob is currently offline XAPBob

 
Lt. Commander

Messages: 957
Registered: August 2012
Thinking on this a little more....
One of my issues with stars! is that even to do a simple -dpf it requires a graphical display.

The Jeffs were economical with bandwidth, but maybe not so much with compute resources (which are now emulated).

Would it be possible to have something that generated a CSV (with quoted strings where appropriate) version of the p and f files, along with a similar 'msg' file. These three could trivially be imported into a spreadsheet for manipulation.

Currently I have to kick off a remote X session in order to extract this data, which rather limits where I can do so...

Report message to a moderator

Re: "message pane" scraper Thu, 01 August 2013 13:29 Go to previous messageGo to next message
XyliGUN is currently offline XyliGUN

 
Ensign
Stars! V.I.P


Messages: 325
Registered: July 2004
Location: Russia, St.Petersburg

XAPBob wrote on Wed, 31 July 2013 21:19
Thinking on this a little more....
One of my issues with stars! is that even to do a simple -dpf it requires a graphical display.

The Jeffs were economical with bandwidth, but maybe not so much with compute resources (which are now emulated).

Would it be possible to have something that generated a CSV (with quoted strings where appropriate) version of the p and f files, along with a similar 'msg' file. These three could trivially be imported into a spreadsheet for manipulation.

Currently I have to kick off a remote X session in order to extract this data, which rather limits where I can do so...

I guess I can export values that are stored inside m file for you, but there is also some data (like planet value) which is not stored, but calculated.
And this one is not so simple, but having planet stat you can calculate planet value in something like Excel. Smile



"Progress isn't made by early risers. It's made by lazy men trying to find easier ways to do something."
Robert A. Heinlein, Time Enough For Love

Report message to a moderator

Re: "message pane" scraper Thu, 01 August 2013 13:35 Go to previous messageGo to next message
XyliGUN is currently offline XyliGUN

 
Ensign
Stars! V.I.P


Messages: 325
Registered: July 2004
Location: Russia, St.Petersburg

XAPBob wrote on Wed, 31 July 2013 02:14
Yes, that would be truly lovely Smile
Obviously convenient for me to it at a local level butbif you feel that that risks the decryption code then importing it into the SAH codebase would work too (optional file attachment on email?)

Does anyone mind to have... let's say StarsPlayerTool, which will verify password and then read m file to extract all the data XAPBob is asking for?



"Progress isn't made by early risers. It's made by lazy men trying to find easier ways to do something."
Robert A. Heinlein, Time Enough For Love

Report message to a moderator

Re: "message pane" scraper Thu, 01 August 2013 14:19 Go to previous messageGo to next message
m.a@stars is currently offline m.a@stars

 
Commander

Messages: 2765
Registered: October 2004
Location: Third star to the left
XyliGUN wrote on Thu, 01 August 2013 19:35
Does anyone mind to have... let's say StarsPlayerTool, which will verify password and then read m file to extract all the data XAPBob is asking for?

I want it! Very Happy

There's the small problem of what to do with all the info in the m files that players aren't supposed to know, like CA-learnt hab values and such. Whip

But that could be solved by inserting a "filter" tool between what Stars! gens and what SAH lets players download. UFO abduction



So many Stars, so few Missiles!

In space no one can hear you scheme! Deal

Report message to a moderator

Re: "message pane" scraper Thu, 01 August 2013 16:06 Go to previous messageGo to next message
XAPBob is currently offline XAPBob

 
Lt. Commander

Messages: 957
Registered: August 2012
The tool could simply not extract that 'hidden' information...
Or we could accept that CA can use the friend setting during a turn anyway, and dock them another 20 points Wink

Planet stats (rather than values) would actually be better, allowing allies/enemies to properly calculate habs for each other with low grav).


The hidden info I'd be concerned about is the MT sales pamphlet. For instance the planet stats are already in the extended planet dump...


[Updated on: Thu, 01 August 2013 16:07]

Report message to a moderator

Re: "message pane" scraper Thu, 01 August 2013 18:34 Go to previous messageGo to next message
XyliGUN is currently offline XyliGUN

 
Ensign
Stars! V.I.P


Messages: 325
Registered: July 2004
Location: Russia, St.Petersburg

m.a@stars
There's the small problem of what to do with all the info in the m files that players aren't supposed to know, like CA-learnt hab values and such. Whip
But that could be solved by inserting a "filter" tool between what Stars! gens and what SAH lets players download. UFO abduction

Regarding the issue with the CA having full opponents race block (which includes for example, all of the race file settings, all of the current tech levels, current research points, current research priorities, available MT toys and other stuff) in his M file...

Good news is that the problem is not that big as I thought before.

  • While in M files generated by Stars! 2.6J RC3 and 2.7J RC3 that data is present, and can be pretty easily extracted if you know decryption and the blocks structure.
  • But Stars! 2.6J RC4, which is used by SAH, does not have this problem at all! In case of M file generated by 2.6J RC4 that block is still present in CA m file, but all of the data except hub values are zeroed.


I cannot find a turn generated by 2.6J RC4 with the MT inside to check whether it is also zeroed or not.
Please, if you have one with the MT, send it to me for research.



"Progress isn't made by early risers. It's made by lazy men trying to find easier ways to do something."
Robert A. Heinlein, Time Enough For Love

Report message to a moderator

Re: "message pane" scraper Thu, 01 August 2013 18:59 Go to previous messageGo to next message
XyliGUN is currently offline XyliGUN

 
Ensign
Stars! V.I.P


Messages: 325
Registered: July 2004
Location: Russia, St.Petersburg

XAPBob
The hidden info I'd be concerned about is the MT sales pamphlet.

Indeed as far as I remember there is only trader info and the wormhole info can be extracted by a cheater.
Wormhole info includes: what are the target wormhole, who seen it, and who been through.
Trader info includes: carrying item, and who already met him.
But these ones can be zeroed by a simple tool at the SAH post processing step.
Ron, would you mind to use such a tool?
Cool



"Progress isn't made by early risers. It's made by lazy men trying to find easier ways to do something."
Robert A. Heinlein, Time Enough For Love

Report message to a moderator

Re: "message pane" scraper Thu, 01 August 2013 21:40 Go to previous messageGo to next message
m.a@stars is currently offline m.a@stars

 
Commander

Messages: 2765
Registered: October 2004
Location: Third star to the left
XyliGUN wrote on Fri, 02 August 2013 00:34
I cannot find a turn generated by 2.6J RC4 with the MT inside to check whether it is also zeroed or not.
Please, if you have one with the MT, send it to me for research.

I should have several, somewhere. Do you want them by email? Sherlock



So many Stars, so few Missiles!

In space no one can hear you scheme! Deal

Report message to a moderator

Re: "message pane" scraper Fri, 02 August 2013 01:59 Go to previous messageGo to next message
XAPBob is currently offline XAPBob

 
Lt. Commander

Messages: 957
Registered: August 2012
I can add you to dropnox with access to a game that has had a good number of MT (tech and toy) from SAH...

Report message to a moderator

Re: "message pane" scraper Fri, 02 August 2013 05:24 Go to previous messageGo to next message
XyliGUN is currently offline XyliGUN

 
Ensign
Stars! V.I.P


Messages: 325
Registered: July 2004
Location: Russia, St.Petersburg

It would be nice to have m files of 2 years: one is of year when trader met and one year before. The same with wormhole. In any form: e-mail or dropbox.


"Progress isn't made by early risers. It's made by lazy men trying to find easier ways to do something."
Robert A. Heinlein, Time Enough For Love

Report message to a moderator

Re: "message pane" scraper Tue, 06 August 2013 18:01 Go to previous messageGo to next message
XyliGUN is currently offline XyliGUN

 
Ensign
Stars! V.I.P


Messages: 325
Registered: July 2004
Location: Russia, St.Petersburg

XAPBob, please check this sample. Is this is what you are looking for? Smile


"Progress isn't made by early risers. It's made by lazy men trying to find easier ways to do something."
Robert A. Heinlein, Time Enough For Love

Report message to a moderator

Re: "message pane" scraper Tue, 06 August 2013 18:55 Go to previous messageGo to next message
m.a@stars is currently offline m.a@stars

 
Commander

Messages: 2765
Registered: October 2004
Location: Third star to the left
XyliGUN wrote on Wed, 07 August 2013 00:01
XAPBob, please check this sample. Is this is what you are looking for? Smile

Looks nice! Smile

But on closer review, I get some doubts:

* There's a couple "extra" bytes at start. Are they for formatting or versioning purposes?
* Including the full "message string" at the end of each line seems a bit overkill.
* I can't remember what the "HintId" column is for. Sherlock
* The "recent breakthrough" messages seem to have a different format for some params. Confused

Perhaps you'll like to test against much bigger (and complex) m-files? I should have a handful fat ones still around. Lurking



So many Stars, so few Missiles!

In space no one can hear you scheme! Deal

Report message to a moderator

Re: "message pane" scraper Wed, 07 August 2013 06:25 Go to previous messageGo to next message
XyliGUN is currently offline XyliGUN

 
Ensign
Stars! V.I.P


Messages: 325
Registered: July 2004
Location: Russia, St.Petersburg

m.a@stars
* There's a couple "extra" bytes at start. Are they for formatting or versioning purposes?

Seems like you mean UTF8 header bytes?

m.a@stars
* Including the full "message string" at the end of each line seems a bit overkill.

Sure. It's temporary here just for the debugging purposes and to make it easy to understand what does all these numbers means. But for an automation purposes EventId column should be used.

m.a@stars
* I can't remember what the "HintId" column is for. Sherlock

Currently I always make it 0, but in fact there is a two byte "shortcut" used by goto button and it can reference planet, fleet, technology browser, etc. I'm not sure if it have any usage for automation purposes, but in any case I have to sort it out before dumping into the file.

m.a@stars
* The "recent breakthrough" messages seem to have a different format for some params. Confused

Well, yes and no. Format is the same, but in fact several event params actually use two value args, this is why I currently put them in one Arg, separated by "|" symbol. I'm still thinking on how to correctly dump these ones.

m.a@stars
Perhaps you'll like to test against much bigger (and complex) m-files? I should have a handful fat ones still around. Lurking

Absolutely, just need to make sure it works on all of my games collection. Smile



"Progress isn't made by early risers. It's made by lazy men trying to find easier ways to do something."
Robert A. Heinlein, Time Enough For Love

Report message to a moderator

Re: "message pane" scraper Wed, 07 August 2013 06:31 Go to previous messageGo to next message
XAPBob is currently offline XAPBob

 
Lt. Commander

Messages: 957
Registered: August 2012
Looks good.
The EventId just translates to the Format (I think there is already a released copy of this translation)

The Args then may or may not need a .map, .p or .f to look at (and I presume they have to be exports from the raw .m file.
After that the information could fairly easily be combined.

Excellent Wink



The Args separated by pipes - all seem to refer to "benefits" (i.e. new technologies) - are they listed somewhere, or should we just ignore them Smile
Silimarly I presume the Research fields are 0-5 in the traditional NWPCLB order?



[Updated on: Wed, 07 August 2013 06:41]

Report message to a moderator

Re: "message pane" scraper Wed, 07 August 2013 10:05 Go to previous messageGo to next message
XyliGUN is currently offline XyliGUN

 
Ensign
Stars! V.I.P


Messages: 325
Registered: July 2004
Location: Russia, St.Petersburg

Here is short description on what will be stored for each of the event param in CSV file. Hope this will make things a bit more clear. Smile

\E - environment area value, stored in form Mesure|Value (measures: 0 - g, 1 - °C, 2 - mR, value... seems to be in clicks from the left???)*
\F - ship id, stored in format: DesignId|FleetId
\g or \G - currently exported as an integer value, but it wasn't found in any of the m files
\I - change type (0 = decreased, 1 = increased)
\j - zero based research area id (0 - energy, 1 - weapons, 2 - propulsion, 3 - construction, 4 - electronics, 5 - biotechnology)
\k - technology id, stored in format: GroupId|ItemId (group examples: armour, engines, orbital, terraforming, etc.)*
\l, \L or \Z - zero based player id
\M - mine field type (0 - standard, 1 = heavy, 2 - speed bump)
\n - location, can be in form of either planet id, fleet id or space coordinates
planet is stored in form P|PlanetId
fleet is stored in form F|OwnerId|FleetId
space coordinates is stored in form C|x|y
\O - zero based owner id (the same as player id)
\P - percent value
\S - packet origin (the same as player id)
\V - just an integer value, but comparing to \v it always used as is, since it is already multiplied by 100
\v - just an integer value (note: some events treat it as is, some add '00'/'kT'/'mg')
\X - strange, but \X is always replaced by empty string in any event message, will be stored as empty Arg in export
\z - design id, stored in format: IsStarbase|DesignId (IsStarsbase values: 0 = false, 1 = true)

* Notes:

  • Two lines highlighted in orange - this is where community input would be more than welcome.
  • \E - this value is always stored as integer number and in case of temperature it is "as is" equal to temperature value shown in Stars! But I don't know how to convert this integer value to floating point gravity value and to radiation value. I cannot find it now, but if I remember correctly I have seen somewhere on the forum, how does this value can be converted.
  • \k - I don't have an group/item ids mapping to the technologies/parts at the moment, so it would be nice if someone can collect it later on.



"Progress isn't made by early risers. It's made by lazy men trying to find easier ways to do something."
Robert A. Heinlein, Time Enough For Love

Report message to a moderator

Re: "message pane" scraper Wed, 07 August 2013 10:09 Go to previous messageGo to next message
XyliGUN is currently offline XyliGUN

 
Ensign
Stars! V.I.P


Messages: 325
Registered: July 2004
Location: Russia, St.Petersburg

XAPBob
The EventId just translates to the Format (I think there is already a released copy of this translation)

Yep, this mapping could be found here.



"Progress isn't made by early risers. It's made by lazy men trying to find easier ways to do something."
Robert A. Heinlein, Time Enough For Love

Report message to a moderator

Re: "message pane" scraper Wed, 07 August 2013 10:52 Go to previous messageGo to next message
m.a@stars is currently offline m.a@stars

 
Commander

Messages: 2765
Registered: October 2004
Location: Third star to the left
XyliGUN wrote on Wed, 07 August 2013 16:05
\k - I don't have an group/item ids mapping to the technologies/parts at the moment, so it would be nice if someone can collect it later on.

Perhaps these numbers relate to the corresponding columns in the "allitems table" found inside the exe (or here) that I used as the basis for my (slightly more readable) table dumper? Sherlock



So many Stars, so few Missiles!

In space no one can hear you scheme! Deal

Report message to a moderator

Re: "message pane" scraper Wed, 07 August 2013 11:23 Go to previous messageGo to previous message
XyliGUN is currently offline XyliGUN

 
Ensign
Stars! V.I.P


Messages: 325
Registered: July 2004
Location: Russia, St.Petersburg

m.a@stars
XyliGUN
\k - I don't have an group/item ids mapping to the technologies/parts at the moment, so it would be nice if someone can collect it later on.

Perhaps these numbers relate to the corresponding columns in the "allitems table" found inside the exe (or here) that I used as the basis for my (slightly more readable) table dumper? Sherlock

Hmmm... partially yes, in this MOD file first number could be group id, and the second one is an item id.
Item id seems be exactly the same, except in MOD file it's 1-based, and in event it is 0-based.
As of group id, it seems to be always a power of 2 and it not corresponds to first number as well as power itself is not also corresponds to the first number.
Here is a few examples:
11,6,"Bear Neutrino Barrier" in MOD corresponds to 4|5 in m file. 5 is zero based, so we have to increment it by 1 and 5 + 1 = 6. But group id is not matched.
5,15,"Temp Terraform ±11" in MOD corresponds to 8192|14. 14 is again a zero based, so 14 + 1 = 15 - matched. But group id is not matched.
6,12,"Laser Battery" in MOD corresponds to 32768|11. 11 + 1 = 12 - matched, but group id is not.



"Progress isn't made by early risers. It's made by lazy men trying to find easier ways to do something."
Robert A. Heinlein, Time Enough For Love

Report message to a moderator

Previous Topic: My wheels are turning
Next Topic: Should Stars! files structures be publically available?
Goto Forum:
  


Current Time: Thu Mar 28 17:24:52 EDT 2024