To make sense of the first dozen or so replies, this was original a thread asking if it was okay with the developers and/or publishers. The answer was:

(15-09-2011 09:31 PM)Timo Wrote:  As stated numerous times already, we don't have a problem with modding per se. In fact, we encourage it and I look forward what people come up with. Just be reasonable about it and don't make chunks of game code available publicly.

So they don't mind it; we just can't upload slabs of reversed code. Works for me. Before we begin, please read the following disclaimer and a couple of notes below it.

Disclaimer I am not a programmer, I am not in IT, I am not working in a field which is directly related to IT. I do not work for, with, against, between, or otherwise attached to Kalypso or Haemimont. I am doing this for the love of the game. The information here may be wholly or partially inaccurate. It works for me, but your experiences may differ.
The header reads: 1B 4C 75 61 51 00 01 04 04 04 01

Sources: A No Frills Guide To Lua 5.1 VM

3RD PARTY TOOLS

• HPK Extractor;
• A hex editor (whatever works for you);
• A Lua 5.1 binary for running 'ChunkSpy' (not luac.exe, lua.exe) & ChunkSpy (use 5.1);
• (Optional) a lua compiler decked out the right way (slserpent's is here) -- note that you do not need to recompile things to get the game to run them!;
• A compiled version of LuaDec 3.2 (get changes to source (luaconf.h) and the binary here).
  

LuaDec is actually bordering on useless. For those wondering, yes I have written my own decompiler since October last year, and no, it isn't public for various reasons.

LuaDec is what SimCutie is using, and what others are using. Be warned it dumps some pretty broken code and you're better off learning how to use ChunkSpy.

I don't see why the dev's would have a problem with players modding for the game, Lets face it if it wasn't for mods the civilization game wouldn't be on there 5th release, modders can add a lot to games and the gaming community extending the life and fun of the game.



1st request.. fix the roads... lol

There's nothing wrong with the roads ... at all. They're also a bit outside of what I'm looking at at the moment. I've got the housing information down pat, but I need to look at a few more things (like why my "can this be upgraded" stuff is broken).

I also just don't want to tread on the toes of devs by releasing anything, or releasing it in the wrong format.

Exactly, take a look at Age Of Pirates 2 City Of Abandened Ships. We have made the game far more stable and added so much to the game that its practically a new game. So modding a game is or should be seen by game develipers as a extention to what they have done, in less you get something like Rome Total War where you have lots of different mods all offering roughly the samething but slightly different to each other, then i would say no for T4 its a bad idea because people should either release mini mods and then the modding team include them in a combined mod pack (if they offer anything of value to the game) This would make the gameplay far better and give the fans of T4 a more injoyable game (not that it isn't injoyable already).

Can you share the savegame for the 7200 population island? I'd love to see how it runs!

Just so everyone knows, I was asking if the devs would have a problem with my releasing anything (i.e. don't want to tread on the DLC).

Also, I wanted to know what the appropriate format is. I'm yet to see if there's any sort of mod loading support (though I suspect I found something that might do), so I was wondering if they had an opinion on this -- I'm not going to release "Game.hpk", maybe individual files, but that's all.

(15-09-2011 01:55 PM)assen Wrote:  Can you share the savegame for the 7200 population island? I'd love to see how it runs!

It crashes as you load it. I hit save mid-immigration, and it really doesn't like that. Also, wrong thread

As stated numerous times already, we don't have a problem with modding per se. In fact, we encourage it and I look forward what people come up with. Just be reasonable about it and don't make chunks of game code available publicly.

Thanks Timo, just wanted to be sure I've got a number of mods for a number of other games stashed on disks here because when I tried to distribute them some of the devs gave a less than warm response.

I'll stay away from the hidden buildings, but I might have a roll at playing with disasters and the AI for people and roads if I can spare a fair bit of time.

The only problem I've got now is working out how to distribute it when I've got it working (at the moment the tiered upgrades work in the building, counting every pre-requisite, but when you go to destroy my custom Apartment class it freaks out -- not to mention it's asserting that I need 3d models). The good news is that I've had to reimplement almost everything, so chances are it'll just be one giant mind-boggling file.

(15-09-2011 09:31 PM)Timo Wrote:  As stated numerous times already, we don't have a problem with modding per se. In fact, we encourage it and I look forward what people come up with. Just be reasonable about it and don't make chunks of game code available publicly.

Awesome!!
Would you consider to 'help' the modding a bit for a next release by 'exporting' some of the game rules to user-accessible code, like Civ does with Python/ XML? Or publishing (/leaking/ putting online accidentally) some guides on where the interesting bits are?

Just wondering cause i have lots of concrete ideas but no skillz to decrypt files and hexedit assembly code.

(16-09-2011 02:14 PM)Martijn Wrote:  Would you consider to 'help' the modding a bit for a next release by 'exporting' some of the game rules to user-accessible code, like Civ does with Python/ XML? Or publishing (/leaking/ putting online accidentally) some guides on where the interesting bits are?

Just wondering cause i have lots of concrete ideas but no skillz to decrypt files and hexedit assembly code.

What are you interested in, specifically? While I'm not yet ready to share all of my magical knowledge, I have had some fun.

(16-09-2011 02:19 PM)Swixel Wrote:  What are you interested in, specifically? While I'm not yet ready to share all of my magical knowledge, I have had some fun.

I want an F3 button too!

I think my other post here http://forum.kalypsomedia.com/showthread.php?tid=11132 gives a small hint on the direction, im not after adding one individual buildings, but more after integrating the economic model to enable El Prez to be much more powerful and get choice between the current dark-red communist government (all houses state-owned? all wages payed by government? then why is the Capitalist faction happy, why do they exist at all?) and total extreme freedom (including the freedom to die of hunger and not afford healthcare). You're El Presidente, you decide where the balance is

I would like an economic simulator that shows (potentially) housing bubbles, banana republics, hyperinflation, unbearable public and government debt spirals, stockmarket crashes and the effect of that all on individual tropicans. Or the effect of just a small increase in tax on higher-incomes.

Ofcourse most of that 'engine' should be hid under the bonnet, and relevant to players only if they want to use it. Ie., you have no risk of housing bubbles in current Tropican economics, so switching on the "housing privatization" edict can have dire consequences if you mismanage. The choice is yours!

Yeah ... unfortunately exposing various parts of the engine hurts my brain. What you want to do I could probably do some of (if not all of), if I had the time and brain power spare. The solution to your query isn't so much rewriting, but instead attaching to existing monthly code queries (e.g. hook onto rent).

The biggest problem you'll face is spending -- unless you tax their income on the way in (i.e. report a false payment value in the UI), or make sure they don't feel they're getting that (i.e. lie to them), then tax will break their wealth stacks.

You could fix it, but you'd need to rewrite everything that's tied to spending.

(16-09-2011 02:46 PM)Swixel Wrote:  Yeah ... unfortunately exposing various parts of the engine hurts my brain. What you want to do I could probably do some of (if not all of), if I had the time and brain power spare. The solution to your query isn't so much rewriting, but instead attaching to existing monthly code queries (e.g. hook onto rent).

The biggest problem you'll face is spending -- unless you tax their income on the way in (i.e. report a false payment value in the UI), or make sure they don't feel they're getting that (i.e. lie to them), then tax will break their wealth stacks.

You could fix it, but you'd need to rewrite everything that's tied to spending.

Yes, Tropicans need to learn to spend their money (and accept taxes as a fact of life).
Therefore i see it as a 3 step progress in functionality:
1. show more *information* on tropican family wealth, income and spending (currently there is very little information, but by observation i have an idea on balances). These calculations would run monthly after wages have been payed.
The Almanac should show a full report as well, both on Revenue and a Balance sheet, with year-on-year history and more (bigger!) graphs. If you've ever played RailRoad Tycoon 3, you know what a financial statement looks like!
2. make them spend around 80% of their income, currently they only spend on rent (30%) and entertainment (8%), add to that taxes (say 20%), food (say 10%), healthcare (per visit or flat fee/insurance) (say 10%). The rest is savings (if any) for leaner times.
3. give Tropicans more options, ie buying their house and getting a mortgage.
4. add macro economic options wrt public sector/ private sector/ finance sector

Ive made some calculations that show this is possible within the current "behaviour" of tropicans wrt rest, healthcare, food needs/food price etc, too much to type here (noone would read it). But getting the information out to see would be a great help already.

Building ownership isn't something I can see as being easy. At the moment you can flag them as private or public, though with enough time we might find ways around that.

Splitting income requires rewriting vast amounts of code, as the behaviour of the individual (regarding spending) isn't unified as far as I can tell. I'm not sure where you get 8% for their entertainment costs either. Each of the divisions is in a different place -- the only locked down to a single file is housing, where rent is caculated in an interesting way.

Public/Private could be done, finance ... I have no idea. You could already implement a stockmarket on top of this (not a goods market, a "privatised stock" market). Assigning ownership is doable, but it hurts my brain at the moment.

Showing the information may be possible, making the assumption that they actually have pools of wealth (which they don't ).

Fundamentally speaking, this won't work. Pay isn't doled out, their spending habits are split on a theoretical pool of wealth, functioning as fractions. Pooling the wealth means another integer store (4 bytes) per person on the island, and code to handle it...

Anyway, I don't mean to come off as mean or unwelcoming of ideas. At the moment I'm just looking into a few custom housing options based around the apartment. I haven't got my build menu working yet, but I do have access to the hotkey binding menu and hotkeys (confirmed working ).

The engine restricts what we can achieve in many ways, but at the same time it's probably one of the most moddable things on which I've ever tinkered. The downside is processing power required by lua, the upside is nearly infinite integration and modification. I have a couple of "real world" style ideas, but I need to think them through a bit first.

What i did was observe a *stable* island population of around 250 people. On average on my islands (i reviewed a few) you see something like (per 250 Tropicans
- 50 outside of workforce
- 50 on food (not making any money)
- 50 on infrastructure (teamsters/ construction, not making any money)
- 50 on government jobs/ services (priests, military, bureaucrats etc, not making any money)
- 50 actually producing stuff and making money (factory workers or maids or entertainers, depending on island type).
(So only 20% is producing the Tropico GDP! therefore more cost-assignment is necessary through food and payment of services).

From this observation you can tell how many % Doctors you need to prevent Tropicans missing their services (dont open up all work spots on a hospital if your population is only 100).
We know many meals per year they eat. We know how many food is subtracted from the marketplace in exchange for a certain amount of meals, we know what corn costs per 100 items, therefore we know the cost of a meal

So what i mean to say is, that in Coding, the Tropicans pay their food in $ per meal, they pay their tax the same way they pay their rent (per month), they pay their entertainment and healthcare per visit. But on the whole, on averaging all tropicans over a few years, you will see around those percentages, assuming "normal" levels of cost and wage.

Anyway i think i've written enough on how i'd like to see enhancements/ mods to the economic model, its time to get down to the nitty gritty of how to get some of that into the game! If at all possible

--edit 1--
the 8% for entertainment came from the fact they spend max one month pay on entertainment, and they visit entertainment once a year, so 1/12th of their income is spend on entertainment.
--edit 2--
Swixels, you dont come off as "mean or unwelcoming of ideas" at all, i mean, the fact you're replying constructively is most welcome

ok i was taking a look at modding a little bit. I was able to change some values of buildings for example make the cathedral have more workers and more visitor spots, bigger area of effect etc. pp.

Do u think its possible to add new buildings somehow and give them own products? or just add another crop you could harvest. like marihuana or something
then you would need to add a new product to the game too so you can even export it.
im just curious what you could possibly do

(16-09-2011 03:56 PM)Martijn Wrote:  What i did was observe a *stable* island population of around 250 people. On average on my islands (i reviewed a few) you see something like (per 250 Tropicans
- 50 outside of workforce
- 50 on food (not making any money)
- 50 on infrastructure (teamsters/ construction, not making any money)
- 50 on government jobs/ services (priests, military, bureaucrats etc, not making any money)
- 50 actually producing stuff and making money (factory workers or maids or entertainers, depending on island type).
(So only 20% is producing the Tropico GDP! therefore more cost-assignment is necessary through food and payment of services).

From this observation you can tell how many ...

It is facinating to read your commentaries - really.

You are exercising some interesting statistical views of a population. Do you intend to manipulate it or just take what the existing game generates? The existing game (IMHO) does not create a population to which you can apply some assumptions which seem implied in your questions.

Example: your observation notes do not mention the gender distribution and age distribution which are essential to the Tropico franchise ambiance. (edit) Oops! I forgot to mention the class=education structure of three levels.

I do not wish to limit your interest in modding, but it seems rather futile to code-up 'family wealth' when the game family really doesn't represent much of anything.

FWIW

(16-09-2011 03:56 PM)Martijn Wrote:  --edit 1--
the 8% for entertainment came from the fact they spend max one month pay on entertainment, and they visit entertainment once a year, so 1/12th of their income is spend on entertainment.
--edit 2--
Swixels, you dont come off as "mean or unwelcoming of ideas" at all, i mean, the fact you're replying constructively is most welcome

@1: Not really. Time != willingness to spend money.

@2: Good to know; late at night I tend to writer colder/less feeling replies.

(16-09-2011 04:15 PM)realspike Wrote:  Do u think its possible to add new buildings somehow and give them own products? or just add another crop you could harvest. like marihuana or something
then you would need to add a new product to the game too so you can even export it.
im just curious what you could possibly do

Yep. At the moment I'm very iffy on handing that out particular knowledge (due to the DLC buildings). I'm already working on a new building, but from what I can understand we cannot (at this stage) import models into the game without moving around their entire (or a large amont of their) model collection. (I have an idea as to how to get around this, but it might open up some bad things were I to achieve.)

As for resources, I had a quick spin the moment I found what I needed. I added "Swixel's Magical Fruitjuice" to the almanac. It threw about a billion errors into my logs. I did make some modifications earlier to change output types, etc., but nothing major yet.

The Almanac should be accessible, but it's low priority at the moment, as the file in which that data is stored is really annoying to read (it's long).

I vote that they put "Swixel's Magical Fruitjuice" as a product for the Pub, with inputs of Papayas.

J

Hello there, I've started to sift through the lua files myself, and being a programmer (although in uni) I might be able to sift through the files fairly quickly.

My main limiter is time, I'm about to start my quarter and I expect to be very busy. I also would like to say to everyone that modifications to existing parts of the program are a lot easier than adding new features and adding new models generally is quite hard.

The first thing I'm going to look into is seeing if I can't get tropicans to walk more often. I had some that would drive 10x the distance it would take to walk.

edit: oh jeez, lua is going to take a little getting used to. I've worked in functional languages before, but this is something new for me.

edit2: if I try to reverse this by hand I'm going to go crazy, alternatively chunkspy complains about a bad header when I just try to feed in the lua files, am I doing anything in particular wrong?

(18-09-2011 08:59 PM)Raptoer Wrote:  Hello there, I've started to sift through the lua files myself, and being a programmer (although in uni) I might be able to sift through the files fairly quickly.

I'm not a programmer; but Lua is like structured English, so while it'll help you, it shouldn't stop others. ( I'm saying this so others don't get put off with the idea that it's only possible for programmer .)

(18-09-2011 08:59 PM)Raptoer Wrote:  My main limiter is time, I'm about to start my quarter and I expect to be very busy. I also would like to say to everyone that modifications to existing parts of the program are a lot easier than adding new features and adding new models generally is quite hard.

Adding new features is pretty trivial at this point. Once you find the name (e.g. Apartment) and how it's defined (DefineClass), the only thing left is to work out how it's called -- i.e. when it's called.

Adding new models is currently impossible unless we can reverse the granny2binary format (and we know that exists in part because it says so in the logs!). This appears to be native to Haemimont/Kalypso in some way, so we can't look outside for help.

Once extracted, they're fairly clean splits of meshes, animations, etc., and the textures are elsewhere. We can probably install our own pretty easily if we can at least pack them, at I've already found a point where external injection is possible, beyond the core three files.

(18-09-2011 08:59 PM)Raptoer Wrote:  The first thing I'm going to look into is seeing if I can't get tropicans to walk more often. I had some that would drive 10x the distance it would take to walk.

That one will be surprisingly annoying if you don't just overwrite the file in question.

(18-09-2011 08:59 PM)Raptoer Wrote:  edit: oh jeez, lua is going to take a little getting used to. I've worked in functional languages before, but this is something new for me.

Yeah ... I've worked on reversing Erlang before for giggles, and that blew my mind. Then the language syntax blew my mind. Though the workflow of functional languages is insanely easy because it's logical and the manner in which they work is like maths

some people naturally get programming, others have a harder time, but eventually get it. I've seen studies where they give people with no programming experience a test involving a fake programming language and there tended to be a 50/50 split in people who picked it up second nature and people that needed a lot more time to understand the concepts.

Ignore what I said before about chunkspy not working, I just had to add --auto to get it to spit out the lua. I'm gonna write a script that takes every file and runs it through chunkspy and outputs the results into a file.

Quote:My current thought is that taking the bytecode from ChunkSpy and then recompiling it may be more reliable than LuaDec's behaviour, but that is an inordinate amount of work. At the moment I'm just reversing and rewriting.

I can try to write a script that takes the chunkspy output binary and recompiles it for every file. My scripting is a little rusty, but it shouldn't be too hard.

Well that'd be a decompiler, really. The real issue is that that won't help too much -- the data will come out in a mangled format unless you work on reconstruction techniques.

If you want properties you can just make a simple mod (i.e. init.lua inside an HPK) with one line: print(Apartment)

It won't give you functions, but it'll give you the properties in the file. You can print any table the same way, but it's hit and miss.

I was interpreting your statement about the bytecode from Chunkspy as being about using chunkSpy's rewrite function to make new bytecode, perhaps it would be more stable.

I finished writing a java script (tiny program, just 50 lines) that just runs chunkSpy on every file in every directory recursively.

I'm providing it here -
http://www.mediafire.com/?7kcob42filu939w

Note that you use this at your own risk. I've made some effort to make sure you can't mess anything up too badly, but seeing as it recursively enters directories make CERTAIN that you're running it in the right directory or else it might try to go through your entire hard drive.

I've run it on my Game folder, it took about 30 seconds to process everything and produced a folder of ~111MB (original and chunkSpy files). I just added a 'O' to the end of the file names, for 'output'. This will create a file with the extension .luaO for every lua file the program finds. I didn't put in a special case for the chunkSpy.lua, so it will also produce a chunkSpy.luaO.

Just copy the .jar and the .lua (just chunkSpy) into your Game directory (or a sub directory) and open your command prompt to that directory. Then run the jar with the command "TropicoChunkSpyLua.jar <lua5.1.exe>". Replace <lua5.1.exe> with the path of your lua exe. It by default will try to run "lua5.1.exe" which will work if you've put the lua exe into your path, otherwise the program will die on the first lua file complaining that it can't find your lua exe.

Even if nobody else uses this I still liked making it, and it will save me time running the files through chunkSpy. I actually started off doing things in this terrible way by taking the output of chunkSpy and writing it to a file myself, instead of just telling chunkSpy to write it out. As I said before this little program is supplied as is, with no guarantee of anything. I also included the source in case you want to make sure that it isn't a malicious file.

As for tropico, I wonder if I could make the cars ignore each other so as to prevent traffic jams. All the cars would just clip through each other. You would still be limited by how many people each garage can handle, but there would be no massive traffic jams.

edit: haha! it works! I changed the collision radius from 2000 to 5 and now the tropicans drive like madmen! Although their movement is not consistent... Cars will speed along and then come to a very abrupt stop behind the other car, sometimes clipping in, sometimes just sitting behind. Now intersections are much better, tropicans don't wait nearly as long when they go through intersections and sometimes multiple cars will navigate the intersection at the same time (though not conflicting).

(19-09-2011 03:34 AM)Raptoer Wrote:  I was interpreting your statement about the bytecode from Chunkspy as being about using chunkSpy's rewrite function to make new bytecode, perhaps it would be more stable.

I used this ...


Though I edited ChunkSpy.lua so that it matches the right stuff... add --auto if you haven't.

Anyway, to run the Python code for those who are interested:

~5 seconds later it's done.

(19-09-2011 03:34 AM)Raptoer Wrote:  As for tropico, I wonder if I could make the cars ignore each other so as to prevent traffic jams. All the cars would just clip through each other. You would still be limited by how many people each garage can handle, but there would be no massive traffic jams.

If you poke around Games\Classes\Units\Cars.lua ... you find a lot of fun stuff. What you want isn't easy by any means, but I can think of a few ways it may be possible. Your real issue will be that I haven't hammered down loading sequence issues yet (i.e. pre/post DefineClass issues). If you look at the basic modding tutorial thread you'll see variables are split for what appears to be no reason, but it fails miserably if you move them.

(18-09-2011 08:59 PM)Raptoer Wrote:  ... The first thing I'm going to look into is seeing if I can't get tropicans to walk more often. I had some that would drive 10x the distance it would take to walk. ...

Sorry, I can't resist. -- Take away the ability to transmogrify into a car from some of the avatars\occupations\units (whatever they are called). For example, why should uneducated Farmers, Maids, Barmaids and such like, choke the roads?

(19-09-2011 01:54 PM)CoconutKid Wrote:  Sorry, I can't resist. -- Take away the ability to transmogrify into a car from some of the avatars\occupations\units (whatever they are called). For example, why should uneducated Farmers, Maids, Barmaids and such like, choke the roads?

Actually, I'm testing that on a per-profession basis at the moment. I'm thinking it'd be a nice addition ... but people starve walking places because I'm testing on old game which really wasn't designed for it

(19-09-2011 01:57 PM)Swixel Wrote:  ... but people starve walking places because I'm testing on old game which really wasn't designed for it ...

Makes Marketplaces more interesting, eh?

BTW, how long does it take a unit to starve? On the T3 board, hitter described the bag of food comprising more than one meal that units picked-up and carried with them. Only husband & wife shared by teletransport -- no food was stored at residences.

Has T4 introduced a new food system that starves people quicker even though all the food production buildings hold 200 units (each comprising several meals) in reserve to feed the people?

#2 BTW: What player who is building a population to shoot-up past the stated game limit is concerned about the random person who dies from starvation?

Hrm? I think the issue was that I messed up the population limit so my marketplaces don't carry enough food, so people appear to be walking about 10x as far as they should to try to find food.

And I'm really not from a game mechanic point of view, it just feels inhumane to watch 400 people die every year.

Anyway, given the 20,000 object limit the engine has (and for good reason!), I need to find a way around that, probably by making "SUPERMARKETPLACES" or something, which shoot fireworks, travel through time (forwards only), and are equipped with more than 1,000 food (or maybe hack pubs to feed people).

EX: Is there any way to a garage to make money?