Threaded Mode | Linear Mode

FordGT90Concept · 19-04-2012 01:46 PM

This was posted previously in the Tropico 3 forum but it has been updated for Tropico 4 and Modern Times so I'll stick a link in this forum as well:
http://www.techpowerup.com/forums/showth...p?t=113705

Basically it packs and unpacks HPK files.

Let me know, here or there, if there's an HPK file it can't handle. Game.hpk had two buggy "files" (Game\CommonEntities.txt and HGO\SRP\srp-init.bat) in it that had to be hotfixed. There may be others.

Edit: The only difference between checking the Tropico 3 and Tropico 4 radio is that Tropico 4 takes the filestamps and multiplies by 2000. That is to say, it makes the time stamps on extracted files correct.

Swixel · 19-04-2012 09:02 PM

Very nice, Ford Smile

While I've not said it directly, I've greatly enjoyed ab/using this for repacking. Good news for people not willing to do bad things to their windows install, Expansion.hpk extracts Smile

The only problems I've had with your tool is the output of LZSS and LZIS, but I managed to raw dump them a while ago using some wonderfully hacky C. (There's no real joy to be had in these anyway.)

simcutie · 20-04-2012 12:56 AM

But extracted lua files are not normal compiled lua binary. very similar to normal .luac but bit different. So normal lua interpreter (v5.1.4) (VM) can not execute them. ( bad header in precompiled chunk ) How to cope with it? I even ported luadec and lua library from lua 5.1 source code to Win32. But it also refuse to load the custom lua compiled format. What is exact difference? I will modify interpreter and luadec. The ...\Tropico 4\Hedgehog\lua51.dll is not useful for linking.

Swixel · 20-04-2012 01:02 AM

T3 and T4 both use 4-bit ints instead of doubles (8 bit floating point ints). I posted that a while ago in http://forum.kalypsomedia.com/showthread.php?tid=11145 and I think Snake posted it in the T3 forums somewhere too.

LuaDec will hit register issues on everything even remotely complex.

EDIT: Ford, I know it's not normal ZLIB, and I couldn't get ZLIB to play nice for me, so is it some sort of chunked decompression (based on the XenTaX discussion for an older Haemimont Game)?

I've just been using detours and the game's Lua VM to extract the data ... I used this primarily to repack. My own extractor (based on the reverse ASM) failed on anything ZLIB related.

FordGT90Concept · 20-04-2012 03:57 AM

LZSS and LZIS? I think I remember seeing them a long time ago but don't remember much more than that.

It uses a custom ZLIB header:
string IdentifyingChars
uint InflatedLength
int ChunkSize (always 32,767)
int[] OffsetCollection

I used ICSharpCode.SharpZipLib.Zip.Compression.Inflate to decompress each chunk. I think I tried System.IO.Compression and it failed so I had to use zlib code.

If you're coding with a .NET compatible language, you could import HPK Archiever.exe as a resource, create your own instance of HpkFile and you have access to all the read/write methods inside. I made everything useful public.

I don't use disassemblers or anything of that sort. I just look at the binary (Hex Workshop) and try to figure out what it means. XD

simcutie · 20-04-2012 04:12 AM

(20-04-2012 01:02 AM)Swixel Wrote: T3 and T4 both use 4-bit ints instead of doubles (8 bit floating point ints). I posted that a while ago in http://forum.kalypsomedia.com/showthread.php?tid=11145 and I think Snake posted it in the T3 forums somewhere too.

LuaDec will hit register issues on everything even remotely complex.

Thanks for your info. I got my LuaDec now working correctly with above info. The de-compiled output seems to be quite mangled with local variable names but it is enormously easier to read than dissembler output. Smile

Swixel · 20-04-2012 04:28 AM

@Ford:
Thanks Smile

LZIS is used for things like save files, I forget where/why LZSS is used, but objdump puts the three signatures together.

LZSS is just a dictionary based compression algorithm I hate to play with, and without remembering where I saw it, I can't do too much with it.

The ZLIB stuff seems thoroughly messed up, but ta. I'd worked out the chars + length + chunks, and tried a few things with the offsets, but the C zlib library I used failed.

I'm using objdump, but to get the gist of it I poked around the header by hand as well.

I got my data out writing a simple loop in Lua to traverse every directory and to write them out to AppData/DataOut/, so the game did it for me. This is why I haven't done too much with the format thus far, and why when zlib misbehaved I just threw the towel in because I didn't need it Wink

The other method (which I really enjoyed) was breaking the protection, DLL injecting, finding the extraction calls, then rerouting the output to a directory. That one is just plain stupid though, as you end up with a really broken system if someone you find malicious code Wink

@simcutie:
Depending on the file you're seriously better off reassembling with ChunkSpy than you are with LuaDec (even 3.2, which I've had compiled for a while). If you need tables, you can just dump them too, or use ipairs or pairs on their structures.

You'll also be missing documentation on the Lua C-API connected functions, which if you get wrong, can stop the game from launching. So get ready to start CTRL+SHIFT+ESCing your games out Wink

FordGT90Concept · 20-04-2012 04:57 AM

Ah, I think I remember looking at LZSS and tried to decompress it but didn't have any luck.

Swixel · 20-04-2012 07:09 AM

Also going to put this here ...

Code:

MyOutputFolder = "AppData/Extracted/"

-- I like using own debug file :)

local SwixelDebug = io.open("AppData/swixel_log.txt", "w")

-- Configure; I hate reloading bugs

function InstallCopyFunctions()

    ExtractAllData = function(folder)

        io.createpath(MyOutputFolder .. folder)

        SwixelDebug:write("Extracting data ...\n")

        copy_dofolder_files(folder)

        copy_dofolder_folders(folder)

        SwixelDebug:write("Extracting done ...\n")

    end

    copy_dofolder_files = function(folder)

        io.createpath(MyOutputFolder .. folder)

        local files = io.listfiles(folder, "*", "")

        table.sort(files, function(a, b) return string.lower(a) < string.lower(b) end)

        SwixelDebug:write("File Count: " .. #files .. "\n")

        for i = 1, #files do

            local file = files[i]

            SwixelDebug:write("Copying " .. file .. " to " .. MyOutputFolder .. file .. "\n")

            CopyFile(file, MyOutputFolder .. file)

        end

    end

    copy_dofolder_folders = function(fld)

        local folders = io.listfiles(fld, "*", "folders")

        table.sort(folders, function(a, b) return string.lower(a) < string.lower(b) end)

        SwixelDebug:write("Folder Count: " .. #folders .. "\n")

        for i = 1, #folders do

            local folder = folders[i]

            SwixelDebug:write("Extracting folder: " .. folder .. "\n")

            ExtractAllData(folder)

        end

    end

end

-- Only fire once

local FiredOnce = false

-- UA Set Mode

OnMsg.UASetMode = function(actions,mode)

    -- Fire until fired once

    if not FiredOnce then

        -- If we boot, trip it ...

        if(mode == "Boot") then

            FiredOnce = true

            -- Run the thread to bypass it

            CreateRealTimeThread(function()

                -- Setup the functions

                InstallCopyFunctions()

                -- Make the base folder

                io.createpath(MyOutputFolder)

                -- Extract

                ExtractAllData("Game")

            end)

        end

    end

end

Drop that in a file, then, appropriately, use the function call:

Code:

ExtractAllData(MyFolder)

e.g. "Maps" or "Game" or "ex" (Expansion data).

FordGT90Concept · 29-04-2012 05:06 PM

1.0.3 released with a lot of bug fixes including the ability to extract challenges.

jboom91 · 30-04-2012 01:04 AM

(29-04-2012 05:06 PM)FordGT90Concept Wrote: 1.0.3 released with a lot of bug fixes including the ability to extract challenges.

OMG thanks! Must check this out.

Threaded Mode \| Linear Mode Tropico 3/4 HPK Archiver
Author	Message