Conquest of Elysium 4

Conquest of Elysium 4

Marlin 29 Jun, 2016 @ 7:28am
Extracted from coe4.exe: Monsters, weapons, spells & items
Almost two months after I released the data for classes and recruitment, the time is due for some more:
    Download[www.dropbox.com] the Conquest of Elysium 4 complete definitions of all monsters/units, weapons, spells and magic items in the standard game.
I actually included the files for class and recruitment data too – extracted fresh from v4.18 but identical to the ones I previously posted from v4.16, I think, except only for the addition of two recruitment entries specific to the Nexus.

Like before, the format is c4m files (plain text files) with pseudo-mod code, and, for the best reading experience, I recommend a text editor with CoE syntax highlighting, such as the schemes I have made for Gedit and for Notepad++ – both of them incidentally just now updated for v4.18 – or, if nothing else, the syntax highlighting of just a Unix shell script. (The last one will at least make comments and string literals stand out from the rest.)

There is also a spreadsheet containing, for now, data for weapons and spells and... not much else. I was struggling a bit with the format for in particular the spell data. Presenting the standard class and recruitment data as “CoE4 mods” worked decently well, I think, as those things are actually moddable, and real mod commands are available for most things. The same goes for weapons and monsters, but the c4m file with spells ended up a bit dull looking. (As it's almost entirely consisting of commented out pseudo-mod commands, the shiny, colorful c4m syntax highlighting I made doesn't do much good.) So I ended up stealing an idea by Colonel Dracula joins the Navy – who is apparently working on a release of extracted ritual data of his – of presenting some things in a spreadsheet instead.

(The simple spreadsheet I made is in two file formats: The ods file is the standard OpenDocument file I first created. The xls file is the same thing converted to the Microsoft proprietary Excel Binary File Format – for those using Microsoft applications, as I'm unsure how well those yet really support the OpenDocument standard.)
________________________________________

With the class and recruitment data, I listed every numeric ability code and bitflag number I encountered in the executable. (Lists in comments at the beginning of each c4m file.)

The corresponding lists I have for the monster definitions, on the other hand, are still a bit messy, and whereas there are only some 30 class ability codes and a few recruitment bitflags, there are more than 200 monster ability codes, I think, and potentially 128 different monster bitflags (though not all of them used). So, at this point at least, I have skipped that part. Where I think I've found corresponding mod commands, I've had my extraction script just use those commands, only, without their numeric representation in the executable. Only those bitflags and ability codes for which I couldn't find mod commands are numerically presented.

Needless to say, I haven't been able to figure out the purpose and meaning of every ability code or every bitflag. In some cases I may even have guessed wrong. If you think you know the purpose of such a code or flag, or if you spot anything that doesn't look right, do tell me about it.
________________________________________

This time, I asked Johan beforehand if he’s OK with data from the exe file being posted, and he graciously gave me his go-ahead. (He wants it to be clear, of course, that the stuff is coming from Conquest of Elysium, but considering there is a browser with Dominions data, he doesn’t see a problem with something similar for CoE.) So a big thanks to him for that.

Also, thanks to Johan for fixing and adding to the mod commands in v4.18, allowing my c4m file with extracted unit data to look a bit better than I first thought it would. (I am particularly happy about the new meleeweaponspec command, and the fix to meleeweaponbonus.)

😀
________________________________________

Edit: Changed link from MediaFire to Dropbox, since Steam now blocks MediaFire.
Last edited by Marlin; 22 Oct, 2017 @ 1:39pm
< >
Showing 1-15 of 20 comments
Superb work. Thank you so much, Marlin! I've only had a chance to glance through everything so far, but it looks great! I'm fighting not to let myself get caught up in it, so I can stay focused on finishing the ritual data in as timely a fashion as possible.

One small thought that jumped out at me, as I've never seen it mentioned anywhere: in my experience, magic items which summon something "at the start of battle" do so on round 0, as opposed to round -15. This is first thing in many battles, but quite a bit after assassination and siege attacks get started, which could come as a nasty surprise if anyone's unfamiliar with how they work. Think it's worth mentioning?

Also, if you don't mind, I've got a little technical question for you: do you know if the game internally uses the same ID number for each monster as is shown with ctrl+i? Obviously, they'd be in hexadecimal, but do you know if they're otherwise altered or offset? I ask because I found that the ritpow values are internally +500 from how they're documented in the mod manual, and I'm wondering if something similar is going on with monster IDs. I'm trying to figure out how Druidism's Minor and Major Animal Summoning rituals determine the types and numbers of units summoned (they do it using the specpow legacy command instead of through addstrings, which is how every other summoning ritual I've looked at does it), and in pursuit of that, I've resorted to using a debugger to break on access of the specpow command and trace up to 100,000 operations after that. Yet, when I've searched through the traced operations for the hex value of the ID of the monster type summoned, I've always either gotten nothing or only what appear to be false positives--no hits where the value originated from a static location in memory, which is how I would most expect the lists of possible summon results to be stored. If I've been searching for the wrong value, that might at least explain why I've been unsuccessful.
Last edited by Colonel Dracula joins the Navy; 29 Jun, 2016 @ 12:26pm
Marlin 29 Jun, 2016 @ 1:58pm 
Good catch, Colonel Dracula, regarding the “start of battle” for item effects. I hadn't really tested that part much (and for some reason haven't thought about it when playing). But, yes, the “start of battle” for items is indeed the first ordinary battle round, after the siege phase is done. This is true both for summoning and for any spell casting too, “at start of battle” – though the latter does not seem to be used for any item in CoE4. (It was used for the fear of the Pocket Lich in CoE3.)

This should definitely be mentioned with the item data, yes. I will update it for the “next release” (whenever that will be).
________________________________________

Regarding monster indexes, although I cannot say for certain, I would be surprised if any offset is to be applied to them. If absolute indexing of monsters is used, I would assume the number you get with Ctrl+I – which also is the number you'll find in the upper right of each monster definition in my coe4_units.c4m file – should be taken as is.

The caveat is, I don't think I have seen any reference of monsters by absolute indexes, only by relative offsets, or by name – or through reference to some “tag” on the monster. In a few cases, montag is used, while mirrors refer to a phantasm value of the monster.

My base assumption here would be reference by name, except, of course, that the specpow command seems to take a number for argument.

Weapons certainly are referred to by absolute index, without any offset. (Such references are found in data for monsters, spells and items, and also within other weapon definitions.) Though, in weapon definitions, for an extra effect, the index of another weapon can occasionally be negated (to indicate that the extra effect is always applied, rather than just if the target is affected by the first one).

The +500 to ritual powers thing (and, yes, I have noticed that too) is, I'd assume, just a way turn them into ability ID codes that don't clash with other ability codes. In the class data that I did first, I found codes 200-229. In the monster data for CoE4, I have found ability codes 1-199 but not anything that clashes with the codes that I found in the class data, then the 500 range for ritpows and an 800+ range for some abilities. (In CoE3, there are also some 600 and 700 ranges, in addition to the 800 range, and I suspect all of these were originally for modding only.)

(Well, to muddy the waters a bit further, I did find an offset also being used with the nametype command – The value given with the nametype command/ability is internally stored as the plain, documented value +100 – As to why that is, I have no idea.)
Alright then, I just don't know why I haven't been successful with the debugger. Maybe 100,000 operations wasn't enough to cover the distance between finding specpow on the ritual and determining the units to summon, but that would be awfully surprising. I'm out of ideas for now, so the exact effects of Druidism's Minor and Major Animal Summonings will just have to remain uncertain for the time being.

I don't think I'll delay over the remaining unknown commands, either. I double-checked the other day, and I had indeed already accounted for every command documented in either the modding manual or the modding documentation thread or located in the ritual section (addresses 0xbb173c to 0xbb1b57) of the command name table in the executable. If you're interested in adding them to your list, the named ritual commands number from 900 (0x384) to 996 (0x3e4), with about a dozen of those numbers apparently not assigned to any named command. I can post the list of commands and their IDs if anyone's interested. The last 8 commands ("free" through "noundead") in the ritual section of the command name table are not given ID numbers, but rather they are flags, each with a specific reserved bit at a particular offset (0xe8) in each ritual's data block. There are 8 remaining unknown commands used in the ritual data (997-1005, sans 999). I haven't looked terribly closely at them yet, and I won't delay release in order to do so, as all the questions I've wanted answered have been answerable without them, which suggests they aren't terribly important (at least, not to me).

A few things I do want to take some more time to work on:
  • Continuing to add comments to help reduce the need to refer to the modding manual to look up how each command works.
  • Writing an introduction and some explanations.
  • Investigating and testing what appears to be a bug on the Send Lesser Horror ritual.

I haven't decided yet whether I'll wait to include my summon rituals spreadsheet in my initial release. I'm doing it by hand because, unlike text files, I never took the time to learn how to output to a spreadsheet, so it'll take a while to finish. Probably, I should just bite the bullet and learn how to automate it (especially to make it easy to update), but that might well take even longer.

Marlin, would you consider sharing your list of monster ability ID codes at some point? There are several ritual commands in the documentation thread with <fx nbr> as their parameter. For at least two of them, choosefxtarg and fxreq, the <fx nbr> parameter refers to a monster ability ID code. As examples:
  • The ritual Stygian Bath has choosefxtarg 90, and 90 is the ability ID for "troll"; this is how the game determines eligible targets for the ritual--if they have "troll", they're eligible.
  • The ritual Hide Heart has fxreq 3, and this is how the game determines who can use the ritual. As shown in your data, ability 3 is not named, but it is present only on the Bakemono Sorceror and Bakemono Great Sorceror. Given that, its use in fxreq 3 for Hide Heart, and the fact that no one but those two monster types can use Hide Heart, I'd say ability 3 clearly represents "valid user of Hide Heart".

Other commands with an <fx nbr> parameter may also refer to ability IDs (though rebatefx25 and rebatefx50, at least, do not--they of course instead look for a matching value as the parameter of a rebate command on a monster), and I suspect it would be useful to modders to have a list of values with which to use these commands.
Marlin 30 Jun, 2016 @ 8:33pm 
Heh! I was wondering about the references to “fx” in rituals. It didn't occur to me that it would be the ability codes that I have just extracted from the monster data. (Obviously, although I have been using the word “abilities” for these codes or IDs, the term used by Johan himself might well be effects, for all I know.) Great find, Colonel Dracula.

In that case, yes, these numbers would indeed be of more general interest to modders than I realized. So, I have now tidied them up a bit and quickly put together (If and when I make another release of my first spreadsheet, this one should be included there.)

Not that I intended to keep them for myself. I just didn't know quite what to do with them. Trying to list all of that at the beginning of the c4m file for monster data seemed messy. (The monster bitflag numbers I have will follow eventually, but those might be of less general interest.)

Originally posted by Colonel Dracula joins the Navy:
I haven't decided yet whether I'll wait to include my summon rituals spreadsheet in my initial release. I'm doing it by hand because, unlike text files, I never took the time to learn how to output to a spreadsheet, so it'll take a while to finish. Probably, I should just bite the bullet and learn how to automate it (especially to make it easy to update), but that might well take even longer.
For the spreadsheet included in my batch above, I actually did just have my Python scripts output to text files, plain csv files. Meaning I had to manually fix the formatting (column widths and text alignments) and some other things in the spreadsheet application (LibreOffice Calc) afterwards. (Yeah, that means it won't be quite as painless to generate a new version of it as it is with the c4m files, of course.)
________________________________________

Edit 1: Updated link to second little spreadsheet. It now also contains the bitflag numbers found in monster type definitions.

Edit 2: Changed link from MediaFire to Dropbox, since Steam now blocks MediaFire.
Last edited by Marlin; 22 Oct, 2017 @ 1:42pm
Awesome, Marlin. Thank you!

By the way, nofxherereq also shows up in the executable's ritual data with an ability code as its parameter, so we can add it to the list of commands confirmed to use them.

Originally posted by Marlin:
The monster bitflag numbers I have will follow eventually, but those might be of less general interest.

Maybe so...unless, as I'd like to hope, they happen to be the same bitflags referenced by the "ben..." or "res..." ritual commands. The modding manual mentions a few of the values and effects associated with those commands, and I notice that none of those effects are listed with ID numbers in your latest spreadsheet, implying that they (except Twiceborn) are among the effects implemented via bitflags in the monster data. If the same effects are treated as bitflags in both the monster data and these ritual commands, it's possible (though by no means guaranteed, of course) that they might have corresponding values in both places. Any chance you see any correlation?

Originally posted by Marlin:
For the spreadsheet included in my batch above, I actually did just have my Python scripts output to text files, plain csv files.

That's a very good idea. I might try that until and unless I learn to get my C# apps to work directly with Excel. Thanks!

Well, I got about 2/3 of the way through writing the text and switch statements for my ease-of-use comments today, but obviously, I didn't finish. Unfortunately, I don't expect to get an opportunity to work on this again over the long weekend here in the USA. I should be able to get back to it on Tuesday.
Marlin 2 Jul, 2016 @ 6:53pm 
Hmm, regarding “res”, yes, it turns out it's even in the nice CoE4 modding manual we now have, if you read it carefully. I missed that. While extracting monster data, I have been thinking of those flags as “misc. flags” rather than resistances, as they determine a wide variety of properties, including for instance the number of item slots of a unit, or whether it can lead troops if a commander. But it seems that resistances is the official term for them.

Alright, I have updated the extra little spreadsheet to also contain all the bitflag numbers found in monster type definitions. (Apart from the “resistances”, there are also the move flags, many of which were documented in the CoE3 modding manual, but which are considerably expanded for CoE4.)

Link in my previous post updated.
________________________________________

The “ben”, on the other hand, I would assume refers to the special benefits, which aren't found in monster type definitions. (Presumably there is instead such a bit field in the data for each individual unit while playing.)

Basically, these are documented in table 4 on page 20 in the modding manual. Except that there seem to be a couple of errors in that table. I believe the correct special benefits table looks as follows:
№ Special Benefit bitmask in decimal notation 0 Invulnerability 1 1 Armor +1 2 2 Fire Immunity 4 3 Cold Immunity 8 4 Poison Immunity 16 5 Shock Immunity 32 6 Magic Resistance +2 64 7 Luck 128 8 Etherealness 256 9 Strength +2 512 10 Fire Shield (3) 1024 11 Air Shield 2048 12 Regeneration 4096 13 Awe +1 8192 14 Command Missiles (unused) 16384 15 Unrecognized benefit 15 32768 16 Blessing 65536 17 Armor +2 131072 18 Pierce Resistance 262144 19 Slash Resistance 524288 20 Blunt Resistance 1048576 21 Displacement 2097152 22 Non-Magical Invulnerability 4194304
Last edited by Marlin; 2 Jul, 2016 @ 7:45pm
hueytuckers 3 Jul, 2016 @ 7:43am 
Marlin,
Thanks for all the work you put into this.
Thanks for the resistance bitflag values and the corrected benefits table, Marlin! Those should open up even more opportunities for modders. Also, thanks again for everything; it can't be said enough!
Xelos 3 Nov, 2016 @ 8:12am 
Thank you! We need a post that has all this data (maybe with dracula's ritual data) aggregated in one spot and pinned.

For a modder this is extremely helpful for a. learning and b. mass changes.


I noticed in the recruitment lists the selectclass 0 for specific terrain types. How does this work linking the terrain to unit recruitment?
Last edited by Xelos; 3 Nov, 2016 @ 8:49am
HelhiganM 20 Nov, 2016 @ 6:58am 
this should be stickied
Lost 2 Jan, 2017 @ 6:05am 
can we please get a new link?
Xelos 2 Jan, 2017 @ 11:45am 
Originally posted by lo5t1e:
can we please get a new link?

Also could we get the monster ability ID codes to use with choosefxtarg.
Here's a re-upload of Marlin's most recently published files:

[removed]

Edit: Google broke my links, and they aren't needed anymore, anyway.
Last edited by Colonel Dracula joins the Navy; 11 Nov, 2021 @ 2:22pm
Xelos 8 Mar, 2017 @ 6:25pm 
Is the unit/monster extracted data supposed to be missing alot of elemental units? For example I don't see many elemental warriors: Mist Warrior, Mud Warrior, Magma Warrior, Smoke Warrior.
Marlin last extracted this data for game version 4.18. The units you mention were added in game version 4.20, so they're not included in the extracted data.
Last edited by Colonel Dracula joins the Navy; 8 Mar, 2017 @ 7:51pm
< >
Showing 1-15 of 20 comments
Per page: 1530 50