If you want to play Hostile Waters: Antaeus Rising (HW:AR) in a custom or a high resolution you have to edit the registry options table for the game, because high resolutions are not available by the Configuration Tool "HostileSetup.exe". Although the game is installed already, the installation starts again every time you start the game within the steam client. That seems to be an annoying bug of the steam version. So you should ignore the steam client and just start the game from the installation directory. The default installation directory is:

C:\Program Files\Steam\SteamApps\common\Hostile Waters Antaeus Rising

Like many old games this game needs admin rights to run. Therefore right click onto the HostileWaters.exe file to open the context menu, then go to Properties -> Compatibility -> Execute as administrator: on -> adopt -> ok. Now you can start directly with the HostileWaters.exe without using the steam client.

I found out, that admin rights are not required, but these cannot harm.

Right click on HostileSetup.exe. Go to Properties -> Compatability -> Change compatibility mode to Windows XP (Service Pack 3) -> switch on "Execute as administrator" -> adopt -> ok.**

When you start HostileSetup.exe and click on Hardware Setup, the high resolution screen modes occur now. Choose the one you like, 4:3 is recommended, because widescreen 16:10, 16:9 ratios cause top and bottom cropping.

If the Windows XP compatibility mode would have been set by the installation instead of Win98, it would have saved plenty of my time. Thanks Valve! (not!) :-(

All thanks go to Ew|mp3, who mentioned that issue!**

You can setup the game by editing its windows registry options table for HW:AR. There you can set custom resolutions, which are not available in the configuration tool (HostileSetup.exe). Start the Windows registry editor by searching for regedit.exe in the start menu search bar or execute "regedit" by the Windows task manager.

[color=red]Warning![/color]: If you're not familiar with registry editing, don't mess around with it. If you don't know what you are doing exactly, you should better not do it. Just follow my instructions carefully!

On a 64 bit Windows sytem navigate to:*
HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Rage\HostileWaters\Options

On a 32 bit Windows system navigate to:*
HKEY_LOCAL_MACHINE\SOFTWARE\Rage\HostileWaters\Options

You probably want to change the resolution according to your High Definition screen. In the registry directory mentioned above, find the entry (name) "ResolutionX". Doubleclick on it to change the value. Change the base from hexadecimal to [color=red]decimal[/color] first! After that enter your desired x-Resolution. If you have a full HD 1080i/p screen, it should be 1920 for example. Click ok and verify the changed value in the table. First should be the hexadecimal value and the second should be the decimal value (in the brackets).
Do the same for the entry "ResolutionY". The resolution y value should be 1080 for the example.
Now find the entry "Colour Depth" and check if the decimal value is 32. If it is not, you probably want to change it to 32 in the same way you changed your resolution. Remember to choose decimal base!

If you start the game now, it will be in 1920x1080 resolution with 32 Bit Color Depth (highest). Unfortunately the game is optimised for 4:3 resolutions only, although widescreen resolutions are available in the configuration tool. I recommend to use an according 4:3 resolution, because the picture is cropped in widescreen resolutions. If you choose 16:9 or 16:10 ratio, some menu buttons in the game will be outside of the screen and therefore unavailable. Maybe you rather like to play it with 1440x1080 4:3 ratio according to the 1080i/p example.

For a better visual quality you should turn on 32 Bit Textures and Antialiasing in the registry options. Find "Textures32" and "FullScreenAntiAlias" and change their values to 1. The option to switch the 32 Bit Textures on or off can be found in the configuration tool, but the Antialiasing option exists only in the registry options.* Don't setup 32 Bit Textures in the config tool if you have already entered your custom resolution in the registry, otherwise the tool will change it to the selected screen mode. Just cancel the hardware setup tool if it is open, don't click ok.

*) http://www.play-old-pc-games.com/2014/04/23/hostile-waters/
**) https://steamhost.cn/steamcommunity_com/app/267980/discussions/0/540738051970281700/#c540741769596357030