I will try to recreate these situations. If you could, there is a log file in %appdata%/spaceengineers/storage that will contain some extra info. If you can post that to pastebin.com after recreating this (I assume you can recreate this all the time) that would help. I will work on trying to recreate the situations.

Oh yeah, pretty much on command. I'll go ahead and get that ready once I'm done with a segment of my station I'm working on (I have to do a save backup and then reload the dedicated with the mod). I'm going to check to see if the log is there from the last time still, though, real fast.

Thanks for the fast reply, btw! Great mod, looking forward to using it.

Edit: The log is there, but what I'm going to do is wipe that log and create one for each bug, that way it's isolated and clear. Noting my steps and which bug the log is for.

Alright I've got those steps together for you, as well as the log:

I'm going to do both bugs in one log, as it will work fine. I'll do the discarding bug, then the duplication bug.

In case it makes a difference: I am running this on a dedicated server (64bit) hosted on my PC, running the 64bit version of the game as well.

1) Start with 5 containers:

• Ingot Storage 1 - Has various ingots, no iron ingots
  
• Ingot Storage 2 - Has gravel [2.2m]
  
• Ingot Storage 3 - Has iron ingots [10.8m]
  
• Ingot Storage 4 - Has iron ingots [5.97m]
  
• Ingot Storage 5 - Has iron ingots [14.76m]

2) Rename container "Ingot Storage 1" to "Storage: [Ingot]" - It pulls the iron from container 5, 4, and 3 but only 1.43m of it ends up in "Storage: [Ingot]" and the rest is nowhere to be found. The end result of this:

• Storage: [Ingot] (formerly "Ingot Storage 1") - has various ingots + 1.43m iron ingots.
  
• "Ingot Storage 5" has 5.00m iron ingots, nothing else
  
• "Ingot Storage 3" has 1.130m iron ingots, nothing else

So the previous total of the iron ingots: 31.53m, the current: 7.56m

I double checked, none of them are in the assemblers or arc furnaces, or any other storage (I did a global search for Iron Ingot, and it's only in "Ingot Storage 5", "Ingot Storage 3", and "Storage: [Ingot]" as listed above.

Duplication Bug - Continuing from above

Now, directly after that and saving the log file for the above, I did this:


3) Rename the following containers:

• Rename "Ingot Storage 2" -> "Storage: [Ingot]"
  
• Rename "Ingot Storage 3" -> "Storage: [Ingot]"
  
• Rename "Ingot Storage 4" -> "Storage: [Ingot]"
  
• Rename "Ingot Storage 5" -> "Storage: [Ingot]"

4) At this point, the mod moved the ingots from storage 3 and 5 into one of the "Storage: [Ingot]" containers. It did this correctly, totalling 5.00 + 1.30 == 6.130m iron ingots. So, currently, only two of the "Storage: [Ingot]" have stuff in them, three are empty.
5) More the 6.130m iron ingots to another container, an arbitrary container, "Component Storage c-1". Now, the component storage only had room for 5.45m ingots, the results are as follows:

• Three of the previously empty "Storage: [Ingot]" have 5.45m iron ingots in them.
  
• The "Storage: [Ingot]" I got the 6m now has 6.131 (noting it has slightly more than I pulled out of it).

The new total: 23.911m (Was 7.56m)


And the log for these steps:

http://pastebin.com/sjyze0AR (I put a note in there noting where the discarding bug ended and the dupe bug begins.)


As a side note:

Also I've noticed this didn't pull any iron ingots from the valid conveyor connected arc furnaces. Are those implemented along side the refineries for pull requests from the containers?

If not, they should. I, and I know some others, use them to act as a kind of buffer for the iron ore, so the refineries can focus on more important ores.

Hope this helps!

Edit: Had to fix a placeholder I had forgot to replace with a number.
Edit: Added a note that I am running on a dedicated server.

Oh, dedicated server. sigh.

Here is one issue. dedicated server inventories have this nasty habit of not syncing. You are actually running into an inventory bug in the game, and not from my mod.

You want to see the fun happen? Turn off my sorting, and take an item, and drag it to the end, then do that again for something else in the inventory. Keep doing it until you have no more inventory.

I will test these in dedicated, but dedicated is so bugged, that it's hard to keep up with trying to fix bugs in it. I actually run a large dedicated server, and have cautioned my users from using my own mod, and only sort things in small amounts, as inventory that moves from one place to another often, tends to just go poof with no reason.

That being said I will try to recreate this locally on a dedicated, as the duping bug is interesting. I believe the sync issue is a lot larger than I thought. I can see in your log the issue which is pretty nasty. I believe what is happening is:

Cargo A has Iron Ingots
Cargo B checks Cargo A and sees ingots and takes all ingots from Cargo A
Cargo C checks Cargo A and sees ingots (desync) and takes them from Cargo A
Cargo D checks Cargo A and sees ingots (desync again) and takes them from Cargo A

I can fake fix this issue by letting things sync before pulling from a cargo container again (how often does inventory sync happen? I don't know to be honest, but it better be faster than a couple seconds!)

I am changing one small way I'm moving inventory that I'm hoping helps with the first issue you have, but it's hard to say on a dedicated. The new version will be out tomorrow, and I'd like you to test that. I don't think the second issue will be fixed by then, but I will look at it.

I am as interested as you to get this working well in dedicated, but it's not simple.

I think I see the problem in your log with stuff disappearing. It looks like if I move multiple amounts of items INTO the same inventory or OUT of the same inventory in a dedicated environment, it loses sync.

So for example:

Cargo A has ingots
Cargo B takes ingots from Cargo A
Cargo C takes ingots from Cargo A (desync here, A shouldn't have any left, but is detected in API as STILL having items - dupe occurs)

Cargo A has ingots
Cargo B has ingots
Cargo C takes ingots from Cargo A
Cargo C takes ignots from Cargo B (desync here, C will only result in having partial amounts either taken from A or B, but not both combined)

So the fix for dedicated appears to be that I can only modify an inventory ONCE per sort cycle. Ug.

Yeah, that makes a lot of sense actually. Sounds similar to issues I had making threads sync up when I was learning it in C#.

I have noticed a lot of sync issues in the dedicated server, though, now that you mention it. In fact, I posted a bug report on the fact that the server seems to delete floating objects like it should, but the client ignores that under certain cases. If you want an extreme example, set the object limit to 32, launch the server in SE Server Extender so you can see the floating objects, then take a large drill ship with a lot of drills (mine has ~200) and watch the floating objects list. You will see it creates hundreds of objects every second or two, and deletes them, but the client just leaves them all there and it ends up lagging out the sim speed until you reload. I hope they fix that stuff (in due time, of course, just when they do their optimizations).

Any idea on an ETA for this fix? I'll give it a try again once it's ready to be tested.

Again, thanks for the hard work and great mod! Also looking forward to using the new refinery pulling features. That will be useful for splitting up some of my ores, and tossing stone out of an airlock.

I will try to throw in a mini-fix soon after I get a few other things out of the way, probably today or tomorrow. I will change it so that an inventory can only be modified once per update. This may cause other unforeseen issues due to multiple assemblers / refineries, and trying to pull from them, but it'll be safer for dedicated.

Thanks!

Your work on this is greatly appreciated.

If I might ask, what kind of issues are you thinking? So that I might be prepared.

Thanks

I guess he means that only one Assembler/Refinery will be cleared per cycle instead of all at once. This could lead to a possible block because:

• Ref 1 and Ref 2 create Iron Ingot faster than any sort cycle
  CYCLES:
  
  1. Cargo (Ingot) sees the Iron Ingot in Ref 1 and takes it, stops also his cycle becaus of one action per cycle
     
  2. Cargo (Ingot) sees the Iron Ingot in Ref 1 and takes it again, leaving Ref 2 alone
• Ref 2 will fill up eventually and stop working because of full output storage

Doing everything in one Cycle would need a way to populate the info of an empty storage before the next item in the current cycle starts working.

• Cycle start
  
• Cargo 1 (Ingot) checks all Ref/Assembler for items it wants
  
• Cargo 1 (Ingot) takes out Iron Ingot from Ref 1, Ref 2 and Ref 3
  
• Changed amounts get populated to the datastorage
  
• Cargo 2 (Ingot) checks all Ref for items it wants and does not see the items taken out by Cargo 1 beause of updated storage values
  
• Changed amounts get populated to the datastorage

Ah, yeah that makes sense.

Both methods actually could pose their own problems. The block issue you mentioned already, but the second would have a downside under these conditions:

Take Assembler 1, Refinery 1, Cargo 1 and 2. Let's say that Refinery 1 is chewing away at Iron. Cargo 1 doesn't have enough room in it to fit all of the ingots that Refinery 1 generated since the last cycle. At the same time, we're making steel plates in a decent sized bulk order (This will cause the assembler, as far as I know, to almost constantly grab iron as it has space to do so).

• Refinery 1 is chewing away at iron.
  
• Cycle start
  
• Cargo 1 checks Refinery 1 and Assembler 1 for items
  
• Cargo 1 takes iron ingots from Refinery 1 - Now let's say it couldn't fit enough in it to take all the ingots, so there are some left.
  
• *datastore update* - At this point the datastore would reflect either how much Cargo 1 took, or more likely how much is left in the various containers/refineries/assemblers that were checked. For sake of argument, let's say it reflects that Refinery 1 should have 100 iron ingots left in it.
  
• Meanwhile/in parallel - Assembler 1 is chewing away at some steel and says "I need more ingots" and takes 10 ingots from Refinery 1 - Now Refinery 1 has 90 ingots, but because it was the game's base conveyor system, the mod's datastore is not updated.
  
• As the cycle continues - Cargo 2 looks at the refineries/assemblers and sees that Refinery 1 should have 100 ingots from the datastore. Now, from what I understand in this case, on the dedicated, the API would be wrong. Plus, due to the assemblers taking ingots in parallel with a cycle, the datastore is out of sync as well.

At least from the logic I can see, that could pose a problem. Now you could simply disable conveyor system on the assemblers, but despite even being able to push to them using the mods, them using the conveyor system is nice (You can queue up anything and they will just suck up the mats).

The only way I see around this would be separation via properyly ordering the conveyor chain. You would want it Refineries <-> Storage <-> Assemblers. The refineries would get the ingots pulled from them, and pull the ore from storage. The assemblers, assuming the storage isn't out of ingots, would pull it's needs from the storage as well, and everything would nicely get back into storage. This is how I have my base setup, but I can't test it yet with this (Though I might fire it up in single player to see how it responds).

It's really unfortunate that the dedicated servers have sync issues. However, it is alpha, so expected.

Your example can be countered fairly easy.
Setting amount to each storage item, split the total amount up by the number of possible entries.

(assuming each item has the same volume requirement, needs adjustments to the volume values of course but this is just to give a general example)
Let's say we have 8 different Ingots/Ore and our Assembly is holding 2400l while our Refinery holds 4000l so putting:
Ore:500 on the Refinery
Ingot:300 on the Assembler
And putting Ore/Ingot on the Cargo containers.

This should eliminate what you mention above by emulating the Conveyer System usage with the Sort mod while having the original "Use conveyer system" disabled. As this will hold enough resources for every production and will fill it up similar to the conveyer link system. I just hope they had this mod to the game itself. Dedicated Cargo containers for different things.

Originally posted by Autositz:

Your example can be countered fairly easy.
Setting amount to each storage item, split the total amount up by the number of possible entries.

(assuming each item has the same volume requirement, needs adjustments to the volume values of course but this is just to give a general example)
Let's say we have 8 different Ingots/Ore and our Assembly is holding 2400l while our Refinery holds 4000l so putting:
Ore:500 on the Refinery
Ingot:300 on the Assembler
And putting Ore/Ingot on the Cargo containers.

This should eliminate what you mention above by emulating the Conveyer System usage with the Sort mod while having the original "Use conveyer system" disabled.

That's true, but you would have to have it put the minimum required number of every ingot, for each of the components you want. For instance, steel plates require 22 (if I recall), so at minimum you would need 22 in it at all times, preferably more (or it will stop until the system places more into it). The point being, though, is that you have to specify each ingot type you intend to use, or it will shove 300 of the first one it gets it's hands on in there (since you use the Ingot wildcard/global). For instance, on the assembler: [Iron:60,Gold:20,etc] (Not sure what the actual numbers would be, but you get the idea). That way no matter what you intend to build, it always meets the reqs if you have the ore.

That being said, the materials aren't *that* heavy, so it would probably work fine.

The wildcard works for EACH match, NOT for the wildcard summary itself!
My example above would sort 500 respectivly 300 of EACH resource matched by the wildcard and not 500 or 300 in total with random values.

Originally posted by Autositz:

The wildcard works for EACH match, NOT for the wildcard summary itself!
My example above would sort 500 respectivly 300 of EACH resource matched by the wildcard and not 500 or 300 in total with random values.

Ohh, my mistake. That makes, thanks for the clarification.

Yeah that would work great, in that case, easily. I would be completely happy disabling the base conveyor system in favor of this. This would completely automate me being able to dump stone because I don't want it, split up my refining between various refieries for specific ores, and keep the assemblers stocked at all times and ready to go for any order.

Hey, not trying to pester or anything, but I was wondering if you had a fix up at all?

Again, not rushing you or anything, mostly just anxious to start sorting my ore and components with this, and was wondering if I should give it another test yet (I'm guessing no, as I didn't see anything in any change logs for it)

Again, thanks for the great mod!