ATMA 5.04 and new DropCalc

[hakai_no_tenshi]

I figure this is about the best forum here to post this.

The release of ATMA 5.04 is done and can be found on the official ATMA website at http://www.luminatrix.com/ATMA .

ATMA 5.04 introduces a few new features as well as bug fixes since 5.03:

BUG FIXES:
----------

1) Mercenary resist display has been fixed. ATMA was picking up the wrong entry in hirelings.txt especially for Expansion characters. This resulted in a wrong calculation for the mercenary resists.
2) IAS display for items with the IAS property (e.g Immortal King Stone Crusher) that have been socketed with IAS runes/jewels has been fixed. This was due to an error in the ATMA config files.
3) A bug in the loading code for player ears has been fixed. This occured when the name of the ear was of a certain length.
4) A stash allows you to drag/drop items from both the picture area as well as the tree list. A check is always made to ensure that the item being dragged into a stash is of the same version. However, this check was only done on the tree area and not the picture area which allowed one to drop 1.09 items into a 1.10 stash. If this happened, the stash is corrupt. This has now been fixed.
5) The display for jewels and charms now shows if they do not meet character level requirements.
6) There was a bug in the code that would sometimes cause the game to randomly delete an item especially if it was moved from an equipped slot into inventory. This was the infamous "disappearing item" bug and it was due to a flag not being reset on the item. This has now been fixed.
7) The background for the picture of an item in a stash is now colored to distinguish the quality of the item.


NEW FEATURE:
------------

ATMA 5.04 now includes a DropCalc which Thrugg and I (Hakai) have been working on. This calculator supports versions 1.07-1.10 and includes many features.

1) Full support for Diablo 1.07, 1.08, 1.09 and 1.10.
2) Correct scaling of creature mlvl and TC upgrade in Diablo 1.10 (Nightmare and Hell).
3) Sorting of results by name, probability or creature location.
4) Display of probabilities as fraction, percentage or odds.
5) Sorting of probabilities in ascending or descending order.
6) Ability to search by monster or by item
7) Full support for "players X" and character MF
8 ) Both item and monster search allow you to customize the parameters by which you choose to search
9) Changing either your MF or # of players requires the user to hit the "Search" button.

The DropCalc is provided "as-is" with a few caveats. The internal mechanics of the drop system is a very complicated one such that depending on certain "branch" points, the drop probabilites need to be computed differently. In mathematical lingo, a DAG (Directed Acyclic Graph) would need to be constructed to represent the drop sequence for each item. After doing an in-depth analysis of the drop tree, Thrugg and I came to the conclusion that all the other drop calculators out there are probably wrong. The problem is that depending on the path taken during a drop sequence, you can hit several branch points which will change the mechanism of the drop probability. Unfortunately, to properly calculate the final drop probability, you need to store the drop path for every contribution from a TC to every item it can drop. When this is done, you would need to compute your chances of the item being of a certain quality and then reconstruct the total probabilities for all items by going back up the tree and taking the number of picks at each level into account. What ATMA does right now is to compute the drop probability for any one possible drop of any creature. Trying to take the full number of drops into account will involve significantly more data storage and slow down the calculator by many factors. In the meantime, you as the user know that if Pindle has a 1:10000 chance of dropping item X on a given drop, he will have close to a 1:5000 of one per kill. Hence it is up to the user to correctly interprete the results especially when comparing two monsters (e.g Baal and Pindleskin) who have different drop mechanics.

Based on new info from Jarulf and Ruvanal, it has been discovered that Pindleskin and Threshsocket are not as "nerfed' as one would expect. The basic idea that Blizzard had to "nerf" Pindle and Threshsocket was to drastically reduce the odds of them dropping uniques and sets. In order to implement this, a new TC was created for them. Every TC has an associated bonus to the chances of unique/set/rare/magic items dropping and this is where Pindle and Thresh got hit hard.

Every TC is defined recursively in terms of "lower" TCs such that you need to traverse the drop tree to determine what item gets dropped. However, the game keeps a running check of the maximum bonuses. Because the TC that Pindleskin and Threshsocket drop from intersect paths with the TC that Snapchip and Frozenstein drop from, the Pindleskin TC gets it's bonuses upgraded to what Snapchip/Frozenstein would have. This means that Pindleskin drops exactly like those two while Threshsocket has a slightly worse chance than Pindle because his mlvl in Hell is 84 (this affects the chance for a given quality) as opposed to Pindleskin who is mlvl 86 in Hell.

The DropCalc is not a stand-alone app as it uses the same config and data files as ATMA. It makes no sense seperating them as you need to download everything anyway.

UPDATE:
-----------
I have just recently developed a new way of computing the probailities based on drop paths which does not add too much more to the original compute time. This method also takes into account the number of picks for a TC such that the final displayed drop chances are "per-kill" and not "per-drop" as I had it before. The new display will show the drop chances for the possible paths e.g for runes from Countess, there will be one entry from her regular drops and one from her Rune drop. I am still in the process of testing the numbers from this and will update when it is done.

--T

[hakai_no_tenshi]

After working out a few more details, I believe I have an accurate version of the DropCalc which calculates the drop chance for items per monster kill. I took into account the different branch paths such that for n branch paths, if the probability of an item (base, unique, set ..) dropping from each path is p_i (i=1..n) and the number of picks per path is c_i, then the total probability for at least one of that item dropping is:

p_total = 1 - PI( (1 - p_i)^c_i )

where PI(x_i) is the usual product notation i.e x_1*x_2*x_3*....*x_n

I have implemented this in a test version of ATMA and would like any one who is numerically inclined to test the numbers. The link to this test version is here . Give it a whirl and let me know. If the numbers come out right, I'll put this up as the revised version 5.04

--T

[Samka]

Great! ATMA was already a terrific product, and the drop calculator will be a really useful feature.

[Ruvanal]

I have not looked at this point (do not have the time to try to check), but did you take into account that there is a maximum of 6 items that can be dropped from a single initial call to the first TC? In other words there is a 6 item cap on the drops of the act end bosses and the countess (the only ones that I recall off the top of my head that end up with more than 6 picks for items).

[Jarulf]

Not sure if this is of any interest. But When I have been calculating drop probabilities, I have been doing it the following way. It handle all type of pick combination of positive and negative values nested. It do NOT currently handle the very important "max 6 items" caps WHEN there is a negative pick value. For positive ones only, one can just scale all values down to 6 if it becomes larger than 6. I think one can add a parameter to count picks and simply move drops to a "drops above 6 items" category but I have not had time to implement it yet. Also if one want to add quality calcs, I have not done it since I was unsure how one would handle sub TCs having their own unique/set/rare/magic values. In addition, if different sub TCs have different such values, what happens?

Anyway, what I did was the following. FIrst, I load all relevant txt files (of course). I also have one big data matrix where for each TC, I will have a list of each item storing the probability values. I simply calculate it for all TCs and keep the values in memory. If memory is a problem, one can do it on a "desired TC" basis of course.

The algoritm is basically to go through the TC, entry by entry and keep track of the probabilities until one reach base items. This is done by a recursive function that takes a probability value as its entry (and some other stuff). When we start of, we enter with a prob=1.0 value. When then call the function for any sub TC, and pass along prob=prob*abs(picks)*(chance/totchance), that is, we calculate the probability of that specific being picked and multiply by the probability that we have so far (that is having taken the path so far to the sub TC we are at). We also multiply by the picks value since if it is larger than 1, we will have multiple entries form here on). When we reach an entry that is instead an item, we store that "prob" value on the item (adding up all such prob values reached). We also store a total probability value which would indicate number of items. If this total value is larger than 6, we can scale all values down so that it becomes 6 (for all poistive picks only). This works great from what I have seen with the exceptions I have yet to implement mentioned above.

I only have an older version of the code here (but it is quite similar to what I have now) so I will paste it here. Note that the code has a bunch of things that are remnants from older things I have been doing so might look strange at times. But it is more to get the general idea:

Code:

void markitem(int tcslot, int islot, double chance)
{
    if (islot!=-1)
    {
 tdrop[tcslot].fidrop[islot]+=chance;
 tdrop[tcslot].idrop[islot]+=1;
    }
    else
    {
 tdrop[tcslot].fnodrop+=chance;
 tdrop[tcslot].nodrop+=1;
    }
    tdrop[tcslot].fdrop+=chance;
    tdrop[tcslot].drop+=1;
} // markitem


Post comment:
Here I add the probability to the item (or to no drop for no item given). It seems I also keep track of how many times the item has been "added" to. Oh well, something I don't know if I use for anything, don't think so. At the end is the total probability which give the number of items you normally get from a monster.


void doonetc(int maintcslot, int tcslot, double chance)


Post comment:
the maintcslot is the entry TC I am calculating for, it is the TC I will store info in for each item. The tcslot is the current TC I am going through. chance is the current probability that we will come to that TC in a drop.


{
    // can be called recursively
    
    int        inbr;
    int        islot;
    int        nbrdrop,totdrop;
    struct tcdatatype    *tc;
    struct tcitemtype    *item;

    tc=&tcdata[tcslot];

Post comment:
This is the data structure where I keep treasureclass.exe info.


    // empty TC, nodrop=100%
    if(tc->nbr==0)
    {
 // set full chance to no drop
 markitem(maintcslot,-1,chance);
 return;
    }

Post comment:
tc->nbr would be the number of entries a TC has.



    // multiple items (can be merged with abs() )
    if(tc->picks>1)
    {
 // random picks
 chance*=(double)tc->picks;

    }
    if(tc->picks<-1)
    {
&nbsp;// negative picks
&nbsp;chance*=-(double)tc->picks;
    }
//    if(tc->picks>0)
//    {
// &nbsp;markmaxitems(maintcslot,tc->picks);
//    }

    // nodrop
    if((tc->nodrop!=0) && (tc->picks>0))
&nbsp;markitem(maintcslot,-1,chance*tc->nodrop/(double)tc->totprob);

    // Loop through all entries in TC
    nbrdrop=0;
    totdrop=0;
//    if(tc->picks<-1)
&nbsp;// negative picks
// &nbsp;totdrop=-tc->picks;


Post comment:
Below I loop through all the entries in the TC. Although my data handle it, this drop calculations does not handle unique and set items in the data fully. It do handle uniques though (so that the anihilus is handeled) by simply marking the base item of the unique as droping.


    for(inbr=0;inbr<tc->nbr;inbr++)
    {
&nbsp;if((tc->picks>0) || (nbrdrop<-(tc->picks)))
&nbsp;{
&nbsp;    item=&(tc->items[inbr]);
&nbsp;    nbrdrop+=item->prob;
&nbsp;    switch(item->type)
&nbsp;    {
&nbsp; &nbsp;case 0 :
&nbsp; &nbsp;    // an item
&nbsp; &nbsp;    markitem(maintcslot,item->slot,chance*item->prob/(double)tc->totprob);
&nbsp; &nbsp;    break;
&nbsp; &nbsp;case 1 :
&nbsp; &nbsp;    // another TC
&nbsp; &nbsp;    doonetc(maintcslot,item->slot,chance*item->prob/(double)tc->totprob);
&nbsp; &nbsp;    break;
&nbsp; &nbsp;case 2 :
&nbsp; &nbsp;    // unique item
&nbsp; &nbsp;    islot=getislot(udata[item->slot].abr);
&nbsp; &nbsp;    markitem(maintcslot,islot,chance*item->prob/(double)tc->totprob);
&nbsp; &nbsp;    break;
&nbsp; &nbsp;default :
&nbsp; &nbsp;    // something else
&nbsp; &nbsp;    break;
&nbsp;    }
&nbsp;}
    }

Post comments:

Below is were I scale for the 6 cap.


    // modify for more than 6 items (positive picks)
    if((tdrop[tcslot].fdrop>6.0) && (tc->picks>0))
    {
&nbsp;for(islot=0;islot<nbrislot;islot++)
&nbsp;{
&nbsp;    if(tdrop[tcslot].idrop[islot]>0)
&nbsp;    {
&nbsp; &nbsp;tdrop[tcslot].fidrop[islot]=6.0*tdrop[tcslot].fidrop[islot]/tdrop[tcslot].fdrop;
&nbsp;    }
&nbsp;}
&nbsp;tdrop[tcslot].fdrop=6.0;
    }
} // doonetc


void dotcprob(void)
{
    int &nbsp;tcslot;

    for(tcslot=0;tcslot<nbrtcslot;tcslot++)
    {
&nbsp;doonetc(tcslot, tcslot, 1.0);
    }
} // dotcprob

Here I simply loop all TCs and do the calcs for each one.

Have fun. Critisism or ideas are welcome.

[Ruvanal]

Jarulf:

Also if one want to add quality calcs, I have not done it since I was unsure how one would handle sub TCs having their own unique/set/rare/magic values. In addition, if different sub TCs have different such values, what happens?

It seems that you forgot about the answer from this thread that you started.
http://phrozenkeep.it-point.com/forum/view...pic.php?t=19359

The current path to an item will end up using the highest of each of the values from the TCs that were traversed to to get to the final item (code).

There were also a couple of threads that dealt with the probabilities of of a drop like the Countess's with its negative starting pick and more than six item potential before being finished. Unfortunately the best one that I saw on the probability of her drops was on the diabloii.net 1.10beta forum; but that in gone now.

[Jarulf]

Jarulf:

Also if one want to add quality calcs, I have not done it since I was unsure how one would handle sub TCs having their own unique/set/rare/magic values. In addition, if different sub TCs have different such values, what happens?

It seems that you forgot about the answer from this thread that you started.
http://phrozenkeep.it-point.com/forum/view...pic.php?t=19359

The current path to an item will end up using the highest of each of the values from the TCs that were traversed to to get to the final item (code).

Hmm, so different items in a TC can then use different values. And actually, the same item can use different values, depending on which path it took. OK.

Silly TC system :)

[hakai_no_tenshi]

While going through the drop chances for Tyreal's Might, I noticed two bugs:

1) For the released version, only one type of minion showed up as being able to drop it in the Pit. This was due to me misunderstanding how the "minion1" and "minion2" columns in monstats.txt work. I've fixed that now and will upload later.

2) For the test version posted above, I forgot to take the monster grade and difficulty level into account which resulted in some drop pathways being merged with others.

Both have been fixed and will be uploaded later.

--T

[Ruvanal]

Hmm, so different items in a TC can then use different values. And actually, the same item can use different values, depending on which path it took. OK.

Silly TC system :)

Actually it is a very robust system. Capable of far more than was actually used in the v1.10 release or in what have 'seen' being used in the mods described at Phrozen Keep.

For the TC table that is used in the v1.10 release, it looks to me like that in most cases that any path that will result in a weapon or armor from a starting TC will traverse the same path through any of the TCs that modify the cu, cs, cr and cm values. The same starting TC may have a bit different path to the miscelaneous items though, but it will probably make no difference. From what I can see (without spending a couple of hours pouring over the entire table), in all but a few cases the highest settings for the cu, cs, cr and cm values will come from the initial TC that is called to generate the drop. Those few exceptions look to be the ones for a handful of SuperUniques like Pindleskin wih their <SU-TC>x listings that branch to the "Uitem" listings which in turn boost the cu and cs values back up to what the other SuperUniques are also using.

[Jarulf]

Hmm, so different items in a TC can then use different values. And actually, the same item can use different values, depending on which path it took. OK.

Silly TC system :)

Actually it is a very robust system. Capable of far more than was actually used in the v1.10 release or in what have 'seen' being used in the mods described at Phrozen Keep.

I was mostly refering to the fact that it is quite complex and that I had not remembered it all :)

Yes, currently, for most ers it is really not a problem. Only a handfull (or less) monsters would have a negative pick at any palce other than the main TC of them for example.

[Samka]

Sorry about the OT, but all of this talk about TC paths is confusing me. My question is : do any TC drop probabilities change in proportion depending on the player count?

[Jarulf]

Sorry about the OT, but all of this talk about TC paths is confusing me. My question is : do any TC drop probabilities change in proportion depending on the player count?

Well, no, only the "nodrop" (if present) part would change.

[Samka]

Well yes, but presumably the other TC probabilities would be adjusted to fill that gap. Would they all be adjusted equally then?

[Jarulf]

Well yes, but presumably the other TC probabilities would be adjusted to fill that gap. Would they all be adjusted equally then?

No, they remain the same. The "total" probability, that is the one ofr each entry plus the no drop will change though. Basically, think of it as if the game recalculate the value in the no drop column on the fly for each drop based on number of people in the game.

[Jeger]

ATMA is not for BattleNet, right?

[Pansesus]

ATMA is not for BattleNet, right?

Actually you can use ATMA for open battlenet or TC/ICP games, as the d2s files from which are saved in your computer.

[hakai_no_tenshi]

I have not looked at this point (do not have the time to try to check), but did you take into account that there is a maximum of 6 items that can be dropped from a single initial call to the first TC?&nbsp; In other words there is a 6 item cap on the drops of the act end bosses and the countess (the only ones that I recall off the top of my head that end up with more than 6 picks for items).

I am currently working on doing this properly ... it's amazing what a pain in the royal behind her TC is but I believe the others are as accurate as they come. Anyway, Thrugg just sent me email outlining what would have to be done:

t's easyish if you know the right numbers at the right time.&nbsp; It might
be trickier depending on what you are storing at the time you need
to do this calculation.

This will be easier with an example, so I might as well use a topical
example - Countess Rune.&nbsp; This has 3 picks, at players 1 it has 5/20
no-drop and 15/20 going on to Runes 12.&nbsp; And let's say the chance of
getting a Tir assuming we go to Runes 12 is 1/5 (I know it isn't
but to make the numbers easier).&nbsp; With complete freedom (ie unlimited
drops allowed), the chance of getting a Tir from the whole TC is
(1 - (1 - (15/20 * 1/5))^3) = 0.385875
However, this number is counting the possibility that you get two
non-Tir drops and then a Tir with the third drop.&nbsp; Suppose you
had a max of 2 drops to come, you have to discount that possibility.
The chance of that is ((15/20)*(4/5))^2 * ((15/20)*(1/5)), or
more neatly (15/20)^3 (4/5)^2 (1/5).&nbsp; This is very similar to binomial,
the difference is you only want to count the exact non-Tir non-Tir Tir
combination (not "Tir non non" or "non Tir non") so there is no
(3 choose 1) factor at the start.

Pr (Tir | 3 drops remaining)
= (1 - (1 - (Pr(drop) * Pr(Tir | drop)))^3)
= 0.385875

Pr (Tir | 2 drops remaining)
= Pr (Tir | 3 drops remaining)
&nbsp; - Pr (non non Tir)
= above
&nbsp; - (Pr(drop)^3 * (1-Pr(Tir | drop))^2 * Pr(Tir | drop))
= 0.331875

Pr (Tir | 1 drop remaining)
= Pr (Tir | 2 drops remaining)
&nbsp; - Pr (non Tir anything)
&nbsp; - Pr (nodrop non Tir)
&nbsp; - Pr (non nodrop Tir)
= above
&nbsp; - (Pr(drop)^2 * (1-Pr(Tir | drop)) * Pr(Tir | drop))
&nbsp; - (Pr(nodrop) * Pr(drop)^2 * (1-Pr(Tir | drop)) * Pr(Tir | drop))
&nbsp; - (Pr(nodrop) * Pr(drop)^2 * (1-Pr(Tir | drop)) * Pr(Tir | drop))
= above - ((1 + 2 Pr(nodrop)) * Pr(drop)^2 * (1-Pr(Tir | drop)) * Pr(Tir | drop))
= 0.217125
("anything" includes no-drop, non and Tir)

Pr (Tir | 0 drops remaining)
= Pr (Tir | 1 drop remaining)
&nbsp; - Pr (Tir anything anything)
&nbsp; - Pr (nodrop Tir anything)
&nbsp; - Pr (nodrop nodrop Tir)
= above
&nbsp; - (Pr(drop) * Pr(Tir | drop))
&nbsp; - (Pr(nodrop) * Pr(drop) * Pr(Tir | drop))
&nbsp; - (Pr(nodrop)^2 * Pr(drop) * Pr(Tir | drop))
= above - ((1 + Pr(nodrop) + Pr(nodrop)^2) * Pr(drop) * Pr(Tir | drop))
= 0 (as you would expect).

You could go in the opposite direction instead of course, start with
0 and add combinations on.&nbsp; When I do such a thing by hand I start
from both ends, because it gets more complex as you get towards the middle,
but a computer can't do that as easily.

--T

[Jarulf]

What is your actual algorithm for going through TCs calculting the probabilities? (If you want to keep it generally secret, perhaps discussions with PM would be OK). I think the recursive way I have done it is good, simple and works. For the negative pick AND going above 6 items, it need to be adjusted (I think it handle all other cases except possibly handling quality determination if it various with how you get to an item).

The way I would solve it if I had time, would be to have an additional parameter passed that would calculate the chance of an item droping from a sub TC (which would be the chance of a non "no drop". Keeping track of this and summing it up as we go (for negative ones, otherwise we adjust at the start so we get a 6 total chance). If we then get to a situation were we allready are above 6, that chance above would be the chance that we have to have the drop as a "nothing" instead of an actual item. That is, we multiply the chance for a drop by the chance we have not yet reached 6 items). I think it would not require too much editing of the algorithm I use. Remember, I want to simply calculate the probability of a "per kill" chance for each item type.

If you do something very different, it may need some other way to handle it. It just seems you are attacking the problem slightly different than I do and perhaps a bit too complicated, but perhaps you can keep track of more things too :)

At least we can compare results. I think for "normal" easy cases, we (Thrugg and I) allready agree on probabilities and thus we should have it correctly donw, both of us.

[Thrugg]

The algoritm is basically to go through the TC, entry by entry and keep track of the probabilities until one reach base items. This is done by a recursive function that takes a probability value as its entry (and some other stuff). When we start of, we enter with a prob=1.0 value. When then call the function for any sub TC, and pass along prob=prob*abs(picks)*(chance/totchance), that is, we calculate the probability of that specific being picked and multiply by the probability that we have so far (that is having taken the path so far to the sub TC we are at). We also multiply by the picks value since if it is larger than 1, we will have multiple entries form here on). When we reach an entry that is instead an item, we store that "prob" value on the item (adding up all such prob values reached). We also store a total probability value which would indicate number of items. If this total value is larger than 6, we can scale all values down so that it becomes 6 (for all poistive picks only). This works great from what I have seen with the exceptions I have yet to implement mentioned above.

There is a difference between what you are calculating and what Hakai wants.
By simply summing the partial probabilities you will get a number which is effectively the expected number of drops of item X from monster Y.
This is not exactly the same as the chance that monster Y will drop item X.

As a simple example, suppose these TCs

subTC1 1 ItemX 1 ItemY 1
subTC2 1 ItemX 1 ItemY 1
DropTC -2 subTC1 1 subTC2 1

The expected number of ItemX to drop here is exactly 1, 1/2 via subTC1 and 1/2 via subTC2. If you create 100 drops of DropTC you expect around 100 ItemXs.
However, there is no certainty that ItemX will fall on a single drop - in fact 1/4 of the time you will get 2 ItemYs.
The actual chance of ItemX dropping on a given instance of DropTC is not (1/2) + (1/2), but in fact (1 - (1 - 1/2)*(1 - 1/2)), as Hakai outlined in his post above.

For very very small probability drops, the two numbers are practically identical. The difference is the chance of getting 2 of the item at once, for example. Not very significant for Windforces. But more of an issue for Super Healing Potions from Pindle or TP scrolls from Duriel.

Which is more useful is also hard to answer. If you want to know your chance of finding a Windforce if you kill Pindle 1000 times, the chance per kill is what you need. If you want to know how many Um runes you expect from killing the Countess 1000 times, the expected number per kill is.

For a drop-calc that is to go to the general public, who won't know or probably care about this distinction, I think "the chance of one dropping per kill" is easier to understand than "the expected number of drops per kill". But as we have discovered it is a little harder to work out :P

BTW Samka, yes, the relative ratios of actual dropped items do not change with number of players, only the no-drop number changes; so for example a Wraith will drop 75 no-drops to 4 Good TC drops to 25 Magic TC drops in a single player game, but 31 to 4 to 25 in a two player game. The odds of a Good drop go up from 4/104 to 4/60, but there is still 4 Good drops for every 25 Magic drops.

Oh, and, hi all :)

[Jeger]

Are the actual treasure dropping routines/equations discussed anywhere? The sites that I have found speak in generalities and sometimes show tables, but don't give the actual calculations. [I might like to develop my own calculator.) :P