Stamina Breakpoints?

Last week I investigated two different situations where Simcraft was giving a user stat weights that seemed wrong.  These turned out to be pretty interesting, and are situations that other players could encounter at some point, so I decided it was worth writing a blog post to describe what happened.

The First Puzzle

The first quandary was posted on maintankadin, as a user asked why they were getting stat weights that looked like this:

Mysterious stat weights

What is this I don’t even….

You can imagine why the player was confused.  Stamina, armor, and mastery were all causing their TMI value to increase (which is bad). And it’s hard to imagine a situation where any of those three would have a negative effect on your survivability.  Remember that these are stat weights, so they’re generated by adding a free 1000 of each stat and comparing to a baseline measurement.  It’s not a simple case of these stats being less effective than other stats, it literally means that adding 1000 stamina made him more likely to die!  Since when will adding extra mitigation or health get you killed?

Well, it turns out, there’s a situation where that’s exactly what happens.

While looking into this problem, I first double-checked all of the TMI calculations in the code.  Everything was still working exactly as it should, so the problem wasn’t with the model.  It had to be related to the inputs and outputs.   So to test, I tried adding a free 1000 stamina to the player’s character with the tabard slot.  And all of the sudden, his stat weights looked fine:

yuko_1kstam

That looks better! But… why?

This suggested that we’re hitting something of a stamina break point.  You’re probably familiar with the concept of haste breakpoints or crit breakpoints – points when all of the sudden, a certain stat becomes more or less valuable due to a mechanical reason.  In fact, you’re probably even more familiar with a few other stat breakpoints: hit caps, expertise caps, and the block cap that we all danced around in Cataclysm are all examples of stat break points.

But…. stamina?  What in the world could cause a stamina break point?

So I ran a few more simulations and scrutinized the results, looking for anything in the report that changed significantly from one run to the next.  Here’s what I came up with:

          DP uptime   SotR uptime  HP_T15_4pc    TMI     Stat Weights
Base        25.82%       86.50%      136.07     611.6    Funky   (Sta/Armor negative)
+250 stam   25.81%       86.54%      136.00     574.2    Funky   (Sta/Armor negative)
+500 stam   25.46%       83.22%      134.26     819.5    Normal  (Hit>Exp>Sta>Mast>Armor>Str>Haste)
+1k stam    25.08%       83.18%      134.24     713.2    Normal  (Hit>Exp>Sta>Mast>Armor>Str>Haste)
FS->DP      24.59%       74.50%      129.50     676.4    Normal? (Hit>Exp>Haste>Sta>Str>Mast>Armor)

When we went from +250 to +500 stamina, we had an abrupt increase in TMI and suddenly got different stat weights.  The source seemed to be a sudden drop in SotR uptime, to the tune of over 3%.  But that drop didn’t come from a change in our usual holy power generators.  In fact, it came from the T15 4-piece bonus!

You can probably already guess the mechanism that we’re seeing here.  The tier 15 4-piece bonus grants one holy power for every 20% of your health taken as damage during Divine Protection.  That 20% threshold depends on your stamina, though.  So if you increase your stamina, it takes a slightly larger hit to qualify for that holy power gain.  In this case, by adding ~300 stamina, we were creating a situation where an attack that was granting Holy Power before was no longer doing so, leading to a survivability loss and strange stat weights.

Keep in mind that the data Simulationcraft was giving was correct given the model we’re using.  By adding 1000 stamina, the player’s survivability was going down.  Bizarre, but true.  However, in practice this issue is essentially irrelevant.  Before we get to why that is, let’s take a look at the second puzzle.

The Second Puzzle

The next puzzle was a little more subtle.  Trompu posted in the comments of last week’s Simulationcraft 530-6 post saying they were observing strange results.  In this case, the calculated stat weights against the LFR boss looked like this:

palarus_funky

Again with the funky stat weights….

So of course, my first guess was that the T15 bonus was at fault.  But when I looked at the report, it was clear that he wasn’t even using the 4-piece bonus.

In retrospect, there’s something else on this plot that should have been a clue that the problem was elsewhere.  The mastery score isn’t negative.  That’s not sufficient evidence that the problem isn’t the 4-piece, but it certainly makes it a less likely cause.

So I dug around the reports for a while.  A long while.  In fact, after an hour or so I got a little embarrassed that I couldn’t figure out what was causing it.  Nothing seemed that significantly different in the reports.  It was clearly some sort of stamina break point, but nothing I could find in the report pointed at what game mechanic was triggering that abrupt dependence on player health.  Defeated, I e-mailed Meloree asking if he could think of anything.

Five minutes later, I got a reply that made me feel like a moron.  As Mel kindly pointed out, I had tweaked the action priority list to perform a shifting queue by using the conditional:

shield_of_the_righteous,if=(holy_power>=5)|(incoming_damage_1500ms>=health.max*0.3)

Which, quite clearly, depends on player health.  Oops.

So in this situation, the break point wasn’t in the game mechanics at all.  It was in the logic we told the simulation to obey while casting spells.  Once we added enough stamina, we stopped using an SH1 queue and simply started spamming SotR at 5 holy power (an “S” queue), because the boss was no longer hitting us for more than 30% of our health!

The important point here is that these stamina breakpoints aren’t always due to a specific mechanic.  Any time we have some sort of dependence on player health, we can get strange discontinuous effects.

Stamina Breakpoints – Are They Real?

To illustrate what’s happening here a little more clearly, let’s look at some of the data from the second puzzle.  I this case, I’ve plotted TMI against the additional stamina I was providing via the tabard slot.

palarus_tmiplot

TMI plotted against added stamina.

In this plot, adding stamina has a very clear benefit on either side of the discontinuity, meaning that it’s still providing added survivability.  It’s only at the discontinuity itself that it’s behaving badly.  Unfortunately, it behaves really badly, giving a fairly significant increase in TMI.  But there are a few reasons that this result isn’t as significant when you play as the plot makes it appear.

For this particular example, the break point was caused by the action priority list logic.  In other words, we were telling the simulation to play very differently after we added ~760 stamina.  But in reality, players don’t play like that.  You’d be very hard-pressed to find a player who can discern between a hit that’s 30% of their health and one that’s 29% of their health.  A real player would likely hit the SotR button in both of those situations, not just in the first case.

But there’s a more subtle reason that this effect isn’t going to happen in-game, and it has to do with what I said earlier about Simcraft’s results being “correct, as far as the model goes.”  You see, when we defined the TMI bosses, we gave them a base damage but no range.  So they always hit for exactly the same amount.

In fact, one of the options in SimC is to use an average damage amount rather than a range.  By default this option is turned on, so your abilities will always do a fixed amount of damage.  So even if we had defined a range, the boss would’ve been hitting for exactly the same amount each time.

Of course, there’s an additional complication: we didn’t define a boss’s damage range because as of version 530-6, there isn’t a way to do it with the action priority list!  This wasn’t a big deal in the past because very few players used SimC to simulate tanks, and DPS players don’t care how hard the boss hits.  But now that tanks are using SimC, it’s suddenly become more important.

Why? Well, because when the boss hits you for a variable amount of damage, it “softens” these stamina breakpoints, often to the point that they no longer exist.  Since the boss doesn’t always hit for exactly X, the stamina break points are no longer concentrated at an exact stamina value.  Instead, they’re spread out because sometimes the boss hits for slightly more or less than the damage value associated with that break point.  Mathematically, this is basically performing the convolution of the boss’s damage distribution with the plot above.

If you take a look at some logs, you’ll see that the average boss’s damage range is huge, often larger than +/- 20% of the mean damage value.  That’s a large enough range to completely obliterate this type of sharp break point.  Rather than a sharp loss occurring at x=760  stamina, it spreads that loss out over a range of tens of thousands of stamina.  That makes stamina slightly less valuable in that range than it normally would be, but by a relatively small amount.

So in practice, you shouldn’t have to worry about whether you’re crossing a stamina break point, because the effects are almost never as abrupt as we’re seeing in these examples.  The only case where it will happen is with an attack that does a very specific amount of damage with no range variation, and those encounters are fairly rare.  Incidental absorption bubbles (ex: Illuminated Healing) are often going to be enough to make even those break points inconsequential.

Solution

What do we do about this problem? Simcraft is giving us results that, while correct for the model, aren’t really reflecting the reality of tanking.  Well, the answer is pretty obvious: we force the boss’s attacks to vary in size.

So that’s exactly what I did.  I’ve added code to force the boss’s melee attacks to vary by +/- 20% of the base value regardless of whether the global “average_results” option is set or not.  That should automatically smooth these sorts of effects under the vast majority of circumstances.  I’ve also added code that allows you to define an attack’s damage range in case you don’t like the default 20%.  I’ll provide more documentation on those details in the version 530-7 blog post when that version is finally released.

In any event, this was a fascinating look into a subtle aspect of the game’s mechanics.  The fact that these break points exist, even if they’re relatively rare and unimportant in practice, is just another example of the depth involved in modeling tanking.  And another reminder that we occasionally have to deviate from the traditional methods to deal with that depth, because sometimes it invalidates a simplifying assumption that made perfect sense in another context.

This entry was posted in Tanking, Theck's Pounding Headaches, Theorycrafting and tagged , , , , , , , , , , , , , . Bookmark the permalink.

12 Responses to Stamina Breakpoints?

  1. flosch says:

    Heh. Halfway through the post, I thought “why can’t he just add probabilistic triggering of SotR, weighted by damage?”. Then you end up randomizing boss damage (which I had totally forgotten was static by default), something similar but arguably better. Though I guess it’ll mean you’ll have to simulate longer to reach stable results that aren’t skewed by potential outliers? (Which of course also would hold true for randomized SotR; just wondering in general.)

    • Theck says:

      Yeah, it means you’ll have slightly larger variation at any fixed iteration size. But it’s not that bad, 1k-10k iterations still gives you very reasonable results.

  2. queldan says:

    Quick thought – by instinctively normalizing the bump, wouldn’t that imply that Sta’s real value is slightly lesser than it seems on Sim? And that we’ll see it when the boss’ damage gets normalized?

    • Theck says:

      In the vicinity of that break point, yes. Rather than having a strong positive value on either side of the discontinuity and a negative value when you cross the discontinuity, it will just have a slightly lower value whenever you’re near the break point. When you’re farther away from that break point, it’ll have almost no effect.

  3. Raed says:

    Are you stacking haste just for the dps increase? I notice on Ask Mr Robot that it has me around 20/21% haste but stacking more stam at that point. My rotation is pretty solid so I have solid uptime of SS and SotR. Thanks

    • Theck says:

      I (and many other tanks) stack haste for both DPS and Survivability. Haste’s survivability benefit makes the opportunity cost of trading stamina for haste very low.

  4. Marsbubble says:

    Theck, you could go further by adding multiple sources of boss damage. I think a more realistic situation would be not only consistent attacks for an avg. amount (that varies), but also a special ability that generally causes a spike (trip punc, snap bite, etc). If you don’t include the spikey sorts of damage that exist in a real raid setting, the model is going to be artificially biased towards smoothing. I haven’t gotten into the sim code so If you are already doing this, disregard.

    • Theck says:

      The problem with adding large spikes is that it makes the metric much more volatile and harder to compare between classes. If you have a large damage spike then the bulk of your TMI score will hinge on how well you prepare for that spike in the APL. You could also cheese it by just throwing every cooldown you have at it, which isn’t very realistic either. And some classes have an inherent advantage for that spike depending on the damage type – paladins have a pretty strong advantage for physical damage, DKs probably have the best toolkit for magical damage.

      There are also quite a few arguments that large, predictable spikes aren’t generally the cause of death for good tanks. I’ve provided many of those arguments in earlier blog posts, but in short: the spikes you prepare for are fairly binary. You either use AM successfully and live, or screw up and die. And the sim doesn’t screw up unless you write a bad APL, so the information you get by including large spikes isn’t actually that useful.

      Everyone has to deal with boss melees and ticking DoTs though, so those are good ways to test spike susceptibility in a general sense.

      That doesn’t mean you can’t put together a spike-laden boss and try to optimize your TMI against it. It’s easy enough to give the boss a spell_nuke or melee_nuke ability and tailor the APL to that boss. But the results are going to be a lot less general.

      • Çapncrunch says:

        Hmmm… I understand your points, that if implemented poorly the addition of burst damage could lead to a model that biases the results in a particular manner, especially with the differences between tanks. However, I don’t necessarily agree that this information wouldn’t be useful, if interpreted and used correctly. I agree that there probably isn’t a good way to implement bursts that will be useful in a “general” form, especially across multiple tank classes. However, I still think it would be useful in the sense of “this boss does this type of spike every so often, what stats are better for this situation, and/or which tanks are better suited for the fight”.

        That sort of question is still something that people are concerned about, how to handle the specific in addition to the general. As long as people understand when looking at more specialized sim results not to apply them to a general case, the information gained is both useful and important. Now the number of possible variations to look at probably goes beyond what we can reasonably expect to be supplied for built-in boss models (especially when you consider different spike types/sizes/frequency and then across the different raid size/difficulty settings). Ideally it’d be nice if there was a set of boss options that could be used to adjust the boss’ melee damage, magic damage, and spike size/type/etc, but I know that’s probably a lot to ask for (so I won’t), so we’ll probably just want to edit the boss ability list when interested in this sort of thing.

        However, perhaps I could ask for maybe a quick run-down on how to put together a boss profile for those of us less SimC-savy? While I haven’t been very exhaustive in my search, just about everything I’ve found is about customizing player-related events rather than boss abilities (probably because the boss’ ability list isn’t particularly important for dps sims which have probably gotten the most attention in the past).

  5. Çapncrunch says:

    Uh, I just had an interesting result in SimC, that’s related to my past concerns regarding Mastery’s value in the sim.

    After a discussion on the official forums regarding the value of haste particularly in regard to less skilled players (the person doubted that setting the player skill to “fire is hot” adequately represented just how poorly some people could be), so to try to emphasize the passive benefits of haste I re-simmed my character with a single change to the ability list: I commented out SotR entirely. (I realize that this still leaves the rotation assuming proper uptime of SS, which some players do not, and that removing SotR also takes away the value of grand crusader procs from avoidance stats, but figured it should still be a good demonstration of the passive value of haste vs avoidance). And as I expected Haste’s value dropped a lot (and lol, over 200M tmi against a 15n boss at 531 ilevel 😛 ), but it still crushed the other secondary stats, but the worrysome thing was the value I got for mastery.

    Here’s the weights I got:
    Stam: 1 (of course)
    armor: 0.47
    strength: 0.34
    haste: 0.32
    dodge: 0.22
    parry: 0.18
    mastery: 0.00

    According to the sim Mastery has apparently no benefit whatsoever without SotR. It didn’t even get a pink bar in the scale factor chart. I double checked and I definitely had mastery checkboxed to be weighted, and it has a value in the “error” row of the chart (while the excluded stats have no error given). This seems very….odd. It suggests that adding 1000 mastery worth of block chance did absolutely nothing. My previous concerns were that the SotR component of mastery was being undervalued without any big hits for it to mitigate, but this result almost shows the opposite: mastery’s passive value is only coming from SotR and block% seems to hold no value at all.

    Now I considered that maybe it was an anomalyse result, or maybe the benefit of just 1000 mastery worth of block was too small to show up, so I redid the sim with scale_delta_multiplier=3 in the overrides tab to see if adding more mastery would give it a measurable result, but I got the same thing: haste still crushes dodge/parry and mastery still comes up with a 0.00 weight (and again no pink bar in the scale factor chart which). I just don’t understand it, I mean I haven’t included crit since you first posted the SimC 101 article, but from what I recall crit still gave some results, even if they were sometime negative, it still had a pink bar, and statistical noise still often made it non-zero, yet mastery seems to just flatly come out as exactly zero when SotR is ignored.

    Frankly I’m just confused. The results definitely show that the sim IS blocking, and the correct block% is in the stats section, but apparently increasing block% is having no apparent effect on survival.

    • Theck says:

      Yeah, about two weeks ago I found a fairly significant bug with blocking. Blocks were being rolled, but the mitigation wasn’t being applied properly. It’s fixed in 530-7, which was released today.

Leave a Reply to flosch Cancel reply