Last week I investigated two different situations where Simcraft was giving a user stat weights that seemed wrong. These turned out to be pretty interesting, and are situations that other players could encounter at some point, so I decided it was worth writing a blog post to describe what happened.
The First Puzzle
The first quandary was posted on maintankadin, as a user asked why they were getting stat weights that looked like this:
You can imagine why the player was confused. Stamina, armor, and mastery were all causing their TMI value to increase (which is bad). And it’s hard to imagine a situation where any of those three would have a negative effect on your survivability. Remember that these are stat weights, so they’re generated by adding a free 1000 of each stat and comparing to a baseline measurement. It’s not a simple case of these stats being less effective than other stats, it literally means that adding 1000 stamina made him more likely to die! Since when will adding extra mitigation or health get you killed?
Well, it turns out, there’s a situation where that’s exactly what happens.
While looking into this problem, I first double-checked all of the TMI calculations in the code. Everything was still working exactly as it should, so the problem wasn’t with the model. It had to be related to the inputs and outputs. So to test, I tried adding a free 1000 stamina to the player’s character with the tabard slot. And all of the sudden, his stat weights looked fine:
This suggested that we’re hitting something of a stamina break point. You’re probably familiar with the concept of haste breakpoints or crit breakpoints – points when all of the sudden, a certain stat becomes more or less valuable due to a mechanical reason. In fact, you’re probably even more familiar with a few other stat breakpoints: hit caps, expertise caps, and the block cap that we all danced around in Cataclysm are all examples of stat break points.
But…. stamina? What in the world could cause a stamina break point?
So I ran a few more simulations and scrutinized the results, looking for anything in the report that changed significantly from one run to the next. Here’s what I came up with:
DP uptime SotR uptime HP_T15_4pc TMI Stat Weights Base 25.82% 86.50% 136.07 611.6 Funky (Sta/Armor negative) +250 stam 25.81% 86.54% 136.00 574.2 Funky (Sta/Armor negative) +500 stam 25.46% 83.22% 134.26 819.5 Normal (Hit>Exp>Sta>Mast>Armor>Str>Haste) +1k stam 25.08% 83.18% 134.24 713.2 Normal (Hit>Exp>Sta>Mast>Armor>Str>Haste) FS->DP 24.59% 74.50% 129.50 676.4 Normal? (Hit>Exp>Haste>Sta>Str>Mast>Armor)
When we went from +250 to +500 stamina, we had an abrupt increase in TMI and suddenly got different stat weights. The source seemed to be a sudden drop in SotR uptime, to the tune of over 3%. But that drop didn’t come from a change in our usual holy power generators. In fact, it came from the T15 4-piece bonus!
You can probably already guess the mechanism that we’re seeing here. The tier 15 4-piece bonus grants one holy power for every 20% of your health taken as damage during Divine Protection. That 20% threshold depends on your stamina, though. So if you increase your stamina, it takes a slightly larger hit to qualify for that holy power gain. In this case, by adding ~300 stamina, we were creating a situation where an attack that was granting Holy Power before was no longer doing so, leading to a survivability loss and strange stat weights.
Keep in mind that the data Simulationcraft was giving was correct given the model we’re using. By adding 1000 stamina, the player’s survivability was going down. Bizarre, but true. However, in practice this issue is essentially irrelevant. Before we get to why that is, let’s take a look at the second puzzle.
The Second Puzzle
The next puzzle was a little more subtle. Trompu posted in the comments of last week’s Simulationcraft 530-6 post saying they were observing strange results. In this case, the calculated stat weights against the LFR boss looked like this:
So of course, my first guess was that the T15 bonus was at fault. But when I looked at the report, it was clear that he wasn’t even using the 4-piece bonus.
In retrospect, there’s something else on this plot that should have been a clue that the problem was elsewhere. The mastery score isn’t negative. That’s not sufficient evidence that the problem isn’t the 4-piece, but it certainly makes it a less likely cause.
So I dug around the reports for a while. A long while. In fact, after an hour or so I got a little embarrassed that I couldn’t figure out what was causing it. Nothing seemed that significantly different in the reports. It was clearly some sort of stamina break point, but nothing I could find in the report pointed at what game mechanic was triggering that abrupt dependence on player health. Defeated, I e-mailed Meloree asking if he could think of anything.
Five minutes later, I got a reply that made me feel like a moron. As Mel kindly pointed out, I had tweaked the action priority list to perform a shifting queue by using the conditional:
Which, quite clearly, depends on player health. Oops.
So in this situation, the break point wasn’t in the game mechanics at all. It was in the logic we told the simulation to obey while casting spells. Once we added enough stamina, we stopped using an SH1 queue and simply started spamming SotR at 5 holy power (an “S” queue), because the boss was no longer hitting us for more than 30% of our health!
The important point here is that these stamina breakpoints aren’t always due to a specific mechanic. Any time we have some sort of dependence on player health, we can get strange discontinuous effects.
Stamina Breakpoints – Are They Real?
To illustrate what’s happening here a little more clearly, let’s look at some of the data from the second puzzle. I this case, I’ve plotted TMI against the additional stamina I was providing via the tabard slot.
In this plot, adding stamina has a very clear benefit on either side of the discontinuity, meaning that it’s still providing added survivability. It’s only at the discontinuity itself that it’s behaving badly. Unfortunately, it behaves really badly, giving a fairly significant increase in TMI. But there are a few reasons that this result isn’t as significant when you play as the plot makes it appear.
For this particular example, the break point was caused by the action priority list logic. In other words, we were telling the simulation to play very differently after we added ~760 stamina. But in reality, players don’t play like that. You’d be very hard-pressed to find a player who can discern between a hit that’s 30% of their health and one that’s 29% of their health. A real player would likely hit the SotR button in both of those situations, not just in the first case.
But there’s a more subtle reason that this effect isn’t going to happen in-game, and it has to do with what I said earlier about Simcraft’s results being “correct, as far as the model goes.” You see, when we defined the TMI bosses, we gave them a base damage but no range. So they always hit for exactly the same amount.
In fact, one of the options in SimC is to use an average damage amount rather than a range. By default this option is turned on, so your abilities will always do a fixed amount of damage. So even if we had defined a range, the boss would’ve been hitting for exactly the same amount each time.
Of course, there’s an additional complication: we didn’t define a boss’s damage range because as of version 530-6, there isn’t a way to do it with the action priority list! This wasn’t a big deal in the past because very few players used SimC to simulate tanks, and DPS players don’t care how hard the boss hits. But now that tanks are using SimC, it’s suddenly become more important.
Why? Well, because when the boss hits you for a variable amount of damage, it “softens” these stamina breakpoints, often to the point that they no longer exist. Since the boss doesn’t always hit for exactly X, the stamina break points are no longer concentrated at an exact stamina value. Instead, they’re spread out because sometimes the boss hits for slightly more or less than the damage value associated with that break point. Mathematically, this is basically performing the convolution of the boss’s damage distribution with the plot above.
If you take a look at some logs, you’ll see that the average boss’s damage range is huge, often larger than +/- 20% of the mean damage value. That’s a large enough range to completely obliterate this type of sharp break point. Rather than a sharp loss occurring at x=760 stamina, it spreads that loss out over a range of tens of thousands of stamina. That makes stamina slightly less valuable in that range than it normally would be, but by a relatively small amount.
So in practice, you shouldn’t have to worry about whether you’re crossing a stamina break point, because the effects are almost never as abrupt as we’re seeing in these examples. The only case where it will happen is with an attack that does a very specific amount of damage with no range variation, and those encounters are fairly rare. Incidental absorption bubbles (ex: Illuminated Healing) are often going to be enough to make even those break points inconsequential.
What do we do about this problem? Simcraft is giving us results that, while correct for the model, aren’t really reflecting the reality of tanking. Well, the answer is pretty obvious: we force the boss’s attacks to vary in size.
So that’s exactly what I did. I’ve added code to force the boss’s melee attacks to vary by +/- 20% of the base value regardless of whether the global “average_results” option is set or not. That should automatically smooth these sorts of effects under the vast majority of circumstances. I’ve also added code that allows you to define an attack’s damage range in case you don’t like the default 20%. I’ll provide more documentation on those details in the version 530-7 blog post when that version is finally released.
In any event, this was a fascinating look into a subtle aspect of the game’s mechanics. The fact that these break points exist, even if they’re relatively rare and unimportant in practice, is just another example of the depth involved in modeling tanking. And another reminder that we occasionally have to deviate from the traditional methods to deal with that depth, because sometimes it invalidates a simplifying assumption that made perfect sense in another context.