The last few posts have garnered a lot of comments, and some of them have indicated that there’s some misunderstandings about the simulations we’re performing. This set of simulations started back in September, and has been evolving ever since. And the rationale behind how we perform the simulations is spread out over 6 months of posts. It seemed like a good idea to consolidate that information in one place.
Basic Modeling Problems
First, let’s consider the point of modeling. Obviously we want to generate data that helps us make more informed decisions about how we gear and play. For modeling to be of any benefit, it has to generate information that’s useful. And generally, that means that you want the model to be accurate.
Most people would assume that means you want the model to be perfect – all-inclusive, with tanks and healers that play perfectly and react with mechanical precision. But in fact that’s not the case. Consider what would happen if we did have perfect tanks and perfect healers. Your perfect tanks would execute their rotation flawlessly, never miss an encounter cue, and would always apply their mitigation skills in the most optimum way. Your perfect healers would react flawlessly to spikes in damage intake, always select the correct heal at the right time, and would be as mana-efficient as possible. In your perfect model, you’d have at least enough healing to cover damage intake given perfect conditions, so your perfect healer would never run out of mana. And the net result would be a tank that never dies.
Unfortunately, that perfect simulation wouldn’t be very useful. If the tank plays perfectly and never dies, then it barely matters how they gear or what they do. They’ll be unkillable with any reasonable gear set. Which highlights the flaw with a perfect simulation: it’s not accurate. It doesn’t reflect reality. Because in real encounters, players make mistakes and tanks die. And there’s a fairly high correlation between those mistakes and tank deaths. Which means that if we want a good, useful simulation, we need to account for that imperfection.
Unfortunately modeling imperfection is almost as hard as modeling perfection is. For example, let’s consider how many ways there are to model an imperfect tank.
- Active Mitigation Usage – You could do it by varying the precision of their active mitigation. For example, a sloppy player might be macro-ing Shield of the Righteous to Crusader Strike – essentially the “S” rotation we’ve used in the past. On the other extreme, you could model a near-perfect tank that pools holy power and only blows SotR early during what seems to be a spike – the “SH1″ and “SH2″ rotations we’ve used. And there are hundreds of variations in-between that differ in effectiveness based on the logic used to control SotR usage. But which one of those best models a real tank?
- Rotation – Maybe your tank is perfect and follows CS>J>AS>SS to the letter. That’s pretty easy to model. On the other hand, maybe your tank slips up and prioritizes J over CS once in a while by accident. But how often do they do that? Are they just as likely to push AS ahead of CS as well, or forget to cast SS when it’s available? Again, we have an infinite number of variations based on the probability distribution you use to determine whether the player makes these errors.
- Latency – Maybe the player lives next to the server farm, and has sub-millisecond latency. On the other hand, maybe they’re playing from Australia, and regularly struggle with pings of 350 ms or greater. Cast queuing helps quite a bit to mitigate these problems, but it still affects events that you need to react to in-game, because even with perfect reactions you’re behind by your latency value. How much latency should our imperfect player have to deal with?
- Player Reaction – And even if you live next to the server farm, nobody’s reaction time is instant. So a player’s response to an event will always be delayed somewhat based on their reaction time and attentiveness. They may react more slowly to an event if they were watching some other information stream, like DBM timers or their action bars. How often does our imperfect tank stand in the fire?
It should be evident from this that there’s a huge parameter space to work in here. And the problem is that it’s very, very difficult to determine what’s “average.” A reaction time of a full second is certainly below average, but there are players who operate at that capacity. A completely perfect rotation is possible, but awfully difficult – arguably difficult enough that even the best players can’t manage it. Everyone slips up once in a while.
Meloree posted a great comment on the last article that I’d like to quote, because it seems fairly relevant here:
As far as imperfect play goes – during wrath/Cataclysm the best I managed to do over the course of a full fight was roughly a 1.53s GCD average over 10 minutes. Thats roughly 98% efficient, and it’s pretty close to an upper bound – not many people ran a tighter rotation than I did. ….
I regularly saw parses from good tanks – in well progressed heroic mode guilds – who weren’t better than ~80% efficient at their rotations. ….
Being more than 90% efficient in your rotation places you in the upper half a percent of tanks in WoW.
So if even the greats are struggling to beat 95% efficiency, what’s our estimate for an average tank? 90%? 80%? 70%? More to the point, do we really want to be running simulations for an average tank? The readership of this blog in particular is decidedly skewed towards above-average. Not many tanks of average or below-average play quality actively seek out blogs to try to improve their play in the first place. And of the ones that do, how many are willing to struggle through my verbose and technical blog posts? Probably a small minority.
More problematic yet is the matter of healers. How do we model an imperfect healer? They have many of the same complications that tanks do. Latency and reaction time are obvious factors one can tweak. But the difficult part is really the healer “rotation,” because there really isn’t one in most cases.
For starters, their rotation varies based on the situation – they may be focusing on single-target healing if they’re assigned to a tank, or on area-of-effect healing if they’re assigned to the raid. And in a 10-man raid, often one healer will be switching back and forth between those two play styles constantly based on the demands of the encounter. So the type of healing you get depends not just on your play, but on what’s going on with the rest of the raid as well.
Further, a healer’s play style is far more reactive than tanking or DPS play. They may have a basic rotation they use as a default during less dangerous periods, but they switch gears and change cast priorities during a crisis. They’ll use different spells if the tank has four or five GCDs to live than they will if death could happen in two or three. And while healers don’t have active mitigation, they do have cooldowns they can throw at the tank if the shit really hits the fan.
So there’s a huge number of variations in how you model an imperfect healer. Is the healer a little slow to switch modes when the tank takes a spike? Or do they over-commit to expensive heals and run out of mana too early? Or do they just not make effective use of their emergency buttons? We could model a “dumb healer” that just spams their mana-efficient heal on the tank regardless of what else is happening, but that’s probably not an accurate model of even the worst healers.
And just as for tanks, each healer plays a little differently. The model of one imperfect healer may not give accurate results for a different healer. Not to mention the inherent variations between healers of different classes with slightly different toolkits. The results may be totally different for a healer that relies heavily on HoTs than they would be for a healer that heavily leans on absorption effects or direct heals. How do you build a model that’s general and widely applicable when the inputs are so widely varied?
The short answer is: you don’t!
Rethinking The Approach
It’s clear that there’s way too much variation involved here to write the sort of modeling software that most people think of when they talk about wow. We could certainly write an equivalent to Simcraft, given enough time and effort, but it isn’t clear that it would give us much in the way of useful results. Tracking things like “how often does the tank die” would vary significantly based on the details of your imperfect healers and tanks.
Not to mention that, while we could model a healer any way we wanted, in practice we have no control over how they play. We might even have a different healer from week to week or encounter to encounter. Tying ourselves down to a particular healer model is doomed from the start to be too specific – it just won’t give us results that we can apply everywhere. We need something that is both more general and simpler to implement.
So we go back to the drawing board. All the way back to the beginning, in fact. We reconsider what survivability means at a fundamental level. What is the root cause of tank death? I’ll put forth the following hypothesis:
Hypothesis: Tanks are in the most danger of dying when their instantaneous average damage taken per second (DTPS) greatly exceeds their instantaneous average healing received per second (HRPS).
Now, that certainly shouldn’t be very controversial. I’ve basically said “you die when you don’t get enough heals,” but put it in a more technically rigorous form. We all know what it means to not receive enough healing. That happens when our healers are distracted or otherwise incapacitated – in other words, when they make a mistake.
And we’re also familiar with the first half of that statement. Our time-averaged DTPS goes up when we take a lot of damage in a short period of time. In other words, a damage spike. To illustrate that thought, let’s consider some data. Here’s what the DTPS profile may look like for a tank over an entire encounter (this happens to be Council of Elders):
This is the broad overview, showing how damage fluctuates over long periods of time. We have extended periods of high intake and periods where the tank isn’t actively tanking anything. Rather than look at the broad view, let’s narrow our focus to the section in the middle, with the large spike followed by three smaller spikes. That looks like this:
So over a two-minute period, our tank sees massive fluctuations in damage intake. It drops as low as 40k DTPS, and spikes as high as 150k-230k DTPS for 5- to 10-second periods. The danger periods are obviously these peaks, where damage intake abruptly increases – these are the moments when we’re most at-risk of dying.
From the same log, the tank healers’ average healing throughput is about 100k HPS. Of course, during this period it gets ratcheted up to around 220 HPS (and other healers chime in to help). But then again, this is from a log where the tank didn’t die. If the healer hadn’t cranked his output up fast enough to meet this new level of damage intake, we’re very likely looking at a dead tank.
Let’s assume for the moment that we don’t have any control over our imperfect healers. Let’s also assume that they screw up every so often during the encounter – maybe they fail to react to a spike quickly enough, or maybe they just stood in fire too long and have to move or break off of the tank to heal themselves. If one of those screw-ups coincides with one of the damage spikes, we’re probably going to die.
If we accept the premise that we have no control over what our healers do, just what they perceive, then what can we as tanks do to minimize the chance of this happening? This is what a scientist would call an “overlap integral” problem. We can increase our survivability by reducing the likelihood that a spike and a healer screw-up overlap in time. And how do we do that? Well, consider what would happen if we had a way to perfectly smooth our damage intake, such that instead of the pink line, it looks like the blue line on this plot:
Suddenly everything changes. First, we don’t have any big spikes to worry about – our mean intake is a little over half of our peak intake (and higher than our average), but it’s very steady. So it’s far less likely that the healer’s screw-up coincides with a period of extremely high intake, because there simply aren’t any periods of extremely high intake. More subtly, and perhaps more importantly, the fact that there aren’t any big spikes that require the healer to change modes means that it doesn’t matter if they’re a little slow in doing so. In practice, that’s a big deal because the “slow to react” healer model is far more likely to cause the sort of overlap that kills you than random healer movement is. So our “perfect smoothing” mechanism turns a tank that dies some of the time into a tank that never dies.
Now of course, this is an unrealistic model – we have no way to perform this perfect smoothing (outside of Photoshop, at least). But it gives us a clear goal to aim for. As tanks, we can’t make our healers play better. But we can give them an easier damage profile to work with: one that doesn’t require rapid fluctuations in healer throughput.
So I propose the following corollary to our hypothesis:
Corollary: If spike damage is dangerous, then minimizing the frequency and magnitude of high-damage periods is the most direct way to increase survivability.
This is a nice, tidy little statement. It narrows our focus from an all-encompassing simulation with many variables to a much smaller subset of parameter space. Instead of worrying about what the healer is doing, we just concern ourselves with the damage intake profile we receive. In essence, we’ve removed healers from the equation.
Of course, without healers, we don’t have much use for health either. If we’re only concerned with the plot of damage taken per second, it doesn’t matter how much health we have. So if we’re ignoring healers, and we’re not tracking tank deaths, then keeping track of the tank’s current hit points is no longer necessary, and we can add “tank health” to the list of things we can cut from the simulation.
Now, that sounds awfully controversial when taken in isolation. What use is a simulation without healers or tank health? But it follows rather naturally from the corollary. We’ve been conditioned to think that for a simulation to be useful, it must model everything down to the last detail with near-infinite precision. But that just isn’t always the case. It’s great if you want details about a specific situation. But not if you want general, portable results that apply to a great variety of situations. By eliminating healers and health from the equation, we can focus our efforts on variables we can control and make sure our modeling of those factors is as accurate as possible.
Building the Simulation
Now that we’ve defined the problem, we can go about building the simulation. This part is pretty boring unless you’re enthralled by mathematical detail and code, so I’ll try and summarize it as briefly as possible. We want our sim to do the following:
- Simulate a tank being attacked by a boss
- Track the damage intake as a function of time
- Take the resulting series of damage events and perform calculations on that sequence
Each of these steps has a number of details to consider. For example, we need to choose the boss’s damage so that we can appropriately assign Vengeance values. We’ll also need to estimate the player’s gear so that we can appropriately calculate avoidance and block chances. And of course, later on we’ll want to be able to vary both of these parameters.
Likewise, since we’re tracking damage intake, we need to properly calculate mitigation and absorption effects. Since our mitigation depends on the tank’s rotation and active mitigation usage, we need to model all of that as well.
The simulation I’ve written does all of those things and more. It’s a Monte-Carlo style simulation, which means that it simulates combat the same way the game calculates it – making rolls for each event as they happen. So for example, it cycles through a loop, incrementing time until an event (like a boss attack) occurs. It then rolls to see if that boss attack is avoided. If so, it rolls again for a grand crusader proc. If not, it rolls to see if it was blocked, and so on. It works the same way for our rotation, following a priority queue (which we can modify) to cast spells and update the system according to their result.
We run it for a very long time (10k minutes of combat, generally) to smooth out random fluctuations and get stable, statistically significant results. And the output is a string of numbers that looks something like this (but a lot longer):
[... 100 55 0 70 0 100 55 55 38 0 100 55 38 100 0 0 0 100 55 55 100 0 ...]
Where each number represents an amount of damage. So 100 would be a full boss attack taken to the face, 70 would be a blocked attack, 55 would be an attack mitigated by a 45% mitigation Shield of the Righteous, 38 would be an attack that was blocked and mitigated by SotR, and so on. In practice these values vary a bit more because of absorption effects (Sacred Shield), but that’s the basic gist of it.
Once we have that string of events, we perform some post-processing on it. We perform a moving average to generate the sort of data you would see in a World of Logs plot like the one I’ve shown above. We then take that moving average and calculate how many of its elements exceed a given threshold – say, 100% of the player’s health. This is the data I present in the tables. In essence, it’s telling you the two things we care about most: how many spikes are there, and how large are they?
A Word on Healing
One drawback of this style of modeling is that without player health tracking, healing becomes awkward. That wouldn’t be a problem if it weren’t for the fact that we have a non-trivial amount of self-healing abilities. For example, how do you model a Word of Glory or Seal of Insight heal if you can’t… well… heal?
The answer is that you treat them as short-term absorption bubbles. If WoG heals us for 200k health, we instead give ourselves a 200k absorption bubble. We keep the duration short because this healing is only relevant over short time windows. In the case of WoG I’ve been using 3 seconds, which is probably too generous; for Seal of Insight I’m using 1.5 seconds. It wouldn’t make sense to grant a 200k absorption bubble that lasts for 10 seconds, because in 10 seconds a real healer would react and top us off. So the absorption needs to only apply to boss attacks that happen shortly after the heal occurs. Another way to think of this is that, by not modeling healers, we’re assuming perfect healers that never let you die, and they react within a few seconds. Thus, if the absorption bubble isn’t used up within a few seconds, it just gets wasted (i.e. turns into overheal).
Which brings up another issue with healing in this model: overhealing and efficiency. We need to make assumptions about how much overhealing occurs. For Seal of Insight, you can think of this as trying to estimate how many procs occur when we’re already at full health since our simulation doesn’t have health. But the issue is a little more subtle than that, even.
To illustrate why, consider the following example: lets say you take a 200k attack from a boss, and react by casting a 200k WoG on yourself. Your healer also drops a 200k Holy Light on you in response to the attack. Which one of you overhealed, and how efficient were you?
Logs will base the answer on whomever acted first. But that doesn’t cut it for our analysis. Either way, there was 200k worth of overhealing happening in that scenario, and whether it was my WoG or the healer’s Holy Light, one of them was wasted. No matter who the combat log credits with the overhealing, I would have been better off letting the healer top me off and using that holy power on a Shield of the Righteous instead.
I tend to take a rather pessimistic view on healing efficiency for that reason. I generally assume that the healer is going to do what they do regardless of whether I WoG or not, because they can’t assume that I will WoG at any given moment. And likewise, I can’t assume they’ll stop healing me because I know where my WoG button is. Since neither of us can reliable predict the others’ actions, we’re bound to cause a good chunk of overhealing. This is less true with incoming healing notifications on unit frames, but not all frames have this capability, nor do all healers use it on frames that do. However, since opinions vary on this, I generally present results for a variety of overheal levels when I’m discussing results involving WoG.
A Word on Patchwerk
To wind down this post, I want to have a quick discussion about one of the more common criticisms of this sort of modeling. The simulation is a Patchwerk model, which for newer players means “a boss that blindly melees you and does nothing else.” But real fights rarely look like Patchwerk. Real fights have tank swaps, movement, different phases, magical damage. The criticism I often see is that because of all of those factors we’re ignoring, these simulation results have no relevance to real play.
I think that attitude is a bit narrow-minded, and I don’t think it’s entirely valid. Sure, there are tank swaps. Sure, you’ll use cooldowns during a fight. But you’re obviously not going to die when you don’t have aggro. And you’re probably not going to die while you have Holy Avenger up and are coasting along on 100% uptime of a 50+% damage mitigation buff. You’re not very likely to die when you have a big cooldown running either, for that matter. You’re not even that likely to die right after a tank swap, since you’ll have extra SotR uptime during that period if you’ve pooled holy power.
You’re most likely to die when you get a big damage spike, and that really only happens when you don’t have any of those safety nets. It happens when you just get unlucky and take a couple big unmitigated melees because you’re rebuilding Holy Power. In other words, it happens in the in-between sections of a fight, when you’re essentially tanking Patchwerk. By that logic the simple, Patchwerk simulation much more relevant than it would initially appear. Thus, I think a boss mindlessly hitting you is a pretty good model for the bulk of our death scenarios.
The “big, predictable boss attack” death scenario is pretty dangerous too, and shows up a lot in current content. But I think it’s less dangerous than many seem to think if you’re using active mitigaton properly. Shaving 50% off of the predictable spike, and likely one of the neighboring melees as well, usually makes it quite manageable. Often it reduces the big attack to less than a regular melee swing. And since it’s a predictable spike, your healers know its coming – they have DBM timers telling them that you’re going to take a big spike in X seconds, so they are more likely to use proactive tools, like absorption shields, small cooldowns, or simply pre-casting a heal on you so that it lands right after the large attack does.
In any event, the “big predictable attack” is something I plan on adding to the simulation once I get time – possibly even before the next round of data posts, since the Seal of Insight code is almost finished.
Despite all of that, it’s worth remembering that specific bosses or mechanics can lead to different strategies being optimal. We’re trying to model the most general situation possible so that it’s applicable to as many fights as possible. But mechanics like Dread Thrash can certainly trump steady-state modeling. A boss like Lei Shi forces us to reconsider whether the model is applicable to that particular boss. If we had another Algalon-like encounter where you have three healers spamming the tank just to keep up with the damage, you may care a lot less about smoothness and a lot more about stamina, armor, or even avoidance. This model abstracts all of those things out, so it’s important to keep in mind that this is a guideline for when all of those other effects are already taken care of. A good tank needs to know when the model is appropriate and when to break from the model to handle specific problems.
This post doesn’t really have a conclusion. I haven’t presented anything incredibly new here. Tanks doing high-end content have been focusing on spike damage for years now. But up until recently, we really haven’t had good tools to quantify those concerns. We’ve been able to calculate total damage reduction fairly reliably in the past, and often that metric was used as a stand-in to approximate smoothness. But the simulations I’ve run in the past six months have demonstrated the inadequacy of that approach. A stat like haste can be absolutely terrible at reducing total damage taken while being fantastic at reducing spike events.
I hope that the logic behind the simulation is more apparent now, though. It should, at least, address the common criticism that the results can’t be valid or useful without including healers. While it’s definitely important to consider what healers think and how they react, I don’t think it’s critical to have a specific healer model in mind when answering the question, “what makes me more survivable.” While our simulation doesn’t model healers at all, the thought process that we used to arrive at our observables certainly did. So they are included in the simulation in an indirect manner, in that rather than participating in the simulation explicitly, they determined how we perform the simulation and assess the results. Healers are the lens we look through when we decide what metrics accurately reflect our survivability.