I hinted about a “secret project” in my last post, and promised I’d be more forthcoming this week. It’s about time for the big reveal. But first, let’s take a trip down memory lane.
In The Beginning, There Was Hit
I’ve been playing WoW since release, but I didn’t really get into theorycrafting until the early part of 2009. At that time, protection paladin mechanics were pretty simple. Our rotation was so simple, in fact, that it was a true “rotation” that repeated and was 100% predictable. If you ask paladins from this era, which was the beginning of Wrath of the Lich King, they’ll tell you all about the 969 rotation. You’d alternate between a 9-second cooldown (one of Judgment, Consecration, and Holy Shield) and a 6-second cooldown (Hammer of the Righteous and Shield of Righteousness). That X-Y-X-Y-X-Y casting sequence repeated itself every two cycles, or 18 seconds.
This was also back in the day when threat mattered, so people cared about tank DPS for reasons other than abusing Vengeance (which didn’t even exist yet!). And even though theorycrafting at the time was miles ahead of what it was in previous expansions, it’s not even close to what we have today. Not due to mind share, because there were some smart cookies working on this stuff back then too, but because the tools were much more limited.
We didn’t have AskMrRobot or Simulationcraft, and World of Logs was fresh on the scene. The brains were available, but the modeling tools were mostly limited to hand-calculations and spreadsheets. I think Rawr was the only software tool out there that truly did modeling, and it had limited support for most classes.
The tool that most influenced me at the time was Jonesey’s Prot DPS/TPS Spreadsheet (EJ link). It wasn’t anything too fancy, just a spreadsheet that took your unbuffed character sheet values and calculated DPS, TPS, and stat weights for the 969 rotation. But it was the tool we had, and probably one of the first situations where someone had tried to model prot paladin mechanics as a whole rather than just looking at one part of the puzzle. Unfortunately, while the links remain, the spreadsheet itself seems to have been lost to time. Even I don’t have a copy lying around.
But while spreadsheets are a nice tool, arguably the most common tool back in those days, they’re rather limited. If you code a spreadsheet to calculate DPS, like Jonesey did, it can be difficult to adapt it to do something else. For example, if you want to compare a bunch of weapons, you need to plug in each weapon, note the result, and make a new spreadsheet with those ordered pairs to make plots. And there are even worse examples, like comparing different rotations, which that sort of spreadsheet really can’t do without a complete overhaul.
My first major contribution to theorycrafting was to take Jonesey’s spreadsheet and transcribe it into MATLAB. I chose MATLAB because it was a tool I was familiar with and used heavily during my graduate work, and because it was designed around doing the sorts of things that the spreadsheets of the time can’t. It treats everything like an array (or matrix, hence the name: MATrix LABoratory), and makes plotting intuitive and simple. If you want to find out how DPS scales with, say, strength, all you need to do is change strength from a scalar to an array, and all of your outputs are now identically-sized arrays. Simple as pi.
I first used the MATLAB model to show that, in contrast to the conventional wisdom of the time, hit was actually our second-best threat stat behind strength. The thread I posted those results into (Hit so bad?) ended up going from a discussion about hit into a discussion of the MATLAB model and what we could do with it, and eventually got split off into its own thread. That thread became my WotLK MATLAB thread, and saw that simple analytical MATLAB DPS model grow to encompass everything from stat and talent comparisons to detailed calculations of seal trade-offs and weapon choices.
But that was only half of the story. Inspired by Aldriana of rogue theorycrafting fame, whom I still admire for his incredible thoroughness and attention to detail, I also set myself to testing many of the mechanics in-game, because several of the equations in the original spreadsheet made incorrect assumptions about how certain spells worked. And in the process, we discovered a lot of things that we didn’t previously know about how different spells were implemented and how they interacted. Slowly, the pieces of the puzzle came together.
By the end of Wrath, prot paladins had an incredibly sophisticated tool that put us at the forefront of the theorycrafting community as one of the most-thoroughly-investigated and rigorously-modeled class/spec combinations. Heading into Cataclysm, it looked like the MATLAB juggernaut I had created would guide us for years to come.
And then it all came crashing down.
The Modeling Cataclysm
The beginning of Cataclysm was pretty tame. I had several people helping me work on the code, and we moved the entire project to google code to have a real versioning system. We did our homework building up to it, with a Call to Arms thread requesting various in-game tests on beta servers to make sure we could verify all of the mechanics, including the new Holy Power system and “939” rotation. As a result, we already had very complete models before Cataclysm went live, and the Cataclysm MATLAB thread was off to a good start.
Then, at some point after the release of patch 4.0.3a but before 4.1, they made a massive change – Crusader Strike stopped granting Holy Power on misses, dodges, and parries.
This was a big deal for several reasons. Up until this point, most players had ignored hit and expertise. Even in the “threat crisis” days of Wrath, which Meloree and I are skeptical ever existed (but that’s another blog post entirely), we ran with as close to zero hit and expertise as possible. Suddenly, there was a potential reason to care. Which was actually a good thing from a game-design perspective, in my opinion.
Unfortunately, it was a horrible thing for modeling. Not being able to rely on Crusader Strike’s Holy Power income completely undermined the analytical model we had so carefully crafted over the course of the beta. The nice, deterministic model simply wouldn’t work, so we had to go back to the drawing board. I whipped up a quick Monte-Carlo to handle the rotational dynamics, and was dismayed to find that I had to run it for an incredibly long time to get the statistical accuracy I wanted.
Of course, I’ve gotten much better at these things since then, as my exploits with the defensive Monte Carlo sims have demonstrated over the past year. Looking back at it, I could probably have knocked that run time down by a factor of 10 or more if I were approaching it nowadays. But alas, time travel hadn’t been invented yet, so there was no way for future me to tell past me that he was being obtuse. And boy, was past me obtuse.
Yet, even when things looked bad, luck stepped in. A new collaborator contacted me offering a new approach that might solve our problems. And together we built the current Finite State Machine code that forms the basis of the MATLAB DPS simulations. This new technique treats the modeling much like a Quantum Mechanics problem, and as such gives incredible results – repeatability down to +/- 1 DPS, and accuracy limited only by the inputs you feed it. And it was fast to boot, giving me outputs in under a second compared to the (admittedly, poorly-coded) Monte-Carlo that would take hours.
And so, for the remaining part of Cataclysm, the faithful FSM code carried us through. With this new tool in our back pocket, we seemed unstoppable. And then, with Mists of Pandaria, the unthinkable happened: we started caring about defense.
The Sha of Modeling
You see, in the entire history of paladin tanking before Mists, tank survivability was mostly a passive concern. You shored up your gear in Stormwind or Orgrimmar to be as defensive as it could be, and then stepped into the raid and took a beating. Apart from cooldowns, which were generally off-GCD, your entire rotation was determined by threat and DPS constraints.
Mists gave us the DK treatment, and introduced Active Mitigation. I don’t think it’s unreasonable to say that this has been an unqualified success – tanking has never been as dynamic, fun, or interesting as it has been this expansion.
But for modeling and theorycrafting, it’s a nightmare. We suddenly had more things to keep track of, more variables to consider, and more details to obsess over. That isn’t a bad thing in and of itself, it just means we have more work to do. But over the first few patches, a problem slowly became apparent. The FSM was getting slow.
You see, Finite State Machines operate by figuring out all of the possible “states” you can be in. So for a simple example, let’s just consider Crusader Strike. The simplest state is the one we start in, with Crusader Strike available and no holy power. Let’s call that state $(0, 0)$ to indicate that there are zero seconds left on the CS cooldown and we have zero Holy Power.
If we’re in that state, then we’re going to cast CS, of course, which might take us to a state $(4.5, 1)$ – 4.5 seconds on CS’s cooldown, and one Holy Power. It might also take us to state $(4.5, 0)$ if we miss with CS. And if we allow one GCD to pass, then we may end up in states $(3.0, 1)$ or $(3.0, 0)$ depending on which one of the $(4.5, X)$ states we started in.
The FSM attempts to compile a list of all of these potential states along with the transitions between states (like $(0,0) \rightarrow (4.5, 1)$) and the probability of those transitions. That works very well for small numbers of states (usually called the “state space”). But as the number of things to track gets larger, the number of states gets larger exponentially. Introducing haste effects via Sanctity of Battle only compounded this problem, because it was no longer sufficient to work in simple 0.5-second increments. Worse yet were the one-minute cooldowns introduced by level 90 talents and the 30-second duration of Sacred Shield, each of which is several times worse than simply adding another spell with a short (<15 second) cooldown.
But the real nail in the coffin was Grand Crusader. Specifically, when Grand Crusader stopped being completely offense-triggered and started triggering on avoided attacks. That wasn’t the straw that broke the camel’s back, it was the ten-ton weight that dropped out of the sky and obliterated the poor camel, leaving a smoking crater where the dromedary used to stand.
Since that change, the FSM has gotten slow to the point of being unwieldy. Coupled with a few annoying bugs that cause it to crash once in a while, and it’s hard to produce useful data with it anymore. Refreshing the simulation results has become an effort in frustration that takes several days, and often can’t be done simultaneously with other work. It’s become clear that the FSM just isn’t the right tool for this job anymore.
And more importantly, my exploits in this blog have convinced me that the separation of survivability and DPS simulations is less than ideal. Even though our SotR usage is somewhat divorced from our rotation by being off-GCD, there’s still a small amount of interaction. And, for example, while the sims I provide here can tell us whether the 2- or 4-piece set bonuses give any appreciable survival benefit, they don’t tell us the cost in the DPS arena. Even if using WoG to maintain the 2-piece had come out ahead, we would have had to estimate the DPS cost by hand.
It’s clear that a single simulation that does both is needed. In fact, that’s been clear to me for the better part of six months now, and I spent the first few months of that investigating and debating different options. For example, I clearly have a working event-driven Monte Carlo that does defensive calculations. Do I expand that to include offensive components, which would roughly mean merging it into the MATLAB trunk to replace the rotation module? Do I recode it from scratch in C# to take advantage of the extra speed it would give me (MATLAB is slow for these applications when compared to languages like C# and C++)?
This wasn’t a question that could simply be answered by expedience or numerical accuracy either. Most of my regular collaborators have reduced their involvement in the project, so I’d be doing most of this alone. And that means the time investment I’m going to have to put in becomes a factor. Not to mention that this is all focused on paladins; I regularly get requests to revisit the warrior simulations, or to write something for DKs or monks. If I’m going to write new code, I’d like to be able to re-use as much of it as possible to make those side tasks easier if I choose to tackle them.
I was staring down the possibility of writing a fully-featured event-driven simulation for protection paladin mechanics that may eventually be extensible to other classes. And that was a frightening thought. Remember, I’m not a professional programmer. While I can write usable code, my talents lie elsewhere: in the analysis and interpretation of data, the extensive testing and verification of game mechanics, and in translating those mechanics into a software model.
Over the course of the last few months, the answer became very clear. If I’m going to be at all successful in continuing to theorycraft at the level you all are familiar with, it cannot be with code I am writing exclusively by myself. Luckily, it doesn’t have to be.
After all, it would be awfully foolish to try and write the sort of simulation code that I need when it already exists.
Sorry MATLAB, It’s Not Me, It’s You
The Simulationcraft project, also known as Simcraft or SimC, is exactly what I’m after. It’s an event-driven combat simulator that seeks to model class dynamics. It already has nearly all of the game’s basic functionality implemented, and can handle a wide variety of options. And it’s incredibly fast and well-optimized, partly because there are real computer science types that work on it.
What it doesn’t have is a protection paladin developer.
Or, rather, that’s what it didn’t have. Now it does.
This has been a bit of a challenge, since SimC is written in C++. I haven’t used C++ since college, and had only very limited exposure to it then. Luckily it’s not that far removed from the C# I’ve been using in the FSM code, so the re-learning was pretty quick. I’m still not a C++ wizard, but I know enough to get around and write usable code.
Over the past two weeks, I’ve been heavily updating the protection paladin module of SimC. It was in a pretty sorry state when I got to it. Most of the abilities and mechanics had not been updated since Cataclysm. While it has been “working” for almost two months now thanks to another dev fixing some bugs (before that, it wouldn’t even simulate if you loaded a prot paladin), the results were more or less useless because they didn’t even resemble what happens in-game.
The latest release of SimC (530-3) is mostly correct for basic DPS calculations. It’s still missing a few things, like some DPS glyphs (Alabaster Shield, Focused Shield/Wrath) and T14/T15 set bonuses. Those have all been implemented, but didn’t make the cut for 530-3; they will be working in 530-4. There are other things that I still plan on implementing in the next two weeks, like Glyph of the Battle Healer and correcting the way that SotR mitigation recalculation is performed (i.e. not at all when the buff is extended). I also want to tweak the Holy Prism implementation a bit and clean up a few other oddities. I may even go as far as to add Clemency and Selfless Healer, though both of those are niche enough that I’m not sure it’s worth the effort. But, barring unforseen circumstances, my plan is for protection paladins to be more or less fully supported by SimC version 530-4.
The other side of the coin is the defensive results. SimC currently supports determining tank scale factors based on overall damage taken per second – in other words, on total damage reduction (TDR). If you’ve read anything on this blog, there’s a good chance you already know what I think of TDR metrics. So I’m working with the other SimC devs to implement better tanking metrics. So far, the groundwork has already been laid to perform exactly the same smoothing calculations that I do with the Monte-Carlo code, and I intend to focus on getting those up and running as soon as the Paladin module is working to my satisfaction.
This means that not only will we get the sorts of results from SimC that you’ve come to expect from my MATLAB simulations, but we’ll be able to translate them into numerical scale factors. And as an added bonus, it won’t just be useful to paladins – any tanking spec that has a working SimC module can take advantage of them.
But perhaps the best part is the increase in accessibility. While I’ve made my MATLAB code public for years, very few people have the knowledge necessary to read the code and look for errors, let alone verify that it’s working correctly. Even fewer have MATLAB installed, and thus the ability to run the code themselves. It was a nice gesture to make it public, but ultimately didn’t make much difference and was ineffective from a bug-finding and bug-fixing standpoint.
However, anyone can download SimC, load up their character, and look at the output. The types of comparisons that players often ask for are as simple as tweaking the configuration, which requires next to no coding experience. And best of all, players can load their own character up and calculate results rather than relying on extrapolated results from a small set of arbitrary gear sets that I’ve run calculations with. With potentially thousands of people running simulations and testing hypotheses independently, we’re far more likely to uncover oddities, quirks, and edge cases. And that means we’ll know more about how our class works than ever before.
All in all, I’m very excited about where this is headed. Being able to team up with SimC means that I can spend less time coding and more time doing analysis and constructing tests, and the automatic extensibility to other classes means that anything I do in the tanking department benefits more than just paladins.
What does this mean for the blog, you ask? I’m not entirely sure yet. You may have noticed that the updates have been scarcer and more focused on data dumps lately. Partly that’s because I’ve been busy integrating myself into the Simcraft project, and those posts are less time-consuming than a wordy, game-design-focused post. It takes three or four hours to write a post like this one, and longer for ones that analyze a particular feature or aspect of the game. A “data dump” post might only take an hour or two once the simulation data is done (and that can be run in the background while I do other things).
Partly it’s because of where we are in the expansion. The beta period is where the design-centric blog posts have greatest effect, because we’re more likely to influence changes. During the bulk of the expansion, things are less volatile. While there are certainly topics of relevance that may influence change (see Valor Morghulis for example), they’re less common.
But there’s certainly no shortage of topics I could write about. I imagine some people are curious to hear my thoughts on Flex raids, 5.3 daily content, and the new-and-improved valor upgrade system. I just haven’t had time, and have been struggling to keep up with my own self-imposed goal of one post per week. With any luck, moving to Simcraft will buy me more time in the long run to write those sorts of posts.
The other thing I’m hoping is that a new focus on Simcraft may draw Anafielle out of hiding. I know she runs her character through Simcraft on a regular basis – probably daily – and I suspect she has a lot of wisdom to share about how to interpret a SimC report that would be applicable to both protection and retribution.
I also haven’t decided how I’m going to split myself between this blog and my usual Maintankadin thread format. This expansion, the DPS-focused work has been on Maintankadin, while the survivability-focused content has been here. As with the code, I think there’s a happy medium of unification somewhere. A blog and a forum serve very different purposes and are good at very different things. The comment system here isn’t really ideal for long threaded discussions, for example. But the Advanced Theorycraft and Calculations forum, which I moderate, is. I’m still thinking about the best way to integrate the blog and the forum without abandoning either, but it may be as simple as creating a “blog discussion” thread (or multiple topic-specific threads) in AT&C. Much like the MATLAB threads, results could both foster discussion and lead to new calculations (i.e. a blog post that grows out of the discussion).
In any event, change is on the way, and I think it’s a change for the better. If you have any advice, comments, questions, etc. about Simcraft, the blog, or anything else, feel free to share in the comments. I’ll try to answer anything I can, and would love to hear how you think I can make this blog more effective.
Just keep it low-profile; I haven’t told MATLAB we’re breaking up yet, and it’s better that she hears it from me.