So it’s finally time to unveil a project I’ve been working on intermittently for the last three months. If you recall, in the past I maintained a suite of MATLAB DPS simulations that attempted to determine optimal rotations, stats, glyphs, talents, weapons, and so on. When the Finite State Machine (FSM) rotation modeler screeched to a grinding halt due to a combination of haste effects and long cooldowns, that suite of simulations was put on hold while I searched for a new (and faster) way to run simulations.
And as I’ve mentioned before, Simulationcraft was the solution I settled on. It had the speed and accuracy I needed, and I wouldn’t have to do all the work myself thanks to the extensive set of contributors. It also held the promise of unifying my DPS and survivability simulations into one simulation package rather than maintaining separate code for each. And it even has built-in stat weight generation, so I wouldn’t need to replicate that functionality of the old sims.
I spent a good chunk of the summer and early fall getting Simcraft’s paladin module up to date and inventing and programming a new tanking metric. But that was only the first step. I now had the simulation back-end to do the heavy lifting, but I still needed some way to do batch processing. To re-create the glyph simulation, for example, I needed code that would run Simulationcraft over and over for a bunch of glyph configurations and analyze the results.
There are a number of ways I could go about doing that. I could write simple batch files in DOS, or more realistically in another language like Perl (which I’d then have to learn, since I don’t actually know Perl). But what I really want to put together are giant tables of data and graphs. And there’s one language I know that’s exceptionally good at handling giant tables of data and graphs. MATLAB.
There was a bunch of grunt work involved, like writing functions that handled all sorts of mundane tasks: writing and reading strings to and from text files, regular expressions to pull the data I wanted out of Simcraft’s text output files, code to do simple tasks like making sure the path information was correct, code to automatically handle the caching and regeneration of results when I update to a new version of Simcraft, and code to output data into a table format that I can copy/paste directly into the blog. None of that was very interesting, even though it was probably 90% of the work involved.
The interesting part, which is what I’m going to write about today, is how the code works and the output it produces.
Simcraft operates by reading in an input file containing all of the relevant information about a character. What we want to do is basically tweak portions of that input file to see what changes in the result.
There are a few different ways we could go about that. For example, to test glyphs we could just have a “default_glyph_simulation_character.simc” file and edit that file over and over to change the glyph setup. That wouldn’t be terribly hard to code, but it has a few downsides. The main one is that it can be an issue for caching, which I’ll discuss a little later.
Instead, I went with a very versatile setup where I modularize the input file. In other words, I split the .simc file into component parts: a player section, a glyphs section, a talent section, a gear section, a rotation section, and a boss section. To run a sim, I just stitch the component parts together. For example, I can combine the default player, talent, gear, rotation, and boss sections with various different glyph sections to create my different glyph setups. I can save each of these combinations as a new .simc file, and by labeling them appropriately I’ll have the input file for each individual simulation.
This is useful for debugging, of course, but it also means that if we want to write a different comparison, we can reuse a lot of code. Instead of swapping in and out the glyph component, we might swap in and out the talent component to create a talent sim, using most of the same automation logic.
It’s also helpful for caching the results. One of the downsides of running simulations is that it can take a lot of time. So it’s helpful to keep and reuse results that shouldn’t have changed. If I run a sim with 50k iterations, I want to store the results so I can just call up those results later on rather than having to re-run the whole 50k iteration sim again. This essentially replaces a several-minute simulation run with a millisecond data read operation.
But you have to be careful about how you do that. For example, if the input .simc file changes, we’d obviously want to re-run the full 50k iteration sim to generate new results rather than call up old results that may or may not have any relevance to the current problem. By keeping a separate .simc file for every individual sim in a comparison, I can do that sort of checking and easily call up saved results when they should still be relevant. And of course, it will re-run the sim if it looks like anything important has changed (i.e. any of the inputs or the Simulationcraft executable are newer than the output).
So, to illustrate how this all works, let’s assume we’re writing a glyph simulation. We start by defining defaults for the components we won’t vary. In other words, we start with a default player:
paladin="Paladin_Protection_T16H" level=90 race=blood_elf role=tank position=front professions=Blacksmithing=600/Enchanting=600 spec=protection
and default talents:
and default gear
#T16N Gear Set head=faceguard_of_winged_triumph,id=99128,gems=indomitable_primal_160exp_160haste_270sta,reforge=parry_hit neck=juggernauts_ignition_keys,id=103916,reforge=hit_haste shoulders=shoulderguards_of_winged_triumph,id=99130,gems=320haste_320haste,enchant=180sta_80dodge,reforge=exp_haste back=qianying_fortitude_of_niuzao,id=102250,upgrade=2,gems=160exp_160haste_90sta,enchant=200sta,reforge=parry_haste chest=chestguard_of_winged_triumph,id=99126,gems=160exp_160haste_160exp_160haste_160exp_160haste_270sta,enchant=300sta,reforge=exp_haste wrists=bubbleburst_bracers,id=103738,enchant=170mastery,reforge=hit_mastery hands=handguards_of_winged_triumph,id=99127,gems=320haste_320haste,enchant=170haste waist=demolishers_reinforced_belt,id=103788,gems=320haste_320haste_320haste legs=legplates_of_unthinking_strife,id=104311,gems=320haste_320haste_320haste,enchant=250sta_100dodge,reforge=mastery_hit feet=wolfrider_spurs,id=103880,gems=320haste_60crit,enchant=175haste,reforge=crit_hit finger1=asgorathian_blood_seal,id=103794,gems=160exp_160haste_60haste finger2=seal_of_the_forgotten_kings,id=103796,gems=160exp_160haste,reforge=crit_mastery trinket1=vial_of_living_corruption,id=102306 trinket2=thoks_tail_tip,id=102305 main_hand=siegecrafters_forge_hammer,id=103969,gems=320haste,enchant=windsong,reforge=mastery_hit off_hand=bulwark_of_the_fallen_general,id=103872,gems=320haste,enchant=170parry,reforge=exp_haste # Gear Summary # gear_strength=19365 # gear_stamina=36396 # gear_expertise_rating=5107 # gear_hit_rating=2607 # gear_crit_rating=1112 # gear_haste_rating=15677 # gear_mastery_rating=7602 # gear_armor=60112 # gear_dodge_rating=180 # gear_parry_rating=1526 # meta_gem=indomitable_primal # tier16_2pc_tank=1 # tier16_4pc_tank=1 # main_hand=siegecrafters_forge_hammer,weapon=mace_2.60speed_10257min_19051max,enchant=windsong
and a default action priority list:
actions.precombat=flask,type=earth actions.precombat+=/food,type=chun_tian_spring_rolls actions.precombat+=/blessing_of_kings,if=(!aura.str_agi_int.up)&(aura.mastery.up) actions.precombat+=/blessing_of_might,if=!aura.mastery.up actions.precombat+=/seal_of_insight actions.precombat+=/sacred_shield,if=talent.sacred_shield.enabled # Snapshot raid buffed stats before combat begins and pre-potting is done. actions.precombat+=/snapshot_stats actions=/auto_attack actions+=/blood_fury actions+=/berserking actions+=/arcane_torrent actions+=/avenging_wrath actions+=/holy_avenger,if=talent.holy_avenger.enabled actions+=/divine_protection actions+=/guardian_of_ancient_kings actions+=/eternal_flame,if=talent.eternal_flame.enabled&(buff.eternal_flame.remains<2&buff.bastion_of_glory.react>2&(holy_power>=3|buff.divine_purpose.react)) actions+=/shield_of_the_righteous,if=holy_power>=5|buff.divine_purpose.react|incoming_damage_1500ms>=health.max*0.3 actions+=/judgment,if=talent.sanctified_wrath.enabled&buff.avenging_wrath.react actions+=/crusader_strike actions+=/wait,sec=cooldown.crusader_strike.remains,if=cooldown.crusader_strike.remains>0&cooldown.crusader_strike.remains<=0.35 actions+=/judgment actions+=/avengers_shield actions+=/sacred_shield,if=talent.sacred_shield.enabled&target.dot.sacred_shield.remains<5 actions+=/hammer_of_wrath actions+=/execution_sentence,if=talent.execution_sentence.enabled actions+=/lights_hammer,if=talent.lights_hammer.enabled actions+=/holy_prism,if=talent.holy_prism.enabled actions+=/holy_wrath actions+=/consecration,if=target.debuff.flying.down&!ticking actions+=/sacred_shield,if=talent.sacred_shield.enabled
We then come up with a list of all of the different glyph combinations we’re interested in and create .simc component files for those as well. For example, there’s an “AS_AW_DA.simc” file that just contains:
and similar files for every other combination we care about. We then piece together a complete .simc file from the default components and one of the glyph components, and run that sim to get our .html and .txt output files. And then we do it again for a different glyph component file, and then again, and so on until we have results for all of them.
The last part is just collecting and displaying the data by reading those output files, searching for the relevant information, and arranging it in data tables or graphs. That’s mostly done by filtering the text output files with regular expressions, and isn’t all that interesting. However, the results it spits out are interesting.
Below is the data from the first run of the completed glyph comparison. The defaults being used are all shown above except for the boss component, which is just the TMI standard T16N25 boss. This is a list of every possible glyph combination using the following glyphs:
AS – Alabaster Shield
AW – Avenging Wrath
BH – Battle Healer
DA – Devotion Aura
DP – Divine Protection
FW – Final Wrath
FS – Focused Shield
HW – Harsh Words
IT – Immediate Truth
WoG – Word of Glory
There are a few omissions here. Some glyphs are basically useless for simulation (Holy Wrath, for example), so they’ve been ignored. Double Jeopardy is missing because it’s not programmed properly in Simcraft at this point – something I hope to remedy during the holidays. I should also note that the Harsh Words glyph doesn’t do anything in the default setup since Eternal Flame is the chosen talent. I can fix that in a variety of ways, the easiest of which is probably just to add an APL entry to offensively cast WoG if the glyph is present.
But otherwise, that list should cover all of the major glyphs that affect DPS and survivability. I’ve ignored minor glyphs since none of them have a significant impact.
Below is a sortable list of the data. Since it’s long, I’ve spoilered it so you can open and close it. While I haven’t included error metrics on the table, the maximum DPS error in this data set is 88, which is less than 0.005% error. Note that “E” stands for an empty glyph slot.
Rather than dig through all of that data to come up with important conclusions, I’ve also programmed it to generate a table showing DPS for single-glyph configurations. That table is shown below. DPS error data is provided here, along with the DPS difference between that configuration and having no glyphs (“Delta”). Delta is thus the DPS gain due to adding that glyph in isolation, to within +/- the error (“Err”).
This table basically shows us that Focused Shield is the largest DPS gain we can get against single targets by a large margin. Coming in second place is the Glyph of Word of Glory, thanks to all the EF casts we’re using in this profile, followed by Alabaster Shield. Final Wrath is a distant fourth, and pretty much nothing else has a significant effect on our DPS output.
Some of the deltas are a little bigger than the “Err” column even though they should have literally zero effect (ex: Immediate Truth given that we’re using Seal of Insight), which suggests that the error bounds SimC is reporting probably aren’t generous enough. I don’t remember whether it’s reporting a 95% CI interval or something else, so I’ll probably have to dig through the statistics module and figure out what I need to do on my end to get more realistic error bounds.
Anyway, we can also make two other useful tables out of this data. The first would be to sort it in order of descending DPS to get the top 10 DPS combinations. We should expect that FS+WoG+AS is on top, followed by FS+WoG+FW. And indeed if we ask MATLAB to generate that table, we find:
I wouldn’t trust the TMI results to better than +/-50% here because we’re clearly running into the “self-sufficiency” problem I discussed in an earlier blog post. In other words, I doubt the difference between the top 6 DPS specs is at all significant, it’s probably just noise. On the other hand, the significant jump we see when using Battle Healer is real. I’m also not 100% sure what’s causing the higher TMI for the FS/HW/WoG combination – I’m guessing it’s a bug in how SimC handles Harsh Words and Eternal Flame (likely guess: it’s automatically casting WoG offensively when EF is cast, but still granting the player the HoT?). Something to add to my holiday to-do list, I guess.
Finally, we could also make a “Best TMI combinations” list:
I’m not sure there’s much to learn from this particular table. DP is really the only big survivability glyph we have since Devotion Aura isn’t on the default APL. So this list is essentially “10 random configurations that include DP.”
For reference, all of the results of these simulations are hosted on the matlabadin project in the “trunk\simc\io\” folder. So if you’re curious about any of the individual simulations, you can just look up the “glyph_X_Y_Z.html” file corresponding to that sim and see exactly what the setup and results were.
I’ve also written the talent comparison; it works basically the same way the glyph one does, but cycles through all the possible talent combinations. I’ve only considered the ones that have an effect on combat (L45, L60, L75, L90). The max DPS error on this table is 84, again less than 0.005%.
The default glyph configuration for these sims is
though after looking at the results of the glyph comparison, maybe it should be FS/WoG/DP. Or FS/WoG/AS to try and cut down on the self-sufficiency problem, though that would also affect Unbreakable Spirit’s valuation significantly.
SH – Selfless Healer
EF – Eternal Flame
SS – Sacred Shield
PU – Hand of Purity
US – Unbreakable Spirit
CL – Clemency
HA – Holy Avenger
SW – Sanctified Wrath
DP – Divine Purpose
ES – Execution Sentence
LH – Light’s Hammer
HP – Holy Prism
In this case, rather than picking out “single-talent” combinations (since those really don’t exist), I’ve picked a handful of relevant ones for a shortlist.
Here we see that Eternal Flame consistently beats Sacred Shield by a large margin for survivability (TMI in the hundreds vs. TMI in the tens of thousands). The slight DPS gain of EF over SS is due to GCD clashes (remember, Glyph of Word of Glory isn’t chosen in the defaults). Unbreakable Spirit is basically a no-brainer thanks to Divine Purpose, so there’s no reason to vary that. Within a group, Divine Purpose consistently gives lower TMI than the other two L75 talents. I stuck with Light’s Hammer across the board so that we could compare the L45 and L75 talents more directly, though I should probably add a few more combinations to this list so it highlights the difference in the L90 talents. Luckily we get some of that from the next two tables: Top 10 DPS and Lowest 10 TMI.
This list suggests that Holy Avenger and Holy Prism are the two dominant DPS talent choices this tier. The L60 talent is irrelevant, and the L45 talents are similarly irrelevant in this table because of the lack of Glyph of Word of Glory. Since nothing in the APL utilizes Selfless Healer yet, that’s basically an “empty” talent choice. Combinations with EF and SH are interchangeable in regards to DPS because neither costs us any GCDs; SS combinations don’t show up because it does cost GCDs and pushes back DPS abilities.
From this list, it looks like in T16 normal gear, HA>SW>DP for DPS in the L75 slot. This is a bit surprising, as I expected Divine Purpose to have a better showing here. I haven’t quite figured out the rationale for why it’s performing so poorly for DPS in these sims.
Holy Prism rising to the top is also a bit of a surprise, but it makes sense. We can cast three Holy Prisms every minute compared to a single Light’s Hammer or Execution Sentence. Three Holy Prisms has always been more damage than either of those two alternatives, even early in the expansion. However, those three Holy Prisms cost three GCDs. When we were using Sacred Shield as our go-to L45 talent and getting Grand Crusader procs from attacking, we simply didn’t have those spare GCDs. Switching to Eternal Flame and losing some Grand Crusader procs opened up enough GCDs that we can fit Holy Prism in very seamlessly.
I should also note that the default APL may not be properly optimized for Holy Prism yet. That’s another thing we’ll have to refine once I have time to write the rotation comparison. But that should only make Holy Prism better, not worse.
As far as the last two L90 talents, we can get a little bit of information from the next table.
Now, this is the table for lowest TMI, but the first thing I want to point out is about L90 talents and DPS. The first three rows of this table are identical except for the L90 talent, and it’s clear from those rows that Holy Prism has a significant lead in DPS (around 11k DPS). Execution Sentence comes in second and Light’s Hammer a close third, separated by only about 550 DPS. I’m hesitant to put too much stock in the survivability value of the three talents given that both Execution Sentence and Holy Prism are always being cast offensively here (though Holy Prism still heals you via the secondary effect, obviously). The TMI spread is also fairly small, so I hesitate to trust the order anyway.
However, turning our attention back to survivability, the dominance of EF+US+DP here is pretty clear, sweeping the top three spots. Swapping DP for HA results in a small decrease in survivability, mostly through a loss of SotR uptime. Swapping HA for SW is another clear loss, and losing US for Clemency in the last row is another clear loss. Note that the default APL doesn’t use Hand of Purity, otherwise I suspect that EF+PU+DP+LH would have taken that last spot. Add another thing to my holiday “to-do” list.
Realistically, I need to fix the “self-sufficiency” problem before I can rely on these TMI lists. I may use some trickery to /cancelaura Vengeance periodically to try and reduce the problem. We’ll see though – if the beta for Warlords of Draenor comes out any time soon, I may just start focusing on that since we’re basically done with content for this expansion anyway.
In short, I now have the ability to automate comparisons using SimC, much in the same way I used to do with my old MATLAB DPS simulations. I’ve gotten the first two done (glyphs and talents), and they mostly confirm things we already knew.
The glyph simulation reinforces that Divine Protection is the only glyph that has a large survivability benefit (and again, that’s situational – on a fight with a big magical burst, you still wouldn’t use it). It also confirmed that our best DPS glyphs are Glyph of Focused Shield, Glyph of Word of Glory, Glyph of the Alabaster Shield, and Glyph of Final Wrath, in that order.
The talent simulation reiterated that Eternal Flame is stronger than Sacred Shield for raw smoothness, and that Unbreakable Spirit is strong when you’re using Divine Protection on cooldown. In the L75 talent category, it showed us that for DPS, Holy Avenger > Sanctified Wrath > Divine Purpose, but for survivability Divine Purpose > Holy Avenger > Sanctified Wrath. And finally, in the L90 category it suggests that Holy Prism is significantly better than Execution Sentence or Light’s Hammer for DPS if you’re using Eternal Flame. It didn’t tell us much about their survivability value, though.
My next task is probably to get more of the simulations online. The weapon simulation should be relatively easy, the rotation simulations less so (but arguably more interesting to write). I also want to make some refinements to the settings for these two existing simulations based on some of the things I’ve noticed while writing this blog post, and probably based on things that other people notice and post in the comments. I’m happy to entertain feedback on what I can improve here, since these sims are clearly still in a fairly rough stage of development.
In parting, I want to leave you with an interesting thought though. Nothing about the code is all that paladin-specific. I’m bolting together .simc files that all contain paladin “stuff,” but the bolting together part is mostly class-agnostic. Why is that important?
Well, consider: what if someone were to write default component .simc files for, say, a Frost mage? With a few minor tweaks, this automation suite could then run nearl identical simulations for a Frost mage. Or a Protection warrior, or Blood DK, or… you get the idea.