MATLAB Automation Code

So it’s finally time to unveil a project I’ve been working on intermittently for the last three months.  If you recall, in the past I maintained a suite of MATLAB DPS simulations that attempted to determine optimal rotations, stats, glyphs, talents, weapons, and so on.  When the Finite State Machine (FSM) rotation modeler screeched to a grinding halt due to a combination of haste effects and long cooldowns, that suite of simulations was put on hold while I searched for a new (and faster) way to run simulations.

And as I’ve mentioned before, Simulationcraft was the solution I settled on.  It had the speed and accuracy I needed, and I wouldn’t have to do all the work myself thanks to the extensive set of contributors.  It also held the promise of unifying my DPS and survivability simulations into one simulation package rather than maintaining separate code for each.  And it even has built-in stat weight generation, so I wouldn’t need to replicate that functionality of the old sims.

I spent a good chunk of the summer and early fall getting Simcraft’s paladin module up to date and inventing and programming a new tanking metric.  But that was only the first step.  I now had the simulation back-end to do the heavy lifting, but I still needed some way to do batch processing.  To re-create the glyph simulation, for example, I needed code that would run Simulationcraft over and over for a bunch of glyph configurations and analyze the results.

There are a number of ways I could go about doing that.  I could write simple batch files in DOS, or more realistically in another language like Perl (which I’d then have to learn, since I don’t actually know Perl).  But what I really want to put together are giant tables of data and graphs.  And there’s one language I know that’s exceptionally good at handling giant tables of data and graphs.  MATLAB.

There was a bunch of grunt work involved, like writing functions that handled all sorts of mundane tasks: writing and reading strings to and from text files, regular expressions to pull the data I wanted out of Simcraft’s text output files, code to do simple tasks like making sure the path information was correct, code to automatically handle the caching and regeneration of results when I update to a new version of Simcraft, and code to output data into a table format that I can copy/paste directly into the blog.  None of that was very interesting, even though it was probably 90% of the work involved.

The interesting part, which is what I’m going to write about today, is how the code works and the output it produces.

Automating Simcraft

Simcraft operates by reading in an input file containing all of the relevant information about a character.  What we want to do is basically tweak portions of that input file to see what changes in the result.

There are a few different ways we could go about that.  For example, to test glyphs we could just have a “default_glyph_simulation_character.simc” file and edit that file over and over to change the glyph setup.  That wouldn’t be terribly hard to code, but it has a few downsides.  The main one is that it can be an issue for caching, which I’ll discuss a little later.

Instead, I went with a very versatile setup where I modularize the input file.   In other words, I split the .simc file into component parts: a player section, a glyphs section, a talent section, a gear section, a rotation section, and a boss section.  To run a sim, I just stitch the component parts together.  For example, I can combine the default player, talent, gear, rotation, and boss sections with various different glyph sections to create my different glyph setups.  I can save each of these combinations as a new .simc file, and by labeling them appropriately I’ll have the input file for each individual simulation.

This is useful for debugging, of course, but it also means that if we want to write a different comparison, we can reuse a lot of code.  Instead of swapping in and out the glyph component, we might swap in and out the talent component to create a talent sim, using most of the same automation logic.

It’s also helpful for caching the results.  One of the downsides of running simulations is that it can take a lot of time.  So it’s helpful to keep and reuse results that shouldn’t have changed.  If I run a sim with 50k iterations, I want to store the results so I can just call up those results later on rather than having to re-run the whole 50k iteration sim again. This essentially replaces a several-minute simulation run with a millisecond data read operation.

But you have to be careful about how you do that.  For example, if the input .simc file changes, we’d obviously want to re-run the full 50k iteration sim to generate new results rather than call up old results that may or may not have any relevance to the current problem.  By keeping a separate .simc file for every individual sim in a comparison, I can do that sort of checking and easily call up saved results when they should still be relevant. And of course, it will re-run the sim if it looks like anything important has changed (i.e. any of the inputs or the Simulationcraft executable are newer than the output).

So, to illustrate how this all works, let’s assume we’re writing a glyph simulation.  We start by defining defaults for the components we won’t vary.  In other words, we start with a default player:

paladin="Paladin_Protection_T16H"
level=90
race=blood_elf
role=tank
position=front
professions=Blacksmithing=600/Enchanting=600
spec=protection

and default talents:

talents=312232

and default gear

#T16N Gear Set

head=faceguard_of_winged_triumph,id=99128,gems=indomitable_primal_160exp_160haste_270sta,reforge=parry_hit
neck=juggernauts_ignition_keys,id=103916,reforge=hit_haste
shoulders=shoulderguards_of_winged_triumph,id=99130,gems=320haste_320haste,enchant=180sta_80dodge,reforge=exp_haste
back=qianying_fortitude_of_niuzao,id=102250,upgrade=2,gems=160exp_160haste_90sta,enchant=200sta,reforge=parry_haste
chest=chestguard_of_winged_triumph,id=99126,gems=160exp_160haste_160exp_160haste_160exp_160haste_270sta,enchant=300sta,reforge=exp_haste
wrists=bubbleburst_bracers,id=103738,enchant=170mastery,reforge=hit_mastery
hands=handguards_of_winged_triumph,id=99127,gems=320haste_320haste,enchant=170haste
waist=demolishers_reinforced_belt,id=103788,gems=320haste_320haste_320haste
legs=legplates_of_unthinking_strife,id=104311,gems=320haste_320haste_320haste,enchant=250sta_100dodge,reforge=mastery_hit
feet=wolfrider_spurs,id=103880,gems=320haste_60crit,enchant=175haste,reforge=crit_hit
finger1=asgorathian_blood_seal,id=103794,gems=160exp_160haste_60haste
finger2=seal_of_the_forgotten_kings,id=103796,gems=160exp_160haste,reforge=crit_mastery
trinket1=vial_of_living_corruption,id=102306
trinket2=thoks_tail_tip,id=102305
main_hand=siegecrafters_forge_hammer,id=103969,gems=320haste,enchant=windsong,reforge=mastery_hit
off_hand=bulwark_of_the_fallen_general,id=103872,gems=320haste,enchant=170parry,reforge=exp_haste

# Gear Summary
# gear_strength=19365
# gear_stamina=36396
# gear_expertise_rating=5107
# gear_hit_rating=2607
# gear_crit_rating=1112
# gear_haste_rating=15677
# gear_mastery_rating=7602
# gear_armor=60112
# gear_dodge_rating=180
# gear_parry_rating=1526
# meta_gem=indomitable_primal
# tier16_2pc_tank=1
# tier16_4pc_tank=1
# main_hand=siegecrafters_forge_hammer,weapon=mace_2.60speed_10257min_19051max,enchant=windsong

and a default action priority list:

actions.precombat=flask,type=earth
actions.precombat+=/food,type=chun_tian_spring_rolls
actions.precombat+=/blessing_of_kings,if=(!aura.str_agi_int.up)&(aura.mastery.up)
actions.precombat+=/blessing_of_might,if=!aura.mastery.up
actions.precombat+=/seal_of_insight
actions.precombat+=/sacred_shield,if=talent.sacred_shield.enabled
# Snapshot raid buffed stats before combat begins and pre-potting is done.
actions.precombat+=/snapshot_stats

actions=/auto_attack
actions+=/blood_fury
actions+=/berserking
actions+=/arcane_torrent
actions+=/avenging_wrath
actions+=/holy_avenger,if=talent.holy_avenger.enabled
actions+=/divine_protection
actions+=/guardian_of_ancient_kings
actions+=/eternal_flame,if=talent.eternal_flame.enabled&(buff.eternal_flame.remains<2&buff.bastion_of_glory.react>2&(holy_power>=3|buff.divine_purpose.react))
actions+=/shield_of_the_righteous,if=holy_power>=5|buff.divine_purpose.react|incoming_damage_1500ms>=health.max*0.3
actions+=/judgment,if=talent.sanctified_wrath.enabled&buff.avenging_wrath.react
actions+=/crusader_strike
actions+=/wait,sec=cooldown.crusader_strike.remains,if=cooldown.crusader_strike.remains>0&cooldown.crusader_strike.remains<=0.35
actions+=/judgment
actions+=/avengers_shield
actions+=/sacred_shield,if=talent.sacred_shield.enabled&target.dot.sacred_shield.remains<5
actions+=/hammer_of_wrath
actions+=/execution_sentence,if=talent.execution_sentence.enabled
actions+=/lights_hammer,if=talent.lights_hammer.enabled
actions+=/holy_prism,if=talent.holy_prism.enabled
actions+=/holy_wrath
actions+=/consecration,if=target.debuff.flying.down&!ticking
actions+=/sacred_shield,if=talent.sacred_shield.enabled

We then come up with a list of all of the different glyph combinations we’re interested in and create .simc component files for those as well.  For example, there’s an “AS_AW_DA.simc” file that just contains:

glyphs=alabaster_shield/avenging_wrath/devotion_aura

and similar files for every other combination we care about.  We then piece together a complete .simc file from the default components and one of the glyph components, and run that sim to get our .html and .txt output files.  And then we do it again for a different glyph component file, and then again, and so on until we have results for all of them.

The last part is just collecting and displaying the data by reading those output files, searching for the relevant information, and arranging it in data tables or graphs.  That’s mostly done by filtering the text output files with regular expressions, and isn’t all that interesting.  However, the results it spits out are interesting.

Glyph Comparison

Below is the data from the first run of the completed glyph comparison.  The defaults being used are all shown above except for the boss component, which is just the TMI standard T16N25 boss.  This is a list of every possible glyph combination using the following glyphs:

AS – Alabaster Shield
AW – Avenging Wrath
BH – Battle Healer
DA – Devotion Aura
DP – Divine Protection
FW – Final Wrath
FS – Focused Shield
HW – Harsh Words
IT – Immediate Truth
WoG – Word of Glory

There are a few omissions here. Some glyphs are basically useless for simulation (Holy Wrath, for example), so they’ve been ignored.  Double Jeopardy is missing because it’s not programmed properly in Simcraft at this point – something I hope to remedy during the holidays.  I should also note that the Harsh Words glyph doesn’t do anything in the default setup since Eternal Flame is the chosen talent.  I can fix that in a variety of ways, the easiest of which is probably just to add an APL entry to offensively cast WoG if the glyph is present.

But otherwise, that list should cover all of the major glyphs that affect DPS and survivability.  I’ve ignored minor glyphs since none of them have a significant impact.

Below is a sortable list of the data. Since it’s long, I’ve spoilered it so you can open and close it.  While I haven’t included error metrics on the table, the maximum DPS error in this data set is 88, which is less than 0.005% error.  Note that “E” stands for an empty glyph slot.

Spoiler Inside SelectShow

Rather than dig through all of that data to come up with important conclusions, I’ve also programmed it to generate a table showing DPS for single-glyph configurations.  That table is shown below.  DPS error data is provided here, along with the DPS difference between that configuration and having no glyphs (“Delta”).  Delta is thus the DPS gain due to adding that glyph in isolation, to within +/- the error (“Err”).

Title
Glyph DPS Err Delta HPS DTPS TMI SotR
E 367429 78 0 157433 157771 588.1 73.0%
AS 372376 78 4947 157454 157791 594.9 73.0%
AW 367355 79 -74 157486 157823 836.9 73.0%
BH 367471 78 42 156043 156789 18810.1 73.0%
DA 367334 78 -95 157443 157779 650.6 73.0%
DP 367323 83 -106 149574 149868 213.6 73.0%
FW 370402 83 2973 157422 157756 737.7 73.0%
FS 382948 78 15519 157464 157802 683.5 73.0%
HW 367394 78 -35 157451 157787 574.5 73.0%
IT 367187 80 -242 157468 157805 546.6 73.0%
WoG 374192 78 6763 157464 157801 598.9 73.0%

This table basically shows us that Focused Shield is the largest DPS gain we can get against single targets by a large margin.  Coming in second place is the Glyph of Word of Glory, thanks to all the EF casts we’re using in this profile, followed by Alabaster Shield.  Final Wrath is a distant fourth, and pretty much nothing else has a significant effect on our DPS output.

Some of the deltas are a little bigger than the “Err” column even though they should have literally zero effect (ex: Immediate Truth given that we’re using Seal of Insight), which suggests that the error bounds SimC is reporting probably aren’t generous enough.  I don’t remember whether it’s reporting a 95% CI interval or something else, so I’ll probably have to dig through the statistics module and figure out what I need to do on my end to get more realistic error bounds.

Anyway, we can also make two other useful tables out of this data.  The first would be to sort it in order of descending DPS to get the top 10 DPS combinations.  We should expect that FS+WoG+AS is on top, followed by FS+WoG+FW.  And indeed if we ask MATLAB to generate that table, we find:

Top 10 DPS Combinations
G1 G2 G3 DPS Err %Err HPS DTPS TMI SotR
AS FS WoG 395388 85 0.00% 157420 157757 819.0 73.0%
FW FS WoG 393219 88 0.00% 157469 157805 566.7 73.0%
AS FW FS 391406 86 0.00% 157456 157796 571.9 73.0%
AW FS WoG 390205 85 0.00% 157464 157796 632.7 73.0%
FS IT WoG 390107 85 0.00% 157439 157773 572.1 73.0%
E FS WoG 390094 85 0.00% 157433 157771 536.3 73.0%
BH FS WoG 389977 86 0.00% 156020 156758 18379.6 73.0%
FS HW WoG 389911 85 0.00% 157495 157830 2258.5 73.0%
DA FS WoG 389881 85 0.00% 157418 157752 533.8 73.0%
DP FS WoG 389830 85 0.00% 149573 149868 384.7 73.0%

I wouldn’t trust the TMI results to better than +/-50% here because we’re clearly running into the “self-sufficiency” problem I discussed in an earlier blog post.  In other words, I doubt the difference between the top 6 DPS specs is at all significant, it’s probably just noise.  On the other hand, the significant jump we see when using Battle Healer is real.  I’m also not 100% sure what’s causing the higher TMI for the FS/HW/WoG combination – I’m guessing it’s a bug in how SimC handles Harsh Words and Eternal Flame (likely guess: it’s automatically casting WoG offensively when EF is cast, but still granting the player the HoT?).  Something to add to my holiday to-do list, I guess.

Finally, we could also make a “Best TMI combinations” list:

Lowest 10 TMI Combinations
G1 G2 G3 DPS HPS DTPS TMI Err %Err SotR
AS DP IT 372263 149585 149879 186.2 10.80 5.80% 73.0%
DP FS IT 382505 149600 149895 194.3 13.40 6.90% 73.0%
E AS DP 372551 149533 149828 195.9 18.90 9.60% 73.0%
AW DP WoG 374264 149555 149852 196.7 14.00 7.10% 73.0%
AW DA DP 367330 149520 149816 197.1 18.70 9.50% 73.0%
DA DP WoG 374264 149524 149820 199.3 22.30 11.20% 73.0%
DA DP IT 367326 149549 149845 201.3 16.50 8.20% 73.0%
E DA DP 367376 149505 149798 204.7 36.00 17.60% 73.0%
AW DP IT 367226 149577 149873 205.3 31.50 15.30% 73.0%
DA DP FW 370381 149562 149856 205.5 19.20 9.30% 73.0%

I’m not sure there’s much to learn from this particular table.  DP is really the only big survivability glyph we have since Devotion Aura isn’t on the default APL.  So this list is essentially “10 random configurations that include DP.”

For reference, all of the results of these simulations are hosted on the matlabadin project in the “trunk\simc\io\” folder.  So if you’re curious about any of the individual simulations, you can just look up the “glyph_X_Y_Z.html” file corresponding to that sim and see exactly what the setup and results were.

Talent Comparison

I’ve also written the talent comparison; it works basically the same way the glyph one does, but cycles through all the possible talent combinations.  I’ve only considered the ones that have an effect on combat (L45, L60, L75, L90).  The max DPS error on this table is 84, again less than 0.005%.

The default glyph configuration for these sims is

glyphs=focused_shield/alabaster_shield/divine_protection

though after looking at the results of the glyph comparison, maybe it should be FS/WoG/DP. Or FS/WoG/AS to try and cut down on the self-sufficiency problem, though that would also affect Unbreakable Spirit’s valuation significantly.

Abbreviations:
SH – Selfless Healer
EF – Eternal Flame
SS – Sacred Shield
PU – Hand of Purity
US – Unbreakable Spirit
CL – Clemency
HA – Holy Avenger
SW – Sanctified Wrath
DP – Divine Purpose
ES – Execution Sentence
LH – Light’s Hammer
HP – Holy Prism

Spoiler Inside SelectShow

In this case, rather than picking out “single-talent” combinations (since those really don’t exist), I’ve picked a handful of relevant ones for a shortlist.

Talent Short List
Talents L45 L60 L75 L90 DPS HPS DTPS TMI SotR
311212 SH US HA LH 390997 121134 153348 753032.4 71.0%
311222 SH US SW LH 388631 124118 161326 1080163.3 63.0%
311232 SH US DP LH 384737 122424 152032 500356.3 70.0%
312212 EF US HA LH 391302 153414 153716 423.1 71.0%
312222 EF US SW LH 388657 161300 161688 563.9 63.0%
312232 EF US DP LH 388086 149540 149832 216.3 73.0%
313212 SS US HA LH 382266 107749 109451 25367.7 69.0%
313222 SS US SW LH 379472 114575 117620 37985.6 62.0%
313232 SS US DP LH 376220 106990 108622 13449.7 69.0%

Here we see that Eternal Flame consistently beats Sacred Shield by a large margin for survivability (TMI in the hundreds vs. TMI in the tens of thousands).  The slight DPS gain of EF over SS is due to GCD clashes (remember, Glyph of Word of Glory isn’t chosen in the defaults).  Unbreakable Spirit is basically a no-brainer thanks to Divine Purpose, so there’s no reason to vary that.  Within a group, Divine Purpose consistently gives lower TMI than the other two L75 talents.  I stuck with Light’s Hammer across the board so that we could compare the L45 and L75 talents more directly, though I should probably add a few more combinations to this list so it highlights the difference in the L90 talents.  Luckily we get some of that from the next two tables: Top 10 DPS and Lowest 10 TMI.

Top 10 DPS Specs
Talents L45 L60 L75 L90 DPS Err %Err HPS DTPS TMI SotR
312313 EF CL HA HP 403360 76 0.00% 159434 159823 3014.5 71.0%
312113 EF PU HA HP 403339 76 0.00% 159400 159787 3775.0 71.0%
312213 EF US HA HP 403333 77 0.00% 153305 153621 520.2 71.0%
311113 SH PU HA HP 403248 76 0.00% 111387 160509 8468304.7 71.0%
311213 SH US HA HP 403211 76 0.00% 112976 153377 1180576.5 71.0%
311313 SH CL HA HP 403172 76 0.00% 111397 160498 8471353.9 71.0%
311323 SH CL SW HP 401415 69 0.00% 114830 168578 9243176.7 63.0%
312323 EF CL SW HP 401393 69 0.00% 167421 167950 3624.0 63.0%
311123 SH PU SW HP 401375 69 0.00% 114837 168569 9262060.2 63.0%
312223 EF US SW HP 401307 70 0.00% 161101 161520 772.7 63.0%

This list suggests that Holy Avenger and Holy Prism are the two dominant DPS talent choices this tier.  The L60 talent is irrelevant, and the L45 talents are similarly irrelevant in this table because of the lack of Glyph of Word of Glory.  Since nothing in the APL utilizes Selfless Healer yet, that’s basically an “empty” talent choice.  Combinations with EF and SH are interchangeable in regards to DPS because neither costs us any GCDs; SS combinations don’t show up because it does cost GCDs and pushes back DPS abilities.

From this list, it looks like in T16 normal gear, HA>SW>DP for DPS in the L75 slot.  This is a bit surprising, as I expected Divine Purpose to have a better showing here.  I haven’t quite figured out the rationale for why it’s performing so poorly for DPS in these sims.

Holy Prism rising to the top is also a bit of a surprise, but it makes sense.  We can cast three Holy Prisms every minute compared to a single Light’s Hammer or Execution Sentence.  Three Holy Prisms has always been more damage than either of those two alternatives, even early in the expansion.  However, those three Holy Prisms cost three GCDs.  When we were using Sacred Shield as our go-to L45 talent and getting Grand Crusader procs from attacking, we simply didn’t have those spare GCDs.  Switching to Eternal Flame and losing some Grand Crusader procs opened up enough GCDs that we can fit Holy Prism in very seamlessly.

I should also note that the default APL may not be properly optimized for Holy Prism yet.  That’s another thing we’ll have to refine once I have time to write the rotation comparison.  But that should only make Holy Prism better, not worse.

As far as the last two L90 talents, we can get a little bit of information from the next table.

Lowest 10 TMI Specs
Talents L45 L60 L75 L90 DPS HPS DTPS TMI Err %Err SotR
312232 EF US DP LH 388086 149540 149832 216.3 31.30 14.50% 73.0%
312231 EF US DP ES 388627 149833 150132 272.4 54.20 19.90% 73.0%
312233 EF US DP HP 399801 149415 149725 329.6 65.80 20.00% 73.0%
312212 EF US HA LH 391302 153414 153716 423.1 101.50 24.00% 71.0%
312211 EF US HA ES 391824 153606 153907 442.7 93.20 21.00% 70.0%
312213 EF US HA HP 403333 153305 153621 520.2 49.40 9.50% 71.0%
312222 EF US SW LH 388657 161300 161688 563.9 63.50 11.30% 63.0%
312221 EF US SW ES 388194 161475 161867 735.9 291.20 39.60% 63.0%
312223 EF US SW HP 401307 161101 161520 772.7 155.80 20.20% 63.0%
312332 EF CL DP LH 388118 155544 155885 914.6 84.60 9.30% 73.0%

Now, this is the table for lowest TMI, but the first thing I want to point out is about L90 talents and DPS.  The first three rows of this table are identical except for the L90 talent, and it’s clear from those rows that Holy Prism has a significant lead in DPS (around 11k DPS).  Execution Sentence comes in second and Light’s Hammer a close third, separated by only about 550 DPS.  I’m hesitant to put too much stock in the survivability value of the three talents given that both Execution Sentence and Holy Prism are always being cast offensively here (though Holy Prism still heals you via the secondary effect, obviously).  The TMI spread is also fairly small, so I hesitate to trust the order anyway.

However, turning our attention back to survivability, the dominance of EF+US+DP here is pretty clear, sweeping the top three spots.  Swapping DP for HA results in a small decrease in survivability, mostly through a loss of SotR uptime.  Swapping HA for SW is another clear loss, and losing US for Clemency in the last row is another clear loss.  Note that the default APL doesn’t use Hand of Purity, otherwise I suspect that EF+PU+DP+LH would have taken that last spot.  Add another thing to my holiday “to-do” list.

Realistically, I need to fix the “self-sufficiency” problem before I can rely on these TMI lists.  I may use some trickery to /cancelaura Vengeance periodically to try and reduce the problem.  We’ll see though – if the beta for Warlords of Draenor comes out any time soon, I may just start focusing on that since we’re basically done with content for this expansion anyway.

Summary

In short, I now have the ability to automate comparisons using SimC, much in the same way I used to do with my old MATLAB DPS simulations.  I’ve gotten the first two done (glyphs and talents), and they mostly confirm things we already knew.

The glyph simulation reinforces that Divine Protection is the only glyph that has a large survivability benefit (and again, that’s situational – on a fight with a big magical burst, you still wouldn’t use it).  It also confirmed that our best DPS glyphs are Glyph of Focused Shield, Glyph of Word of Glory, Glyph of the Alabaster Shield, and Glyph of Final Wrath, in that order.

The talent simulation reiterated that Eternal Flame is stronger than Sacred Shield for raw smoothness, and that Unbreakable Spirit is strong when you’re using Divine Protection on cooldown.  In the L75 talent category, it showed us that for DPS, Holy Avenger > Sanctified Wrath > Divine Purpose, but for survivability Divine Purpose > Holy Avenger > Sanctified Wrath.  And finally, in the L90 category it suggests that Holy Prism is significantly better than Execution Sentence or Light’s Hammer for DPS if you’re using Eternal Flame.  It didn’t tell us much about their survivability value, though.

My next task is probably to get more of the simulations online.  The  weapon simulation should be relatively easy, the rotation simulations less so (but arguably more interesting to write).  I also want to make some refinements to the settings for these two existing simulations based on some of the things I’ve noticed while writing this blog post, and probably based on things that other people notice and post in the comments.  I’m happy to entertain feedback on what I can improve here, since these sims are clearly still in a fairly rough stage of development.

In parting, I want to leave you with an interesting thought though.  Nothing about the code is all that paladin-specific. I’m bolting together .simc files that all contain paladin “stuff,” but the bolting together part is mostly class-agnostic.  Why is that important?

Well, consider: what if someone were to write default component .simc files for, say, a Frost mage? With a few minor tweaks, this automation suite could then run nearl identical simulations for a Frost mage.  Or a Protection warrior, or Blood DK, or… you get the idea.

This entry was posted in Tanking, Theck's Pounding Headaches, Theorycrafting, Uncategorized and tagged , , , , , , , , , , , , , . Bookmark the permalink.

37 Responses to MATLAB Automation Code

  1. Zothor says:

    Definitely curious about the weapon sims… I’ve turned down the heroic tanking axe twice to keep my normal haste mace because my understanding was that weapon damage is a minimal factor in our damage scaling… but that was all well before the 5.4 nerfs.

  2. Dalmasca says:

    I really love your approach here and how it can be translated to other specs/classes. Keep up the great work!

  3. I’m confused as to how ES beats HP for TMI in the last chart. ES should have no survivability effect, since you’re using it offensively. HP will at least heal you. Its possible that with the APL you’re using HP is pushing back something ‘more important’.

    • Theck says:

      You can take a look at the APL yourself (it’s earlier in the post), HP isn’t pushing back anything survivability-related. The reason is more innocuous – the error margins on TMI in this case are greater than the difference between the two values. As I said in the paragraph immediately following that last table, I don’t really trust the TMI differences separating our L90 talents.

      Most of that is due to the “self-sufficiency” problem I’ve described in other blog posts. We’re out-healing the boss, so a lot of the TMI data gets fuzzy. I’d trust a difference of a factor of 2-3 or more, but not much less than that. In other words, going from 200 to 500 or 500 to 1000 would be significant; going from 200 to 300 probably isn’t.

  4. Schroom says:

    The one thing that bugs me in those simulations, is that ES does snapshot, so very skilled players might want to delay ES, when they know stuff like “my AP trinket should procc any second now” or “I’ll have a huge vengeance spike incoming”, even taunts have a huge impact on this (vengeance downtimes e.g.) also boss encounters have a lot to say here, see malkorok solo soaking, thok solo tanking, blackfuse and so on. The sim only shows what happens if you play it “by the book” but not if you want to use your full potential. I would find interesting to know, how much AP delta i need vs CD delay for a snapshot ES to beat HP.

    • Theck says:

      Given that the player has max vengeance pretty much the whole time, there’s no such gaming to be had in this sim. And keep in mind you can make similar snapshot decisions about HP (although at a larger cost for delaying it).

  5. Schroom says:

    Yes, the problem in a “real environment” is that our vengeance peaks aren’t very likely to occur every 20 seconds. But a vengeance peak happening once a minute, and pretty regularly in the same interval (as bosses also follow time base mechanics) is much more likely.

    There of delaying ES, and using it at a vengeance peak is much more practicable then doing the same thing with HP. In fact because the Cooldown of HP is so much shorter, it would be more unforgiving for Timing it too much, and overshooting our window.

    That said, as you pointed out, your model uses max vengeance all the time. And this is, as we know, causing some Problem (like the Problem with EF healing more than dmg inc) and also here, it is falsifying our results. For the ES vs HP case, you show that “3xHP > 1xES” but only if we are using a constant vengeance level for the whole time, now if that vengeance would oscillate, and you use one HP at the peak and the other 2 on cooldown, versus using that one ES on peak. Your Thesis might (!) be proven wrong.

    So I’m going to take a leap and Question your model : )

    The thing is, the goal of the sims are, at least how I saw it, to simulate real events, in order to get some artificial data, so we can predict what happens if. And from here on we can read out stuff like skill X does more DPS than skill Y.
    So, the main thing causing this problem is that we use vengeance like a constant. But vengeance is not a constant, it is a variable that changes all the time.

    This morning I had an idea about how we could change your model to account for this, and giving us results that are way closer to reality.

    Why not use a pulse-code modulation to read out our results? We put out sims for different states of vengeance. Spread out all of those sims and read them out with a pulse code modulation, which frequency we also can control.

    I think that would be an amazing model.

    I did a quick graph to show exactly what I mean: http://img834.imageshack.us/img834/4414/sz29.png

    So in this example we have 3 sims. One with 0 vengeance, one with max vengeance all the time and one with avg vengeance all the time (of course we could add more stats to make it even more precise) now with a sin(x) function we read out those files, which gives us the illusion of different vengeance states, but always comparable, without having to recalculate every one of those sims all the time.

    • Theck says:

      I would hesitate to call it a “Thesis” – this is a model for a very specific situation, namely solo-tanking a boss that just melees you. There are certainly differences between that model and a real boss (the Vengeance fluctuation thing being one major example). However, bosses vary quite a bit, so any model you make will be inaccurate for most of the bosses in a tier. We’re just looking for the most generic steady-state approximation we can find.

      The sinusoidal setup you have is probably overthinking it. There’s an APL line that we can use to /cancelaura vengeance, so we can just use that to more accurately simulate the rise/decay dynamics of Vengeance. SimC doesn’t have tank swaps built-in yet (another thing on the to-do list), which would give us even better approximations. I’m hoping to have something like that implemented before 6.0, but probably not in time to be of use for 5.x.

      • Çapncrunch says:

        Unfortunately Theck, I’ve been attempting to work with SimC’s version of /cancelaura to see if I could remedy the self-sufficiency issue for my simulations, however the results have been less than compelling. For starters this is the line that I add into my action priority list (I put it right after startattack, though I don’t know if its position should really matter):

        actions+=/cancel_buff,name=vengeance,line_cd=35

        And I’ve toyed around with using different cd periods for it, but although the vengeance graph clearly shows that the command is working, the health timeline chart and overall HPS shows it having little impact on the self-sufficiency issue, and the stat-weights it generates still tend to come out garbage (my last sim I ran had nearly every stat with a negative weight). Even putting my 561 tank up against the Q17 boss resetting my vengeance every ~30s still has me outhealing the boss (though my tmi does skyrocket, but the stat weights still looked pretty unreliable).

        I’d have to double check but I think that to have a real impact on the self-sufficiency issue would require canceling it every 10s or less, which seems like overkill as real life vengeance variations are not that frequent. It starts to make me wonder if the bigger offender of the issue is maybe EF rather than vengeance, though I’d have to run a few sims without EF to see if it still gives garbage results or not.

        • Theck says:

          Well, the source of the problem is partly that EF scales very well (or else there’s a bug with the EF code – but I couldn’t find it last time I checked, it seemed to be ticking for appropriate amounts).

          What if you try against a T16N10 or lower boss? If the issue is extreme scaling with Vengeance, then a low-Vengeance scenario might give better results. The TMI values will obviously be very low, but they might be more consistent?

          • Çapncrunch says:

            It’s been a while since I’ve used the T16N10 boss, but I do tend to try the T16N25 boss frequently as the sort of “lower bound” for my tests, I just used the Q17 boss as an example of the other extreme (honestly, I’m probably self-sufficient against the T16N10 boss anyways due to outgearing it at 561 ilevel).

            My point regarding EF is that perhaps its the difference between actual health gains vs damage prevention that is affecting the TMI calculation. I’m running a few sims now comparing SS vs EF, resetting or not resetting vengeance and a few different boss difficulties. Too bad it’s hard to be sure how trustworthy the results are without generating stat weights, they really up the simulation time (especially on my little 2-core). So far simply replacing EF with SS seems to give real results even without resetting vengeance.

          • Theck says:

            From the standpoint of the calculation, heals vs. absorbs is pretty much irrelevant. It’s subtracting healing out of the damage taken during the time window, so being hit for 500k and healed for 50k within a 1-second time bin is identical to being hit for 450k after a 50k absorb.

            The only difference is that absorbs are guaranteed to be in synch with your damage events, while heals may lead or trail by a small amount of time. But since we’re interested in a 6-second time window that shouldn’t be a noticeable effect.

            I think the reason you get useful results with SS is because SS is that much weaker. IIRC it doesn’t scale as well with AP as EF does. So when you talent EF, you’re self-sufficient. If you talent SS, you aren’t.

            The self-sufficiency problem is basically that if you’re very likely to offset all of the damage you take in a 6-second window with healing in that same window (i.e. because EF ticks keep topping you off), then the TMI will be dominated by whatever random spikes you get. And most of that is occurring within the first 20-30 seconds when you don’t necessarily have a big EF active. Using glyphed Divine Purpose on the pull helps a little bit with that, but doesn’t really eliminate the problem.

          • Çapncrunch says:

            So far still only done some SS tests, but I have found it interesting that even using SS I’m still self-sufficient, for example against the T16N25 boss I’d be pretty self-sufficient without any lvl45 talent (SoI, 2p bonus and holy prism splash account for almost all of the boss’ damage), but the stat weights reliably come out where you’d expect them to be (except that when I tested SS with resetting vengeance I got a negative weight for strength, but that could just be that by making vengeance non-constant that the ap benefit from str becomes lost in the noise)

            Which supports my idea that the issue has more to do with EF and healing than it does with self-sufficiency.

    • Zapelm says:

      Any chance choosing “Helter Skelter” fight style (or something like that) could bring anything more valuable than “Patchwerk”, or just a different POV? AFAIK it includes stuns (-> spikes), movements and boss immunities. It would maybe require to adjust the standard rotation since a player would not spam all cds in that scenario. Don’t know if it is buggy or should be discarded anyway..

      • Theck says:

        Different yes, more valuable probably not. The stuns and movement will de-emphasize active mitigation and rotational effects. The problem is that the largest spikes are going to happen during those stun/movement periods, and there’s very little you can do to counter them. So it’ll just increase randomness and noise.

  6. Schroom says:

    yeah well maybe not “Thesis”. English is a foreign english to me. so, please excuse :) but you get what I mean.

    and yeah maybe it is over thought, I just remembered PCM from my school days and thought this would be one possibility to get more “down to reality” results. as you say every model will be innacurate, but one can try getting closer, and the steady-state approximation is just not close enough to reality IMO. yes with your model HP as a talent is bether then ES for DPS. In a real boss encounter, it is a completely different kind of monster.

    for clarification, I’m so reluctant to this, as I’m maintaining a prot paladin guide myself and offer help to a lot of Prot paladins from the German community:) and a beginner might be bugged hearing “use HP over ES for singletarget dps” from someone because he read it here, without understanding the context (which is that persons fault I agree, one should not shout what one does not understand) and then having a hard time reading elsewhere use ES for this boss to have a big nuke and huge burst.

  7. Duncan says:

    Theck,

    How would a non-programmer help?

    I mean, I have plenty of Excel experience writing various functions, etc…etc…

    I’d like to do begin something for my main’s Class, a Hunter.

    Thank you!

    p.s. You’re welcome to email me directly if you like.

    • Jackinthegreen says:

      Usually the way non-programmers can help is by testing stuff in the game and comparing it to the simulations. SS’s GCD cast time not being affected by haste was one example where there was a difference between what the game was doing versus what the sims were doing.

      • Duncan says:

        Well, I’m semi-technically proficient and am up for learning something new.

        While testing is definitely helpful, I had hoped to move in different direction, if only to learn something new. =)

        -Duncan

        • Jackinthegreen says:

          You might be able to look over at coursera.org to see if they have any C++ classes going on if you want to become more proficient. Khanacademy might also have something. Your local library might also have some C++ books.

          Being semi-proficient can work. About a month ago I got looking through the DK module and found that Scent of Blood was being treated as 100% “proc” instead of the proper 16.66 PPM it actually is, so I put in an issue report and it was fixed that day.

    • Theck says:

      If you’re talking about helping with SimC, what Jack said is good advice. Alternatively, just scrutinizing SimC output files (i.e. run your character through and see if the results make sense and/or match what you’d get fully buffed against a dummy or Patchwerk boss) is helpful if you can spot bugs/errors.

      Regarding the automation, I’m not sure yet. Eventually I’ll need simc components for any class I intend to run through. So for example, splitting up a simc file into player, glyphs, talents, gear, etc. But before I go that route I need to make some decisions about naming conventions and such to keep everything straight.

  8. Caltiom says:

    Great work Theck. I have seen a couple of people interested in doing batch simulations for various permutations ( some over at mmo-champion.com even thought they could iterate through action priority list permutations ), but I think you’re the first who has done it with such a systematic approach.
    And even thought there are various special output formats like xml, csv, etc.some simple regexe on the text output is all you need ;)

    First, may I ask how you did the sortable tables here in your blod post? Is that a functionality of your blog software, or did you use something explicitly? The idea has been raised a few times around and would greatly benefit SimC itself ( and I could think of a few other places to use it as well ).

    When you mention possible problems with statistical confidence measurements you’ve got my attention. First, one has to be careful to use ~sqrt(2)*dps_error to estimate the confidence of the difference between two simulations – you can add the variation because the simulations are independent, but you still need to take the variation of both into account. In the example you mentioned that still results in a too big dps delta, if the glyph really does not have a buggy implementation.
    ( btw if you want to check the hypothesis that eg. dps(simA)>(simB) it is a lot more efficient to look at the distribution of the iteration-wise delta of the two simulations, though that would require a bit of work. But for the amount of simulations you’re doing that might still be interesting, to reduce compuation time needed. )
    If we really have a wrong error estimate, that is a very serious problem. The measurement is one of the most important values SimC has, because otherwise we could not at all estimate the amount of iterations needed to get a certain statistical confidence level. A simple test would be to run the same ( e.g. empty glyph ) simulation N times, and check if the expected 0.95*N simulations are within the confidence interval.

    Unfortunately I can’t find the single glyph simulations in your matlabadin project. Could you please upload them, assuming I’m not just too stupid to find them.

    Oh and I’m curious: How long does it take you to do a 50k simulation, and what number of sims do you think you’ll realistically include in your batch process? Under the assumption of a long term analysis, where you’d repeat things after every major patch / bugfix.

    • Theck says:

      The sortable tables are a feature of the table plugin I use for WordPress, EasyTable:
      http://wordpress.org/plugins/easy-table/

      I’m not suggesting that SimC has any problem with its statistical error calculations. Just that I don’t know off the top of my head exactly what measurement it is that SimC is spitting out. So I’m not sure it’s fair to say that mean +/- error is reasonable for the purposes of seeing if two simulations are statistically significant or not. The math is a little difference depending on whether the Error result is e.g. the 95% Confidence Interval, Standard Deviation, Sigma, etc.

      The single-glyph sims are of the format glyph_E_E_X (E stands for empty). For example:
      http://code.google.com/p/matlabadin/source/browse/trunk/simc/io/glyph_E_E_AS.html

      I’ve been using 4 threads, and it looks like a single 50k-iteration simulation takes about a minute on my machine. Which syncs up with my recollection of the whole process taking about an hour or two. I left it running unattended while I was doing other stuff, so I didn’t check on the exact timing.

  9. Shadanah says:

    Thanks for the meticulous calculations!

    Question: to what degree do you suspect your theorycrafting has impacted Blizzard’s decisions regarding changes in WoD? Specifically, I’m thinking about the changes that make “necessary” secondary stats less of an issue and that add personal-preference-tertiary stats. Follow-up: what changes will you, askmrrobot, icy-veins, etc. have to make to provide a broad stratagem for your followers? Or do you believe the changes will be less-than-cataclysmic (pun intended)?

    • Theck says:

      It’s hard to speculate on that. I doubt that my theorycrafting in particular had a lot to do with secondary stat changes; I’m just one of many people echoing sentiments that are in-line with the direction they’ve chosen to go. I’m sure it was more of a consensus by the developers that the community in general shared those sentiments, and that they were legitimate problems.

      As far as what I need to change? Not much, I expect. I don’t think we’ll see sweeping changes to our base mechanics, so my focus as usual is going to be to work on making sure we have accurate models with which to judge performance.

      I’ve never tried to make this blog a suitable resource for a beginner tank – IMO that’s what sites like Icy Veins are for, and that’s why I consult for them to help make sure their intro advice is accurate. AMR probably has the largest amount of change coming into the next expansion thanks to itemization tweaks, but they’ve already announced their desire to get into the combat log analysis business.

  10. Çapncrunch says:

    Theck, I’ve been thinking about the dps order for our level 75 talents, and I actually think that they make sense the way the sim compared them, and it’s probably correct that DP is the lowest. The only thing DP gives us is extra holy power (sort of) but doesn’t really contribute to dps (aside from the extra SotR, but all of the talents do that). Compared to HA, which provides similar SotR usage/uptime but also increases the damage of our generators by 30%, which means it gives extra dps on top of the extra SotR usage. And SW contributes to dps by giving us higher uptime on AW which would increase dps.

    I’m actually a little surprised that SW has such a clearly lower SotR uptime compared to the other talents, because from what I can see it shouldn’t be that much lower. DP averages out to a 33.33% increase in finisher usage after factoring in its ability to proc off of itself (though the 4p provides DP slightly more uptime since it can proc off of the free EFs). HA on the other hand increases our holy power generation by 200% with a 15% uptime which averages out to 30% more finisher uses. Now for SW it should be a 28.33% increase which should put the gap between it and HA similar to the gap between HA and DP. I got this number by first comparing the standard rotation to what the rotation becomes during SW, our standard rotation generates 5hp every 9gcds, while during SW we generate 5hp every 4gcds, which is a 170% increase, and it has a 16.667% uptime which averages out to 28.33%

    I have a possible question, does the sim correctly increase the duration of AW when SW is chosen? I ask because if I redo my SW numbers with a 20s duration (11.11% uptime) that gives SW an average increase of 18.889% which seems to match your chart better (ie if ~70% SotR uptime is obtained with ~30% increase in holypower/finisher usage then the “base” SotR uptime would be ~53%, and applying a 18.9% increase to that gives it ~63% uptime which is about what your sim shows SW at). Though if that was the case then I don’t see why SW would be coming out ahead of DP in dps.

    Or did I miss some other obvious reason for why SW has such lower SotR uptime compared to the other 2?

    • Theck says:

      SimC should properly model SW’s duration increase. i think your math is just wrong.

      w/o SW: 5 HP every 9 GCDs, or 5/(9*1.5)=0.3704 HP/sec
      during SW: 5 HP every 4 GCDs, or 5/(4*1.5)=0.8333 HP/sec

      Time-averaged: 30/180*0.8333+150/180*0.3704 = 0.4476 HP/sec, which is a 20.83% increase over the “w/o SW” value, not a 28.3% increase. Did you drop a zero by accident?

      • Çapncrunch says:

        Yeah, I found my error. I kept my math in terms of gcds instead of seconds and kept them as fractions instead of decimals (5/9 and 5/4). And when I converted them to a common denominator I put 54/36 instead of 45/36 which lead me to seeing SW as a larger increase in HP generation than it really is.

        Though it seems a little strange that they’d make SW generate such a disparately different amount of HP compared to the other 2 talents. I would assume that the intent was that because it boosts damage/healing (and healing received) that it would make up for generating less HP. But that explanation doesn’t really make sense when HA generates just as much (practically) as DP in addition to buffing the damage/healing of HP generators.

        Sort of funny how SW is so poor for prot while it’s HA that is so poor for ret.

        • Theck says:

          I think it’s just that they want to make sure there are at least 2 viable options for each spec. SW just happens to be the odd man out for prot, even after making J generate two holy power per cast.

  11. Aribethe says:

    People need to take sims and adjust for reality. I’m assuming Patchwerk sim means 100% uptime on the boss for the tank (meaning the boss is meleeing the tank 100% of the time). If i’m wrong please correct me.

    However, under that assumption, in reality most boss fights have tank swaps, which result in lets say 50% uptime. Holy Power generation will be lower due to less GC procs. Hence, the SotR uptimes are probably a bit higher than reality.

    The other things to take into account is that the TMI values will change in the sense of when we are in danger of dieing. Dieing while offtanking is unlikely and usually I’m spend Holy Power putting EFs on the other tank or raid members. THe level 75 talents are ones that will change in value here. DP is mostly the same regardless of uptime. The other two SW and HA have a greater value in my eyes in that they can be used when you are actively tanking the boss. I hope people don’t use them during offtanking. HA and SW are cooldown based TMI and will have a higher value in the sense that their uptime is when you are in danger.

    If the sim could simiulate tank swaps, and put in rules where you use HA or SW only when actively tanking, then I believe their TMI value would outclass DP. If this is already being done, then please ignore me =P…

    Another thing affected would be glyph choice. You will get less AB procs when you’re not tanking, since you have to block for them. Assuming 50/50 tank uptime, this would lower the value of AB by half. Which I believe would push FW above AB in value.

    Another question I have would be what is the formula for Holy Wrath that is being used? There has been a tooltip error for the whole xpac on Holy Wrath. It hits far harder than the tooltip suggests. I calculate it at 4300 + 91% Attack Power (note not Spell Power). This should move it up in priority, even during execute mode. My priority for fillers is: GC > Holy Prism/Execution Sentence (If I have it) > AS > HW > HofW (under 25%) > C (if not already down). This would cause more usage of HW during the last 20% where Glyph of FInal Wrath is in effect. Otherwise, most filler GCDs are filled with HofW because of its short CD. Holy Wrath at times accounts for more damage than SotR on my damage breakdowns. AB is the lowest in my priority for Glyphs.

    Also just to emphasize it, because many may not realize it, there rotation has a &!ticking for Consecrations action. You can’t have 2 Consecrations down at a time. The graphic remains, but the first one stops ticking and even worse if you clip a tick its even worse because it doesn’t tick for another second.

    • Çapncrunch says:

      The sim does currently model solo tanking, which has lead to vengeance related issues, though that’s something that I believe Theck on his to-do list.

      The GC thing though isn’t that big a deal, as even when simming a solo tank situation GC only accounts for maybe 10% of our holy power. So assuming we only tank half the fight that’d only be about a 5% loss in holy power/uptime. And while a 5% loss in holy power may be significant, another way to look at it is that the sim is saying that we have ~70% uptime while tanking the boss, because honestly the time that we’re not tanking doesn’t matter. I’d imagine that when/if Theck gets tankswaps worked into the model he’d probably wind up making the model ignore the time we spend not tanking in the TMI calculation because that time is irrelevant to our survival concerns (it really only matters for cds and vengeance).

      And while I do in principle agree with your point regarding HA/SW vs DP with tank swaps, you are overlooking something else, which is that DP has a “smoother” effect on survival than HA/SW. For example if there’s a tank swap every 30s then that means you’ll have HA up for one of your turns tanking (during which you’d be virtually invincible) but it wouldn’t do anything for you during your next turn tanking, whereas DP will benefit you all the time, so you’re comparing a relatively consistent benefit of DP to having high and low points. Now if the tankswaps were a minute apart then you’ll have HA up for every single turn, but usually taunts come more frequent than that. Not to say that DP is better, just that the comparison isn’t so straight forward.

      You do probably have a point regarding Alabastar Shield, because Final Wrath is already so close to it, that if we factor in downtime off the boss Final Wrath will probably pull ahead.

    • Theck says:

      The Holy Wrath formula in SimC is the correct one, not the tooltip version.

  12. Aribethe says:

    I realize that DP will be smoother in a sense. But the TMI’s are very close and assuming 30s tank swaps would double HA’s uptime, which would bring things closer. The other thing that tank swaps will add is the ability to pool Holy Power until you taunt. In the case of tank swaps, you can use HA for one turn then use CD’s for the other. Even breaking AW and HA so you can have extra healing for the turns HA is down.

    The point was more that you need to look at the sim knowing its assuming 100% actively tanking the boss and make decisions based on that.

    If SimC is using the correct value of Holy Wrath, should it not be higher in priority than Hammer of Wrath as it hits harder?

    • Theck says:

      Yes, the default rotation should probably be CS>J>AS>HW>HoW>Cons. I haven’t written the rotation sim, which should automatically answer that question for us.

  13. Aribethe says:

    To further clarify what I’m trying to get at, its more look at the fight and use the sim data accordingly to help you make decisions.

    Say 30s tank swaps or greater, I’d lean towards HA. 15 or 20s swaps maybe not so much and I’d go with DP.

    Or heavy add fights, Alabaster Shield might be favorable over Final Wrath.

  14. Pingback: 5.4.2 Rotation Analysis | Sacred Duty

Leave a Reply