Armory Error 503 - Server Busy

Mar 3, 2008 at 3:48 PM
This has been bugging me for a while.

I've done some digging on the WoW forums and I think I've the answer - Blizzard is throttling Armory access by IP banning folks who mine information off of it. Some thoughts on the forum is about 4-5 XML page views per minute. And, once you exceed that, they block your IP for about 12 hours - giving you the same 'Server Busy' error.

This runs in-line with what I've been seeing on my development machine - it will successfully obtain the roster and scan about 2-3 members before getting the 503 error.

I've been thinking this morning about how to deal with it - and I have an idea...but wondering what your thoughts are on this.
1 - Make this a roster only list and abandon the stats/items. This means a single page lookup and no worries.
2 - Create a second, throtteled process. This would mean that I would set the current service to only scan the guild XML file and build up that list. I would then have a second service that ran every 30 seconds or so, and would pick up one character per run. This would mean that it would take about 2 hours to refresh a guild roster, but it would work and get around the Blizzard IP-ban (unless you have multiple services running on the same IP address).

Mar 4, 2008 at 6:38 PM
Edited Mar 4, 2008 at 6:39 PM
Unless this is information specifically from Blizzard regarding usage, I would doubt this is really what is going on. This can be invalidated by running a test on the server during the day when the server isn't expected to be busy. I can pull up 20+ xml sheets in a minute for 10 to 20 minutes at a time without problems.

What is a problem with the armory is that the service is extremely flaky. Often times the site just doesn't come up at all, or the browser has some rendering issue and locks up. I understand that accessing the armory is a big use case behind this project, but I've often viewed the sole reliance on this use case for data gathering as a weakness.

What I would suggest is to revisit the fundamentals of this module. Right now, if the armory service isn't available, this module is frankly worthless. Instead of being highly dependent on the service, the module should be independent and use the service as a consult if it is available.

Take for instance my DKP system's relationship with WowHead. If WowHead isn't responding to item requests (fortunately almost always does), it's no big deal. The creation of an item and the association of a DKP value with it does not depend on the item existing in WowHead's database or even being able to contact the wowhead servers.
Mar 4, 2008 at 8:27 PM
In it's current implementation, you're right that it has a sole dependency.

The code, however, is more flexible. The original code this is based on pre-dates the armory -- and could be loaded either by hand or uploading an LUA from a custom WoW UI Mod one of my guildees wrote (we called it 'Guild Basics' - it exported list of players, last played, player notes, and officer notes to an LUA file).

What other methods of population would you like to see?
Mar 4, 2008 at 10:51 PM
Ah yeah, this si the part of this project that I often forget. I tend to assume the module is as functional as the code.

I think supporting a minimal set of manual edits should be supported and might include such things as...

1) Adding removing player names from a roster.
2) Changing any data item displayed on the guild roster's main view (not the player detail view), so things like level, spec, class, professions.

Supporting player initiated uploads from game dumps would also be good, but not sure it's as important as manual administrative control.

Supporting different types of Armory fetch would be good.

Might not want to fetch stats for alts. Lvl 70 characters shouldn't have to worry about level changes. Profession switching isn't that common. Talent changes can be common, but don't necessarily correspond with a change in the player's reason for being in the guild. (My priest will occassionally switch between healing and dps depending on what the need is, but I'm always considered a healer when it comes to my position in the guild and for recruitment needs. So even though I become shadow for a run, the guild roster shouldn't show me as a shadow priest.)
Mar 7, 2008 at 3:44 PM
BTW - I tested this a couple days ago - a run of the roster and about ~350 characters killed my access from one IP address, while I could remote desktop into another machine and use that public IP's just fine.

And, in discussion on the company DL for other players of WoW - other folks coding against the Armory confirm the same behavior when doing large amounts of queries. :(

The fun part will be trying to determine the bounds.
Mar 7, 2008 at 3:49 PM
Edited Mar 7, 2008 at 3:50 PM
And thanks for the thoughts - one thing I was asked for was the ability to say 'show me guild chars, sorting by +healing and mp5' -- which I think runs in line with your 'not all stats are needed' thought.

What about if I allow inline manual editing (of someone's own character(s) and for 'certain DNN roles' (officers))? And, at the module settings area, it will have the option, by roster for (a) display roster, (b) manual editing, and (c) LUA loading (which hasn't been implmented yet). How would you ideally like to see data entry (especially around 'add new player')? Admin screen? Inline?

This also means some DB col changes to track char status (manually loaded, armory loaded, manually deleted, etc) - as this has the potential to create conflicting states, depending on how things are modified. But this sounds like a good idea, the manual changes are extremely easy - as the business logic is all there, it's just lighting it up in the ascx page.
Mar 10, 2008 at 12:14 AM
In the v1.3.x release, I've gone and added in display of 'last loaded' and 'last updated' values.

My thoughts around manual data entry in the v1.4.x release are two-fold:
- On the character stat page, providing a 'Modify Character' button/hyperlink, which is only visible to the owner/officers (much like the 'main character' drop-down box today). Clicking this will bring up a panel below the 'character sheet' that will allow the following to be edited:
  • Change level
  • Numeric spec (three text boxes: xx / yy / zz) that will enforce logic to not all more points than level - 9, if over level 10
  • Change professions (drop-down box) and levels for those professions (1-375)

- I will create a new page that has a link similar to the 'Control Panel' link today, which may be called 'Roster Data Admin' or something (any thoughts on name?), when clicked, it has a datagrid view that is editable:
  • Can add/delete characters
  • The list will show: name, guild rank, level, race/sex, class, spec, profession1, profession2, IsPrimary flag, drop-down for main, drop-down for DNN user
  • Can modify name, guild rank, race/sex, class, level, spec, professions, main char flag, dnn user, and main char (to map an alt to a main)

This should allow some bulk-editing (which would be done by officers), as well as the one-offs (which would be done by the char owners)...and should cover all of the major bases.
Mar 14, 2008 at 8:11 PM
I'm going to start coding v1.4.x this weekend. My hope is that this will be a 10-hour development release (should I call it a 'sprint'?) - and completely tackle the data loading aspect, per above.

I had originally been hoping to work on it over this week, and post tomorrow - but I've been caught up getting things wrapped up on my current role as I prep to change teams here.

It looks more likely that I'll get it posted mid-week this week (hoping for Thurs, Mar-20). I will then be back on the east coast visiting family from Mar-21 (huzzah for the SEA->BOS red eye!) to Mar-27. I will try to get some work done on v1.5.x during that week, but it probably won't be posted until mid-April. Which may be good, I'm predicting some bumps in the new data loading logic, so we may see a v1.4.1 or v1.4.2 as we move along to v1.5. :)
Mar 17, 2008 at 3:30 PM
v1.4 Weekend Progress:
This weekend I dove back into the Armory loading logic and made the following changes:
  • I've created base abstract classes for roster loading and character entry loading
  • The base classes have proper hooks into the DataLoad logging, which is much cleaner than before
  • I rewrote the logic for the roster loading code to better handle errors on the character
  • The logic now (in the current state of the code) does an entire parsing of the guild roster while only loading the one XML file from the WoW Armory site (because it parses only membership, and not any character stats) - this yield parsing the entire roster within a couple seconds

I am now working on parsing the character entries. I've reformatted the code for the char stat parsing, but need to start hooking the code snippits back to the system so that it's executed. I am going to try and get this working this afternoon.

The major upside to the code refactoring is that it would be easy to write other services to sit on top of the base classes, if others were interested in writing services...and this also sets the stage for my LUA parsing plug-in model - 90% of the code and logic will work with an LUA parsing model.

I am still torn how I will set this up - I will configure the existing service class to behave as it works today - parsing the entire roster and then iterating through the character entries. I will also probably create two services - one for the roster, and one for the char entries, and enable them to be set up on their own in case users want to split the scheduled services. And, for those wondering, the 'existing service' will simply call the two separated ones.

Also - as a side note, I've cleaned upsome logic in the module around the data cacheing - removing some of the superflous try/catch blocks (now using an if statement instead), and also doing some checks around data cache schema aspects (such as relations) to help prevent some of the hidden exceptions that were going on in the background.
Mar 21, 2008 at 3:26 PM
After a lot of playing around with the code - I have the character retrieval working. The logic for 'retrieve and process x characters', and then be able to call it later to pick up where it left off is working out just fine...technically.

The issue now is one of logic - the roster retrieves fine, but how to walk that tree is presenting me some issues. With the WoWArmoryParser library I'm using, it attempts to retrieve a character XML file, but if it doesn't exist, it sends back an empty character entry - which happens to be the same result if the Armory has gone into 503 mode (and this behavior differs from the roster retrieval, which throws an exception when it encounters a 503).

I will be pushing back v1.4's release until I get back from vacation (next week) - and I am going to address this Armory concern. I will either create a new Armory Parser DLL that uses the LINQ sample code I posted to my blog last month, or else modify the source code for the ArmoryParser DLL I'm now using. Personally, I'm leaning towards using LINQ - but I need to see what mixing .NET runtimes in DNN does to the server.

This also potentially means creating yet another table to track DNWWGRSettings - to track things like 'number of characters to retrieve per Armory run' and 'Show Character sheet or have link to Armory on list' and the like. I don't want to create another table, but it's either that or store it in an XML file on the file system...which could introduce it's own problems.
Apr 4, 2008 at 1:17 AM
Yes yes yes - another date has come and gone. I tried working on the code while I was on vacation, but a virtual image on an external hard drive only gets you so far!

I mentioned this in another thread, but I've started a new role within MSFT - I'm now the product manager in the .NET Framework product marketing group, working on Windows Workflow Foundation (WF). This week has been busy fixing a couple starter kit sample bugs and getting logistics and demo scripts settled for TechEd 2008 (I'll be working the WF booth for part of the TechEd - Developer conference, if anyone else will be attending).

At any rate, this is floating to the back-burner while I get real life in order. I'm hoping to get a v1.4 cut out at the end of this weekend (since most of it's coded), but it may be next weekend. But this may take a long pause after that...just too much going on. If someone else is interested in picking this up in the lull, I'm more than happy to support and work with them - just let me know.

But this is stable (as stable as the Armory anyway ;-) ) - so I don't feel too guilty letting it sit until June once I get v1.4 out.
Apr 14, 2008 at 7:02 PM
I've currently changed the update schedule on my website to run once every 13 hours with no retries. Would this work as a workaround before v1.4 is available? (We've recently had another guild merge into us, and have yet to see the new members showing up on the list.)
Apr 16, 2008 at 1:03 AM
Okay - a reason to tackle this coding effort. :)

I'll get v1.4 moving then. I've been having a couple side conversations and will try to put a governor screw of sorts into the code and see if it clears all the entry pulls.

The current code will definitely pull in al the member stats, but it would fail on character stats.

What I may do to REALLY fix your issue is put in a custom armory parser DLL to better pull the XML files (I think 3-4 changes to his DLL could drastically reduce this issue). I've been trying to reach K, but he seems to have gotten involved in real life.

I'll wrap it up and get something out - I won't have time to crack open the code until this weekend. But I'll get the environment back up on Friday (I had redone my laptop for the new role -- too much beta code clogs the gears) and get something coded/up on Saturday.
Apr 20, 2008 at 7:26 AM
As an update - I think I nailed this one.

I've got the code done; I just need to package up the SQL file for v1.4.0 and make some presentation changes (I now have two separate data loads happening - one for the roster and another for the character entries - and I need to ensure that is communicated in the control panel page).

I'm also packaging in a 'branched' version of the Armory parser, as mine will throw the 503 errors from the DLL and the code handles it in escalating degrees...which should prevent this from happening again. The downside of the code changes is that it will take much longer for the roster to fully parse the stats for a guild (~15-20 minutes), but it seems to prevent the 503 errors from happening...which I'm hoping is an improvement for you guys.

I'll get a v1.4 package up tomorrow.
Apr 21, 2008 at 4:52 AM
I posted up code into the v1.4.0 release under 'Planned' releases.

Take a look and let me know what you think. I had a hiccup on one DNN portal with the installation - an odd error about a FROM message, but everything seemed to go okay afterward. Please let me know if you find an issue with it. If it hits you also, let me know and I'll dig in deeper.
Apr 21, 2008 at 1:56 PM
Good to see you got around those pesky 503s, hope the advice helped. :D
Apr 21, 2008 at 2:39 PM
It did!

I minimized some of it by hitting the entire roster first (and adding shell 'entries') and then doing a second pass to fill in the details - but the delay between xml pulls seemed to do the trick. Thanks for the tip!
Apr 23, 2008 at 3:12 AM
I know I as well as others are probably anxious to see 1.4 ready for install :) hows pesky RL treating you?

Apr 23, 2008 at 4:02 AM
2 notes on this:
  • There is a bug in v1.4 -- if the roster pull returns a 503, it doesn't do a load and actually blanks out the entire roster. (oops!)
  • RL is busy :)

And, for those interested, v1.4 is up for playing around - you need to click on the 'planned' release link on the 'Releases' tab. So you need to:
  1. Click on the 'Releases' tab
  2. Click on the 'Planned' link in the upper-right
  3. There is then a 1.4.0 link to click on

All of that being said, I'll get a 1.4.1 posted tomorrow night with the fix. I'm going to give the app a chance to do a second pull and see what happens on my live site. ;)
Apr 24, 2008 at 6:07 AM
Okay - I found the FROM error in the SQL - it's been fixed and shouldn't be bothering anyone.

I've posted 'v1.4.1' and posted it in the released section. I've also incorporated feedback from Bill into this release to add some 'color'. This can be overriden by creating a CSS entry in your portal's CSS file (typically portal.css) and overriding what I've put in there. In the packaged CSS file (module.css), I have placed suggested colors for light and dark backgrounds, but you should feel free to experiment using one of the many color wheel sites on the web.

Please let me know what you guys think!