Games

I don’t want to jinx things but the queues appear to just be… gone. I spent some time popping around various zones and I did not encounter a single “wall runner”. For those who might not have been following along this saga, the term wall runner is used colloquially in New World to refer to someone who is exhibiting a behavior designed to evade the AFK Timeout functionality. There were various schools of thought here, but the most basic seemed to be toggling the run hotkey on and then pushing up against some obstacle that would keep you running into a corner. Moving around the zones you would find people in random corners of the world furiously racing against some rock, or gate… all in an attempt to make the game think they were actively playing.

Yesterday Amazon patched in some updated tech designed to detect these folks and punt them from the game. Then magically it was like we had enough breathing room to simply not have to worry about login queues at all. I popped in yesterday morning to take a few screenshots and unsurprisingly there was no queue. Then I logged in after work… and still no queue. Throughout the course of the night, I logged in and out a few different times, with one of them being around 8 pm and still no queue at all. In fact, it showed that our server population was only sitting at around 1700 players. So I guess the question is, was all of the activity by folks afking to reserve their place in line?

I think you have a few things at work. Firstly like every game, the newness has worn down a bit and those folks who took off days from work to play 24/7 have run out of vacation time. Secondly, there is the challenge that it was impossible to play on the busy servers reliably, so if our little group is any evidence… many folks voluntarily rerolled elsewhere and started from scratch because they had not made enough progress to make that painful yet. Thirdly, of course, there is always a significant drop-off in a new game after a week or so, and we are seeing that play out here. For all of those folks who crushed the server on the opening days, there are going to be a chunk that for one reason or another didn’t really find what they were looking for. New World is not going to be the ideal game experience for everyone, and as a result, you are going to have a significant chunk of players that just completely bounce off the title no matter how good it might be.

All of that said… I really do think the improved detection made a significant difference and the early signs that maybe just maybe folks were going to get banned for the behavior. What we are left with is a situation where those original few servers now don’t have queues but are still in a locked-down state to prevent new players from rolling on them. Similarly, we have a fragmented community of folks waiting on the server transfers to allow everyone to once again play together. That said I find myself questioning what the next best steps are going forward. I am left with the majority of a community on one server that has no queues, and an isolated but growing number of people taking up residence on other servers. Is it the best choice to try and get everyone on Minda, or is it to abandon ship and latch onto a completely new server?

If we united the community, we have maybe in the realm of fifty players. Coordinating that many people landing at the same destination is going to be a bit of a challenge. At least with the loss of the queues, it takes away some of the immediacies of needing to do this. My hope is we can take a bit of a wait and see as the server transfers open up. I still think that the server status website is our best bet in gauging active populations, but I do wish it had a 48-hour peak number or something to that effect so that you can see what the potential for a server load looks like, not just what the active one is. In the meantime, I am nearing level 40, and probably need to stop what I am doing for a bit and focus on some faction currency given that I would love to buy the level 40 set of faction gear.

I do feel a little bit bad though in that the worst seems to be over… and now a handful of folks have scattered to the wind. I hope we can reunite the community and get everyone back together on either Minda or some new destination. I will of course keep folks updated as to where we decide to land. For the short term however I want to see how folks shift before choosing a destination to leap to.

This weekend I was not certain how much time I would get to play, because the weekend tends to be when we do things. For the most part, I got to hang out and play New World all Saturday because it was raining buckets outside, and neither of us had the desire to leave the house. Sunday however it became rapidly clear just how untenable our current server situation is. Because of just the nature of how our Sundays go, I spent time in queues three different times for a grand total of four and a half hours spent watching this box tick down. Essentially if you cannot log in before noon, you are not going to have what I would call a reasonable experience. Thankfully my machine is capable of running multiple games at the same time so on the longest of these queues… 2.5 hours, I spent time over in Final Fantasy XIV doing roulette.

One thing that I can attest to with a fairly high degree of confidence is that the tool we have been using to estimate how long a queue is… is pretty freaking accurate. The three queues that I entered yesterday were pretty much dead on for what the queue estimates were at the time of me logging in. These are some sample queues from the site at the time of writing this, and I sorted them by the most players in a given queue. This more or less tells the tale of New World right now. Notice that a few servers have seemingly been bumped up to 2250 players as a test, but I am starting to doubt that they plan on scaling these up very much higher. In fact 2000 players seems to be an important number for the design and stability of this game. I provided some server data on Friday when I wrote about the game and since then 131 more servers have been added… only further increasing the fragmentation of the player base. The current data centers look something like this:

• Frankfurt, Germany – 230 Servers
• Arlington Virginia, United States – 200 Servers
• Sydney, Australia – 70 Servers
• Umatilla Oregon, United States – 64 Servers
• Sao Paulo, Brazil – 44 Servers

Amazon has made some attempts to flag players who are trying to get around the AFK timeout and force them out of the game. Cities right now are rife with players running in place against obstacles or auto-attacking while standing still via a macro. So while I am happy to see them making strides in keeping this from happening, I also don’t think it is going to really go very far in solving the problem. The core issue is that the day one, day two, and probably even day three servers are way too populated.

Amazon has officially locked many of these overly populated servers so that new characters cannot be rolled on them. However, this action probably came a little too late for it to make a difference either. If you have a fixed server size in mind as they did… maybe stop character creation at 2.5 times that number? On opening day there were servers with queues that were upwards of four times the size of the total number of players allowed on a server. Ultimately I am just not certain that staying where we are is going to be tenable much longer. There has been a discussion that Amazon is trying to rush a solution to allow players to migrate elsewhere, and right now I am thinking that is our next best hope. Once this opens up, find a smaller server and then coordinate a move to that location so that we can recreate the company again on the other side.

There is a lot of really fun group content available in this game. I spent some time over the weekend closing “Rifts” aka corruption breaches and they were great. I also got pulled into a random group with players from various factions and did Amrine Expedition the first dungeon. All of this is pretty awesome content, but also things that would have been so much more enjoyable had I been able to do it with my friends. The core problem with the game right now is actually getting players online. It is making all of the core systems function a little worse as a result. For example, I watched a video from a YouTuber that talked about how hard it is to make the War system work right now because they just can’t reliably get players online at a fixed time in order to meet up and defend a territory.

There are times I am committed and connected to the community on a server. For example, I would refuse to ever move away from Cactuar in Final Fantasy XIV, because that server is just phenomenal. In New World, however… I think pretty much every server is a dumpster fire right now and Minda is not really a place I care about significantly. I am deeply connected to my character and would not want to re-roll elsewhere. I am deeply connected with the people I am playing with… whenever we can actually play together. I am not however connected to any given server, so far as I am concerned our best bet is to move. This is going to still have some pain associated with it because I am in a place where I need those tier 4 crafting machines, but I can deal with being delayed if it means we can all play the game in a reasonable manner.

I think the plan going forward is to wait for the server transfer tool to open up, and see what it does to server populations. Then find a relatively low to medium population server and transplant our little group of players. The players that rerolled on low pop servers are seeing a lot more of these sweet sweet loading screens than those who have stuck by Minda. At level 33, I am too deep to contemplate starting over, so my last hope is for a server transfer.

Good Morning Friends. I find myself still here and still playing New World. I did however take a break and play some Final Fantasy XIV… while the New World client was up and idling in the background. Yesterday I logged in over lunch to about a thirty-minute queue, and that was enough anxiety to have me just leave the client open all day long. After yesterday’s blog post, I took to “the twitters” and threaded up some commentary for anyone who might not read my blog. My advice this morning is still pretty much the same as that thread, if you are not already bought into the game then I would highly suggest just waiting to pick it up until the server congestion situation is more reasonable.

The New World team took to their Twitter account and posted this message, trying to put a positive spin on the current situation. The tweet talks about how many servers they created on the fly and how they are working to increase capacity on each server. It also goes on to say that if people just pick any available server now they are working on a way to get folks transferred onto a server with their friends. The problem here is this fundamentally is not how an MMORPG works. You need those friends the most in that early rush as you are attempting to do dungeons and collaborate on getting folks geared. Even if you are not actively playing with other folks, there is something necessary about just knowing that friendly voices are out there waiting on the other end of guild chat. There is something deeply flawed in the design of this game and I will go into detail a bit but first, we need to talk through some numbers.

Right now at this very moment, we have the following server capacity:

• United States East Region – 158 Servers available
• South America East Region – 35 Servers available
• Europe Central Region – 195 Servers available
• Asia/Pacific Southeast Region – 40 Servers available
• United States West Region – 49 Servers available

Looking through this you notice some strange data points. First off the EU region has significantly more servers than I would have expected at this point based on the sizing data from other MMORPGs. The United States West Coast region seems grossly undersized to what I would normally have expected based on my experience with other games as well. In grand total, you have 477 servers and if my math is correct a maximum capacity of 954,000 players could be playing the game at any time. There is a maximum single server capacity of 2000 players based on the metrics we have available to us, and there are many realms that have more players in the queue than actively playing on the server.

In yet another tweet thread, I surmise that there is something about their design model that is fundamentally easier to scale horizontally than to scale vertically. This 2000 player limit seems to be important, as does the 100 players in a single company. That tells me this game was more or less designed for 20 companies to be duking it out for the resources of Aeternum, which makes me wonder if New World more or less started its design cycle as a slightly larger scale Battle Royale game, and morphed into an MMORPG game as players violently rejected its PVPcentric design. We know without a shadow of a doubt that AWS can scale upwards extremely well, and you can very well keep throwing resources at a problem by adding more processing power, memory, and disk space ad infinitum. So that tells me there is something on a software level that makes it a challenge to scale above 2000 players in a single server.

Let’s talk about that 2000 player number a bit. Once upon a time both Everquest and Dark Age of Camelot showed the current player population of each server on the login screen. Quite frankly this was something they were both proud of because you have to understand… coming from games with a maximum player population of 16 in a single match, having thousands of players was quite the feat. The average peak for those servers generally speaking was in the 2000-3000 player range. As we move forward to the modern era of MMORPGs, I have heard that 6000 players are the point at which the login queue starts to kick in on a World of Warcraft server.

Final Fantasy XIV requires a bit more factoring to come up with some sort of estimate. Based on the player knowledge I have… most people do not use the Steam client, and I would guesstimate that at most 1/5rd of the player base connects that way because you have a huge group that plays through their console. There are 68 total servers in FFXIV, and Steam recently reached a peak concurrency with the game of about 67,000 so for sake of making things simple let’s call that 1000 steam players per server which again is going to be a deeply flawed number. If we are taking as an assumption that the number represents a fifth of the player base then my guess would be that would give you around 5000 concurrent players per FFXIV server.

So you might be asking yourself… Bel, why did you go through all of that contorted math that you know is going to be completely wrong? Just trying to paint you a picture of what the modern MMO landscape looks like as far as player population caps, and how 2000 per server seems extremely small based on the demand already existing in mature games. In an immature game, there is going to be even more of a burst of players that needs to be dealt with as folks are currently hungry for any new games coming out regardless of genre. New World also is a special game, because it represents the first completely new “Western” MMORPG in years. Western players have developed a pretty sour taste in their mouth for the microtransaction hellscape that are Korean and Chinese localizations. Based on the recent flurry of activity in Final Fantasy XIV, there should have been signs that the community was primed and ready to latch onto something new and fresh given that the entire World of Warcraft player base seems freshly unmoored.

So once again I fall back on, that there is something specific to their design model that indicates 2000 players as a specific unreachable boundary. Otherwise knowing what I know about AWS, it would have been far simpler to add additional processing power to the servers that were already provisioned than to keep spinning up new nodes. There has to be some reason on the software side that they are not doing this, however, and it makes me question if they are confident that they can scale upwards. Alpha and Beta testing didn’t really give them an idea of what a real-world player demand looks like. We went into those tests just happy to be playing, and not much caring if we were playing with our core group of friends. However, an MMORPG launched is a totally different beast, and players organize into dense clusters as everyone tries to make landfall on the beach that their friends already arrived on. Having more servers doesn’t really do much to dilute the demand placed on those first few servers.

Now we are going to swing back around to why the servers per region numbers look fucked. Amazon shot themselves in the foot with this launch and rolled the game out in a manner as to encourage players to dogpile a limited set of servers. Amazon rolled the servers out by region in a staggered manner… causing the EU and NA East servers to fill up completely because once players are established somewhere, there is entirely too much inertia to get them to move elsewhere when more appropriate servers are opened. The schedule looked a little something like this:

• European Servers Opened at 11 pm PDT on 9/27
• South American Servers Opened at 4 am PDT on 9/28
• Asia/Pacific Servers Opened at 4 am PDT on 9/28
• North America East Servers Opened at 5 am PDT on 9/28
• North America West Servers Opened at 8 am PDT on 9/28

The first servers to come online were in the European Data Center, and there was a mad rush to get in and reserve your name. This is another flaw that added to the problems we are dealing with, but the fact that character names are globally locked and not tied to a specific server. As result, there were a lot of folks that popped into those servers to create a character reservation, and likely a handful of people who just wanted to play the game period regardless of server environment stuck around and started playing the game. The next big avalanche came when the East Coast servers opened, and since there is a relatively insignificant difference for most players between the two environments… every single major guild chose to start the game at this point.

Regardless of how painful the launch might have been for some players, the staggered launch and the fact that there was a land run on virtual real estate… aka player and company names made everything that much more important that you got there first. The problems that we are dealing with in this New World launch are absolutely problems of Amazon’s making, in part due to sheer lack of experience in dealing with the MMORPG player base. There is only about a 100 ms ping difference between me playing on my NA East server and playing on a European server, which means for most people the choice of where they landed was largely meaningless. New World was treated like a highly localized experience when time and time again the players have proven that they are a global audience.

So here we are with this botched launch and questionable game design… that we could easily walk away from were it not for the fact that when you ARE able to get into a server the game is damned addicting. New World is a great gameplay experience when it is operating under optimal conditions. My hope is that they have engineers working on how best to scale the servers, and as one of my guild members commented… if they doubled the capacity per node most of the queues would go away instantly. I think they need to do precisely that and at the same time offer free server moves at a company level. Let entire companies transplant themselves on greener pastures, because quite frankly unless they specifically are holding territory there isn’t a lot connecting people to a specific node at this point.

I hope they can make this work because the game legitimately is good. We haven’t had an MMORPG launch that was this solid in a very long time. My advice stills stand for anyone waiting and watching from the wings. Keep waiting. If they manage to pull out of this tailspin I will be the first to raise notice that it is time to start paying attention again. Until then, however, all that waits for you is a whole lot of frustration and anxiety buffered only by just how good the game actually is.