Yesterday I mentioned that I thought I had a blog post in me about games testing and NDAs, and this morning we are going to see if that is true. For those who are uninitiated the NDA stands for a Non-Disclosure agreement, and if you have reached this step in life without having already signed one count yourself lucky. As a software developer by trade, pretty much every job has required one from me so I was more than familiar with them when I also started signing them for games tests. Since I am talking about material that cannot effectively be spoken about… I decided to mock up the experience of playing something heavily governed with the above image filled with watermarks. Currently I am engaged in three separate alpha tests and it seems as though the winds have changed. For awhile it was en vogue to do all of your testing as a public alpha that allowed those who were in to effectively act as free advertising for the game drumming up hype along the way.
The problem with that however is that I am not entirely certain it ever worked as intended. YouTube has basically made a cottage industry of mocking games that are not quite ready for prime time, which in truth should be any alpha or beta test. Originally those were times for the game to find itself and having a limited testing group helped to prune things that were not working and hone in and polish the things that were. However within the last decade these shifted from being a development mechanism to taking on a bunch of different purposes. You had some games where the alpha or beta served as an extended demo period… take for example Fallout 76 that went into “beta” on October 23rd and the final game “shipped” on November 14th. Having been around software development for going on three decades now… there are no meaningful changes that can be made in such a short period of time.
Another case that has sprung up are the games that began selling access to testing in the form of “early access” or “founders packs”, which amount to you helping to fund the development of the game and in theory helping to shape the features as they are being put into the game. I’ve purchased a number of paid alphas over the years, because in some cases especially with Indie games it gives you the opportunity to lock the game in at a bargain price. After all when I bought into Minecraft it was less than $10 and that certainly was an investment that paid off over time. However all too often games languish in early access more as a means of getting their shit together and keeping websites from publishing “official reviews” of the title since they can keep claiming that it is still in beta. Rust for example went into early access in December of 2013 and finally launched in February or 2018… which maybe seems like an excessive amount of time in testing. The other problem with early access is you are effectively squandering whatever hype you might have had upon launching the game… because effectively in the eyes of your players you launched a buggy game when you opened initial access.
The biggest problem with public testing is that while it allows you to develop a bit of a grass roots community on platforms like YouTube or Twitch… it also means that at any moment you could be subject to the same sort of blooper reels that effectively killed Mass Effect Andromeda. Within days of that game launching a number of the issues were cleared up and I found it to be an amazingly fun experience. However once the glitch videos started circulating it not only killed the game… but effectively killed the franchise. Opening your game to the world is effectively playing with fire and once you get to a certain level of hype there are going to be folks all too willing to shit on your game to make a buck and rake in the views.
So effectively what I am seeing as a result is that more games are going underground and slapping an NDA on so that they can safely get on with the business of testing. I was one of the very early tests of Elder Scrolls Online, starting with the very first external test in February of 2013 and continuing right up until the April 2014 launch date. During that time I watched that game change significantly sometimes for the better, sometimes for the worse but it was legitimately a testing process where we provided feedback and the developers reacted to it. That is how testing should be and started out with limited engagement testing every few weeks and then eventually worked up to being on for longer periods of time.
I’ve been in extremely bizarre alpha tests as well. One company required me to effectively fill out a contract that included the NDA as part of that, and not only sign a digital agreement but to print out a document and get it witnessed and notarized. In addition to that I had to send in a photocopy of my drivers license. I’ve had others where the game was technically restricted but I never actually wound up agreeing to anything and had a key just show up in my mail box with no mention of the NDA in the letter. I personally tend to take the NDA pretty seriously, and in twenty years of testing maintained a firewall that keeps information about said games out of the things I am willing to talk about publicly. I am only really comfortable talking about this in general right now because I am in three separate testing processes and there is no real way of you guessing which games that might be.
Sure it sucks that I can’t talk about the things that I am really enjoying, because there are things I could hype about each of them. There are also things that I would complain about each of them, and that NDA is effectively buying the company time to fix or at least mitigate those problems. I for one am happy to see closed testing returning to seemingly being the norm, because after years of public testing I am not sure the hype generated ever was worth the issues that arose from it. There will always be people willing to break NDA like the individual who streamed Anthem and got some reportedly major circumstances for doing so… but then later was confirmed to not actually have any games on their Origin profile to start with. The thing is though… breaking an NDA is against the terms of service of most digital distribution platforms and in theory you could lose whatever account you used to cause the break on. It would never be worth me risking my Steam account for example, just in the hopes of getting a few more eyeballs to find this blog or my neglected twitch stream.
To wrap this up… I am very much in favor of games testing starting to go dark again. That said my job is also not tied to talking about games. This is a thing I do for fun and as a hobby, and I have gone out of my way to not actually make any money from this blog in spite of the regular stream of folks who want to advertise on it. Were this my daytime gig I might feel completely different, because then I would be grasping at things to fill the current 24 hour a day gaming news cycle. Everyone loves seeing a sneak peek at games, and when you have alpha access and can take some really cool screenshots or videos to embellish your prose, it makes for a compelling user experience.
This is just my take, but I am absolutely open to other ideas. Are you in favor of games testing going dark or would you prefer that testing remain open and public?
I’m ambivalent. I am all in favor of a return to closed alpha and beta testing with limited participation and extended duration. On the other hand I also like the open development system seen in Early Access. What I don’t like is when the two are conflated.
Both ways are problematic. If you test your game behind closed doors then release it as a finished product you take the chance of it being received very differently than you expected because, in my experience at least, the players who stick it out through months or even years of genuine closed development bear very little ressemblance to the eventual audience who will pay for the game. ESO, which you mention, is a very good example. That game was panned on release and only went on to become the success it is by effectively changing many of its core systems to suit the live market. Had it used open development those changes would have happened at a much earlier stage.
Then there’s the “la la not listening” version. The original version of FFXIV, which I beta-tested suffered from developers who literally ignored all feedback from testers throughout the whole process. That launched so disastrously it had to be scrapped and re-written from scratch. Had it been in open development it would have been impossible to pretend all was going to be fine.
On the other hand, Early Access/Open Development opens the way for all kinds of scams, lazy shortcuts and wild changes of direction in response to bad publicity or player complaints. It also runs a huge risk of players becoming bored with the game long before it officially launches and the final release being a complete anticlimax as most people ignore it altogether.
As a blogger, I find being under NDA intensely irritating. It’s all very well companies protecting themselves against negative reporting but I’m in a closed alpha under NDA that I’d love to post very positive things about and especially post some of the hundreds of gorgeous screenshots I’ve taken but I can’t. It’s hard to see why marketing departments would want to suppress positive feedback. I’d be more than willing to submit my posts for approval before publsihing – maybe that would be a good compromise.
I am a huge fan of NDAs in testing. Yeah, it sucks when you enjoy one or get invited and can’t talk about it. I do like that most of them allow you to discuss it with other people who are taking part and if you know your friends well enough you can figure out if they are maybe, possibly, involved and through a little cat and mouse get it confirmed.
NDAs are important because… game shave bugs. ALL games have bugs. The only way to find most bugs is by seriously putting in play time and developers just don’t have that time. I don’t mind dealing with huge bugs during testing, some of them are even humorous.
My main gripe… and it’s a small one because I fully understand it… is the inclusion of *extremely* pronounced watermarking on alphas. It is difficult to pay attention to things with big text swirling around the screen. I’m also part of three right now and have done a lot of them over the years. Most have been enjoyable…and all have been educational.
I would prefer a return to the old way. No hints, no advance showings, locking out the data miners, which is a whole other discussion, and final release when it’s ready, not to meet player demands.