Why My Sites Went Dark

Tales of the Aggronaut circa May 2010

When Tales of the Aggronaut was born, I was in the fortuitous situation of knowing someone who worked for a web hosting provider. It was someone I hung out with on the regular and even occasionally raided with, so it was super simple for me to ping them via Google Chat when I needed help with something. However things happened and that friend left the web host, which caused me to see significantly degraded support and performance. It was around this time that they also went through a round of restructuring which saw me not only paying more for my hosting plan but also having to swallow bandwidth and disk space usage overages.

The final straw that broke the camels back is that I went through a period where every morning the hosting environment would be too unstable for me to actually publish content. The problem with the arrangement with my original host is that when they first started out years ago they saw themselves as a multi tier hosting environment that would eventually play with the big dogs. However as time progressed they put more of their effort into Colocation and Disaster recovery data centers and effectively stopped selling any of the hosting plans. I only got to stay on the service because I had been with them so long at this point, but they were theoretically unwilling to offer much in the way of real support.

I ultimately made the jump when I needed to find hosting for the church my wife attends. Since we would be footing the bill for it, I wanted to find some place cheap, reliable and without the spectre of bandwidth overages and disk usage caps. I had used bad experiences with Arvixe and InMotion various work environments and after a bit of back and forth I finally landed on BlueHost. After seeing how generally good the service was I went through the process of moving my main sites over Aggronaut.com and Aggrochat.com, and for the most part everything has been pretty cromulent. Occasionally there would be some stability issues, but it would always resolve itself quickly.

On May 19th that was not the case and I experienced what felt like a server outage. Between the hours of roughly 6 am to 10 am my sites were experiencing intermittent connectivity issues. They alternated between loading fine and getting an assortment of Connection Reset, Connection Closed and Connection Timed Out errors in the browser. At this same time the CPanel associated with the environment my sites were all hosted on was exhibiting the same behavior telling me that this was a server level thing and not a specific problem with my sites.

This lead me through a Dantes Inferno-esc series of useless web chat sessions for two hours while this was going on. I ultimately went through four or five different chat agents as one would mysteriously get disconnected and another would immediately join in their place… causing me to ultimately start from square one in explaining my case. Finally the last of these said that they would escalate the issue to the server techs and that I should receive an email back explaining what was wrong. Within 15 minutes of ending this chat, the waters had calmed and I assumed that something must have been happening on their end ultimately putting it out of my mind.

Yesterday morning however I woke up to the nightmare scenario of seeing these littering my inbox. Sure enough when I hit any of my pages it threw a 404 error. I was completely dark to the world. I began freaking out, connected into CPanel and it seemed as though all of my files were still there and fine, and it was only after this that I noticed another email sitting in my inbox that arrived about 2 in the morning.

While reviewing your account regarding your website being down.  I noticed a few of your file permissions were strange and there was quite a bit of added files that were not named in a way that one would expect. This prompted me to run a malware scan. When the scan finished, it had found compromised files in your account, and in order to protect you, your website visitors and the stability/reputation of the server, we have temporarily suspended your account until the malware has been cleaned. 

This is just a snippet of the email, but it gets to the important part. Immediately following this is an attempt to upsell me cleaning services through some group that they partner with advisement that there was now a “Malware.txt” file sitting in my site root that would explain what the scans found. My mind immediately went to the worst, because I have had to manually clean a site before. It is a slow and tedious process, but also I remembered that two days earlier I had gone through all of the processes to look for any possible signs of infection. I scoured directories looking for any new files that shouldn’t be there, so I was pretty certain my site was clean.

I download the text file and open it to see this. I feel like at this point I need to back track a bit to explain something. Having gone through the nonsense of having to manually clean a site before, I have both an Application Firewall and an Malware Scanner built into my WordPress installation. I regularly receive emails talking about various threats being blocked. What happens in this case is that the file gets dumped into a quarantine directory for me to inspect at my leisure and this directory is walled off from the world with a .htaccess file. These were the strange file permissions that the tech was talking about and these were the supposed Malware files that they were referring to.

I was furious that they ran a scan, saw something… and immediately jumped to shutting off my account without making any attempt to interpret the resulting log files. There are lots of application firewalls available for WordPress at varying price points, and they all effectively do the same thing. There is a quarantine directory that intercepted files go to so that you can clean them if needed and restore them back to your site. This is something that a hosting environment should have encountered before, and they in fact offer similar tools that you can pay extra to have installed on your sites. Someone ran ClamAV, saw that there was a non zero number of files reported and jumped straight to account suspension. To make matters worse, the “rate how I did” link included in the email did not work and told me that it had timed out already… for an email I had only received 5 hours prior.

This once again lead me to return to Web Chat support, which was as bad as I was two days prior. Effectively I was left with the answer of that it would take at least 24 hours for someone to come along, scan my site again, and restore it. After ranting to fellow bloggers on Twitter about this for awhile, I finally decided to try one last avenue… contacting the Twitter based support account. It claimed to be staffed from 7 am EDT to 2 am EDT, so I figured it was worth a shot. It took about an hour for someone to respond, but what transpired after that was a series of DMs with the main Bluehost account. I was surly as hell at this point because all I wanted to do was scream obscenities… but reading back through my responses I mostly kept my shit together.

I managed to get them to rescan my directory, and got back a messsage saying that I still had infections. This lead to what was ultimately the most hostile reply that I made during the entire exchange. “Seriously? Those went into the trash when I deleted them from before… do you not review these at all?” Apparently when I deleted all of the quarantined files earlier, they automatically went into a .trash folder in the root of my account. So now ClamAV was detecting them there, which again if someone were actually looking at the results other than checking for a non zero number, they would have been able to interpret that.

Once I regained some composure, I made my way back into CPanel File Manager and emptied the .trash folder and got the support agent to scan my account again. To be fair, once I shifted away from using the Web Chat and over to Twitter support, the experience was fairly smooth all things considered. There is an inherent awkwardness of the delay built into asynchronous communication but I figure for all future questions I will go straight to the twitter option. The account was scanned once again and finally ClamAV returned a clean bill of health. The final step involved a frustrating sequence of me firing off authentication tokens via DM because again the delay of the medium caused the tokens to time out before the agent could verify them.

At this point the Support Agent flipped a switch and restored everything back to its previous working condition. I still have no clue why their servers were unstable two days ago, and I likely never will. It was an extremely frustrating experience, but in the end once I switched over to Twitter support I managed to get a resolution. Bluehost is for the most part a fine hosting environment, and I have had more good luck with it than bad. However the support from the Web Chat is fairly abysmal. Any issues I have had before seem to magically resolve themselves shortly after getting off chat with someone who claims that they can’t help me.

At the end of the day however you get what you pay for, and I am not paying much for this hosting so I guess I am getting the overall support I deserve. The twitter folks though are on point and I will be relying on them as my primary vehicle for support from this point forward. I could say “Well I Never!” in a huff and transfer hosting providers… but that is such a colossal pain in the ass to do so and I am super lazy. For now I am just going to move on having vented about this tale and call it good.

A Discord Problem

Me and the Wumpus at Pax South 2017

Today has been one hell of a day, and tomorrow I will regale you all with the tale. However this evening I am going to actually bang out the topic that I originally was going to write about. This is a topic that was inspired personally by Chestnut, who was in turn inspired by GamingSF. Both are posts worth reading so you should totally check those out. Both tell a tale that I find very familiar about how they struggle with establishing a method of using Discord on a regular basis. Personally I find the whole experience of discord and having so many fragmented servers to be a little on the overwhelming side. However first let me show you the level of problem that I am personally dealing with.

A Sea of Server Icons

This image does not represent all of the servers that I have joined. At this very moment I am sitting in 57 different servers, all with their own structures and rules and overlaps between my friend base. Some are social in nature, others are associated with game testing programs I am part of, and others are associated with various projects that I have supported over the years. The biggest challenge with all of this is that they vary in activity, stratification of infrastructure and community focus. There are a few that I have focused on interacting with specifically, but even they overwhelm me as I never can seem to be truly “present” in any of them.

The Blaugust Discord

The saddest part for me is that I feel like I am failing to keep up with even the discords that I started like the one associated with Blaugust. I try my best to keep tabs on the Moogle’s Pom Tavern and Geek to Geek media network but fail miserably at both of those goals. The weird thing is that I am fairly active on Slack with the AggroChat Podcast crew and I use Microsoft Teams all day every day to keep in touch with my coworkers during this time of Quarantine. For whatever reason whoever I have struggled to really find a place for Discord in my life. Maybe if AggroChat were primarily located there it would be a bit easier, but whatever the case I have it up and running in the background but find I only dip my toes into the water on occasion or if I am specifically trying to play games with a specific group that uses it for voice communication.

Snipping from the Chat Mixtape feature request

I think for me, in order for me to really use it well… I need a feature that doesn’t exist yet or nor likely ever will. In general I only care about the general chat channel on a given server, and I wish there was a way to create a “virtual server” of sorts that blended together more than one server. I posted this on the feature request but it didn’t gain much traction, in part because I decided to call it a Chat Mixtape, which is a thoroughly 80’s reference. I just find it too cumbersome to really try and be active in more than one server, but at the same time I find it really hard to jettison older servers in the odd chance that I might need to interact with them. The 57 servers represents what I am in now after having culled a large number of them over the years. On paper Discord is everything I want in a chat client, but in practice I just find it hard to attach for the purpose of anything other than the occasional voice chat session.

Do you also find yourself struggling to stay connected and engaged with Discord? Drop me a comment with your own thoughts, or maybe tell me how you stay “present” on the platform.

Old Man eyes and Game Generations

Unreal Engine 5 Demo on PlayStation 5

This week we were able to see footage of what this generation of video games will bring us on the new consoles with the release of an Unreal Engine 5 demo running on live PlayStation 5 hardware. Before I dig into this topic I want to throw up a disclaimer that nothing I am about to say is meant to discount the hard work or how revolutionary this may be for game developers. It does in fact sound like it is going to reduce the work load significantly allowing them to directly use assets rather than trying to bake them down to usable assets in a game engine. However from a pure visual standpoint, this seems less like a generational leap and more like cranking the settings from High up to Ultra in a video game.

Unreal Engine 5 Demo on PlayStation 5

I am wondering if we have gotten to the point where graphics are good enough that any improvements just end up feeling marginal? We reached this point with pixel graphics during the jump from the 16 bit era to the 32 bit era with games like Castlevania Symphony of the Night, not really looking that much better than games you could have seen on the Super Nintendo. Sure you had a much wider palette and could therefore display a wider variety of colors on screen at once. However if you jump between playing SoTN and a game like Dracula X that released for the SNES there isn’t much of a noticeable difference.

Legend of Zelda: Ocarina of Time – Nintendo 64

We’ve been on this journey with 3D graphics that started out with something as primitive as this, and evolved rapidly over the course of a few generations. We’ve gotten used to each generation being instantly and noticeably better than the previous generation. I am starting off with the Nintendo 64, because while it technically released after the PlayStation it seems like the floor of 3D graphics that were available during this generation.

Vagrant Story – PlayStation

It may be a matter of preference but for me at least games like Vagrant Story showed the PlayStation 1 era to be cable of a superior experience. The Nintendo 64 games felt childish and block, whereas the PlayStation games were capable of conveying rich emotions. Now these could come entirely down to art direction and decisions made by the studios, but for me at least it felt like I was leaping forward a full generation when I moved from the 64 over to my Dual Shock Playstation.

God of War – PlayStation 2

When I moved up to the PlayStation 2 the leap was even more noticeable. The textures were smooth and gone was the blocky nature of the characters. Other than the 4:3 aspect ration of most PlayStation 2 era games, I still consider many of these to be completely playable. God of War was one of the better looking games on the system, and since I have some direct comparisons to games on the PlayStation 3 and PlayStation 4, I figured I would use this as a reference title.

God of War 3 – PlayStation 3

To be honest, looking at these two screenshots, I would argue that this generational leap at least with this title wasn’t as big. Last of Us after all originally released on the PlayStation 3. However just between these two shots you can see higher fidelity of assets and better lighting. It is hard to tell it in a single image, but the main thing that came with the PlayStation 3 was bigger and more open worlds, whereas the PlayStation 2 mostly focused around more static scenes, you had a sense that you were roaming a living environment.

God of War – PlayStation 4

Now I realize this is a completely different type of game to the previous God of War titles, but it no less highlights something important. For me the leap forward that happened between the Xbox 360/PS3 era to the Xbox One/PS4 era was significant. This was the first time we were capable of playing in what felt to be photo realistic worlds, and the games were completely capable of carrying off that illusion. For years PC games had effectively stagnated due being tied to the 360 and PS3, but this is also the generation that 4K became a major player in gaming and with it a higher level of fidelity than we had experienced before. This generation felt like it was the biggest leap to date, for me personally.

Unreal Engine 5 Demo on PlayStation 5

There is no denying that the Unreal Engine Demo footage looks good, but when you compare it directly to an image coming from a game of this generation, I see higher fidelity but not enough to make it clear that we are making a generational jump.

Jedi Fallen Order – Xbox One X

If you compare the above image to a screenshot from Jedi Fallen Order, I am not sure if I could tell they were effectively running on different generations of hardware. I think the challenge there is that we had a half generational jump in the middle of this generation. Notice that this image is from an Xbox One X which is capable of 4K gaming, whereas the base model is still stuck in the 1080p era. I think this generation is going to feel like another incremental bump forward, and not necessarily the radical shift that I was hoping for.

Horizon Zero Dawn – PlayStation 4 Pro

Granted I am cherry picking examples of some of the best looking games from the current generation to use as contrasting evidence, and they are being placed up against a demo reel. However I do feel like it is an important question to ask ourselves as we go into this next generation. If you are already running an Xbox One X or a PlayStation 4 Pro… is this leap worth the $500 to $600 price tag that is going to come with it? For me it likely is because at least on the console front I never took the half leap and am running an original Xbox One and PlayStation 4. So yeah it probably will be a sizeable upgrade, but for those rocking a 4K capable console or PC right now, I am questioning if this is a big enough bump to make it feel worth it?

Unreal Engine 5 Demo on PlayStation 5

All of this said… it could just be that I am getting older and that my eyes are not capable of picking out the same level of visual fidelity that they once were. However if I went into this scenario and you threw screenshots of all of these 4k games in with the Unreal Engine footage, I am not sure if I could immediately tell the difference. Granted all of this is ultimately moot because I am mostly a PC gamer, and maybe it is just that I am used to the fidelity of games that can be produced on that platform already with significantly higher performing hardware than the consoles will likely ever have. Ultimately this has left me with a feeling that the games look great, but that ultimately I expected more in some way that I can’t quite quantify. I feel like we have maybe reached the limits to what is visually possible, and everything moving forward is going to be a battle of margins and not the big leaps forward I experienced throughout my gaming career.

Age of Shareware

Commander Keen 4

I grew up in what I consider to be one of the best eras in gaming, namely the mid to late 80s and early 90s. This was the era of the 8 bit and 16 bit juggernauts by Nintendo and Sega, and while it was years later when I first experienced it NEC as well. However there was something else going on that damned near knocked me out of console gaming entirely. In 1991 my family got our very first computer, a no-name 386 16 mhz with a massive 90 mb hard drive and 2 mb of ram. There was no sound card and we were still several years ahead of CD-ROMs being a thing that you would regularly see in a computer. However the same friend of my dad that used to send him home VHS tapes filled with movies from HBO, used to send me floppies loaded with games for me to play on our new computer. Howard was a member of a BBS, an through that he would get all sorts of things some what I would later learn as “Warez” and others something called “Shareware”.

Apogee ASCII Catalog

For those not old enough to remember this era, the idea was simple. A company would release one fully playable level of a game and distribute it freely on Bulletin Board Systems and FTP sites. Folks would download it, play it, and if they liked it you could buy a code that would unlock the full version of the game and let you play the rest of the levels. They also distributed their entire catalog of games in an ASCII text file format along with all of the pertinent information on how to purchase the games. Being a teenager and not having access to a credit card, it would be years before I was able to play the full versions of most of these games. For example I played Spear of Destiny long before I actually played any level of Wolfenstein other than the first one. The same is true with Doom and having played Doom II before the later episodes, in part because in both of these cases they got a physical release stocked at our local Walmart that didn’t require me to convince my parents to give someone a credit card number over the phone.

Duke Nukem II

The first of these titles that I played were the original Commander Keen and Duke Nukem, and I remember at the time not being able to understand why these games were not released for the Nintendo or Super Nintendo. I was completely unaware of the proud history of effectively home brew game development on Computers like the Atari ST, Amiga and Commodore 64 because I simply wasn’t exposed to it at the time. All I really knew was the original Atari 2600, and then the 8 bit and 16 bit era consoles. It was after I got access to the internet that I more or less descended into the madness of all of the other options and got heavily into the Amiga scene when I picked up a 3000. At this time however it was extremely common place for ALL major game releases to offer a freely downloadable demo. When CD-Rom entered the scene is was extremely common for a Games magazine to have a pack in CD filled with demos of various products that were either out or coming soon.

Playstation Demo CDs

This wasn’t just a computer thing either. During the PlayStation and Dreamcast era, I remember demo cds for both systems in regular circulation. I used to subscribe to a PlayStation magazine, and each month there would be a CD included that had short demos for a lot of the titles that were just about to release. Once you moved into the PlayStation 2 and Xbox era of game consoles, the demos existed but were significantly less common. When you arrived at the Xbox 360 and PlayStation 3 era, the concept of downloading a demo had more or less been forgotten by developers. The era of try before you buy was a thing of the past, and this was also the era of several game releases becoming controversial for not quite living up to expectations.

Nintendo E-Shop with Switch Demos

While none of this is Shareware, I find it interesting that at some point over the last few years something changed. Nintendo Handheld devices have often had downloadable demos to sort of whet your appetite and get you interested in buying the full game. When Nintendo released the Switch I started noticing how many of these first party and major third party games had a fully playable demo that you could download ahead of time. Not only was the game playable, but often times you could pick up where you left off in your save file giving you further incentive to pay some money to continue that gaming experience. For example I absolutely played the Demo for Trials of Mana, and while I decided to start fresh after-all with a different party, I could have easily just picked up where I left off.

Steam Game Demos

This is a trend however that might have been happening under my nose for longer than I realized. Now that I look around it seems like there are many digital storefronts offering demos, and that might be what ultimately changed. Digital distribution, just like in the golden age of shareware, has become more a primary means of getting titles out to the public. It costs money to press a demo cd and distribute it out to the stores, but uploading a demo version to a store front is effectively free. It feels like maybe we are just about to go through a second age of Shareware, and while you are not downloading the games from some University hosted FTP server that you found through Gopher, you are still downloading them freely. So if you are curious what is available in demo form you can check out the following Storefront links that should in theory bring you right to the demo sections.

As someone who often writes impressions of games that he is enjoying, I should start digging up links to see if demos are available for that game. I can write all I want to tell you how cool I think something is, but giving you access to download freely and see for yourself is significantly more powerful.