Testing Times: Part 2 – New generation, same old problems?

Editor's Note: Given that the person being questioned is a tester, it stands to reason that any next-generation console referenced would be in an unfinished state. This is, however, their opinion on how events appear to be shaping up.

'Testers hate developers because they think they don't listen to them. Developers hate testers because they think they don't know anything. And management hates everyone.'

Quality Assurance departments are strange places, unruly yet hardworking little things, stuffed with Peter Pans and ageing trolls, inquiring minds and those after an easy ride.

Territorial, combative, relaxed, fraternal, stressful, monotonous but creative: they're houses of contradiction, relied upon to ensure Blockbuster X is all it can be and yet cast aside at a moment's notice. Testers, and their departments, are caught in the middle of a power struggle between developer's dreams and executive cost-cutting, a weird world of half-importance.

For all their worth, testers are often employed on zero-hour contracts. Their work is seen as unskilled, the digital equivalent of those whose job it is to sit at a production line and weed out defective toys.

At least, that's how it was when I worked in the industry, between 2007-2009. Under the condition of anonymity, I sat down with a senior tester at one of the world's largest publishers – who is working very closely with the next-generation consoles – to see if anything had changed. And, if it had, what effect it would have on the games we were waiting to play.

The first issue we talked about is one of the most contentious: outsourcing of testing. When I was in the industry, this was becoming popular, as publishers recognised the supposed benefits of cheaper labour in foreign countries.

"All our base work is [outsourced] now, or given to test centres at other places. I guess they're probably treated the same way we were back then, so they probably aren't trusted.

"[Foreign or outsourced centres] get a huge [turnover] apparently, where they get lots of people and they lose them straight away. Then they hire new people and they don't know anything and it just keeps going round and round."

Trust – and its related issue of employee retention – are huge issues in QA. Anyone below management or senior tester roles is seen, fundamentally, as disposable bodies in seats: a strange state of affairs given that you're entrusting the quality of your multi-million dollar game to these people.

It creates a strange atmosphere between those on permanent contracts and everyone else; a hierarchical schism not helped by the relative immaturity of these offices as a whole. (When I was at Electronic Arts, testers weren't even allowed in through the front door, instead funneled through a back entrance: a pathetic state of affairs that has thankfully now changed.)

It also feeds into the constant expansion and contraction of teams. It's standard QA practice, yet it creates nothing but more confusion and admin work for those that remain when the axe falls. Testing teams are often required to get to know (and correctly use) fairly complex software for tracking and testing their issues. The average working day is made up of reviewing 'scripts', and testing the specific issues within them. As an example: in one build of a famous open-world racing title, it was possible to drive your car out of the world at specific places. In the next build, you'd have to go back and make sure this couldn't be done.

This, along with ticking off checklists related to stability (does the game crash if I disconnect the controller?) is the lowest level of complexity. Persistence teams (who make sure that XP unlocks and the like work as they should) and UI teams (who test the front end and menu functionality) have to work together on a regular basis, compiling and amassing reams of data. As you can imagine, hiring and firing people isn't conducive to getting jobs done quickly or efficiently, no matter how good the guys are that you're taking on board.

On that note, we then started talking about Xbox One and PS4. The arrival of a new console generation is as exciting for testers as it is for consumers or journalists: new toys, new challenges. It also strains the industry's churn-and-burn, unskilled attitude to its staff, as getting your games in a fit state to ship requires serious skill at all levels. Especially, if like our man says, one of those consoles is in a terrible state.

"The Xbox One is completely not ready" our source told us. "I can't believe they're releasing that console." Usually, testing has a meticulous structure, with deviation limited to set times. With the Xbox One, Microsoft's failings have allegedly given testers freedom to create their own solutions to problems. "[In a way] a lot of the stuff actually feels a lot easier because Microsoft don't seem to know what they're doing."

"Far from being the well-oiled process that the launch of the Xbox 360 was, Xbox One is the exact opposite. When we got Xbox 360 for test, [the machine] was done. The console was ready and everything. Whereas the [Xbox One] is completely not ready.

"In one instance, a tester had to write their own software to get Microsoft's machine to do what they wanted. We have to use [archaic] command line prompts to do anything on it. I think there's some guy on a certain title who [had to] write a graphical user interface for it himself!

"[I've seen] stuff that [state the Xbox One] was meant to come out like, next March, which makes sense because that's when everything's now dated for. They didn't have background music running [until recently]. Things that are on the 360 just aren't on the Xbone. I've no idea why."

Such stories aren't new – when I worked on early-ish releases for Playstation 3, many of the games and tools were in an appalling state (a personal favourite being when you fired a gun in a well-known multiplayer shooter, black blobs would come out instead of bullets). But they are important reminders of the importance of having a motivated and stable QA department.

As of now, however, it seems too many publishers find it easier to simply chop and change personnel, throwing more manpower at a project whenever it approaches crunch (EA once even went as far as to handing out recruitment flyers to people in pubs), the infamous software engineering silver bullet that rarely works. The sheer bureaucracy of organising, funding and running these departments also has a negative effect on the games themselves.

"I always think that QA holds back so much of games, things like LAN support and stuff they won't allow because that's going to require too much testing. But end users would love to have that."

It's difficult, with budgets being as they are, to see how this will ever change. Our man said that things had improved, with testers being allowed to bring in their own games consoles to play at lunch and access the internet. But he caveated it by saying that was because there are so few testers left that those remaining were fairly senior anyway.

It's a strange world, and one that doesn't look like it'll change anytime soon. "I reckon they'll just keep doing what they're doing."