Respawn Entertainment gives more detail about the Cloud and Titanfall
Let’s talk about the Xbox Live Cloud
Posted by Abbie Heppe on Jun 24, 2013
Hi everyone! I’m Jon Shiring, and I’m an engineer working with the Cloud technology that you’ve heard about for Respawn’s game Titanfall. I have seen a lot of confusion online and I think it’s worth explaining more about what we’re doing on Titanfall and also more generally about Cloud computing and dedicated servers.
First, let’s take a step back and dive into the common multiplayer design and talk about why Dedicated Servers are better.
Player-Hosted Servers
The vast majority of games will pick a player and have them act as the server for the match. This means that all of the other players talk to them to decide what happens in a game. When you shoot your gun, the server decides if that is allowed and then tells everyone what you hit. Let’s agree to call this system “player-hosted” for simplicity.
What kinds of problems do you get with player-hosted servers?
What if one player has great bandwidth, but it’s laggy? Games are having to choose between different player hosts, and have to make hard decisions about which one should be the host, with two different measurements – bandwidth and latency. Sometimes it will pick a host who has good bandwidth, but whose latency isn’t ideal. But we don’t want the game to make compromises on lag and we really want the game to feel the same every time we play. We really don’t want to worry about this stuff – we just want to play and have the game feel good.
What about host advantage? The player-host has the game running locally on their machine, so they get super low latency access to the game world. You’ve probably seen this in action as some player seems to see you long before you get to see them or their bullets hit you before yours hit them. That sucks. Nobody should have an artificial advantage in a competitive multiplayer game.
What if the player-host is a cheater? Since the host gets to make decisions about kills, XP, and unlocks and such, it’s really bad if they abuse their power to wipe out your stats, or they cheat by flying around maps and insta-killing people. It’s infuriating, in fact.
What if the host disconnects? In the “best case” for this, you can do host migration if there’s another player who has enough bandwidth and everyone else can talk to them. If you hit that jackpot, you can migrate from the old host to the new one, which pauses the game and then unpauses when the new player-host is ready to start acting as the server. This isn’t a fun process, and it can fail.
What if the host’s bandwidth disappears? The game tested the host’s bandwidth at some point and decided that they had enough to host. But someone at their house is now torrenting files and their roommate is streaming Netflix. That “great” bandwidth the game detected earlier is now awful bandwidth, and the other players are lagging halfway through a match.
What if some players can’t talk to the host? You know all that “Open NAT” stuff? Your home internet router is generally trying really hard to keep bad people out, and games are sort of a weird case where the game is trying to get your router to cooperate and let other players create connections INTO your network. Games need to check if every player can talk to the host and if one can’t then that host won’t work. It makes matchmakin slower, and we hate that. Also, by telling you to open up your router, the game is asking you to reduce the security of your home network in order to make the game work. It would be great if you didn’t have to compromise your security in order to play games.
What if nobody has enough bandwidth? You got a great group of players together, but nobody has enough bandwidth to actually host a game. You can work around this by compromising your matchmaking a little to make sure that each lobby has a player in it that can be a host. But we don’t actually want compromised matchmaking, so this isn’t a good fix.
What about players who are paying for their bandwidth or have bandwidth caps? If you have a bandwidth cap on your home internet connection, or even worse, you’re paying for your bandwidth, what happens when you play a game and later find out that the game thought you were an awesome host? Your home internet connection is now slow or you have a huge bill waiting.
So if I’m hosting, my machine is doing all this extra work on behalf of everyone else? Yes! You are doing more work on your CPU than all of the other players are. This means the game isn’t as cool looking as it could be and everyone else has extra CPU just sitting there. Or worse, their game actually looks better than yours! We think the game should be consistent on every machine in a match. Don’t punish the host with a worse game or leave all of that extra CPU sitting empty on the other players machines.
Okay, so player-hosted servers have a lot of downsides. So why do so many games use them? They have one really big upside – it doesn’t cost money to run the servers! Running hundreds of thousands of servers can be extremely expensive. EXTREMELY expensive. Like “oh my god we can’t afford that” expensive. So your player experience gets compromised to save (large amounts of) money.
Dedicated Servers
Dedicated servers are when a computer sitting out on the internet handles all of the host duties, leaving every client free to just be a client.
- You can get even more CPU on your dedicated servers to do new things like dozens of AI and giant autopilot titans!
- Suddenly you have no more host advantage!
- Bandwidth for the servers is guaranteed from the hosting provider!
- You can use all of the available CPU and memory on the player machines for awesome visuals and audio!
- Hacked-host cheating isn’t an issue!
- Matchmaking can be lightning fast since it’s guaranteed that everyone can connect to your servers.
- And since the servers aren’t going to go disconnect to watch Netflix, you don’t need to migrate hosts anymore!
- The player experience is so much better. This sounds awesome!
But it costs a LOT of money.
This is something I have worked on for years now, since coming to Respawn. A developer like Respawn doesn’t have the kind of weight to get a huge price cut from places like Amazon or Rackspace. And we don’t have the manpower to manage literally hundreds-of-thousands of servers ourselves. We want to focus on making awesome games, not on becoming giant worldwide server hosting providers. The more time I can spend on making our actual game better, the more our players benefit.
I personally talked to both Microsoft and Sony and explained that we need to find a way to have potentially hundreds-of-thousands of dedicated servers at a price point that you can’t get right now. Microsoft realized that player-hosted servers are actually holding back online gaming and that this is something that they could help solve, and ran full-speed with this idea.
The Xbox group came back to us with a way for us to run all of these Titanfall dedicated servers and that lets us push games with more server CPU and higher bandwidth, which lets us have a bigger world, more physics, lots of AI, and potentially a lot more than that!
......