GovHack 2017

Nice.

GovHack is wrapped up for 2017. It was finished on the weekend, but Monday and Tuesday were a waste - I have been too miserable and sick to blog about it, and only just got a burst of energy to break through the mehffort. Our entry, Death Who? (Colonial Edition), is a virtual card game based upon real lives recorded in Tasmanian historical records. I wanted to share what I’ve learned about making a super hacky multiplayer game backend from scratch in 46 hours, in case you want to try the same thing.

The past few years I have flown down to Hobart to team up with old friends. As usual, I stated up-front that I wanted no share of any prizes we might win. I’m literally in it for the fun of hanging out with my friends.

The mix was a bit different this year. Our team of 8 consisted of me, Hannah, Paris, Mars, Jon, Tim, Seb, and Shea. Our new team members Hannah, Mars, and Shea all brought their own different strengths this year; Hannah and Shea worked on a web-based component, which is unusual for our team. Previous regular Rex went and did a solo entry which you need to check out.

Similar to past years, I made a backend for the game in Go. Each time this has been a server of some kind, depending on what is needed. There are a few preliminaries that can be done while the team settles on a direction, but it’s often better to have the decisions settled on Friday night, get a good night’s sleep, and then get cracking on Saturday morning. This time, we did the latter.

Constraints

  • Serve a game based on some government-provided data.
  • Work with multiple players.
  • SLO: It has to work for demonstration purposes (what might be called “best effort” SLO, but I have a whole other rant about that).
  • No need to worry about bandwidth, CPU, memory, disk usage, or any of that as long as it works for demo purposes.
  • Players can cheat and hack the server as much as they like - demo purposes, see above.

As such, the server doesn’t have to be Google-class production-grade stuff. It just has to work for 30 minutes. Nevertheless, it’s relatively easy to do the right things in Go. I think it’s still running even now…

Hello, GovHack

The designers of Go had web servers in mind, so adding one feels natural. I generally jump straight to a skeleton main.go in a server directory in the GitHub repo without thinking much more about it:

package main

import (
    "flag"
    "fmt"
    "log"
    "net/http"
)

var httpPort = flag.Int("http_port", 23480, "Port the webserver listens on")

func main() {
    flag.Parse()

	// Set up HTTP handlers
	http.HandleFunc("/helloz", func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprint(w, "Hello, GovHack 2017!\n")
	})

    // Start listening on HTTP port; block.
	if err := http.ListenAndServe(fmt.Sprintf(":%d", *httpPort), nil); err != nil {
		log.Fatalf("Couldn't serve HTTP: %v", err)
	}
}

If you look carefully through our repo history, you’ll notice that flag.Parse call was missing until Sunday. I frequently forget the basics!

The beauty of running a web server is that it lets you (1) provide a way of checking that the server is still running, and (2) provide a way of inspecting the game state at any point without logging—just leave a browser tab open on localhost:23480/something and refresh when you want to take a peek. So let’s set up a handler for that:

	http.HandleFunc("/statusz", func(w http.ResponseWriter, r *http.Request) {
		// TODO: write the game state to w
	})

If you’re wondering about the z at then end of /hello and /status: I really can’t help putting it there. It’s a Google-ism.

State

It’s best to agree upon the basic structure of the state-space with the people making the client before going any further. In this case, we have a multiplayer card game with two types of cards. We whiteboarded some states:

  • Lobby: the players are joining the game, the game hasn’t “started”
    • When someone pushes the “start” button, transition to…
  • In game: the game is underway
    • Players take turns. Once one player plays, go to the next player.
    • Once all players have had a turn, begin the next round.
    • Once all rounds have been played, transition to…
  • Game over: the winner is proclaimed.
    • The game can be returned to the lobby state by pushing a button.

Start setting up some types to track the state. I like to do this in a separate package, but there’s no compelling reason to be so neat in a jam/hackathon.

package game

// Statum is some fake latin I made up
type Statum int

const (
    StateLobby Statum = iota
    StateInGame
    StateGameOver
)

type State struct {
    State     Statum          `json:"state"`
    Players   map[int]*Player `json:"players"`
    WhoseTurn int             `json:"whose_turn"`
    Clock     int             `json:"clock"`
}

type Player struct {
    // ... snip ...  
    Score       int     `json:"score"`
}

While it’s no big deal in Go to handle sending/receiving different JSON types nested inside some kind of genericised “message” type by adding MarshalJSON/UnmarshalJSON methods, to make it easier for clients I recommend avoiding doing that. In this case some parts of the state only have meaning depending on other parts of the state (e.g. WhoseTurn and Clock only mean something when State == StateInGame). To make it even easier we are also sending the entire game state to the client—we don’t care about cheating (it’s a hackathon).

Since this is a multiplayer game, be sure to smother state mutations in mutexes. sync.RWMutex is great and lets you get better performance in some cases but a sync.Mutex would be fine (it’s a hackathon). It would be possible to use the new sync.Map here, but we have to guard more than just the map from concurrent access and I like concrete types, so embedding a mutex makes the most sense.

type State struct {
    //...snip...

    // Non-exported fields for bookkeeping
    mu     sync.RWMutex
    nextID int
}

func New() *State {
    return &State{
        Players: make(map[int]*Players),
    }
}

func (s *State) AddPlayer() int {
    s.Lock()
    defer s.Unlock()
    id := s.nextID
    s.Players[id] = new(Player)
    s.nextID++
    return id
}

func (s *State) RemovePlayer(id int) {
    s.Lock()
    defer s.Unlock()
    delete(s.Players, id)
}

Player is defined in a way that its “zero value” is a sensible default. State is almost but not quite as simple, hence the New (it contains a map which we’d like to just use, but nil maps don’t work that way). It starts in the lobby state and supports arbitrarily adding and removing players. You could use a slice for storing players but then you will either have to handle nil “holes” in the slice, or fiddly logic with reslicing. Just use a map (it’s a hackathon). I might use a slice with nils next time.

Serving a game

The next step needs you to settle on the communication between the game client and the server. We chose what seemed like the easy thing which was to send JSON messages over TCP. Typical “serious” game servers often implement a compact binary protocol over UDP. It’s straightforward to do JSON and TCP in Go, and you can see how I did it this time in server.go. The example TCP listener in the net package documentation is a nice starting point. However, some notes.

Firstly, an infinite loop will… infinitely loop, blocking its goroutine indefinitely. Since this binary is also running a web server, one of the two has to be executed in a new goroutine. (Best practice is to have the goroutine created only in the main func so it’s obvious what its lifetime is, but this is a hackathon.)

Secondly, unless the server is synchronous (in the sense that there are no messages sent from the server that aren’t in response to something from the client), you need 2 goroutines per connection: one for receiving and one for sending. One client can affect all the other clients, so this was needed. We planned for the server to just spam the clients with state objects as it pleases.

Thirdly, it is very important that goroutine leaks are avoided: they might be lightweight but they consume memory and CPU cycles, after all. Contexts are great for this, especially in larger server projects. Here they serve the purpose of keeping the sending and receiving goroutines organised. When the context is cancelled, the connection can be closed and both goroutines can end. Additionally, the context can hold some per-player state. For a while I was using the context to hold the player ID, but instead went for an explicit parameter. Instead of a cancellable context, it is pretty much equivalent to give it a “quit” channel that gets closed for cancellation, but I use contexts all the time at work and didn’t bother to think about it much (it’s a hackathon).

Notifying all the clients

The goroutine handling outbound data (the imaginatively-named handleOutbound) is notified by a channel closing when it is time to transmit the game state, but this bears a little closer examination since there’s a great time-saving upside to this: there is no need to implement a registry of things to send notifications to, Go can handle it.

Firstly, remember that all reads on a closed unbuffered channel finish straight away and get the zero value. The outbound handler is an infinite loop around a select waiting on that channel or on the context. Here’s the part in State:

type State struct {
    // ... snip ...
    changedNote chan struct{}
}

func (s *State) Changed() <-chan struct{} {
	s.RLock()
	defer s.RUnlock()
	return s.changedNote
}

func (s *State) notify() {
	close(s.changedNote)
	s.changedNote = make(chan struct{})
}

Every time Changed is called it returns a channel whose sole purpose in life is to be closed in the future by notify. When the state changes (and it should only be changed by methods that do the correct locking), notify is called, closing the current channel and replacing it with a new one. (Calls to notify need to be guarded by the mutex.) Anything that is interested in the state can then just call Changed, and proceed once the returned channel is closed.

However, it’s a hackathon - why not do something really cheap and use a timer to spam updates every second (or something?):

func (s *Server) handleOutbound(conn net.Conn) {
    for range time.Tick(time.Second) {
        s.state.Dump(conn)
    }
}

This is fine, as long as steps are taken to avoid the goroutine leak (hint: it should end when the context is done, which means selecting on both the ticker and <-ctx.Done()). But it didn’t occur to me to do this at the time. Using channel-closing as a snappy notification system for arbitrarily many clients is a technique I’ve used a lot, and feels very natural in Go.

One thing that’s important for development speed is to give the client developer (Jon) a simple message to send the server that does nothing, to ensure communication works from the client side without much effort. Here’s Action:

type Act int

const (
	ActNoOp      Act = iota
	ActStartGame
	ActPlayCard
	ActDiscard
	ActReturnToLobby
)

type Action struct {
	Act  Act `json:"act"`
	Card int `json:"card"`
}

The zero value for Action has Act = ActNoOp, so, sending the empty JSON object {} works as a no-op message. This also helps manually testing the server: you can netcat/telnet into it and manually enter {} (or real actions as JSON).

Unit testing

Test what it does, not how it does it.

It’s a hackathon: if you don’t have time, don’t bother with unit tests, and just test manually. However I can hardly live without at least one or two unit tests. By writing a unit test against your actual API you force yourself to understand some implications of the API design.

Go doesn’t come with a mocking framework. You don’t need one. Run and test the actual server:

func TestGame(t *testing.T) {
	s := server{}
	r := &response{}
	if err := s.listenAndServe("localhost:0"); err != nil {
		t.Fatalf("Couldn't start: %v", err)
	}
	defer s.Close()

    // Connect player 0
	conn0, err := net.Dial("tcp", s.Addr().String())
	if err != nil {
		t.Fatalf("Couldn't connect: %v", err)
	}
	defer conn0.Close()

	send0 := json.NewEncoder(conn0)
	recv0 := json.NewDecoder(conn0)

    // ... snip...

    // Play a game!
	for i, p := range actions {
        // Send an action as one of the players.
        // Check each player receives the state.
        // Check the state against the desired state.
    }
}

All that really has to be faked is the card deck: you don’t want a flaky test because your virtual players got dud hands. So the game state uses a deck that you give it satisfying a Deck interface, and it has two implementations: the real deck which can be Shuffled, and a fake RiggedDeck which is the same but calling Shuffle has no effect.

Adding the, y’know, game

Adding the game logic isn’t all that interesting: with the above in place, it would be possible to make it into almost any kind of multiplayer game (with tweaks). The biggest tweak would be reducing the outbound data to only partial state updates, but that adds unnecessary complexity (it’s a hackathon).

Implementing the rules of the game is a lot like careful state bookkeeping. Having clear delineation of states helps a lot. There are explicit and implicit actions, e.g. player 1 plays card 3, versus player 2 has disconnected. It is important that concurrent actions don’t corrupt one another - use mutexes and the race detector (go test -race ...). It’s also important to think about what states could be “black holes” (often related to implicit actions). For example, if someone disconnects during a game, then the game shouldn’t wait for them to play. Or another example: if a player successfully connecting requires the game to be in the lobby state, and all the players disconnect, then the state should reset to the lobby state so people can rejoin.

Fortunately it’s a hackathon, so it’s allowed to have bugs galore, but I don’t think there are many. 😜

The game is data-driven (a bunch of historical data is churned into game cards). Data wrangling was done by Seb and Tim. We agreed early on that the data should be a JSON-formatted array of objects, which is not hard for them to encode and the server to decode. Loading the data is straightforward (hello my old friend json.NewDecoder) but the magic is in creating the cards out of them. (A few loops though.)

Conclusion

It’s a lot of fun working with such talented people in a hackathon environment. Making a good game server involves a mix of planning, technique, experience, design, teamwork, and communication. I’m once again looking forward to GovHack next year!