I recently left Crowdstar just shy of my three year anniversary. Before the memory of those three years fades into the crunch of trying to make my own games, I thought I’d look back at the best and worst parts of those three years. Before I get into that, though, it might be worth a little bit of background.
After Piqqem (my first job out of school) folded in January of 2010, I resolved not to get another job and to dive full time into indie game development. I’d only been a half-time employee for the last year anyway, spending the rest of the time focusing other random pursuits (iphone and xbox indie versions of Filler, a review site for other indie games, a couple of new Flash games, a movie showtime search engine that got a nice write-up on TechCrunch but was too expensive to run). One month into “going indie,” a recruiter emailed me pitching three different companies: AuroraFeint, Sibblingz, and Crowdstar. The story was the same on all three: game related, YouWeb funded, growing like a weed, desperately in need of engineering bodies. Instead of getting back to the recruiter, I looked up Peter Relan (the founder of YouWeb) and shot him an email asking how you get into the program.
We chatted on the phone and he thought I should have a chat Suren and Jeff, the two founders of Crowdstar. What actually happened ended up being more of a 3-4 hour “interview” of sorts. I talked with a couple of their Flash guys for awhile about what they were working on and games I’d made–one of them later told me that it wasn’t a real interview, but a “push-on” (their attempt to recruit me). After chatting SQL and ruby awhile with the guys at Sibblingz, I finally went into a big glass office with Jeff and Suren. Jeff immediately asked for a resume (which I hadn’t brought) and ended up settling for a LinkedIn profile chock full of all the bullshit made up websites and companies I’d been tinkering on for the last year or two.
We talked a bit about my background, but the thing that stands out the most was one question Jeff asked me: “Would you rather make a game that a million people play and like or 100,000 people play and love?” Crowdstar was clearly already in the business of the former, but up until that point I hadn’t really considered it. Only one month into my “make games to support myself” experiment, I was still making games that I wanted to play (and not really considering games as an economic vehicle). I gave him the 100K answer, which is clearly what he wasn’t looking for. Then again, I hadn’t signed up for an interview–I thought I was coming up to Burlingame for a chat about social games in general.
Peter invited me over to his house a day or two later and offered me the job of lead dev on Happy Island. His pitch was simple–working alone or even starting a new game company would put me way behind where Crowdstar was already, and he was willing to make me “CEO” of a game and let me run it like my own startup. It was a tempting pitch, but at first I was mostly in it for the money–I told him I wanted half a percent. Playfish had recently sold to EA for $300 million dollars and social games were exploding, so I figured if Crowdstar could grow to that scale and exit within a year or two I could clear $500k-$1mm over the life of a 4-year vest. I’m not sure he heard me right, because he said yes and asked if I could start the next day. I said yes.
As it turns out, Happy Island was in flux. Originally developed by Sibblingz, it was a flash game with a Rails backend. My first day was a Wednesday. Sibblingz was due to roll off the game on Friday, and Crowdstar had not a single rails engineer. As you might guess, that first week or two was crazy as hell–setting a good tone for what would follow. A rough timeline:
Rather than go through it all chronologically, I’m just going to ramble a bit about the ups and downs–mostly ups!
Before joining Crowdstar, I had experience with all the relevant technology–Ruby on Rails, MySQL, Memcached, Flash… but never at scale (well, maybe a little on the Flash side). The day I joined, Happy Island had 2.5 million DAU (still climbing on it’s way up to 3 million peak). The other web apps I’d worked on maybe peaked at around 100-200 DAU. When Filler launched and hit #1 on Digg back in 2008 (when Digg was cool!), it peaked at around 200k players per day. Not even 10% of what Happy Island was doing in normal day. On the web side, the biggest I’d done was a single cache server, a single app server, and a single MySQL box. Island was running multiple memcached boxes, a bunch of sharded, replicated MySQL instances, and a boatload of app servers (each running still more rails processes through Unicorn).
The first few months were crazy–coming up to speed on a new codebase (both client and server), writing sharded database code, adding new features, fixing bugs… and all the while monitoring the live deployment and waiting for things to go wrong. Operating a game at that scale basically requires you to think like web-dev MacGuyver. Something is always breaking or going down or getting hot, and hotfixes deployed in the heat of the moment are more like duct tape than carefully engineered responses. Crowdstar had (and still has) a great ops team, and I probably learned as much in my first two months at Crowdstar as any other time in my life.
Most of the hard work (architecting and scaling up) had already been done on Happy Island, though–I was mostly a caretaker. When I later rolled off and started up the It Girl team, we were building from scratch. I was lucky to work again with the guys at Sibblingz (now Spaceport), who are much better engineers than I am. For It Girl we ditched MySQL and Memcached entirely, instead building the entire backend on top of Redis (including an in-house ORM instead of ActiveRecord). Redis has since gone in a slightly different direction (everything in memory instead of just the keys), but for our purposes it was fantastic.
It Girl never got to the scale of Happy Island and we had a few quirks pop up with Redis, but all in all it was a mostly smooth ride from 0 to tens of millions of users. Not having to think about which shard your data is on, along with not having to worry about caching, simplified development tremendously (and is one of the reasons we switched to Membase for future games).
After It Girl, I jumped on to another Facebook project that eventually got cancelled. By that time the company had shifted to mobile, and that shift brought a lot of changes to how we architected our games. Because it’s harder to rely on an internet connection and harder to mess with the in-memory data of a cell phone, mobile games tend to be a lot more client-authoritative. With simpler back-ends, there’s much less demand for server-side engineering.
Going from such a frenetic scale and pace (multiple deploys per day and constant monitoring) to something that felt almost pedestrian was a radical change. Part of this was due to the way mobile games operate. There’s no such thing as “push it and see what breaks”–mostly because of Apple’s onerous submission process (with 6-7 days being required for any client-side update). From what I know of console development, most mobile studios operate a lot more like our AAA cousins: develop -> QA -> release. It’s certainly a “healthier” way to run a business, but in many ways it felt like the taming of the Wild West.
At Crowdstar I was always the “rails guy” at a company mostly filled with PHP guys. The back-ends have now gotten simple enough that there’s not really a compelling argument for using Rails (actually, Closet Wars is a hybrid of Sinatra and Rails, but still) over PHP. Most of the back-end operations consist of either “get me this key” or “put this data in that key.” Had I stayed on, what was initially pitched as a Rails & Flash job would’ve been PHP & C++ going forward.
I’ve noticed that my experience with scaling Facebook games has affected how I view other companies outside of Crowdstar. Being an engineer in Silicon Valley, I’m constantly bombarded by recruiters via email and LinkedIn. Before working on scale projects, I wouldn’t have thought twice about the size of a company and instead just looked at what they do. Now I’m a little reticent to even consider joining a company that doesn’t have a crapton of traffic–it’s less fun! (Not that my own projects will have significant traffic any time soon).
Likewise, it seems like there’s a perception of hiring people with scale experience that reminds me a lot of the AAA console industry. Most job reqs at companies like EA will explicitly state that one shipped title is required, but there’s no way to get that one shipped credit without getting hired by someone with no credits. It seems like a lot of companies are looking for people with scale experience and not willing to hire those without it. In that sense I’m incredibly lucky to have been baptized by fire at Crowdstar.
Once I had a handle on actually running Happy Island at scale, I got to start playing around with the product a little more. None of the Crowdstar games at that time had time travel cheats (i.e. skip ahead 8 hours), which made it difficult to do some gameplay testing. There were serious problems for high level users, so I took it on myself to rebalance the entire game’s economy. After building a “simulator” version of the game where I could play two-to-three sessions “per day” just by pressing a button restart the session, I played dozens and dozens of playthroughs and eventually created a pretty tight escalating economy. It wasn’t perfect, but I think it was pretty fun (and certainly extended the life of that game). Even though my title was “lead dev” at the time, I was in fact more like a producer–coming up with new features, balancing the economy, running the revenue reports… as well as a good bit of code.
When we later added the actual job title of producer (while I was on It Girl), I became the producer for It Girl (and the unshipped title and Closet Wars). Up until recently, though, titles didn’t mean a whole lot at Crowdstar. Being entrepreneurial (read: curious and willing to poke my nose into places I probably shouldn’t), I wore almost every hat possible: client-side engineer, server-side engineer, QA (before we had a “QA Department”), product manager, content manager, producer, game designer, data analyst.
Working at a startup with more tasks than bodies affords someone who likes to get their hands dirty (me!) tons of opportunities to move around and try new stuff. This was probably most true on It Girl–as we “grew up” as a studio and people took on more well defined roles a lot of that flexibility went out the window. Even on my last game, though–Closet Wars–I was putting a lot of time into being producer, doing game design, and engineering. One of the reasons I never really wanted to go into AAA games was the fear of being a cog in a great machine–the fact that I was still able to wear quite a few hats on team sizes approaching 20-25 people has done a little to allay that fear.
The downside to having more tasks than bodies is that there never seem to be enough hours in a day. This was especially bad when I first joined–we were still in hyper-growth mode and I was new to a lot of the tech. Because I had as much equity as I did (and, to be honest, because I wanted so badly for us to succeed), I always felt like I needed to be the last one out the door. For a long stretch on It Girl, I’d say I was putting in 80-90 hour weeks. I wasn’t the only one doing it, though–I remember many 2-and-3 a.m. nights with the Sibblingz guys. We had this sort of visceral feeling that the future of the company rested on our shoulders–I don’t know how true that is, but we definitely felt it like a force weighing down on us.
Those kinds of hours weren’t sustainable, obviously. After Crowdstar raised a ton of V.C. money and started shifting more into a “stable studio” mindset, the hours definitely calmed down into a more normal 10-to-7:30ish, 50-hour Silicon Valley work week. Even working 10-to-7:30 on most days, though, I had to tack on an extra half hour to each side for my commute. Factoring in 8 hours of sleep and a bit of time to make dinner, I essentially had 1 hour in the morning and 3 hours at night to fit in everything in my life outside of Crowdstar.
It took me most of that first year to “finish” ColorTangle, the game I’d been tinkering with when I started CrowdStar–a game that I’d estimated needing another month to finish. And it probably was about a month’s worth of work–just crammed into a few minutes here and there on the fringes of 8-10 months. With tinker time going out the window, I also cut back on almost all of my leisurely pursuits: exercise, moviegoing (and I’m a film major!), reading books, and playing other video games. I’m a firm believer that game development is a creative pursuit, and when your world narrows to the point that you’re not consuming any media and keeping up with your body, that creativity suffers.
Once the baby came (February 2012), that claustrophobic feeling of never having enough time doubled or tripled (and especially never having time for myself). I’m sure that happens to all new parents, but I was on the verge of quitting for almost all of 2012 to try to win back some of that time. It certainly doesn’t help that I have an almost-neurotic desire to stay on top of all the current tech trends and try out every new gadget and technology that comes my way. What kept me going (besides my wife telling me not to quit) for most of that year was the fact that I had my own team (Closet Wars), we were making something that I legitimately thought was going to be great, and my team was awesome.
In that sense, I’m really excited to be working from home now–the hours and the commute were my #1 excuse not to exercise (except “i’m tired” or “i’ll start next week” or “slept funky last night”…), so hopefully I’ll be able to repurpose some of those hours into dropping 20 pounds this year (my pre-Crowdstar weight–damn you catered lunches!).
It’s one thing to read about A/B testing best practices on Hacker News, and quite another to run tests yourself on a massive scale and see what kind of impact they can have. When I joined Crowdstar, they were already A/B testing a little bit of stuff–but usually only running one test a time and modding on the user ID (which, at our scale, was probably random enough but not as statistically rigorous as it could’ve been).
While we were launching It Girl, it became pretty obvious that there were a lot of knobs and levers to tweak. The current mod-on-the-user-ID system would only let us run one or two tests simultaneously, and it was pretty brittle at times (was it user that end in 0-4 or 5-9 that were the control group?). I built a new system for It Girl that was not tied to user ID and was event-based, meaning we could split users into test groups at various steps in the user life-cycle. This was probably my favorite bit of code I wrote at Crowdstar, and now that I’m out I’d like to re-write it some day for my own personal use.
As an example–when we launched the boyfriends feature ,we knew we wanted users to have a minimum number of “clique members” (friends in the game) before they could access the feature… as well as a minimum level in the game. We were able to A/B test:
Between those tests and dozens of other tests we ran on price points, progression speed, and monetization features, I’d say that the A/B test system doubled or tripled the LTV on It Girl and taught us all a tremendous amount about player behavior. Maybe this goes hand-in-hand with running a product at scale, but it was also a hell of a lot fun to have access to that much data (in near real-time) and that much flexibility to test things.
When I joined Crowdstar, we were given two weeks of PTO (I think it’s up to 17 days now, which is better but still not great), which is totally standard for a new company. I travel a lot, so I was always scraping the bottom of the barrel when it came to PTO… and on several occasions either went way negative or took long stretches of unpaid time off.
Because all of the games we made were games-as-a-service, the usual lulls in the game product cycle just don’t exist. There was always something important going on, so picking a time to skip out for a week or two was always tough. Compounding my lack of time off was the fact that my wife works at EA–where she gets 4 weeks of PTO, the entire week of Christmas, and usually a couple of weeks of under-the-table comp time whenever they finish a game.
I had a bunch of people at work say things like: “how do you have so much time off?” — and I would always answer the same way: “I don’t.” I’m lucky that financially I could take the unpaid time (and was in good enough standing with the company to get away with it), but I always thought it was a downer that the majority of people worked so hard and got so little time off to show for it (though most people probably don’t travel as much as my wife & I).
To be fair, though, one mitigating factor for the lack of PTO was the company trips. My first year we went to Italy, renting a villa in Umbria and hacking for a week while surrounded by gorgeous sunflower fields (with day trips to Venice, Rome, Siena). My second year we had a hack week in Maui, and we just recently came back from Puerto Vallarta. These trips were awesome, but ostensibly still “work trips.”
When I joined the company, most people were either flash engineers or server-side engineers. There were certainly other full-stack devs on board (most of the dev leads, in fact), but there wasn’t really anything pushing people to do both. On my teams I tried to push everyone into learning ruby and actionscript (as applicable). Ruby is fun and weird and awesome, and actionscript is easy if you’ve ever done any other dot-syntax-based-language ever. Having a developer who can implement an entire feature end-to-end just saves so much more time and stress over having to have two people agree on a spec and implement in parallel (for small stuff anyway, obviously that doesn’t scale).
There were some exceptions–we had a few truly truly great flash guys where it actually was faster to keep them in Flash and let other people support them. Even those guys, though, probably got more exposure to ruby than they would’ve liked (anyone can use the console, right?). That sort of education and guidance (mostly on the ruby side, but even in Flash) was the most rewarding part of being a dev lead (or even a producer) to me.
Even being a dev lead itself is pretty fun. Once a team gels pretty well, you get a pretty good sense for what each team member is capable of (or capable of being pushed into). I found the whole thing to be kind of like a puzzle–given the following pieces, how could I best match the tasks to the people to play on all of their strengths and push them forward? Maybe it’s because my mom is a teacher, but there’s something really satisfying about developing other people around you.
Once we shifted to mobile, though, I switched more onto product/design/project management and less of a lead developer–not having the really really deep knowledge of C++ and mobile that I had on the Flash side, I never had the confidence to try and be a leader there. That freed me up in some ways to focus more on design and product, but it was also pretty frustrating at times.
While teaching was fun, there were parts of being a dev lead and product lead that I wasn’t so well suited for. I think I’m probably a horrible micro-manager. I can fit the whole product in my head at once, but I’m not so good at writing all that stuff down into a useful document and delegating. Because there’s no central repository for how things should work, people have to keep asking me how it would work. Some people will just go for it (which is sometimes awesome and sometimes not, but generally preferable). I ended up having a finger in every pie, but not really feeling like I had full control over any of the pies (I mean–I knew what was going on in all of them, but didn’t have enough time to fully understand any one pie).
Because I keep the whole product in my head, I have a pretty good sense of how I want everything to work (both at an experience level and technically). Trusting someone else to do something “right”, where “right” means how I would have done it, is next to impossible. The trick, which I was getting better at but never fully mastered, is to just let your people figure it out and make mistakes as they go. Back in the It Girl days, Pete Hawley (now at Red Robot) told me that the crux of it was basically to hire really good people you could trust and then get the fuck out of their way. I think I could get there eventually, but for now I’m still too close to the development and the architecture side of things. It would grate on me at times when looking at code people had checked in because that’s now how I would’ve done it–but most of the time it would be a perfectly reasonable solution.
At other times on mobile, I felt like I had nothing to do at all. As a producer, though, that’s the state you should strive to be in–if you have nothing to do, that means you’ve got your whole team right where they should be doing what they need to do. As we got closer and closer to launching Closet Wars I was able to jump on bugs and help fix things. Bug-fixing, which can feel like more of a junior task sometimes, is actually a great thing for a used-to-be-programmer-but-now-manager to help out with. Fixing bugs is much much easier than adding features, as it mostly involves tracing logic errors and correcting them instead of architecting whole systems. Because I might be called away at any time if a fire came up, it wouldn’t be fair to the devs for me to take on a big feature–more often than not they’d end up stuck waiting for me to finish it.
I saved this one for last, because it’s probably the most important. My background is mostly in making heavily procedural, art-light games–mainly because I suck at art and work mostly alone. I remember having a conversation on a Flash forum before I joined with an artist who was looking to partner up with a developer and split everything 50/50–my theory at the time was basically that art was a commodity resource and I’d never give up more than 10 or 20% for an artist. I should find that dude and apologize.
The absolute coolest thing about working at Crowdstar was working with the amazing artists there. Our art director had incredibly high standards for artists. It could be frustrating at times when we needed a body in the door for UI or other art positions (I remember interviewing a couple that I was blown away by that he said “meh” to). In the end, though, it definitely shows in the games. It helps that the games we built were essentially content-delivery vehicles (lots of digital goods and weekly releases), but I don’t think I would’ve stayed nearly as long as I did without some of the artists we had.
For nearly three years straight, I’d constantly get blown away by the things they came up with. You’d think after creating 5,000 garments for It Girl there would be absolutely nothing left in the tanks creatively, but I think the artists both on It Girl and Closet Wars (now Top Stylist) got stronger and stronger as they went. The stuff those guys churned out on a regular basis was just fun to watch.
Coming from a place where I wanted to make games but didn’t know any artists, I always remember being frustrated at how I was going to get around it… art was a barrier that prevented me from making what I wanted to make. Now, I have a pretty large network of artist friends who are willing to give me recommendations to their artist friends (or take on contract work themselves). And having worked closely with really really good artists for several years in a row, I feel like I can communicate a lot better with artists when I do bring them on board. Just because of that, I feel like I’m no longer limited in the kinds of games I want to make (which I think is huge!).
I debated whether to include this section or not–I’d say my three years at Crowdstar was more or less 33 months of good times and three months of sour grapes. I worked on Closet Wars from November of 2011 through around October of last year, when the decision was made that it wasn’t hitting its performance metrics fast enough.
Even before the axe fell, though, cracks were starting to form. There had been a round of layoffs in the summer (which mostly unphased me, because I was having so much fun on Closet Wars), which spooked a lot of our best people into finding new jobs. My team mostly held together through the worst times, but by September or so we’d lost our lead artist, our UI artist, and were about to lose one of our best engineers. Zynga’s stock was in the shitter, which was making the whole industry look bad. I considered leaving then, but Peter and Jeff talked me into staying through the end of the year and seeing Closet Wars through.
A couple of weeks after I agreed to stay on, the decision came down to scrap Closet Wars and repurpose it into Top Stylist. A new game designer was brought on, and I was asked if I was willing to produce the hatchet job (that’s probably too harsh–Top Stylist is a pretty well designed game, it’s just not my baby). I’d only taken two weeks off when my son was born, and in California you can take up to 8 weeks of paid family leave. I thought a better solution would be to take a month off, come back refreshed, and then figure out where I could be helpful.
When I got back in early November, everything seemed different. I’m not sure if it was the month off or not, but it was almost like they didn’t know what to do with me. I was given the vague assignment of researching new tech. Mostly, though, I was just ignored. Not being on a team and not being assigned anything meant that most days I just sat there and did nothing. That’s fun for like a day, but I had just had four weeks off and I was raring to get to work on something. Small things would come up–ruby questions, questions about how we’d built things or how the art pipeline worked–but certainly not a full day’s work. It almost felt like a game of chicken, waiting to see which would happen first: getting fired or quitting.
I was travelling for a lot of December and January, and upon returning let Jeff know I was ready to leave. He asked if I’d stay on and finish one last project, but in the end the pieces for that project didn’t quite come together quickly enough and we kind of synched up and decided to call it a day (or three years!). In the end it’s kind of a silly way to go out, but I was ready to do my own thing and Crowdstar is a much different company than the one I joined three years ago. No longer really a “growth” startup, they’re in full-on game studio mode. They have a clear direction they’re headed and need people to fit into pretty specific roles to get them there. If I’d stayed, I probably would’ve dropped back down to being just a “server dev” or a “client dev.”
The only thing I really regret is that I was never able to do anything with all those stock options (which will just expire in 90 days). I could exercise them, but locking up that much cash in a studio that’s raised a ton of VC, over which I no longer have any influence, and which has no guarantee of an exit seems pretty silly when I could just invest that cash in myself and my own projects. It’s a little crazy to think the thing that acted as such a good carrot during the long hours of the It Girl days will come to nothing, but that’s just part of the startup game.
The last few months were a little strange, but overall I think it was a great decision to jump on when I did. It’s easy to wonder what would’ve happened if I’d stayed the course on doing indie iphone games back in 2010. Comparing what I know now about game design, free-to-play, art, operating a game as a business, even technology–I still love some of those old designs I was working on, but they probably would’ve been disasters. I’m really excited to get a “second chance” at some of those designs, but revisiting them through the lens of my Crowdstar experience. It should be a fun ride!