Interop 2012 Product Preview

"Hybrid is the new black. I sleep well at night knowing that I've got both [public and private cloud resources,]" declared Allan Leinwand, Zynga's CTO of infrastructure engineering, during a keynote address at Interop 2012 on Tuesday.

Zynga developed its own hybrid cloud model, based on Zynga's zCloud in Zynga data centers and public Amazon Web Services, via trial and error, Leinwand says.

Many companies can place at least some less-sensitive workloads out in the public cloud, Leinwand told Interop attendees.

As he amplified his remarks in an interview with InformationWeek editors immediately afterward, he also made it clear that getting to hybrid operations required a substantial investment that not every enterprise is going to be able to duplicate.

Zynga is the supplier of Farmville, CastleVille, Words With Friends, and more recently through the Omgpop acquisition, Draw Something. When Zynga finds game activity is reaching unexpected peaks, it offloads some of the traffic onto Amazon Web Services EC2. "The public cloud will become the shock absorbers for your business," Leinwand predicted in his keynote at the Las Vegas event.

[ How did Zynga shift from 80% dependence on Amazon Web Services to 20%? See Zynga Trades Amazon Cloud For In-House Servers. ]

"We spent a lot of time looking at public clouds to try to understand how they work. Our servers are almost the same as theirs," he said. Zynga purchases zCloud servers from Dell and data center server design specialist, ZT Systems. Among other things, Zynga is a heavy of user of disk drives built into server racks close to CPUs, as opposed to Fibre Channel storage arrays or other detached storage.

Zynga also is able to order arrays of servers that are designed to perform particular functions in Zynga's data centers. The company's ability to order hardware optimized as Web servers, data base servers, and application servers allows it to operate with a higher degree of efficiency in its own data centers than it could on Amazon servers. Leinwand repeated his statement, first made at CloudConnect in February, that zCloud server does the work of three Amazon EC2 general purpose servers.

Some of that gain also reflects a heavy investment in application optimization. "Love your app," Leinwand advised the Interop crowd. Zynga dissected how its game logic ran in the public cloud before designing its zCloud data centers and found many ways to eliminate bottlenecks and improve performance, he said.

"We had to take time and look at process threads, how PHP intersects memory and CPU, whether Apache Web servers were caching data properly or not," he recalled. "We didn't say our application is big and heavy and we need to throw more hardware at it."

With a programmer-intensive staff, Zynga could identify parts of games that looked light and efficient to programmers, but seemed to run slowly in spite of it, and troubleshoot issues from there. After months of work, the operation of the zCloud along with Amazon workloads still holds surprises for its operators. "It's not a monotone. There's a lot of peaks and valleys in our infrastructure. We wake up each morning to see what this day is going to bring," he said.

Engineers make up 40% of Zynga's staff. They are engaged in building out the private cloud infrastructure, testing PHP code, and inventing monitoring tools for the first time for new features and functions that are constantly being added to five online games. Zynga is a heavy user of Linux, Apache, and Membase open source code.

"Fortunately, we use a lot of open source code. We can get in there and tweak the code," he said.

One of the things that Leinwand disclosed is that Zynga seeks out skilled members on open source projects as prospects for its engineering staff. "We have folks who are well versed in the Linux kernel, the internals of PHP, Membase, and the most efficient ways of allocating memory."

"They're not average programmers. They're knee-deep in the projects they're working on," he noted.

At the same time, Zynga is not looking for detail workers but innovators with big ideas. "We want people who want to work on the opportunity to scale up. If scaling freaks them out, then chances are they're not our guy or gal," he said.

Leinwand also said in the interview that Zynga is not competing with Amazon when it comes to providing basic infrastructure, even though at the end of March it began offering its own zCloud Platform of game services. A key service is Zynga's ability to analyze "on a large scale" what players are doing in its games. "That isn't easy," he noted. Six third-party game producers have signed up for Zynga promotional services or analytical services since March.

The bulk of Zynga online game activity originates with Facebook users, who play Zynga games using game logic running in Zynga data centers. Some game users log directly into Zynga.com rather than Facebook. Zynga hosts 2.8 million active monthly users in this fashion. Overall, Zynga experiences 292 million monthly users when those originating from Facebook are added in.

The Enterprise 2.0 Conference brings together industry thought leaders to explore the latest innovations in enterprise social software, analytics, and big data tools and technologies. Learn how your business can harness these tools to improve internal business processes and create operational efficiencies. It happens in Boston, June 18-21. Register today!