Facebook’s growing infrastructure spend

On Thursday BusinessWeek reported Facebook is seeking new financing for its data center operation growth in 2009. Facebook continues to add new members and their associated content at an extremely fast pace, with most new growth coming from international markets. Facebook needs to expand its abilities to serve these markets by bolstering current infrastructure offerings and cutting latency to its members through new international points of presence. In this post I will take a deeper look at Facebook’s current computing infrastructure and related expenses and examine likely new areas of investment in 2009.

Facebook members

Facebook currently has over 160 million members in its top 30 markets. Facebook enjoys a 24% market penetration across all 30 countries, including complete domination in Chile and Turkey, where 76% and 66% of all Internet users are members of Facebook. Facebook member numbers are taken from Facebook.com ; total Internet users for each country is as reported by the CIA World Factbook.

Facebook’s current infrastructure serves North America with minimal latency. Future expansion into Europe and southeast Asia are likely as Facebook tries to expand its international audience.

Data centers

Facebook currently operates out of four data centers in the United States : three on the west coast and one on the east coast. Facebook leases at least 45,000 square feet of data center space.

Switch & Data’s PAIX at 529 Bryant Street in Palo Alto is just around the corner from the Facebook offices and a long-time home to Facebook servers. It’s unclear how much of the 100,000 square-foot, liquid-cooled data center is currently occupied by Facebook.

Facebook has been with Terremark’s NAP West in Santa Clara since November 2005. Facebook originally leased 10,000 square feet but may have grown larger over the years. Facebook is still listed as a Terremark customer in Santa Clara but the company might be consolidating its operations into its new local data center.

Facebook geo-distributed its web operations in 2008 with DuPont Fabros’ ACC4 in Ashburn, Virginia. Facebook leased 10,000 square feet in 2007 and occupied the space in 2008 after extensive reworking of the Facebook backend. Facebook shares ACC4 with MySpace, Google, and other competitors.

In January 2009 Facebook moved into its first exclusive data center, Digital Realty Trust’s 1201 Comstock Street in Santa Clara. The 24,000 square feet of data center space operates at a PUE of 1.35, a respectable mark against reported 1.22 marks for Google and Microsoft. Facebook leases the center from Digital Realty as its sole occupant.

Facebook is rumored to be adding an additional 20,000 square feet of data center space in Ashburn, Virgnia in DuPont Fabros’ ACC5. Facebook is expected to move into ACC5 in September 2009 and place new servers online by the end of the year.

Facebook recently announced an international headquarters in Dublin, Ireland that will include "operations support" across Europe, the Middle East, and Africa. A European data center is a likely expansion point for Facebook as they try to solidify their European offerings.

Server loans

Facebook paid for part of its infrastructure expansion through specialized debt financing from TriplePoint Capital. Facebook drew down $30 million in 2007 followed by another $60 million in 2008. BusinessWeek reports Facebook is currently trying to secure as much as $100 million in debt financing for its next round of growth.

Debt financing against physical assets such as servers and office buildings offer lower rates than a traditional venture capital round. Facebook’s server expenditures have a recoverable resale value mapped over a depreciating lifespan, unlike direct and unrecoverable payments to employees and service providers. Lenders such as TriplePoint are a specialized type of real estate investor, a market with huge risk premiums in the current market. Facebook’s $100 million debt financing is bigger than TriplePoint’s typical investments, placing Facebook’s new expansions beyond the investment strategy’s scope during a time of real-estate investment turmoil. Facebook needed to look to other financing operations for bigger infrastructure loans, an expected move for the growing company.

Facebook spent $68 million on Rackable servers in 2007 and early 2008, likely as a result of their Virginia data center build-out. Facebook is also rumored to be a large consumer of premium-priced proprietary hardware NetApp storage appliances and Force10 networking.

Facebook’s debt financing agreement with TriplePoint Capital expired a few months ago, leading the company to seek new sources of financing for its new Santa Clara data center and other expansion plans. Facebook is in discussions with Bank of America for additional loans against this capital expenditure according to BusinessWeek.

How many servers ?

Facebook had over 10,000 servers as of August 2008 according to Wall Street Journal coverage of a presentation by Jonathan Heiliger, Facebook’s VP of Technical Operations. Facebook signed an infrastructure solutions agreement with Intel in July 2008 to optimally deploy "thousands" of servers based on Intel Xeon 5400 4-core processors in the next year.