Tech

Binary Code Lump: Why Facebook is Glitchy Sometimes

You've seen it frequently on Facebook: somebody complaining or freaking out over something that isn't working right. A feature that suddenly vanished. The inability to post a comment, or a reply on a comment. Often the person expresses high frustration and exasperation.

Since Facebook works in an expected and routine manner most of the time, when something goes wrong, we may think it "means" something -- something bad. Are we being punished?

I recall when, for about a month or so, I would get warnings from Facebook along the lines of "Are you sure you want to post this comment? If your comment is inappropriate or irrelevant, you may be blocked from commenting on public posts" or "You have been flagged as posting spam comments, so you cannot post this comment."

All ridiculous accusations with no merit. My comments were always relevant, gentlemanly, diplomatic, positive, and enriching to the conversation thread. Having nothing to sell, I was incapable of spam and felt insulted to be faced with such a wild allegation from Facebook.

It's funny to see the speculations from people who have experienced some dysfunction in Facebook. They may think they were not "politically correct" or expressed some concept that runs contrary to Facebook's political views.

Actually this could be a real problem, as some have voiced concerns over apparent censorship by Facebook of ideological updates, most recently involving the blocking of certain posts or images related to the worldwide protests against Monsanto and GMO produce.

Then there are those who think Facebook is a personal entity who asks them direct questions like "How are you feeling today?", "What's on your mind?", "What costume are you wearing this Halloween?" or "What are you thankful for today?" (the holiday specific user prompts).

It can be a bit creepy when someone starts talking directly, in their status update, to Facebook as a personal entity. Facebook is not a person. It's just a great big pile of binary code, as you'll see in the quote from Ars Technica below.

Before you start tearing your hair out over a Facebook malfunction, or start feeling guilty and blaming yourself, or feeling paranoid -- understand why FB is glitchy.

According to Ryan Paul - Apr 5, 2012 on Ars Technica "A Behind the Scenes Look at Facebook Release Engineering":

The Facebook source code is largely written in the PHP programming language. PHP is conducive to rapid development, but it lacks the performance of lower-level languages and some more modern alternatives. In order to improve the scalability of its PHP-based infrastructure, Facebook developed a special transpiler called HipHop.

HipHop converts PHP into heavily optimized C++ code, which can then be compiled into an efficient native binary. When Facebook unveiled HipHop to the public in 2010 and began distributing it under an open source software license, the company's engineers reported that it reduced average CPU consumption on Facebook by roughly 50 percent.

Because Facebook's entire code base is compiled down to a single binary executable, the company's deployment process is quite different from what you'd normally expect in a PHP environment. Rossi told me that the binary, which represents the entire Facebook application, is approximately 1.5GB in size. When Facebook updates its code and generates a new build, the new binary has to be pushed to all of the company's servers.

Moving a 1.5GB binary blob to countless servers is a non-trivial technical challenge. After exploring several solutions, Facebook came up with the idea of using BitTorrent, the popular peer-to-peer filesharing protocol. BitTorrent is very good at propagating large files over a large number of different servers.

Rossi explained that Facebook created its own custom BitTorrent tracker, which is designed so that individual servers in Facebook's infrastructure will try to obtain slices from other servers that are on the same node or rack, thus reducing total latency.

Rolling out a Facebook update takes an average of 30 minutes—15 minutes to generate the binary executable and another 15 minutes to push the executable to most of Facebook's servers via BitTorrent.

The binary executable is just one part of the Facebook application stack, of course. Many external resources are referenced from Facebook pages, including JavaScript, CSS, and graphical assets. Those files are hosted on geographically distributed content delivery networks (CDNs).

Facebook typically rolls out a minor update on every single business day. Major updates are issued once a week, generally on Tuesday afternoons. The release team is responsible for managing the deployment of those updates and ensuring that they are carried out successfully.

About the Author

Steven Streight is a man of many skills. He’s a talented writer, web content developer, internet marketing consultant and photographer. He’s a trustee on the Peoria Historical Society, a member of SCORE Peoria and the author of the Peoria technology history book, “Bicycle Fever.” In his downtime, he’s hangs out with his beloved Min Pin and tries to get some rest. Considering how involved he is in the community, it sounds like he could use as much as he can get.