When developing a similar technique on my own website, I designed it so it can very easily be disabled server-side to serve more standard pages, without BigPipe enabled. Now, I'm looking for a way to make it crawler-friendly.
The third way would be to stream BigPipe content not using HTML comments as Facebook does, but using <noscript> tags. This would make a pagelet look like :
<noscript id="pagelet_payload_foo">Some content to be indexed here</noscript>
instead of the Facebook's approach:
<code id="pagelet_payload_foo"><!-- Some content to be indexed here --></code>
This looks great, simple, both crawler friendly and user friendly. But this seems a little hackish to me, and does not work in IE 7/8 because the contents of the noscript tag is ignored in the DOM. That would involve some dirty special case for these browsers.
Then, I looked more closely at what Facebook does. Seems like they are doing the same. Pages are optimized in my browser, but are not in Google's cache. I tried to clear all my browser cache and cookies, and requested the page again. No matter what, I keep getting the content through BigPipe. They are not using any cookie-based technique.
Then, the question is simple : How does Facebook do that? Would the first method be considered as cloaking, or does it only work for Facebook because it is Facebook? Or did I miss something else?