Does Google crawl dynamic content?

teletext.ioFor the purpose of validating the technical concepts behind distributed content service Teletext.io, here is a series of experiments to see what gets indexed and what doesn't if javascript and Google Tag Manager are used to inject content and JSON-LD in your page.

  1. Content injection before DOM has loaded
  2. Content injection after DOM has loaded
  3. Content injection by async javascript
  4. Content injection after httpRequest
  5. JSON-LD content
  6. JSON-LD injection
  7. JSON-LD injection, asynchronously
  8. JSON-LD injection after httpRequest
  9. JSON-LD injection with Google Tag Manager
  10. Meta elements injection
  11. Meta elements injection, asynchronous

Update this was posted to Google on Friday the 17th of July, 2015. Monday, the 20th, I checked the results. The management summary: Google crawls and indexes all content injected by javascript, independent of whether it is synchronously or asynchronously injected. However, JSON-LD content fails to show up in the search results.

Back to Centrical home

Experiments

To validate the results of every experiment, I used more or less unique strings to identify content in the search results. View source to find out the exact nature of each experiment.

Injection before DOM has loaded

This experiment writes a string into a DIV element with javascript synchronously.

Test content

For the test to be succesful, the following content should get indexed.

Result

Google definitely indexes this content.

Google result for content injection before DOM has loaded

Injection after DOM has loaded

After the DOM has loaded, this experiment writes a string into a DIV element with javascript.

Test content

For the test to be succesful, the following content should get indexed.

Result

Google definitely indexes this content.

Google result for content injection after DOM has loaded

Asynchronous injection

After a time out of 1000 milliseconds, the experiment writes a string into a DIV element.

Test content

For the test to be succesful, the following content should get indexed.

Result

Google definitely indexes this content.

Google result for asynchronous content injection

Injection after httpRequest

This experiment loads a JSON file and then the callback function injects a DIV with a string using its data.

Code

Test content

For the test to be succesful, the following content should get indexed.

Result

Google definitely indexes this content.

Google result for content injection after httpRequest

JSON-LD

As part of the HTML source, a JSON-LD object was added to the HEAD.

Test content

For the test to be succesful, the following content should get indexed. Look for the name.

JSON-LD in the HEAD

Result

As you can see in the Google Structured Data Testing tool, the content gets recognised. However, there are no results in the SERP yet.

JSON-LD in the HEAD

JSON-LD injection

A JSON-LD object was added to the HEAD with javascript before the DOM is loaded.

Test content

For the test to be succesful, the following content should get indexed. Look for the name.

JSON-LD injection

Result

As you can see in the Google Structured Data Testing tool, the content gets recognised. However, there are no results in the SERP yet.

JSON-LD in the HEAD with javascript before the DOM is loaded

JSON-LD injection, asynchronous

Using a time-out function, a JSON-LD object was added to the HEAD.

Test content

For the test to be succesful, the following content should get indexed. Look for the name.

JSON-LD asynchronous injection

Result

As you can see in the Google Structured Data Testing tool, the content gets recognised. However, there are no results in the SERP yet.

Asynchronous JSON-LD in the HEAD with javascript

JSON-LD injection with httpRequest

JSON-LD data was loaded using httpRequest. Then, the callback function injected it into the HEAD.

Test content

For the test to be succesful, the following content should get indexed. Look for the name.

JSON-LD injection using httpRequest

Result

As you can see in the Google Structured Data Testing tool, the content gets recognised. However, there are no results in the SERP yet.

Asynchronous JSON-LD in the HEAD with httpRequest

JSON-LD with Google Tag Manager dataLayer

JSON-LD data was created from a dataLayer in Google Tag Manager and inserted with a custom HTML tag as JSON-LD on DOM loaded.

Test content

For the test to be succesful, the following content should get indexed. Look for the name.

JSON-LD injection using Google Tag Manager

Result

As you can see in the Google Structured Data Testing tool, the content gets recognised. However, there are no results in the SERP yet.

JSON-LD with Google Tag Manager dataLayer

Meta elements injection

Synchronously, the meta description was replaced by another one containing a unique string.

Test content

For the test to be succesful, the following content should get indexed and shown as the description in the SERP. Compare the unique string. Note: the next experiment might overrule this one.

Result

This experiment was considered a success, because the next experiment succeeded - leading to an overwritten description.


Meta elements injection, asynchronous

Using a time-out function, the meta description was replaced with a new one, containing a unique string.

Test content

For the test to be succesful, the following content should get indexed and shown as the description in the SERP. Compare the unique string. If this experiment is succesful, the previous one will also work.

Result

Google definitely indexes this content. This means the previous experiment is also valid.

Google result for content injection before DOM has loaded

Conclusion

The following can be concluded.

So, very soon, the days of pre-rendering PhantomJs snapshots and serving shadow content to spiders will be over. That makes me happy.

Back to Centrical home