- Search for search-engine bots, either by looking at the URL (in case you used a #! Hashbang) or by checking the user agent of the request.
- In case you have detected the bot, you need to redirect the request to your rendering engine of choice, like Phantom JS etc. This engine should wait until all the AJAX content is loaded.
- Once the content is loaded, you must take the source of the rendered page and output it back to the bot.
This whole procedure is a form of cloaking, which is often refrained in SEO, yet as the content is almost the same as what users would see – this kind of cloaking is considered “ethical”. Therefore the search engines accept it. Google has a whole set of AJAX crawling specification, which covers all the important points one need to take care of. At the end, one needs to either add #! identifier to the URL or use the HTML tag as the page header. This whole process is called pre-rendering snapshot pages and bot-specific rendering. Once you do this, Google gets an alert that the page uses URL fragments and gets them to the index page.
Things to take care of while pre-rendering
There are some potential issues which you need to be careful about. They are:
- Snapshot timing
- Page load time
- Bot detection
- Batch processing
Check the Snapshot pages
As you are using bot detection, it becomes hard to verify whether your process is actually working. One way in which you can check out if the process is actually working is through “fetch as google” feature in Google Webmaster Tools. This process requires a live page, thus it is advisable that you plan accordingly. At present, “Fetch as Google” supports #! only, and not pushState URLs. So if your URLs are static looking, you will not face any problem.
Landing pages and paid search