Experiment: We Removed a Major Website from Google Search, for Science! Posted by Cyrus-Shepard The folks at Groupon surprised us earlier this summer when they reported the results of an experiment that showed that up to 60% of direct traffic is organic. In order to accomplish this, Groupon de-indexed their site, effectively removing themselves from Google search results. Thats crazy talk! Of course, we knew we had to try this ourselves. We rolled up our sleeves and chose to de-index Followerwonk, both for its consistent Google traffic and its good analytics setup—that way we could properly measure everything. We were also confident we could quickly bring the site back into Googles results, which minimized the business risks. (We discussed de-indexing our main site moz, but... no soup for you!) We wanted to measure and test several things: How quickly will Google remove a site from its index? How much of our organic traffic is actually attributed as direct traffic? How quickly can you bring a site back into search results using the URL removal tool? Heres what happened. How to completely remove a site from Google The fastest, simplest, and most direct method to completely remove an entire site from Google search results is by using the URL removal tool. CAUTION: Removing any URLs from a search index is potentially very dangerous, and should be taken very seriously. Do not try this at home; you will not pass go, and will not collect $200! After submitting the request, Followerwonk URLs started disappearing from Google search results in 2-3 hours. The information needs to propagate across different data centers across the globe, so the effect can be delayed in areas. In fact, for the entire duration of the test, organic Google traffic continued to trickle in and never dropped to zero. The effect on direct vs. organic traffic In the Groupon experiment, they found that when they lost organic traffic, they actually lost a bunch of direct traffic as well. The conclusion was that a large amount of thier direct traffic was being attributed as direct—up to 60%. At first glance, the overall amount of direct traffic to Followerwonk didnt change significantly, even when organic traffic dropped. In fact, we could find no discrepancy in direct traffic outside the expected range. I ran this by our contacts at Groupon, who said this wasnt totally unexpected. You see, in their experiment they saw the biggest drop in direct traffic on long URLs, defined as a URL that is at least as long enough to be in a subfolder, like ift.tt/1rl2Ifn. For Followerwonk, the vast majority of traffic goes to the homepage and a handful of other URLs. This means we didnt have a statistically significant sample size of long URLs to judge the effect. For the long URLs we were able to measure, the results were nebulous. While we cant confirm the Groupon results with our outcome, we cant discount them either. Its quite likely that a portion of your organic traffic is attributed as direct. This is because of different browsers, operating systems and user privacy settings can potentially block referral information from reaching your website. Bringing your site back from death After waiting 2 hours, we deleted the request. Within a few hours all traffic returned to normal. Whew! Does Google need to recrawl the pages? If the time period is short enough, and you used the URL removal tool, apparently not. In the case of Followerwonk, Google removed over 300,000 URLs from its search results, and made them all reappear in mere hours. This suggests that the domain wasnt completely removed from Googles index, but only masked from appearing for a short period of time. What about longer periods of de-indexation? In both the Groupon and Followerwonk experiments, the sites were only de-indexed for a short period of time, and bounced back quickly. We wanted to find out what would happen if you de-indexed a site for a longer period, like two and a half days? I couldnt convince the team to remove any of our sites from Google search results for a few days, so I choose a smaller personal site that I often subject to merciless SEO experiments. In this case, I de-indexed the site and didnt remove the request until three days later. Even with this longer period, all URLs returned within just a few hours of cancelling the URL removal request. Likely, the URLs were still in Googles index, so we didnt have to wait for them to be recrawled. For longer removal periods, a few weeks for example, I speculate Google might drop these semi-permanently from the index and re-inclusion would comprise a much longer time period. What we learned While a portion of your organic traffic may be attributed as direct (due to browsers, privacy settings, etc) in our case the effect on direct traffic was negligible. If you accidentally de-index your site using Google Webmaster Tools, in most cases you can quickly bring it back to life by deleting the request. Reinclusion happens quickly even after we removed a site for over 2 days. Longer than this, the result is unknown, and you could have problems getting all the pages of your site indexed again. Further reading Moz community member Adina Toma wrote an excellent YouMoz post on the re-inclusion process using the same technique, with some excellent tips for other, more extreme situations. Big thanks to Peter Bray for volunteering Followerwonk for testing. You are a brave man! Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you dont have time to hunt down but want to read! ift.tt/1rl2Ifh
Posted on: Tue, 12 Aug 2014 07:19:16 +0000
Trending Topics
Recently Viewed Topics
© 2015