bradKELLETT

I’m sure most of us have had the problem – search for something with Google, see a search result appear that is exactly what you’re looking for, only to find that when you click through to the page you get told that you must log in to see the content or (gasp!) pay to see it in full. Well, there is a simple way around this: pretend to be Google!

Sites like these detect when Google is crawling their site and show the full content, but don’t give regular people the same luxury. Sites generally detect this by looking at the user agent, which is a string that identifies what kind of browser/device is requesting the page. Therefore, if we change our user agent string to match that of the Googlebot (the system Google uses to find content on the web), we can see the content that was indexed to make the search result we found.

There are various ways to change the user agent that your browser is reporting, including the great User Agent Switcher extension for Firefox, or the IE7Pro plugin for Internet Explorer 7.

Whichever way you go about it, you want to change your user agent string to:

Googlebot/1.0 (googlebot@googlebot.com http://googlebot.com/)

You may have to restart your browser for the change to take effect, but after that you should be able to go back to the site you were trying to access and see exactly what Google does – the content!

As I announced on Twitter, I have recently joined Cuil as Senior UI Developer. While there are many reasons I chose to come on at Cuil in favor of other opportunities, there is one factor that really won me over: the people.

Cuil has a very flat management structure, and as such everyone has to be very good at a variety of things. There are some incredibly talented people here at Cuil, including ex-senior Google, Internet Archive, and IBM employees (our VP of Engineering was the architect of Google’s large search index, TeraGoogle). Not only is everyone smart, but they are all focused on a very exciting goal, and moving toward it in logical and defined steps. We all share a vision, and that isn’t to be a direct competitor to Google right now – ask anyone here, we know we aren’t there yet, but we have some amazing technology that is set to really change things with time.

I strongly believe that Cuil is the evolution of search. This doesn’t just extend to the magazine-style search results presentation we use now, nor the interesting relevancy and multimedia data we’re splicing into search results, but also to the very cool things that are on the road map. I’m in a very good position to direct the next generation of information presentation, I have a ton of ideas, and am very excited about the future.

Technology, people, and ideas – three things that all add up to a very exciting opportunity.