So I’ve been using HTC phones for several years now and so far I’ve liked every single phone I’ve had. But the Desire was my switch from Windows Mobile to the Android platform, and I was very worried about the switch to say the least.

First impression of the Desire

After switching on the phone for the first time I went through the setup procedure, which is as simple as 1, 2, 3. So thank god for that, it’s a huge step up from Windows Mobile 5.

I have to give HTC one thing, the phone response very well to any time you touch the screen. It’s probably the capacitive screen, but its a whole lot better then what I was used to.  Just after first boot you will probably play around a little with the widgets that are included.

As the Sense interface gives you 7 different tabs that you can put widgets on it will take you some time to configure it to your likings. Which is as much a blessing as a curse, especially in when you are just using the phone for the first time. Luckily the phone comes with several presets for the Sense interface.

My biggest concern was migrating all my data from Windows Mobile to the Android platform. And unfortunately my worries where right. So Google step up to the plate and make migrating from different phone platforms easier. It’s impossible to migrate text messages or call history without doing a lot of manual work.

The good the bad and the ugly

So what’s up with this phone. Is it a good thing to buy or good gift to get . Well it sorta depends on what you are expecting. If you want a phone that gives you a lot of access to your social networks, then yeah it might be something for you. If you want something that just enables you to call and send text message, then please don’t waste your money on the Desire.  It’ll do the job but at a high price.

Having said that here are some of the pro’s and con’s of the phone:

The strange crashes started happening once I found a page that didn’t contain the information I needed and I hit the back button. Every single time I did this Google crashed my tab with some type of cross scripting warning. Even when IE tried to recover the tab it crashed again. After the second crash IE just said, slightly paraphrased, ‘f*ck it the website keeps crashing go somewhere else instead!’.

So here is the steps to reproduce (as it crashed every single time):

• Use Internet Explorer 8
• Have multiple tabs open, I had at least two in a google search result
• I was logged into personalized search, don’t know if it is relevant but hey you never know
• Click on of the search results
• Hit the back button, or the backspace key
• Presto crasho.

Always fun to see how some javascript can crash a Internet Explorer tab. I am at least presuming it is caused by javascript.

We already new that files with .exe, .dll and .lib were being filtered from the search results. Which I think is a good thing, as it protects the visitors from potential harm. But just a few days back I got word that Google is now also banning or blocking pages ending on .0.

Some examples are:

• The wikipedia page Web 2.0 (http://en.wikipedia.org/wiki/Web_2.0) ending in .0 is no longer showing up in the search results, rather the page ending in 2 (http://en.wikipedia.org/wiki/Web_2).
• The Seomoz page for Web 2.0 has disappeared from the results. Instead the summary page is listed.

After some chatter about the issue around the internet and blogs Matt Cutts wrote a quick entry in his blog as to why they have been removed from the search results. Read it at http://www.mattcutts.com/blog/dont-end-your-urls-with-exe/.

So for know try to avoid ending Url’s with .0 or any of the already known blocked extensions.

Before I continue please note that this is not meant to be a replacement for a real search engine like Google. But it may be useful for you to use on your own website.

One of the first things you must do is structure how you want to store the data you index from the web. I won’t go into much detail on this right now since that’s probably a university study on it’s own. But I will go into detail on:

• How to feed pages into an existing index
• Extracting useful links from indexed pages
• Extracting nice information on the page like title, description, etc
• Filtering out useless content of pages

Feeding new pages into the index

The simplest thing should be the manually entering of new pages to index. But if you’re anything like me then you can make something simple something complex. What you need to consider is how you want to do this. Do you want the engine to index only the given page, or perhaps pages it links to as well.

If you have these two possibilities then your engine will look something like this:

class Engine {
  function indexPage($page_url, $subpages_included = false)
  {}
}

Looks simple doesn’t it, well that’s cause so far it is. In the next couple of paragraphs I will slowly keep adding code to this class to show how I build my search engine.

So let’s start by adding the first feature that you will need to have. When adding a new page to the index you will need to get that page. That’s the first thing we are going to be building. Something I build to prevent accessing non existing Urls is a checker that verifies if a page exists. I do this as follows:

function verifyUrl($page_url)
{

Open a connection to the Url provided by the engine (user)

   $real_url = parse_url($page_url);
   $use_port = isset($real_url['port']) ? $real_url['port'] : 80;
   $socket = fsockopen($real_url['host'], $use_port, $errno, $errstr, 30);

If we successfully opened a connection sent the url we need to the webserver.

   if ($socket){
     $sent_header = "HEAD ". @$real_url['path'] .
       " HTTP/1.0\r\nHost: ". @$real_url['host']"\r\n\r\n";
     fputs($socket, $sent_header);

As long as we don’t have the header information we need from the webserver we will keep reading the response.

     while (!feof($socket)) {
       if ($rec_header = trim(fgets($socket, 1024))) {
         $ar = explode(':', $rec_header);
         $key = array_shift($ar);
         if ($key == $rec_header)
          $headers[] = $header;
         else
          $headers[$key] = substr($rec_header, strlen($key)+2);
         unset($key);
       }
     }
  }

Check if the response from the server was that the page actually exists and is OK.

  if (stripos($headers[0], '200 OK')>0)
    return true;
  return false;
}

Now that we have a function to check if Url’s exist and don’t return a 404 (meaning ‘uh I lost the page’) we can start writing the function that fetches the actual content of the url.

Reading the content of the url is pretty simple just by adding the following code:

function getUrlContents($url_string)
{

Prevent the fetching of none existing pages so the index never gets polluted.

  if (!$this->verifyUrl($url_string))
     return;

Now fetch the content and strip some basic information from it so we have some details on the page.

  $page_content = @file_get_contents($url_string, 'rb');
  if (!$url_content) return;

  $urls =array();
  @preg_match_all('/<a[^>]*>/', $page_content, $href_tags);
  foreach ($href_tags[0] as $href_tag) {
     @preg_match_all('/http(s)?:\/\/[\w\-_]+(\.[\w\-_]+)+([\w\-\.,@?^=%&:\/~\+#]*[\w\-\@?^=%&\/~\+#])?/', $href_tag, $url);
     if (isset($url[0]) &amp;amp;&amp;amp; isset($url[0][0]))
        $urls[] = $url[0][0];
     unset($url);
   }
}

Now that all the links are extracted from the page and stored in an array we could use those to fetch those pages, strip the links and continue. This would automatically fill the index using a simple methodology.

From this point on you may want to store the entire content of the page in the index together with the links on the page. So far for the basics on feeding new pages into the index.

More about building a PHP search indexer

For now that is enough on how to build your own search indexer. In the near future I will expand on this article with:

• Extracting useful information from a web page
• Storing the indexed pages in a useful manner
• Calculating static weights of web pages