Keep in mind the average page size is not only the size of the HTML but also adding the external CSS, JavaScript and images. Which could dramatically increase the amount of data the users are downloading. So I started looking in the compression options of Apache 2.0

First I looked on the official Apache page, but as usual the data is probably available but not very user friendly. So after some testing and crashing I found out the following procedure which seems to work fine to enable compression per Virtual Host.

Firstly enable the module that supports compression by executing:

:> ln -s /etc/apache2/mods-available/deflate.load /etc/apache2/mods-enabled/deflate.load

This will instruct Apache to load the needed library (module) for compression using GZip. The second thing you will need to do is add the following lines to every single Virtual Host you want to use compression on.

    <Location / >
      SetOutputFilter DEFLATE
      BrowserMatch ^Mozilla/4 gzip-only-text/html
      BrowserMatch ^Mozilla/4\.0[678] no-gzip
      BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html
    </Location>


This will enable the compression for output (everything sent to the end-user) but not for incoming requests, which in my case is enough compression for right now. It also exludes some browsers that do not support compression.

Now restart or reload your apache by running the statement below and your website should support compression. This will make the loading of pages faster, though the client software needs to decompress the pages from this point on.

:> etc/init.d/apache2 reload

If you have any trouble enabling compression just leave a message and I’ll try and help you where I can.

The strange crashes started happening once I found a page that didn’t contain the information I needed and I hit the back button. Every single time I did this Google crashed my tab with some type of cross scripting warning. Even when IE tried to recover the tab it crashed again. After the second crash IE just said, slightly paraphrased, ‘f*ck it the website keeps crashing go somewhere else instead!’.

So here is the steps to reproduce (as it crashed every single time):

• Use Internet Explorer 8
• Have multiple tabs open, I had at least two in a google search result
• I was logged into personalized search, don’t know if it is relevant but hey you never know
• Click on of the search results
• Hit the back button, or the backspace key
• Presto crasho.

Always fun to see how some javascript can crash a Internet Explorer tab. I am at least presuming it is caused by javascript.

As most developers know that have tried to get their flash pages indexed by search engines this is almost impossible. In the past the search engines were unable to read the content of the flash page. Simply because they don’t understand flash.

For now only the future will tell if this cooperation between Adobe and the search engine will help in getting flash pages in the index. And even better in the index with decent results. But given the dedication of Adobe I have no doubt that it will be a success.

Read the official Adobe press release

Read Matt Cutts reply to the announcement

We already new that files with .exe, .dll and .lib were being filtered from the search results. Which I think is a good thing, as it protects the visitors from potential harm. But just a few days back I got word that Google is now also banning or blocking pages ending on .0.

Some examples are:

• The wikipedia page Web 2.0 (http://en.wikipedia.org/wiki/Web_2.0) ending in .0 is no longer showing up in the search results, rather the page ending in 2 (http://en.wikipedia.org/wiki/Web_2).
• The Seomoz page for Web 2.0 has disappeared from the results. Instead the summary page is listed.

After some chatter about the issue around the internet and blogs Matt Cutts wrote a quick entry in his blog as to why they have been removed from the search results. Read it at http://www.mattcutts.com/blog/dont-end-your-urls-with-exe/.

So for know try to avoid ending Url’s with .0 or any of the already known blocked extensions.

Before I continue please note that this is not meant to be a replacement for a real search engine like Google. But it may be useful for you to use on your own website.

One of the first things you must do is structure how you want to store the data you index from the web. I won’t go into much detail on this right now since that’s probably a university study on it’s own. But I will go into detail on:

• How to feed pages into an existing index
• Extracting useful links from indexed pages
• Extracting nice information on the page like title, description, etc
• Filtering out useless content of pages

Feeding new pages into the index

The simplest thing should be the manually entering of new pages to index. But if you’re anything like me then you can make something simple something complex. What you need to consider is how you want to do this. Do you want the engine to index only the given page, or perhaps pages it links to as well.

If you have these two possibilities then your engine will look something like this:

class Engine {
  function indexPage($page_url, $subpages_included = false)
  {}
}

Looks simple doesn’t it, well that’s cause so far it is. In the next couple of paragraphs I will slowly keep adding code to this class to show how I build my search engine.

So let’s start by adding the first feature that you will need to have. When adding a new page to the index you will need to get that page. That’s the first thing we are going to be building. Something I build to prevent accessing non existing Urls is a checker that verifies if a page exists. I do this as follows:

function verifyUrl($page_url)
{

Open a connection to the Url provided by the engine (user)

   $real_url = parse_url($page_url);
   $use_port = isset($real_url['port']) ? $real_url['port'] : 80;
   $socket = fsockopen($real_url['host'], $use_port, $errno, $errstr, 30);

If we successfully opened a connection sent the url we need to the webserver.

   if ($socket){
     $sent_header = "HEAD ". @$real_url['path'] .
       " HTTP/1.0\r\nHost: ". @$real_url['host']"\r\n\r\n";
     fputs($socket, $sent_header);

As long as we don’t have the header information we need from the webserver we will keep reading the response.

     while (!feof($socket)) {
       if ($rec_header = trim(fgets($socket, 1024))) {
         $ar = explode(':', $rec_header);
         $key = array_shift($ar);
         if ($key == $rec_header)
          $headers[] = $header;
         else
          $headers[$key] = substr($rec_header, strlen($key)+2);
         unset($key);
       }
     }
  }

Check if the response from the server was that the page actually exists and is OK.

  if (stripos($headers[0], '200 OK')>0)
    return true;
  return false;
}

Now that we have a function to check if Url’s exist and don’t return a 404 (meaning ‘uh I lost the page’) we can start writing the function that fetches the actual content of the url.

Reading the content of the url is pretty simple just by adding the following code:

function getUrlContents($url_string)
{

Prevent the fetching of none existing pages so the index never gets polluted.

  if (!$this->verifyUrl($url_string))
     return;

Now fetch the content and strip some basic information from it so we have some details on the page.

  $page_content = @file_get_contents($url_string, 'rb');
  if (!$url_content) return;

  $urls =array();
  @preg_match_all('/<a[^>]*>/', $page_content, $href_tags);
  foreach ($href_tags[0] as $href_tag) {
     @preg_match_all('/http(s)?:\/\/[\w\-_]+(\.[\w\-_]+)+([\w\-\.,@?^=%&:\/~\+#]*[\w\-\@?^=%&\/~\+#])?/', $href_tag, $url);
     if (isset($url[0]) &amp;amp;&amp;amp; isset($url[0][0]))
        $urls[] = $url[0][0];
     unset($url);
   }
}

Now that all the links are extracted from the page and stored in an array we could use those to fetch those pages, strip the links and continue. This would automatically fill the index using a simple methodology.

From this point on you may want to store the entire content of the page in the index together with the links on the page. So far for the basics on feeding new pages into the index.

More about building a PHP search indexer

For now that is enough on how to build your own search indexer. In the near future I will expand on this article with:

• Extracting useful information from a web page
• Storing the indexed pages in a useful manner
• Calculating static weights of web pages

I’m pretty far along but have encountered some problems with indexing web pages. The first was that I somehow was not able to index sub pages successfully. As it turns out this had to do with the fact that my link extractor made a mistake. It included quotes as part of the Url.

The second problem I encountered was that the indexer crawled the same page multiple times in one refresh. First time when they were fetched from the existing index and then once every time another page linked to it. To solve this I added a list of pages already visited, which actually worked .

The last problem, which so far I haven’t solved, is that the indexer seems to be unable to find new pages. This happens when the index reaches a fixed size. I have no idea why at this point. But it may have something to do with a bug in the crawler.

So hopefully I will have all of these problems fixed before my next article on how to build a search indexer.

The opening of the website is going alright, but as soon as I start navigating through my mail Firefox disapears. When I restart firefox it just asks me if I would like to continue my previous session. (Not really:()

The only thing I know for sure is that I can’t read any e-mail for the time being using firefox, well unless I wan’t to watch a crashing browser of course . If any one of you know how to solve the problem please let me know!

Last week I decided that I was going to try and use Google Analytics to help me decide what to do with this blog. You may ask why, but I’m hoping it can help me provide you with better information and me with a better website.

How can analytics help
To be honest I didn’t know where to start so I decided to look at some of the features of analytics. These are the ones I’m hoping will help me:

1. Capability to track users through the website (sort of)
2. Track statistics on users, like location / operating system / browser / etc..
3. Track page statistics, like how often pages get visited and how long
4. Where people came from
5. Give differential reports

Lets walk through how you can use them to improve the website, starting with the first .

Capability to track users through the website
Why is it useful to track users visiting your website. Easy, if you find that your visitors don’t visit a page then you may be able to figure out why. For example can’t they find the link, is it to deep in the website. This kind of information can help refine the navigation through the website.

With analytics you won’t be able to track individual visitors, but you can see the general flow through the website. So if you changed the navigation and visitors suddenly can’t get to a page you have a good idea as to why.

Track user statistics
I hope it’s clear why this helps refine a website. Just knowing where your visitors come from helps you target content. Or blog posts in my case. Just like knowing the browsers they use and the average connection speed helps you refine the implementation of the website.

Analytics offers a very thorough overview of the visitors system. From operating system (eg: windows) to JavaScript support.

Track page statistics
One of the features I particularly like is the capability to track which pages are most viewed. This helps me refine what people are looking for and what type of text and images they like best.

But it also helps with something else. You can roughly determine how long it takes before a page on your website becomes popular. As well as how long it takes for search engines to send some real traffic to it.

Where people are coming from
With this I do not mean the location like ‘US’ or ‘United Kingdom’, but the referrer of the visit. This helps me to determine in which areas my website is currently good. For example right now most of my traffic comes from Google. I prefer that most of my traffic comes from other websites though.

So this tells me I should improve my content to get more links from related websites. As well as improve my lay-out so other website owners are more likely to link to my website.

Give differential reports
And last but not least the differential reports. A differential report is when you compare the current month against the data from the last month. You can have these e-mailed to you, which I like.

You can set these up by going to any page in Google Analytics and then clicking the date in the upper right corner to set a timeframe. You can also set the ‘Compare against previous’ option. This will give you a differential report. (Sorry I can’t give any screenshots, since I’m using the Dutch variant of analytics)

You can then see how you have improved or worsened over the past month.

Well there are a few ways to find out what the people visiting your site are interested in. When you have this information you can start refining the content on your website. If you for instance wrote something about a Ferrari. And you get a lot of visitor searching for ferrari tires, you may want to add some content about that.

Finding the needed information
Lets first go into where you can find the information you will need to refine your website. No matter how, you will always end up trying to pursue your visitors to give more information then they might like.

One way of doing this is by using polls or questionairs. This is a very useful tool, but with one major drawback. If you don’t already have a steady visitor base it won’t work. How often have you entered a website and clicked away an anyoing popup asking you to answer some questions. Most new visitors don’t like questionairs.

A second way is by tracking every move a visitor makes on your website. This will provide information on how they navigate through the site. Which in turn indicates which information they find highly relevant. Big downside is that it requires a lot of disk space. Even worse you have to make some type of mention of this in the privacy policy. Last drawback is that it doesn’t provide information on what the visitor was looking for when he came to your website.

The third and far better way, at least I think so, is to log how the visitor came to the website. Some questions could be:

• For instance did he originate from a search engine. If so what keywords did he search with. What could he be looking for.
• Was the visit from my own website. If so did he use a search feature and what keywords. If not what related page did he come from.
• Was the visit from an external website. What is the content of that page. How does it appeal to the visitor.

This logging can be done quite easily as all the information is stored in the HEAD information sent by the visitors web-browser. (Please note that you can’t fully rellie on this as it can be manipulated). Various other tricks are available as well to get this information, but more on that in a follow up article about data mining.

Using the information to refine the content
Now that you have the information you still need to put it to good use. So lets asume that your website is about cars. And it has a section on Ferrari. You have a lot of content on the various cars. But you don’t have information on the various parts of the cars.

Now in your data-mining (the steps described above ) you find out that a lot of visitors come to your site through search engines. But they are looking for tyres that are for the Ferrari 250GT. Initially when creating the website you never thought this might be useful information.

After making this discovery you could decide to add new content to the pages of the various cars with details about the tyres. How they perform and which are best for which models. By doing so you appeal to your visitors as they now find information they are interested in.

A second type of discovery you may encounter is that your website structure (including navigation) may be wrong. For example people looking for the tyre information never found the content on the website. But it does exists. This may mean they just can’t find any navigation to it. Keep in mind that most visitors only spend a few seconds looking for ‘useful’ information before moving along.

So make sure your information is easy accessible!!!!!

Now that you now some basics of refining your website you can start. But start with some small things first, just to try it out.

So what is it you noticed. Is the text wrong. Does it not appeal to you. Or do you simply don’t like what they say. Maybe even all of the above. It really doesn’t matter since any would drive you away from the website. What does matter is why did it feel off.

Discovery of the problem
In most cases you will have found one simple problem. The creator of the website forgot to keep his audience in mind when writing all of the text. Or maybe he didn’t even consider you part of his audience (bad him ). But what does this mean to conversions of that website. Well for one you will be unlikely to buy anything because you probably don’t trust it completely.

Knowing that text could cause problems with your targetted audience what is it that makes the text on a website right. Well the following list could make a text unattractive for your readers.

• Wrongly using complicated words,
  Using difficult words may be hot in your niche, but it won’t attract the John Doe from the street. Most of them don’t even understand what they are reading. Something that won’t help you selling products to them. Always keep your audience in mind when writing texts.
• Using he, she, you incorrectly,
  This is especially important in other languages then English. If you look at German for example they have ‘Sie’ or ‘du’ which both means you in English. But one is polite and the other isn’t. Make sure you know which your audience prefers and stick with it. Don’t do half with ‘Sie’ and the other have with ‘du’.
• Missing call to action,
  One of the most common mistakes is that the text doesn’t tell your visitors to actually do something. If you want them to buy something don’t stop at how good it is. Also inform them of how to buy it (and don’t use ‘Buy now’, thats just wrong in so many levels).
• Annoying calls to action,
  On the other end of the scope is a to agressive call to action. This will rub your visitors the wrong way. Making them anger or dislike you. Both not good things if you still want them to buy.

Solving the problem
If you know the problem then you can solve it. Right…..

Well not necessarily. The problem may be a bit more complicated. You may not know your audience well enough. Prehaps your audience has changed over time. So make sure to keep updating your information about the visitors of your website.

If you have something to say about this article, don’t hesitate to comment….