Keep in mind the average page size is not only the size of the HTML but also adding the external CSS, JavaScript and images. Which could dramatically increase the amount of data the users are downloading. So I started looking in the compression options of Apache 2.0

First I looked on the official Apache page, but as usual the data is probably available but not very user friendly. So after some testing and crashing I found out the following procedure which seems to work fine to enable compression per Virtual Host.

Firstly enable the module that supports compression by executing:

:> ln -s /etc/apache2/mods-available/deflate.load /etc/apache2/mods-enabled/deflate.load

This will instruct Apache to load the needed library (module) for compression using GZip. The second thing you will need to do is add the following lines to every single Virtual Host you want to use compression on.

    <Location / >
      SetOutputFilter DEFLATE
      BrowserMatch ^Mozilla/4 gzip-only-text/html
      BrowserMatch ^Mozilla/4\.0[678] no-gzip
      BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html
    </Location>


This will enable the compression for output (everything sent to the end-user) but not for incoming requests, which in my case is enough compression for right now. It also exludes some browsers that do not support compression.

Now restart or reload your apache by running the statement below and your website should support compression. This will make the loading of pages faster, though the client software needs to decompress the pages from this point on.

:> etc/init.d/apache2 reload

If you have any trouble enabling compression just leave a message and I’ll try and help you where I can.

So now to repair my mistake I am adding the following features to my already existing mail server. To find out how to setup the  mail server itself read my earlier article.

• Use spamassassin to mark spam in the header
• Change the mail delivery to dovecot LDA, this is needed for step 3
• Automatically move marked e-mails to the spam folder of the user

Sounds easy right. Well it should be ;-).

Install spamassassin and set it up in postfix

The first step we will take is installing and setting up the spamassassin tool. First lets install it with the default options from Debian by running:

apt-get install spamassassin

Next we setup the basics of spamassassin configuration, the file called /etc/spamassassin/local.cf

report_safe        0
required_score     2.0
use_bayes          1
bayes_auto_learn   1
use_bayes_rules    1

# Some basic configuration
score DCC_CHECK 4.000
score SPF_FAIL 10.000
score SPF_HELO_FAIL 10.000
score RAZOR2_CHECK 2.500
score BAYES_99 4.300
score BAYES_95 3.500
score BAYES_80 3.000

# Headers to be added for all scanned messages
add_header all Status _YESNO_, score=_SCORE_ required=_REQD_ tests=_TESTS_ autolearn=_AUTOLEARN_ version=_VERSION_
add_header all Level _STARS(*)_
add_header all Checker-Version SpamAssassin _VERSION_ (_SUBVERSION_) on _HOSTNAME_

So what we’ve done so far is set spamassassin up to automatically learn from mail it receives and to mark anything with a bayes score over 2 as spam. But we still need to integrate it into postfix to get it to work. Add the following to the postfix configuration (/etc/postfix/master.cf):

spamassassin unix -     n       n       -       -       pipe
 user=spamd argv=/usr/bin/spamc -f -e
 /usr/sbin/sendmail -oi -f ${sender} ${recipient}

You will also need to change the smtp line in the same file to the following, this will set up spamassassin as a pre delivery filter:

smtp      inet  n     -     -     -     -     smtpd -o content_filter=spamassassin

This will add the spamassassin service as a unix service through a pipe connection. Sounds complicated but it’s really easy it basically uses a binary file to comunicate between postfix and spamassassin. Make sure the user spamd exists and create the folder /usr/bin/spamc with full access to the spamd user. Also edit the file spamassassin to the following:

# /etc/default/spamassassin
# Change to one to enable spamd
ENABLED=1

# SpamAssassin uses a preforking model, so be careful! You need to
# make sure --max-children is not set to anything higher than 5,
# unless you know what you're doing.
SAHOME="/var/lib/spamassassin/"
OPTIONS="--create-prefs --max-children 5 --username spamd --helper-home-dir ${SAHOME} -s ${SAHOME}spamd.log"
PIDFILE="${SAHOME}spamd.pid"
CRON=0

Make sure to create the directory /var/lib/spamassassin and give spamd full access right to it, otherwise the service might not run properly. Spamassassin is now setup and will start scanning your e-mail. So if all you are interested in is the header being changed to indicate if a message is spam then you are done. If you also wish yo move it to a different folder then continue reading.

Settup Dovecot as LDA (Local Delivery Agent)

Next step is setting up dovecot as the default LDA for postfix. This is needed to let dovecot filter messages into the proper locations. First setup the connection to the between postfix and dovecot by adding the following to the /etc/postfix/master.cf:

dovecot   unix  -       n       n       -       -       pipe
 flags=DRhu user=dovecot:dovecot
 argv=/usr/lib/dovecot/deliver -f ${sender} -d ${recipient}

Please note that we now need to keep track of the fact that dovecot will be the user that is running this part of the mail server. So every file needed, log or configuration related, needs to be owned by dovecot:dovecot. I have had issues with files having the wrong owner so it is important! Before activating the postfix connection lets configure dovecot for LDA by editing the /etc/dovecot/dovecot.conf file. Add the following piece of code in the group auth default >> socket listen:

        master {
          path = /var/run/dovecot/auth-master
          user = dovecot
          group= ssl-cert
          mode = 0600
        }


This will setup the authentication service that dovecot will use for delivering the e-mail to the correct user and authenticating users. Also add the following at the bottom of the file to configure the LDA of dovecot:

protocol lda {
  mail_plugins = cmusieve
  sieve_global_path = /etc/dovecot/globalsieverc/global.sieve
  log_path = /var/log/dovecot-delivery_log
  postmaster_address = noreply@domain.com
}

plugin {
 sieve = /etc/dovecot/globalsieverc/global.sieve
}

Again make sure that the user dovecot has full access to both the logfice and the global sieve directory (which you will need to make) :-). The plugin group is just for safety. I don’t know if it is needed but it doesn’t break anything. Also check the dovecot-mysql.conf that you made earlier to make sure that the uid and gid are those of the dovecot user. You can find this out by running:

:> id dovecot

If this is all setup in dovecot restart it by running:

:>/etc/init.d/dovecot force-reload

Now it is time to wrap things up and activate dovecot as the LDA for postfix, edit the file /etc/postfix/main.cf and change the virtual_transport from virtual to dovecot. Last but not least is creating the globe.sieve file.

:> vi /etc/dovecot/globalsieverc/global.sieve

require ["fileinto"];
if anyof (
    header :contains ["X-Spam-Flag"] “Yes”
) {
    fileinto “Spam”; stop;
}

Now restart postfix and send yourself a test mail. If everything is setup correctly it should be delivered to your mailbox. If not then first check ‘/var/log/mail.info’ to gather some information as to what is going on. If this indicates a delivery was attempted but it failed then check ‘/var/log/dovecot-delivery_log’ to find out more.

I hope you have fun using your spam free mail server. If you have any issues let me know, but please include snippets from the log as this will help me help you

My setup is as follows:

• Postfix as a SMTP service
• Dovecot as the IMAPS service and authentication service
• MySQL as a storage location indicating the virtual email boxes and domains
• PostfixAdmin as a package to manage the virtual domains and boxes

Installing the whole thing

The first thing you might wanna do is install the needed services and applications on your Debian machine. You can install everything using:

apt-get install dovecot-common dovecot-imaps postfix

At this point I am asuming you already have MySQL installed and configured. If not then you should do so, but I won’t include that in this tutorial. The PostfixAdmin needs to be downloaded from http://sourceforge.net/projects/postfixadmin/. You will need to setup this somewhere on the webserver site, either in an already configured apache directory or by configuring a new one.

Setting up everything in MySQL

After everything is installed you will need to prepare everything in MySQL. Connect to your machine and run the following code to generate a database and tables in that database:

create database postfix;
grant all on postfix.* to postfix identified by 'postfix';
grant all on postfix.* to postfixadmin identified by 'postfixadmin';
set password for postfixadmin = old_password('postfixadmin');

This will setup the basic stuff for postfix and postfix admin to work. Next step is setting up the data for postfix. First configure the postfixAdmin plugin by setting some of the settings in ‘config.inc.php’.

Setup Postfix configuration

The next step is configuring Postfix to enable it to sent mails, use the MySQL database and login using the Dovecot application. The latter you will setup in a later stage. First open the ‘/etc/postfix/main.cf’ file and change the entire content to something like:

queue_directory = /var/spool/postfix
command_directory = /usr/sbin
daemon_directory = /usr/lib/postfix
mail_owner = postfix## Host name given by ISP
myhostname = <isp hostname>
mydomain   = <domainname>
unknown_local_recipient_reject_code = 550
debug_peer_level = 1

# Virtual domain administration MySQL
virtual_alias_maps      = mysql:/etc/postfix/mysql_virtual_alias_maps.cf
virtual_gid_maps        = static:104
virtual_mailbox_base    = /usr/postfix/
virtual_mailbox_domains = mysql:/etc/postfix/mysql_virtual_domains_maps.cf
virtual_mailbox_limit   = 51200000
virtual_mailbox_maps    = mysql:/etc/postfix/mysql_virtual_mailbox_maps.cf
virtual_minimum_uid     = 104
virtual_transport       = virtual
virtual_uid_maps        = static:104

# The settings for the SASQL authentication using the autdaemon.
smtpd_recipient_restrictions =
   permit_mynetworks,
   permit_sasl_authenticated,
   reject_unauth_destination,
   reject_unauth_pipelining,
   reject_invalid_hostname,
   reject_rbl_client list.dsbl.org,
   reject_rbl_client sbl-xbl.spamhaus.org
smtpd_sasl_auth_enable           = yes
broken_sasl_auth_clients         = yes

# Setup authentication using Dovecot
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth

In the above configuration you need to manually change the domain name to your primary domain and the ISP domain to the name given by your ISP. Another thing you must change is the number 104 this should be the UID of the postfix user on your system. You can find out what the UID is by running: 

 id postfix

Setting up the Postfix SQL files

The last step in the postfix configuration is setting up the SQL files that instruct postfix on how to load the data from the MySQL database. Run the commands as instructed below.

:> vi /etc/postfix/mysql_virtual_mailbox_maps.cf

user     = postfix
password = <password>
hosts    = <ipaddress>
dbname   = postfix
query    = SELECT maildir FROM mailbox WHERE username='%s' AND active = 1

:> vi /etc/postfix/mysql_virtual_domains_maps.cf

user     = postfix
password = <password>
hosts    = <ipaddress>
dbname   = postfix
query    = SELECT domain FROM domain WHERE domain='%s'

:> vi /etc/postfix/mysql_virtual_alias_maps.cf

user     = postfix
password = <password>
hosts  = <ipaddress>
dbname = postfix
query  = SELECT goto FROM alias WHERE address='%s' AND active = 1 

Change the password and the host name to the ones as configured in your MySQL server. If everything went correct your postfix should be up and running by now. Please note that the logon when sending mail does not work yet as dovecot has not yet been configured.

Configuring Dovecot

Your last step should be easy. Edit the dovecot.conf file into the following.

protocols = imaps
log_path  = /var/log/dovecot_log
log_timestamp = "%Y-%m-%d %H:%M:%S "

default_mail_env      = maildir:/usr/postfix/%d/%n
mail_privileged_group = mail
first_valid_uid       = 100

protocol imap {
  imap_client_workarounds = outlook-idle
}

##
## Authentication processes
##
auth_username_chars = abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@
auth default {
  mechanisms = plain login
  userdb sql {
    args=/etc/dovecot/dovecot-mysql.conf
  }
  passdb sql {
    args=/etc/dovecot/dovecot-mysql.conf
  }
  socket listen {
    client {
      path = /var/spool/postfix/private/auth
      mode = 0660
      user = postfix
      group = postfix
    }
  }
}


This simple configuration will enable dovecot to use IMAPS (you could also add POP or IMAP). The auth default group defines how to authenticate the user. In our case this is using the MySQL database to locate passwords and the directory of the user. We also define a socket in this group that indicates a location where Dovecot should offer the authentication service for other application, being Postfix in our case.

The last step for the dovecot setup is creating the query to find the data. Run the following command and then paste the code below:

:> vi /etc/dovecot/dovecot-mysql.conf

driver = mysql
connect = host=<host> dbname=postfix user=postfix password=<password>
default_pass_scheme = CRYPT
password_query = SELECT password FROM mailbox WHERE username = '%u'
user_query = SELECT maildir, <uid> AS uid, <uid> AS gid FROM mailbox WHERE username = '%u'

You should change the host, password and UID. The UID is the one you previously obtained for postfix.

If everything is done correctly you can start adding domains, mailboxes and aliasses using PostfixAdmin. As well as starting the Postfix and dovecot service. You can do this by running:

:> /etc/init.d/dovecot start
:> postfix start

And you’re all done. Congratulations you now have a mail service running on you Debian machine. If you have any difficulties please let me know and please post a snippet of the ‘/var/log/mail.info’ to help me understand what’s going on. Trust me that I’ve probably seen most of the errors that you can encounter come by.

The strange crashes started happening once I found a page that didn’t contain the information I needed and I hit the back button. Every single time I did this Google crashed my tab with some type of cross scripting warning. Even when IE tried to recover the tab it crashed again. After the second crash IE just said, slightly paraphrased, ‘f*ck it the website keeps crashing go somewhere else instead!’.

So here is the steps to reproduce (as it crashed every single time):

• Use Internet Explorer 8
• Have multiple tabs open, I had at least two in a google search result
• I was logged into personalized search, don’t know if it is relevant but hey you never know
• Click on of the search results
• Hit the back button, or the backspace key
• Presto crasho.

Always fun to see how some javascript can crash a Internet Explorer tab. I am at least presuming it is caused by javascript.

The reason for me not to write one straight away was simple, I first wanna play around a little to find all the pleasant features and those less pleasant ones.

Bigger, better, best

The first thing you expect is this new version to be bigger, better and the best one yet. Well it is bigger in size, with 190MiB for just Java development. So this means a lot more time downloading, but hey that’s a one time thing.

So lets move onto something more relevant. Is it better then Netbeans 6.1. Lets summorize some of the issues I had with the previous version and see if the problem is solved:

1. It crashed when working on larger projects. This issue has been fixed, netbeans won’t crash or hang. Though on occasion the parser may crash. Disabling autocomplete and syntax checking.
2. For some reason previous versions had difficulty displaying the names of parameters when using Java libraries. For as far as I’ve been able to determine this has been solved.
3. The autocomplete completely fails in some more complicated web projects. Well I’m sorry to say this is still true. For some projects not only does autocomplete fail, but so does the syntax checker, both for the JSP files as well as the Java Source files.

So are there any other changes that would make you consider moving from Netbeans 5.5 to 6.5?

New stuff you will like in Netbeans 6.5

Well have there been any changes that would make you wanna switch to the new version.

• The new version actually has some basic debugging integrated of JavaScript. Though I haven’t got it working yet.
• PHP development is now support. Which is a big plus, but it still kinda feels like a cheap addon that doesn’t work perfectly yet. (Maybe in the next release)
• Thread debugging has been improved and made easier

Unfortunattely not all is good in the world of Netbeans. There are also some new bugs, some of which are really annoying.

• For some unexplainable reason adding of libraries by right clicking the ‘Libraries’ in the project view doesn’t work all the time.
• Like I stated earlier the code completion fails in some of the older Java project I’m working on. Not only does code completion fail, but the syntax checker tells me every single line is wrong. Even though the project compiles beautifully

As a final note: is it worth upgrading to this version. Then I’d have to say hell yeah, it’s a lot better then previous versions. If you are a PHP developer however I suggest not using it, for the moment. Especially when you are already using Zend-Studio, which is still a lot better.

Setting up certificate authentication is really simple. Just login to the machine as you would normally. Change to the user you are setting-up a certificate for and enter:

ssh-keygen -t rsa

You will be prompted for a password twice. After this is completed two files will be generated. One being the public file and one being the prive file. You need to download the private file from the server and store it somewhere safe. The public key needs to be copied to:

 ~/.ssh/authenticated_keys

After this you will be able to login to the server using your username and the private certificate. Please note that for Putty you will need to import the generated private key into puttygen and export it into a new private key. This is because Putty does not support the SSH generated private key.

This on its own is not that big of a deal, but I can’t seem to trace down why it’s happening. If I debug the function that causes the error has no issues. No memory violations, but when I continue running (without debugging every single call) then all of a sudden at some point in time I get a memory access violation.

Now I don’t exactly know where it’s comming from. It may have something to do with the fact that the object I’m deleting is a prototype of a prototype. And also occasionally gives me an error during compile time that I have no destructor in the class. But when I compile the files one by one it does not give this error. And the class in question does have a default destructor.

Hmm, guess it’s time for some real line by line debugging and digging into the internet about the obscure error message I got.

So far I ran every website on Debian Sarge, and though it’s stable they choose to no longer developer for it nor support it. So it was time to move to a newer version. Wich I did yesterday. Here’s some of the steps you should take before upgrading to the next version of Debian.

• Make a full backup of any website’s / databases / scripts running (don’t forget your crontab and logrotate files)
• Copy the backup to a directory not affected by the upgrade. In my VPS the provider has one folder which will be left untouched, but also pull the backup to a local system (just in case ;))
• Notify any other users also using your server, don’t want any suprises to potential paying customers.

I was lucky that my hosting provider offers an easy way to upgrade or reinstall a Linux distro on the VPS. Just a couple of mouse clicks and they prepare the VPS for the installation. Please note that this will take up to an hour to complete. Once this base installation is complete you will want to perform the following steps:

1. Un rar the backup of the websites / conf files and scripts (tar xf <tar name>) and move them to the right location.
2. Setup the proper user account /groups to have access to these files (or your users won’t be able to upload or change the website’s any more)
3. Install apache / PHP and mysql by running the following command:
     apt-get install php5 mysql5-server apache2 mysqllib
4. Setup the apache conf file to load in your restored conf files (eg: Include /export/conf/apache/*.conf)
5. Replace the mysql my.cnf with the one you backed up previously
6. Enable the rewrite and ssl modules of apache (I forgot and got some upset calls :()

From this point on your ready to go. All of the websites should be running without any problems. You do need to configure any other tools you had installed, like AWStats / Subversion or anything else. Though these should not require to much configuration as all of the configuration files where included in your backup. At least if you followed a setup similar to my first article on setting up linux.

This time round I’ve been playing around with two seperate versions of custom ROMs for the HTC P3300. The first one I’ve looked at is the ‘Black&Blue 5.0′ ROM, written by Tom. The second one is the ‘Oops Black 1.0′ also written by Tom. If you need information on how to install these ROMs then look at my earlier article and follow those steps, except you don’t install the Windows Mobile 6.0 ROM but one of the two below.

Black&Blue 5.0

Image of ROM Black&Blue 5.0

The main reason for me to use this ROM was that it was build on top of Windows Mobile 6.1. Why was this so important you may think. Well let me list a couple of nice new features in:

• Better usability with out using the styles
• A lot faster

Though it’s not a big list the first one was reason enough for me. Finally a Windows Mobile version that I don’t need a styles to operate. Some of the parts I really liked about this ROM was that it was really fast. And even better really stable. I’ve had no problems what so ever using this ROM.

You can download the ROM Black&Blue 5.0 here.

Oops Black 1.0

Image of the Oops Black 1.0 ROM

Like I stated before the second ROM I tested was a relatively new one written by Tom. The base of this ROM is identical to the Black&Blue one. It’s also running on Windows Mobile 6.1 and has more or less the same applications installed. One huge difference is that this ROM is attempting to emulate the Touch Diamond style.

For those of you interested in this please let me give you a warning in advance. The ROM looks a lot like the Touch Diamond, but it’s behaviour is nothing like it. So if you are interested in the eye-candy this ROM is for you. If you want the functionality behind it then buy the new phone.

You can download the Oops Black 1.0 here.

Warning notice: Please note that both of these ROMs have been written by a 3rd party, which means you’r phone operator will not support you if your phone breaks. As to support from HTC they might, but you did modify the phone without approval.

After some research and some more debugging this only appears to happen when debugging jsp files or jspf files. For a beta that is something I can live with. Beyond that the debugger works like a charm.

I did however found some other issues:

• Randomly crashes when right clicking in the project view, seems to have something to do with the parsing of the tree that renders the window.
• You cannot use code completion inside a switch clause. The autocomplete crashes.
• Random crashes when debugging and selecting static variables for modifying their values in code.

To be fair I have to say I never got Netbeans itself to crash. But you do get an exception window that asks you to file a bug report.