Category Archives: web servers

PERL Search and Replace in Multiple Files – Easy as PIE

I help to administer a blog with about 2000 entries.  Previously the site was managed in MovableType but earlier this year we moved it over to WordPress.  At the time we migrated about 30% of the posts to WordPress (the top trafficked 30%).  It was a slow process and as time has gone by we’ve migrated more and more posts  but it’s a time consuming process.  We need to create the new WordPress post, put a 301 redirect in place and then manually delete the old MovableType post.  And by manual deletion I mean deleting the actual HTML file that MoveableType created for that post.  You see, MovableType produces static files for each and every post, archive page, and index page.  This is a blessing and a curse, it means that the load on the web server isn’t large on heavily trafficked sites, but it also means maintaining a legacy site is a PAIN IN THE A**.  It’s actually such a pain that I have given up with MT itself and now I am working with the static HTML files directly.

This week I needed an easy way of doing a search and replace in all the legacy HTML files, all 2000 or so of them.  It needed to be recursive and ideally it needed to happen without me needing to FTP all the files to a local computer and then FTP them back.  I have command line access to the server the blog is on so I checked to see if PERL offered a way to do what I wanted working on the files in situ.  As it turns out it’s pretty simple.  I wanted to find and replace all internal links that used the old domain name (did I mention this site just changed domain names) with the new domain name.  The PERL command to do this with all the files in a single folder is like this:

perl -p -i -e 's/oldstring/newstring/g;' *.html

The -p means the script we’re running will be put through the C pre-processor before PERL compilation. The -i option means that PERL will edit the files in place. The -e option allows the running of PERL commands from the command line (it doesn’t look for a script file). The actual PERL operator s is the substitution operator while the he oldstring and newstring are regular expressions so special characters need to be escaped appropriately. And finally the operator /g means the command will do a global match. So to replace my URLs I needed something like this:

perl -p -i -e 's/www\.old\-domain\.com/www\.new\-domain\.com/g' *.html

The issue with that is that I needed the script to process sub-directories recursively to search and replace in all the HTML files. That could be done in a few different ways, the immediately obvious were chaining with FIND or GREP. I chose FIND and ended up with a command that looks like this:

perl -p -i -e 's/www\.old\-domain\.com/www\.new\-domain\.com/g' ' `find ./ -name "*.html"`

Ran that from the ubuntu commmand line and thousand or more files were processed in under a second. Very cool.

Preloading Images with CSS

Here’s a helpful snippet that will preload images with CSS. Stops that annoying lag when using CSS to switch images and the image isn’t pre-loaded.

body:after {
	display: none;
	content: url("filename1.png") url("filename2.png");

Renewing my Namecheap SSL Certificate on Linode Apache2 Web Server

This post is something of a reminder to myself. How to renew a NameCheap Comodo SSL certificate on one of my Linode Apache2 web-servers. It’s something I only have to do every few years so I essentially have to re-learn the process every time which is rather tiring. This post will be here for my future self to make the process easier.

1. Renew the SSL Certificate on NameCheap

I use NameCheap for all my domain name registrations and SSL certificates. They have many different types of SSL certificates available from several different providers ranging from simple domain certificates right up to full wildcard organization based certificates. The great thing about any of them is that NameCheap will email you well in advance of them expiring. When they do so it’s simply a matter of logging into your NameCheap control panel and renewing the certificate. A new entry will be created with the status of “Waiting Activation”.

2. Create the CSR File

My host runs apache2 on Ubuntu and OpenSSL. Apache looks for certificates in /etc/apache2/ssl and I usually create a sub-folder for new CSR and key files when the old ones expire. I feel its a bit more secure to generate a new Certificate Signing Request (CSR) whenever a certificate expires, which is why I do this. So I create a folder for the files:

cd /etc/apache2/ssl
mkdir december-2014
cd december-2014

Then create a CSR file using:

openssl req -newkey rsa:2048 -days 1095 -nodes -keyout -out

The openssl req command will then ask for some information to encode with the request. That looks something like this:

You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
Country Name (2 letter code) [AU]:AU
State or Province Name (full name) [Some-State]:South Australia
Locality Name (eg, city) []:Adelaide
Organization Name (eg, company) [Internet Widgits Pty Ltd]:My Company Name
Organizational Unit Name (eg, section) []:
Common Name (eg, YOUR name) []
Email Address []

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:My Company Name

This will create the CSR file ( and a SSL certificate key file ( The key needs to be protected with:

chmod 400

3. Issue the SSL Certificate via NameCheap

The contents of the CSR file are needed to issue the SSL certificate via the NameCheap control panel. View the contents with:


The contents will look something like this:


Copy the contents to the clipboard and go to your NameCheap control panel and select the SSL certificate to activate. Select the correct server configuration (in this case Apache2 + OpenSSL) and paste the contents of the CSR file into the CSR Text field and submit the activate request. NameCheap will then ask for an email address to confirm the issuing of the SSL certificate for the domain. Confirm the request and you’re done with NameCheap for this process.

4. Confirm the SSL Certificate Issue

The SSL certificate I renewed was a Comodo certificate and after a few seconds I got an email from Comodo asking me to confirm the issuing of a certificate. The email looked like this:

Comodo Confirm Issue SSL Certificate Email

Comodo Confirm Issue SSL Certificate Email

It’s simply a matter of clicking on the confirmation link in that email and pasting the “validation code” into a text box on the resultant web page. Once you’ve done that another email will arrive in a few minutes with a ZIP file attached. The zip file contains the following files:

Comodo SSL Files

Comodo SSL Files

The AddTrustExternalCARoot.crt is the CA (certificate authority) root certificate which identifies the company we’ve paid to issue the SSL certificate. In this case, Comodo via NameCheap. You’ll need that file. You’ll also need the www_mydomain_com.crt file which is the signed CSR.

5. Install and Protect Certificate File

Both files mentioned above need to be copied to the folder we created in step 1, namely:


Once there protect the signed crt file with:

chmod 400 www_mydomain_com.crt

6. Create the Certificate Bundle

To complete the SSL chain for Apache you must create a bundle with the files send by the certificate authority. This is done easily with:

cat AddTrustExternalCARoot.crt COMODORSAAddTrustCA.crt  COMODORSADomainValidationSecureServerCA.crt www_mydomain_com.crt >

7. Configure Apache to use New SSL Certificate

The last step in the process is to configure Apache to use the new SSL Certificate. Open up the site configuration file with:

nano /etc/apache2/sites-available/site-name.conf

In my case I looked for the SSL VirtualHost section of the file which looked like this:

    SSLEngine On
    SSLCertificateFile /etc/apache2/ssl/december-2011/www.mydomain_com.crt
    SSLCertificateKeyFile /etc/apache2/ssl/december-2011/
    SSLCACertificateFile /etc/apache2/ssl/december-2011/PositiveSSLCA.crt
    SSLCertificateChainFile /etc/apache2/ssl/december-2011/
    DocumentRoot /srv/www/public_html/
    ErrorLog /srv/www/logs/error.log
    CustomLog /srv/www/logs/access.log combined
    AddHandler cgi-script .cgi .pl

You can see that the SSLCertificateFile SSLCertificateKeyFile and SSLCACertificateFile directives need to be changed to use the new certificate files. I modified them to look like this:

    SSLCertificateFile /etc/apache2/ssl/december-2014/www_mydomain_com.crt
    SSLCertificateKeyFile /etc/apache2/ssl/december-2014/
    SSLCACertificateFile /etc/apache2/ssl/december-2014/AddTrustExternalCARoot.crt
    SSLCertificateChainFile /etc/apache2/ssl/december-2014/

Saved the file, exited Nano and then restarted Apache with:

service apache2 restart

8. Verify New Certificate is Installed

If you’ve done everything correctly it’s a simple matter of checking if the certificate is installed correctly. I did it by navigating to the website in Google Chrome and clicking on the https padlock icon and viewing the certificate information. It looked like the image below and you should see that the expiry date has updated correctly. Job done.

Success - SSL Certificate Installed

Success – SSL Certificate Installed

Bash Shell Vulnerability Found – Patching Ubuntu

The interwebz is abuzz with the latest vulnerability for *nix based systems that us the BASH shell. The so-called Shell Shock exploit allows devious users to run commands against your system using a bug in the way BASH handles environment variables. Ouch. To check if your system is vulnerable run the following in a shell:

env x='() { :;}; echo vulnerable’ bash -c “echo this is a test”

If your system is vulnerable you’re going to see this:

 this is a test

If your system is not vulnerable you’re going to see something like this:

bash: warning: x: ignoring function definition attempt
bash: error importing function definition for `x'
this is a test

There’s a patch available for Ubuntu systems already. All I did was this:

sudo apt-get update
sudo apt-get install bash

And hey presto the system is no longer vulnerable.

How I Reload my iptables Rules on Ubuntu Reboot

There’s a few ways that custom iptables rules can be re-loaded when your Ubuntu server reboots. I’ve chosen to reload mine using the /etc/network/interfaces file. Here’s what I’ve included in that file:

auto lo
iface lo inet loopback
pre-up iptables-restore < /etc/iptables.firewall.rules
auto eth0
iface eth0 inet dhcp

The key line here is the line starting with pre-up. This directs the iptables-restore to reload my rules from the /etc/iptables.firewall.rules file.

Another way of accomplishing the same thing is to create a script file in the /etc/network/if-pre-up.d/ directory and put the following in it:

/sbin/iptables-restore < /etc/iptables.firewall.rules

Then set the permissions on the script file with:

sudo chmod +x /etc/network/if-pre-up.d/your-filename

Setting up Apache Permissions for www-data and an FTP User

A problem I have when setting up a website on a new server (or a new site on an existing server) is sorting out permissions for the Apache user (www-data) and an FTP user. I can never quite remember how to set things up so that I am not continually needing to log into a console to adjust permissions so I can FTP some files to the server. This post is the definitive reminder to myself showing how it should be done. This example applies to Ubuntu but I guess it is equally applicable to other flavours of Linux.

Add the FTP User to the www-data Group

First thing we want to do is add the FTP user (you have created an FTP user haven’t you?) to the www-data usergroup. www-data is the user/group used by Apache.

sudo adduser ftp-username www-data

Change Ownership of Files

The next step is to set the www-data group as the owner of all files and directories in the HTML source directory.

sudo chown -R www-data:www-data /var/www

Grant Group Permissions

Now we want to add write permission for the www-data group for all the files and directories in the HTML source directory.

sudo chmod -R g+w /var/www

Add umask for New Files

The final step is to make a change to an Apache configuration file so that the umask for new files created by Apache is such that the www-data group has write permissions on them. Open /etc/apache2/envvars in your text editor of choice and add this to the bottom of the file:

umask 007

The three octal digits for umask are for the Owner/Group/Others. The 0 leaves permissions unmasked (ie left at read/write/execute) and 7 gives no permissions at all. This would be equivalent to chmod 770. There’s a useful chart here showing the relationship between the binary rwx permissions and the octal numbers used by chmod and umask.

Credit for this must go to the top voted answer to this question on

Checking if Someone is Logged into MyBB

I run a little web forum that uses MyBB. I’ve been building some tools for members of the forum to use and thought it would be good to restrict access to those tools to logged in forum members. Turns out this was VERY easy to do. MyBB drops a cookie for logged in users called mybbuser. This cookie contains data in this form:


uid and loginkey are columns from the MyBB users table. So, checking if someone is logged into MyBB is simply a matter of checking for the existence of the mybbuser cookie which can then be exploded to select the users details from the MyBB table. Here’s some code I knocked up in PHP to select the MyBB username if the user is logged into my forum.

	if (isset($_COOKIE['mybbuser']))
  	        $link_id = mysql_connect($server,$user,$password);
         	@mysql_select_db ($database);	
		$result=mysql_query("select * FROM ".$table_prefix."users where uid=".$user_details[0]." and loginkey='".$user_details[1]."'");
		if ($result)
	  	if (mysql_num_rows($result)>0)


You can now use that username anywhere in the web page. Or use it as a conditional check to display certain content to logged in MySQL users and different content to non logged in users.

Installing Munin on a Linode Node

I’ve followed this guide for installing Munin on my nodes at Linode a few times now. Munin is a remote server monitoring tool that allows you to monitor all sort of things about the health of your servers. Servers you want to monitor are “munin-nodes” and the server where you view all of the reports from is the “munin-master”. It all makes sense but the Linode installation guide fails horrifically if your apache webroot is not the default /var/www that Munin expects it to be. So as a reminder to myself, here’s how I got Munin working on a server where the web root was /srv/www.

First, follow the install guide exactly as instructed by the good folk at Linode. Then modify /etc/munin/apache.conf to remove references to /var/www and change them to your web root of choice. My file looks something like this:

Alias /munin /srv/www/munin
<Directory /srv/www/munin>
                Options Indexes FollowSymLinks MultiViews
                AllowOverride all
                Order allow,deny
                allow from all
    <IfModule mod_expires.c>
        ExpiresActive On
        ExpiresDefault M310

Then I restarted Munin with:

sudo /etc/init.d/munin-node restart

I happen to use my munin-master as a dumb web host so the pretty Munin web pages are just in a folder off of the web root. You could be tricky and put them on their own sub-domain but I’ve chosen not to do that. So they live in a sub-folder and I can access them using this address:

And when I did I was seeing a 403 forbidden error. And when I took at look at the apache error log saw an entry like this:

[Mon Feb 03 04:09:23 2014] [error] [client] client denied by server configuration: /var/cache/munin/

Clearly Munin is still trying to access a folder outside of my webroot. So then it was a matter of adding this new section to my /etc/apache2/sites-available/default file:

        <Directory /var/cache/munin/www/>
                Options Indexes FollowSymLinks MultiViews
                AllowOverride None
                Order allow,deny
                allow from all

Then restarting apache with

sudo /etc/init.d/apache2  restart

Now if I visited the Munin pages using the same address as above I can see pretty graphs telling me all about my servers! w00t!

Installing the RFC 868 TIME Service on Ubuntu

One of my products uses an old OCX control that can query NIST and old RFC 868 TIME servers to get the current time. To make things easier for the users of the product I have a server that provided the RFC 868 TIME service for them to use. The main reason I did this is that service for such an old protocol is a little patchy and I wanted them to have a server they could rely on. Main problem is that the old protocol isn’t included in Ubuntu any more (the newer NTP protocol is supported though with NTPDATE or NTPD). So I’ve had to work out how to enable the RFC 868 Protocol on one of my new servers. Turns out it was pretty simple. First download and install the XINETD super-server.

sudo apt-get install nfs-common nfs-kernel-server xinetd

Then edit /etc/xinetd.d/time to enable the time server. I changed mine to the following because I only wanted to use the TCP version.

service time
        disable         = no
        type            = INTERNAL
        id              = time-stream
        socket_type     = stream
        protocol        = tcp
        user            = root
        wait            = no

# This is the udp version.
service time
        disable         = yes
        type            = INTERNAL
        id              = time-dgram
        socket_type     = dgram
        protocol        = udp
        user            = root
        wait            = yes

Final step was to restart the XINETD super server with:

sudo /etc/init.d/xinetd restart

Note to self. Ditch the OCX control and build something in .NET that uses the newer protocol on port 127.

Installing the PHP MCRYPT Extension on Ubuntu

I use the PHP mycrypt extension to generate some encrypted URLs for one of my online applications. I had reason to test some of this functionality on my local development web server recently and the following error was thrown by PHP:

Call to undefined function mcrypt_create_iv()

Turns out that I didn’t actually have the MCRYPT extension installed on my little Ubuntu web server. Doing so was simply a matter of typing this at a command prompt:

sudo apt-get install php5-mcrypt

And that’s it. No need to re-start Apache or anything else. Groovy.