Apr 09 2016

OpenVPN tricks and traps

Why

Recently I had to update my certificates. As things go, I decided to do a 2 tier CA. Of course, this compilcates the config of systems like OpenVPN. This needed some bending of the configs to get what I wanted.

How

Critical config enrties to make this all work – with comments

ca /etc/openvpn/root-ca.cert <= root CA certificate
cert /etc/openvpn/chain.cert <= certificate chain : server cert + intermediate CA cert
key /etc/openvpn/private.key <= certificate private key
dh /etc/openvpn/dh.pem       <= Diffie-Helman key exchange parameters  
tls-version-min 1.2          <= force TLS 1.2 ciphers

Results

  1. Control Channel: TLSv1.2, cipher TLSv1/SSLv3 DHE-RSA-AES256-GCM-SHA384, 2048 bit RSA TLS v1.2 achieved.
  2. OpenVPN working.

May 11 2015

Debian 8 (Jessie) updates are coming

Debian 8 “Jessie”

As most of you most likely already know, Debian Linux 8 aka “Jessie” has been released in the last few weeks.

This release brings us a host of significant updates: Xen 4.4, apache 2.4, php 5.6.x, postfix 2.11.x and perhaps most controversially a new parallel service start-up utilitiy called systemd

The Linux distribution vendor notice of the Debian 8 release is here. have a read if that interests you.

VRL Testing

We have been testing this release extensively on our test rigs, and, while we still have some tests to complete, the overall upgrade path looks relatively painless. The most significant change which requires an outage is the change to the “allow/deny” syntax in apache vhost stanzas and in the .htaccess file configurations.

These issues are well documented in this article from the Apache Software Foundation. It is well worth reading this article, as there are some traps here for the unwary.

Upgrade Plans

Our current plan is to upgrade and stabilise the hosts that require the least configuration changes to do so. These are likely to be, in the fist instance, the secondary MX and DNS servers and the database server. More details on this in our future posts

As always, our clients will be given advance notice of the planned outages.

Dec 07 2014

Server Outage on 2014-12-10 23:00 AEDT

Vandrad Research Labs need to perform a server reboot to activate a number of kernel and kernel module security and stability patches.

* WHEN: Currently the server reboot work is scheduled to start on
2014-12-10 23:00 AEDT and for all work to be completed by 2014-12-10 23:30 AEDT

* HOW LONG: The outage will require approximately 30 minutes. We will endeavour
to keep the disruption to minimum.

* AFFECTED SERVICES: all services will be affected by this outage.

* WHAT: We will distribute new kernel modules to all Xen VMs and reboot the VM
cluster to bring all systems to the same Linux kernel baseline.

Feb 09 2014

Server outage scheduled for 2014-02-12 23:00 to 2014-02-12 23:30

With the release of Debian Wheezy point release, on 09 FEB 2014, all the
servers have been upgraded to run Debian Wheezy 7.4 and as a  consequence we now
need to reboot the servers to take advantage of the kernel and kernel module
security and stability patches.

* WHEN: Currently the server reboot work is scheduled to start on
2014-02-12 23:00 AEDT and for all work to be completed by 2014-02-12 23:30 AEDT

* HOW LONG: The outage will require approximately 30 minutes. We will endeavour
to keep the disruption to minimum.

* AFFECTED SERVICES: all services will be affected by this outage.

* WHAT: We will distribute new kernel modules to all Xen VMs and reboot the VM
cluster top bring all systems to the same Linux kernel baseline.

Aug 19 2013

About efficacy of the fail2ban daemon:

As many of you know we have been using various countermeasures to stop unwanted individuals  from getting into our servers.  Especially the kind that attempt to brute-force a username/password combination.

To that end we have recently deployed the ‘wp fail2ban’ plugin . The question is does it work?

The answer is below:

--------------------- fail2ban-messages Begin ------------------------
 Banned services with Fail2Ban:                          Bans:Unbans
    ssh:                                                    [  7:4  ]
    wp-auth-fail:                                           [944:6  ]
---------------------- fail2ban-messages End -------------------------

Please bear in mind these are 24 hour stats - just one day of activity... 

Watch this space for some instructions oh how to set-up fail2ban effectively.

Aug 15 2013

Updated Xen Hypervisor, Linux kernel and more

Vandrad Research Labs have carried out major work last night.

Between 21:30 and 22:30 on 14 August 2013 AEST, the main server was upgraded and the following work was done:

  • the Xen server was upgraded to Debian 7.0
  • new version of Xen Hypervisor (Xen 4.1.4) was installed
  • a new Linux kernel (3.2.46) and modules was distributed to all Xen virtual servers.

In addition to the regular maintenance,  we have also noted a marked increase in slow but persistent probes against our hosted web application platforms and server ssh services.

To help curb this a little and keep us a bit safer from brute-force attacks we have deployed additional fail2ban jail recipes and added logging plugins to the various web application and CMS platforms to allow the server level fail2ban to block access from badly behaved IP addresses.

If you would like more details, drop us a line in the comments or contact us by the usual means.

Jun 01 2013

email server upgrade

Our main mail email server has been upgraded:

  1. Linux distribution was upgraded from rom Debian Squeeze (Debian 6.0.7) to Debian Wheezy (Debian 7.0). This means:
    • new libc, kernel
    • new postfix, courier POP and Courier IMAP daemons
    • new bind daemon
    • new ssh daemon
  2. We used this upgrade as an opportunity to migrate the email domains and users database to the new database server; and
  3. We uninstalled all databases form the email server (freeing up RAM, disk and CPU resources in the process).

Overall we are very happy with the result.

May 27 2013

database server build

Long time between updates.

Most recent upgrade to our systems is the establishment of a dedicated Mysql/Postgresql server for use with our WordPress and Drupal CMS servers, and to hold other databases.

So far, We have migrated 18 or so WordPress databases to this server. Next up is migration of remaining WordPress databases, the Drupal instances and other databases to the central server.

To date, this upgrade has resulted in significant improvement to the performance on out main WordPress server, mostly because it allows the web server to act as a web server, and offload all the database queries to a dedicated platform.

Feb 25 2013

Upgrade to Debian 6.0.7 installed

An upgrade to Debian 6.0.7 was installed last night across all servers.

This work  appears to have been successful and no adverse side-effects have been observed as at this time.

Debian 6.0.7 includes an update to the Xen hypervisor and Linux kernel binaries. To activate these, a reboot of the Xen server stack  will be required. This will be scheduled shortly and an email will be sent to all affected clients.

 

Feb 04 2013

mysql backup script updates

The VRL servers now have a new backup script that does:

  • dump of all of the databases from a mysql instance
  • sha256 checksumming of db backups
  • sha256 checksumming tarballs of multiple db backups
  • gpg encryption of the tarballs once they are assembled
  • deletion of backup files and other files older than 14 days

The script needs more work before it will be published in a public git repository

 

 

Jan 05 2013

RKHunter upgrade

Today we have rolled out a new version of the rkhunter package – RKHunter 1.4.0  – to all of our managed hosts.

Its a relatively simple process on the Debian hosts:

  1. Downlaod the rkhunter 1.4.0  (at this time it is 1.4.0-2)  Debain package from your local Debian project mirror.
  2. Verify package hashes and signatures (There are many ways of doing that. If needed I will post an article about this… )
  3. Installtion on all chosen servers (next lot of steps is scripted):
    1. scp the package to each of the target hosts
    2. install package using dpkg, choosing the necessary options for your system when asked if you want to keep current configuration or use the package manitainer provided one.
    3. run rkhunter properties update to prevent false-positive warnings (running  sudo rkhunter  –propupd  does the trick )
  4. Verify that you have a cron.daily job to run rkhunter

This new version of rkhunter introduces a wider root-kit detection library and fisex several bugs which led to false-positive reports in the past.

Oct 10 2012

Advance Notice : October Server Reboot

Due to a number of kernel bugs patched in recent updates, Vandrad Research Labs will be scheduling server reboots in near future. More information as the schedule firms up

Sep 30 2012

Debian Linux 6.0.6 stable release update

Debian foundation have advised that the stable Debian release, Debian 6.0 (aka “Squeeze”) has been released to public.

Vandrad Research Labs staff are in the process of updating our servers.

A list of updated packages and the offcial write can be found in the official announcement on the Debian foundation web page.

Sep 28 2012

Updated VPS price list.

For a long time we have not looked at our prices, and how we structure our products and services.

Having done a through review, we have decided to streamline the services offerings and simplify our pricing. The very first result of this is our updated pricing structure for virtual servers and associated services.

Details can be found here.

Next off the rank will be an update of our shared infrastructure services followed by a review of the professional services/consultancy offerings.

Jun 17 2012

IP address migration completed

All the work that was necessary for IP address migration has been completed (including clean-up works).
As at this time we do not anticipate any further service disruptions, other than the regular security patch updates.

Jun 01 2012

New IP addresses for servers

Our upstream provider have advised that they need to migrate us to a different IP subnet.

Much fun is about to be had by all… some VM reboots required.

Check back on this page for more information…

Apr 17 2012

April server reboot.

A scheduled reboot of the Vandrad Research Labs Xen server stack has been successful. All servers appear to have come back on-line without a problem.

The reboot was necessary to update the kernel and to distribute new modules to all Xen hosted instances.

Feb 10 2012

Debian 5.0 aka “Squeeze” Security support terminated

The Debian foundation have announced that as of 6 FEB 2012 they are terminating the ongoing security support for  Debian 5.0 aka “Squeeze”.  All customers using this release of Debian Linux are requested to upgrade to current stable distribution of Debian Linux – Debian 6.0  aka Debian “Lenny”.

The full text of the advisory can be found here on the Debian news website.

May 06 2011

server stack updates

We are planning to upgrade the XEN server stack sometime in the next two weeks.

Unfortunately this process will involve some down-time for all of our hosted clients.

The upgrade will take the current servers from XEN 3 series hypervisor to a XEN 4  series hypervisor and upgrade the underlying Linux kernel to the current Debian 6 provided versions. As part of the upgrade process we will distribute the new kernel modules to all XEN virtual servers.

The upgrade is driven by the desire to take advantage of the improvements that XEN 4 series hypervisors bring in terms of performance CPU, I/O and RAM and to enable some of the more stable and faster virtual devices and associated management features.

We will be contacting our hosted clients over the next few days to work out the best time for the outage .

Mar 13 2011

Fixing Australian time zones in plone

One of our clients, the Ancient Arts Fellowship have requested a new Plone site.

After some deliberation and several futile attempts to upgrade their current Plone 2.5.5 to a plone 3.x instance we made a collective decision to run with a new Plone 4 setup.

Of course the “curse of the time zone” reared its ugly head and bit us on the bum.

the “Curse of the Time Zone” arises when (for reasons historic and political) there is a collsion between the GNU/Linux timezone names and those expected by zope/Plone framework. This happend because the GNU/Linux servers call Australian and US time zones “EST” and “EDT”… while zope/plone stack calles them AEDT, AEST for Australia and  EDT and EST for USA.

Ultimately the solution is to set a timezone variable inside of the Zope framework to take care of the differences, but that problem is what to call it…  Australia/Canberra and Australia/Sydney did not work . But after a few searches and trial and error we found this magic string  “AEST-10AEDT-11,M10.1.0,M4.1.0/3” it seems to work.

so the process is:

  • build your zope/plone stack.
  • go to base.cf in $TARGET/zinstance
  • add the follwing lines to [instance] section:
      environment-vars =
          TZ AEST-10AEDT-11,M10.1.0,M4.1.0/3

 

Mar 06 2011

Upgrading postgresql databse server software on Debian Squeeze

I am very happy to report, that just as just as was described in the PostgreSQL update post the same process works flawlessly for migrating postgresql  8.3.x to 8.4.x on Debian Squeeze (Debian 6.0)

Jan 31 2011

Server outage 22:30 AEDT, Tuesday 01/02/2011

An outage is required to activate linux kernel patches as detailed in DSA 2153-1 -linux-2.6 security update.

Current schedule is for 22:30 to 23:00 on AEDT on Tuesday 01/02/2011
Service will be unavailable for up to 30 minutes

Nov 28 2010

Outage Advisory 2010-11-30 20:00 to 20:30 AEDT.

Outage Advisory 2010-11-30 20:00 to 20:30 AEDT.

A scheduled server maintenance outage will take place between 20:00 and 20:30 AEDT on 30/11/2010.

We expect that the actual service interruption will take about 10 minutes
because a reboot of the server is required.

Nov 25 2010

Hosting Joint Venture Launched

Vandrad Research Labs are proud to announce that together with Mr Dale Baldwin of dalebaldwin.com we have launched a new web services and hosting joint venture to support Dale and his growing web design and consultancy business based out of Hobart in Tasmania.

Recently Mr Baldwin has advised that this arrangement has allowed him to close two deals in a very short period of time. Excellent work Dale, and we look forward to having our inaugural customers on board very shortly.

Nov 22 2010

Only 6 static IP addresses left…

We did some system accounting last night – we only have 6 more static IP addresses left.

We still have reasonable amount of RAM and a lot CPU cycles and several hundred gigabytes of HDD space in reserve but we can only cater for 6 more dedicated Xen Linux servers, and two of those are currently reserved for existing projects (one server is being built, with another server order expected soon).

This means I am in position to offer only 4 more dedicated Xen Linux servers and after that we are at capacity as far as provisioning of new Xen Linux systems on our main hosting system.

If you are looking for a server to host your websites, blogs, email or e-commerce, look us up at VandradLabs. I am sure we can come up with a hosting solution that suits your needs.

Nov 02 2010

More RAM…

We have ordered more RAM for our hosting servers. This will allow us to host more clients and provide mode dedicated resources per client on each server.

Sep 05 2010

New Blog Crated

Today I created a new WordPress Blog engine instance for “Rational Capital” – http://www.rationalcapital.com.au/ – which is a sceptic and rational thought blog run by a very good friend of ours, a certain Mr Andrew Gould.

We look forward to see what Mr Gould has in store for us.

Aug 22 2010

hosting servers – maintenance window

Server maintenance window required. At the moment scheduled for Tue, 24 Aug 2010 from 20:30 to 21:30 AEST. Email to hosting clients will follow.

Jun 26 2010

Blog Engine Upgrage no 2

Yesterday I reconfigured all the instances of WordPress to talk separate wordpress instances. Each container can be upgraded and themed independently of any other.

Each container can also serve the same content to multiple domain names.

This now has been tested and an in-place upgrade to WordPress 3.0 is working very nicely.

May 10 2010

Blog Engine Upgrade no 1

Just a quick note
blog software has been upgraded to wordpress 2.9.2

So far, no screams of anguish from anyone.