Xenion Hosting

Xenion Updates!

2011-06-19T03:06:00.001-07:00

It seems that although I've written posts, I've not put them to be published! I apologise for this!

Posts will be going out shortly, so please stay tuned!

New Transit Service - IPv6

2011-04-05T06:07:00.000-07:00

I've just turned up another transit feed from WAIA today, thanks to the wonderful help of Joe Wooller. On the plus side, that includes both IPv4 and IPv6, so Xenion clients should expect native IPv6 soon!

WAIX upgrade - gigabit!

2010-11-13T02:30:00.000-08:00

I've just updated the WAIX link from 100mbit to 1000mbit. Clients won't notice the change (unless of course you happen to do a lot of WAIX traffic) but there's now a lot more room to grow in the future.

Since Xenion doesn't charge extra for WAIX traffic - it's up to me to make sure it's fair and equitable! - no-one should have any problems now using or providing local services.

A bit of a change - working on Atheros / 802.11

2010-06-08T21:01:00.000-07:00

One of my current contracts has me working on the FreeBSD 802.11 support (specifically Atheros) chipsets for a local company. I've been adding some missing features to the later chipset support and working out kinks in general.

This has involved spending a lot of time knee-deep in the Atheros driver code. What you would traditionally find in firmware on wireless NICs is implemented in the host driver. The hardware handles packet transmission and reception, doing collision avoidance, encryption/decryption, the radio encoding and decoding; the driver pretty much handles everything else. This includes keeping all that per-state information about wireless stations, handling authentication and roaming, and even includes things like doing radio and baseband calibration.

The most difficult part of this has been the distinct lack of documentation. So far I've made some good contacts in the Linux developer community who include a number of developers employed by Atheros (and have access to documentation!) and I've made quite significant inroads in making the FreeBSD atheros/wireless code a lot more stable.

The wireless/embedded development work can be found at my FreeBSD Wiki site. The majority of the improvements will make it into FreeBSD-9.0.

Xenion is still providing on-going Squid/Lusca/CDN support and development services to a variety of clients. This is not going to change any time soon!

Twitter

2009-09-19T04:10:00.000-07:00

I should've mentioned this a while ago - Xenion status updates are posted on the @xenion_pty_ltd twitter feed. Please feel free to subscribe and post messages/queries!

Squid docs

2009-08-03T21:34:00.000-07:00

I've started writing up some of my notes from Squid consulting into something (mostly) fit for public consumption.

This is partly to aid myself and partly to try and stop others from finding and fixing the same mistakes.

The fledgling documentation dump is here . I'll be adding more to it as I type up more notes and complete more work!

Installing Proxy Cache Servers for Fun and Profit..

2009-07-15T05:36:00.000-07:00

One of my current contracts involves setting up a web cache farm for an ISP on the end of a whole lot of full duplex satellite IP. They initially specced out 5 rather large servers (at least $10,000 each); I think they had a minor heart attack when I reduced that to one server. But then, the cost of bandwidth savings versus hardware (and Xenion's contracting/support rates!) is very minor in the long run.

In any case, it has been a resounding success. I'll summarise how things look at the moment; I'll do up a proper press release sometime later next month.

There's about 15,000 users sitting behind the single proxy cache server, with around 100mbit or so aggregate satellite IP bandwidth. The service uses a slightly modified FreeBSD-7 setup to support fully transparent HTTP interception (both client and server-side IP address spoofing) with a Cisco 3750 providing the WCCPv2 interception.

Tuning the FreeBSD stack (and Linux too, for those Linux people out there!) to effectively scale for satellite IP is no easy feat. It took a bit of time but I have quite a bit of experience in this area so the tuning was quite successful. The issue here is finding the right balance between throughput, scaling and link efficiency. A little bit of first year college mathematics helped me predict some decent settings and they work as expected.

The software is Lusca-HEAD (the very recent version as of this post) - this gives me all the useful Squid-2.7 features, stability and performance with my extras (twiddles for satellite IP stuff, TPROXY support, etc.)

The box itself is a dual dual-core AMD Opteron 270 at 2GHZ; 16gig RAM; Intel 1000/pro NIC, 3ware 9000 series SATA controller with 12 x 500gig 7200rpm disks of some sort. The disks are all mounted individually - no RAID at all. 10 disks are for storage; 1 for OS and 1 for logging.

The box pushes around 80 to 120mbit at peak with a byte hit rate between 20 and 40%. The request rate sits between 300 and 600 requests a second, sometimes peaking to 800 or more. This translates to traffic savings (saving a whole lot of money - satellite transponder space is expensive!) and much improved performance for clients.

It also handles between 10,000 and 20,000 concurrent connections with peaks over 40,000. Yes. 40,000 concurrent connections. I'm not making this up.

The cache size at the moment is around 2TB and 20,000,000 objects. I'm absolutely, positively not filling the disks to capacity for a whole lot of very good reasons. (Hint - don't do it.) I'll be happier to increase the storage to 4TB and beyond once I've deployed COSS for the small objects and tidied up some of the memory usage. The Lusca process is around 4 gigabytes at the present time and 75% that is the storage index and related bits.

Just for interests sake - out of the 20,000,000 objects, around 300,000 of them are larger than 256 kilobytes. The rest are small objects. It is quite scary actually how much of the cache directory is small objects.

I've included some preliminary windows update caching which is providing a 100% hit rate for the update files themselves. It's actually quite scary how simple it was to implement. Shame on you Microsoft for -almost- but not quite getting HTTP caching "right" in the windows updates.

All in all, the client in question is extremely happy about the support, installation and performance of the cache. There's a shortlist of items to do including Lusca improvements and reporting tools so the client can provide further information to his boss about how effective this all is.

why this blog is suddenly a spam blog?!

2009-06-15T00:58:00.000-07:00

So apparently updating all of your labels to be consistent is enough to trigger the spambot logic. I apologize to anyone reading this blog and thinking its spam - honest, it's not. Really! Honest!

Grrr!

New replacement hosting service - hosting-5

2009-06-13T03:16:00.000-07:00

G'day,

I've just deployed a new hosting server (hosting-5.) It's running on the new network setup, running CentOS 5.3 32-bit, and generally seems quite well-behaved.

I'm going to migrate everyone on the old Fedora Core 6 hosting server over to this over the next week and then finally retire it.

EDIT: I've migrated a couple of customer VMs onto it (with their permission, of course!) and it has fixed their stability. Even Ubuntu VMs, traditionally having been very unstable, are now stable once again. Success!

Hosting-4 downtime

2009-06-04T23:47:00.001-07:00

One of the VM servers, hosting-4, needed a spontaneous reboot this afternoon. It seems the Xen management software got very upset after only 360 days of uptime.

On the downside, it means those who are hosted on hosting-4 will suffer a 10-20 minute downtime. It should have been 5 minutes except that the box and VMs have been up so long that fsck is enforcing file system checks.

For example:

/dev/sda1 has gone 361 days without being checked, check forced.

So to make things run smoother, I'm manually starting each VM once the previous one has finished fscking.

On the upside, the server is now running the latest CentOS 5.3 Xen packages which have fixed a fair few bugs.

I apologise for the downtime. I have to say though, 360 day uptime is pretty good. I'll just have to make sure that further downtime is scheduled in advance.

Outage - interstate and international traffic

2009-06-01T08:32:00.000-07:00

There's some issue with my upstream's upstream's interstate link provider. Its affecting interstate and international traffic for my upstream, my upstream's upstream and potentially other providers.

I'll post an update when the problem is resolved.

EDIT: I really should point out that I'll feed twitter updates: http://twitter.com/#search?q=%23xenion

EDIT: The service has been restored at 1:10am. I'll keep an eye on things for a bit longer.

FreeBSD 6.3 and FreeBSD-7 Xen hosting

2009-05-22T09:43:00.001-07:00

I've been playing around with the FreeBSD-7.x and FreeBSD-6.x Xen DomU support (thanks to Kip Macy) and documenting all of the strange bits needed to make a fully working environment.

I've managed to figure out all the right incantations to build the DomU, run the DomU and for the most part, keep the DomU up and running.

I may offer FreeBSD DomU support to Xenion customers with part proceeds being donated back to the FreeBSD project.

Let me know if you're at all interested in this!

Solaris 10, Active Directory, Squid-2.7, NTLM. Eww.

2009-05-22T00:49:00.001-07:00

I've been working on another Solaris 10 and Active Directory + Squid NTLM integration project. I think that I've finally coaxed out the niggling bits from all of this.

In summary (thus far);

The latest Solaris 10 ships with a "sun free software" Samba package with Kerberos and Active Directory already working. Good.

It -may- still have the 8 character password limit in the "net ads join" command (for "logging in" the server into the Active Directory.) Eww.

The Kerberos setup is a bit crack smoking but reasonably trivial. The trick is making sure the realm is setup right (capitalise the realm in the kerberos configs) and that the server queries the Active Directory DNS or things just don't work. (Active Directory DNS is used to discover services - eg ldap, kerberos, wins, etc.)

The default LDAP query results in Active Directory is limited to 1000 entries. So "wbinfo -u" doesn't return all the users from a large Active Directory.

Figuring out why/when to restart winbind; when to purge the winbind idmap/usermap tdb files is very Eww. I need to properly understand what is going on there.

Make sure the damned server is NTP synched to the AD servers.

I need to make certain that the Active Directory Kerberos is returning renewable tickets.

The winbind separator works best when its "+" apparently. Again, not sure why. I need to document all of this.

Having tightly controlled firewalls makes a 1 day job take a week; but it has shown me all the random communication which happens. For example, Samba uses LDAP-over-UDP on this setup to do the initial net join..

There's more to come as I finalise this installation. I'll publish the install guides on my website.

Downtime - web hosting services

2009-05-21T20:02:00.001-07:00

There was a brief outage today on the web hosting services cluster. I've kicked the relevant service hard and things seem to be working again.

I'm looking forward to Sunday's upgrade and shuffle - I'll be able to do a lot more with what I have after that.

Current outage

2009-05-18T09:38:00.000-07:00

I lost connectivity to the data centre a few minutes ago. It looks like all the WAIX participants down there are offline.

.. nope, its back now. I wonder what the problem was. There wasn't a power loss in the data centre so it looks like a problem with the backhaul to WAIX.

Hosting Referral Special!

2009-05-14T10:02:00.001-07:00

G'day everyone,

I'm running a little referral campaign for May and June 2009. Existing clients who bring in three new clients get three months free.

Don't be shy!

Project Updates!

2008-12-05T07:36:00.000-08:00

Here's a few updates concerning my current and upcoming projects at Xenion "Labs".

From the Squid consulting side of things:

I'm putting together a proposal to implement generic Quota support in Squid-2 for an interested party. Hopefully they'll fund both the Squid-side (which will go back into Squid-2 and maybe Squid-3 in the future) and the proprietary code. This way they get their shiny features, the Squid community gets their shiny Quota support, and the company doesn't have to pay for ongoing maintenance of the Quota framework! Winners all around!
I'm doing some installs of Squid for a company supplying internet access via Satellite. This means I'll have to tune Squid to cache a whole lot of popular stuff -and- do some tuning to optimise TCP/IP over satellite. I'll wrap up everything from this project and put it online.
I've got an upcoming project to setup a small Squid cluster for a company providing services via a MediaWiki install. I'm going to leverage the excellent work done by the Wikimedia group and hopefully put all of that up online for others to leverage.

I've also been working on an NNRP proxy for a commercial USENET provider. Yes, people still pay for usenet services. Its quite scary actually! The first phase of that is near completion and I'll hopefully be able to leverage some of that in some open source projects. The NNRP proxy is currently handling almost a gigabit of reader traffic without any particularly scary issues; I'm quite impressed. Hopefully we can scale this software up to proxy around 5 to 10gbit of NNRP traffic and about 5000-15000 connected clients per server. This scales much, much better than the current software of choice in the NNRP world - The diablo reader stuff, or "dreaderd".

I've been working on a few things on the hosting side - mostly relating to some content delivery ideas of mine. Nothing fantastically different there; I'll post some more details when there's something concrete and deliverable.

New Pricing Online!

2008-09-08T22:15:00.001-07:00

I've just put the new pricing online - http://www.xenion.com.au/products/pricing.html .
This includes basic webhosting, domain services, traffic, colocation and virtual private servers.

New Customer: European-based media company CDN

2008-08-23T19:01:00.000-07:00

I've recently acquired a small support contract to help build and troubleshoot issues for a European-based media company. They've built their own simple CDN with servers around Europe to deliver content. I've built and deployed two new Squid-2.7 servers for them which handle their current summer holiday load (20-30mbit at peak) with absolutely no trouble whatsoever.

Based on their existing servers which I'm apparently helping replace, they predict the servers will each peak above 150mbit/sec once the holiday period is over and everyone is back at work. I'm looking forward to it. I've got a few configuration tricks up my sleeve to optimise their setup to dramatically reduce the CPU consumed without any code or system changes. Stay tuned..

Pricing Updates!

2008-07-31T23:15:00.001-07:00

I'm about to publish some updated bulk pricing rates for existing and new clients. I'll also take the oppertunity to tidy up the existing pricing pages so they make more sense.

Stay tuned!

Commercial Squid services - taking off!

2008-07-26T00:37:00.001-07:00

I'm just signing up my newest support client - in Greece, no less. I've got clients in a variety of countries now - Spain, Brazil, Greece, UK, Australia, Switzerland. Shiny!

The client-side delay pools work and the per-connection shaping should shortly appear in a Squid-2.HEAD tree near you.

My work queue looks something like this:

For commercial support clients: update Youtube and Microsoft Updates rules; grr to Youtube for changing their stuff again!

Commercial contract: delaying request forwarding until a certain amount of the request body has been read in. Slightly overdue thanks to college commitments but it'll be done very shortly.

Commercial contract: Sorting out some Squid related sub-contracted work for Yahoo!

Commercial contract: Solaris + Samba + AD + Squid - why the hell is it so painful!

Final testing of my Squid development/testing cluster hardware before I go and install it in the colocation rack. I can hear the Dell and Sun servers from the other side of the house; I think its time they moved into professional hosting space!

More stuff which I can't yet talk about!

Starting August 1, I'm going to be doing a big push for Squid support and hosting services. I now want to grow this into something much, much bigger than just me and I believe I'm now in the position to do it.

Also, I hope to organise some meetings with vendors when I'm in the US in October. I -hope- this includes Sun and Google - specifically for hardware/performance work and video/youtube caching respectively. I'll post more when I have firmer plans.

Hosting environment updates

2008-07-26T00:35:00.000-07:00

G'day,

I'm just finishing the final rollout of the DNS updates on the older servers. Everything checks out fine.

Hosting-4 is handling new clients well. I'll have users migrated off the initial hosting servers and retire the old hardware back into the testing lab by next week.

I'm about to post some updated VM and bandwidth pricing; stay tuned!

Updates - 20 June 2008

2008-06-20T06:32:00.000-07:00

A few updates!

Hosting-4 is online! I've migrated myself and one client over. Everything seems to be functioning normally. I'll be contacting clients over the next few days to migrate them off Hosting-1 in preparation to remove Hosting-1 (and replace it with more up to date hardware.)
I've completed two pending Squid contracts and I'll be shortly beginning a third. The work is being folded into Squid-2.HEAD as we speak. Thanks to everyone who has signed up for support and development work, your business is helping strengthen the Squid open source project!
Adrian needs sleep. Who can recommend some music?

Welcome!

2008-06-20T06:31:00.000-07:00

Welcome to Xenion Live Updates! :)

I decided to create a blog here rather than run and maintain my own blogging software.