For a few months I had no time to invest in puppet and I ended migrating three minor versions of puppet. Namely from 0.24.4 to 0.24.7; three minor versions that moved puppet forward and caused some havoc on my systems.

With the new version Webrick’s performance was even worse than before. Plugins took forever to sync and most client just gave up with a timeout while retrieving plugins.

So the update was also motivation enough to switch to mongrel plus nginx. The instructions at puppet’s wiki are good enough. Once every thing was working properly the performanced skyrocketed, with mongrel my clients fetch and run its catalogs in a fraction of the time required with webrick; Hooray!

Only a issue with my prior client access control, had to disable any IP restrictions on fileserver.conf using Allow *. The issue has to do with nginx and mongrel unable to properly communicate the Ip of the client and seems fixed since a month ago.

The only pitfall on the installation was grub mishandling the Sas6 arrays. Two Raid-1 arrays were configured on the internal SAS6 controller. The two arrays are seen as virtual disk on the Linux side, but their names/order are reversed:

/dev/sda => contains the physical disks 2-3 (second array)

/dev/sdb => contains the physical disks 0-1 (first array)

Both me and Debian Installer got confused with the disk distribution. Grub insisted on being installed on /dev/sda and that device won’t boot the machine. After a few tries (and hours) I got grub on the boot array (the first one); the install was completed by entering the target chroot and manually issuing:

grub-install –no-floppy “(hd1)” 

Lesson learned, your Sas6 won’t look for a boot sector on the second array.

Now, with the machine provisioned and running, I’m itching with curiosity about the lcd display and the ipmi implementation on the machine. Seems that you can output to the lcd from linux in a primitive and crude way.

Working with many development and production some people forgets to check their default charsets; in the end encoding mismatchs crawl. I’ve seen a few programmers falling in this trap.

Last time our php was Latin-1 encoded and the production Apache server used Utf-8 by default. In a shared hosting environments the default encoding can be set from the .htaccess file:

AddCharset ISO-8859-1 .php
AddDefaultCharset ISO-8859-1

Same thing from php:

PHP. Use the header() function before generating any content, e.g.:
header(‘Content-type: text/html; charset=utf-8′);

Links and discussion:

• Basics on http charset parameter at w3c.
• Apache .htacces usage.
• Handy list of IANA accepted encoding names recognized by apache.

As Wouter, I find the biggest drawback to multi-filesystem layouts is lack of flexibility when resources are scarce; scarcity means laptops in the Wouter’s post. I’m thinking of long-lived servers in need of upgrades, in particular those with a life of unplanned and ad-hoc growth.

When scarcity strikes: drama.  The cleanup dance isn’t a solution, everything clogging /var and other undersized partitions is of value; forget about deleting it. On the first storage famines I go resizing the affected filesystems with unused/un-partioned space(yay, I planned for un-planned growth). After the resizing is done you realize that some other partition also need extra storage.

Here comes the cleanup-moving dance. First put data out of /var and system partitions, then update configuration for the affected services or start symlinking to the new directories.

My two worst cases were bacula related. In both bacula’s database needed more space than its current partition could afford:

• Shortly after updating retain period for data backups to two years (ISO requirement). Database moved to a new server where …
• When we moved from cvs to git; again bacula’ db jumped in size. It had to be moved and symlinked to /var.

The web page for Hotel Mayari is simple, the content almost static. Most of the job was mediation, bypassing the limitations frequently found in spanish hosting companies, cooperation, and coercion of past service providers. Some of the troubleshooting included:

• Expiring domain names.
• Unknown hosting capabilities
• Missing database access.

Client had ideas on the looks of the final page, but for every intermediate step they needed constant reassurement. The facilitation done by Visualtis was 10% of technical nature, 30% of hand-holding and coaching, 30% of client and providers negotiation, and 30% of education.

These are some of the things I learned in the first years:

• Give testing priority.It’s hard, and it takes resources but if you don’t commit to proper  testing from the beginning, you never will. The later on you make your commitment the harder it becomes, until it is one of your eternal TODO items.
• Care for your internal communication. Provide spaces and tools for conversation, both real and electronic. People should feel comfortable and the information must be readily accessible and search enabled.
• If you can’t trust your employees, distrust them (deeply).
• Marketing and sales are a priority, you are supposed to run a business for money. The perspective of a sales person helps when searching new venues and income sources.

Last: try hard to make fun. I’m sure our last project benefits from the experience gathered; but the energy people are putting into it and the fun we are getting is something we needed in our past venues. I feel that makes a lot of difference, we really like our Porra!

Some one else sharing experiences at a startup-wannabe?