Interesting projects

The following are short descriptions of interesting projects that I've worked on for various customers.

High volume email

At Alpha Omega Computer Systems, I did programming as well as system administration for our collection of Linux server machines. The domain name of Alpha Omega is ao.com. We obtained the name before America OnLine thought the Internet would catch on.

Unfortunately, the spammers don't always spell correctly. Actually, many other legitimate users don't always spell correctly either. As a consequence, Alpha Omega gets flooded with large quantities of email for which there is no such user. Any single such request doesn't take that much out of the server. However, hundreds of thousand of the requests per day hurts.

So one of my tasks was setting up a back end email server to handle the traffic for the users so they can send and check mail and have consistent response, and a separate front end mail system that faces the Internet and takes the random floods of garbage and filters out the email that is actually of interest to the real users.

Plywood Lathe

The manufacture of plywood starts with peeling logs into thin sheets of veneer. It turns out that there is a surprising amount of equipment needed to pick up logs, examine where the best place to put the log in the lathe, and then peel the log down to a core that may only be two or three inches in diameter.

You can see some startup videos of a lathe in action at: http://www.youtube.com/watch?v=aKRAyoNrFl8 and http://www.youtube.com/watch?v=TX-novpYpCw.

There are about a dozen high performance hydraulic cylinders and motors necessary to control the various pieces of equipment. We have feedback sensors that are accurate to about 1/1000th of an inch. Depending upon the size of the logs going through the lathe, we peel between 5 and 20 logs per minutes.

This control system for this lathe has changed over time. Early versions of the software ran on Z80 microprocessors and PDP-11 minicomputers. Those systems migrated to 68040 processors, and have since migrated to Power-PC processors in a VME rack.

The current software builds a 3D model of the log based upon sophisticated laser cameras that returns hundreds of thousands of distance measurements per log. Our software looks for the largest cylinder of wood that is hiding in this model, and then causes the equipment to place the center of this cylinder at the center of the lathe.

Because the knife is as long as the log is, and is applying considerable pressure, there are rolls that come down on the opposite side of the knife to counterbalance the force of the knife against the log.

As we peel the log, we must ensure that we match the speed that veneer is produced to the downstream production capacity at that moment.

Besides all of the control, we have a web server that generates various reports, diagrams of logs, and other supporting information.

This system is implemented on a pair of Power-PC VME cards running Linux It uses Gigabit Ethernet to capture camera data and to communicate with the rest of the equipment within the mill.

Titanium refining

I helped implement a control system for refining titanium using electron beams (totalling about 250KW) to supply energy. Hot titanium is very reactive, and electron beams need a vacuum to operate, so the actual refining chamber was inaccessible during operation.

My responsibilities on this project involved the user interface necessary for specifying where energy needed to be deposited at what times to accomplish the refining process they were trying to do for a given run.

Ski ticketing system

I helped implement and currently maintain a ski ticketing system used for one of the local ski resorts. This system handles everything from the point of sale terminals, through the automated turnstiles at the base of the ski lifts.

As a portion of the sales process, we interact with the Internet in order to process credit cards. The main interest in this is that the ski resort is not at all close to modern telecommunications, and we have migrated between a variety of connection technologies in order to get reliable Internet to the mountain. A portion of my work has been to ensure that the communication technology used in any given year can interact with the rest of the system.

Email transition

One of the sites that I do consulting work for maintains email for about 1000 accounts. Over the years, I have migrated the email system to a couple of hosting providers, and finally to an in-house computer system. Each of these transitions was planned out to ensure that email was not lost, and that there was a minimum of downtime.

Web based order entry, and production tracking system

I was on a team who build a set of Drupal modules to implement an order entry system for a local company. Besides just accepting orders, the system also needed to track the steps of production so that everyone could see the current status.

Disk drive assembly line (1988)

I was on a team of programmers that implemented a production line for assembling disk drives. This assembly line had about a dozen workstations where robots or screwdrivers on X/Y positioners were used to build, and disassemble the drives.

The manufacturer was building drives by hand, and the rework rate was very severe, and so wanted the automated assembly line to be able to handle disassembling the drives using the same equipment that put the drives together.

The programs that I wrote dealt with the scheduling the work to be done on each work pallet as it wound between the workstations, and making sure that the appropriate step (either assembly or disassembly) was done as needed. The scripting at each step actually consisted of a number of basic operations of that work cell.

In the end, we had a very flexible system that could schedule arbitrary robot actions as appropriate for the capabilities of the robot. Unfortunately, once people were moved away from the work areas, the rework rate went down to almost nothing, so the sophistication of the scheduling wasn't really as important as was first thought.

This system was controlled by a network of computers running QNX. QNX is an operating system that was based on some of the ideas from Unix, folded over with an excellent implementation of a micro-kernel, and optimized for the computers that were available at the time.

Backups gone wrong (198?)

This is a story of how not to do things.

My wife was working for a small company at the time of this event. They had a small Xenix based computer that maintained a database of names and addresses that the buisness used to maintain contact with a large set of clients.

I received a phone call saying that there had been a power failure, and the system hadn't come up. Their normal computer person wasn't available, so my wife asked me to help.

This was a 286 version of Xenix, and the address space didn't allow enough RAM to conduct an fsck run from RAM, and so part of the fsck process was to request the name of a file to use as a scratch file. One of the other parameters was the name of the disk partition to do the fsck on. Because of a misunderstanding of the handwritten notebook of what to do when problems occur, the person who actually performed the fsck ended up using the partition name of the data disk as the scratch file name. This was an unfortunate mistake.

So, I was able to see the above mistake on the screen. An unfortunate mistake, but they had been making backups on a cartridge tape unit once a week. So someone had been thinking ahead. It turns out that one cartidge wasn't enough space, so when backups were done, someone had to sit there and wait for the first cartridge to fill, change to the second cartridge, and then they could go home as it wrote the last little bit out.

I find the last back tape, and try to read it. Failure. As I look around on the file system I note that there is no special file for the cartride tape unit. Instead I find a normal file in "/dev/tape". And it had the contents that should have been on the second cartridge. So the backup had been saving onto a normal file rather than a cartridge for an unknown amount of time. The file that was left was small, and basically useless, and I had no idea how long they had been going through the motions of making backups, but weren't doing anything useful at all.

Once I figured out what the major/minor numbers should have been for the special file to represent the tape drive, I found that the tape unit no longer physically functioned. I took the collection of tapes to another site and read the tapes. I found that the most recent tape was about two years old. This was very disappointing. However, I used dd to cut up the tape file into floppy sized images, and wrote the tape to a stack of floppies.

The original failure destroyed the inodes, but had left the data intact. This was an old V7 style filesystem with direct blocks, indirect blocks, and double indirect blocks. But without the inodes the data was not accessible.

Once I had copied the floppies onto computer on the one good partition, I was able to do a restore of the ancient data. This gave me an idea of what the disk structure was supposed to look like. Next, I wrote a suite of programs to catalog all of the blocks of the damaged disk. This identified blocks that looked like parts of a database file, blocks that looked like they were part of directories, blocks that looked like indirect blocks, etc. The larger files on the system were the database files which tended to only get appended to. By looking for matches between the old filesystem and the current data blocks, I was able to piece the files back together. The smaller files, because they were all contained in direct blocks, had less redundancy and were harder to piece back together.

In the end, I was able to piece the filesystem back together and was able to get the business moving forward again. I then installed a new backup device and instructed the people on how to actually check the backup in a routine fashion. Finally, I transitioned the entire set of programs over to a Linux system with modern tools to help move things forward. Finally, I used rsync to backup the computer to an offsite backup so if their routine computer staff ever lost the computer again I'd have a place to start for recovery.

Backups have to be checked periodically. Ideally, they should be restored on a separate machine to see that the entire procedure works as documented.