The -5 board is quite a bit more complicated than earlier models due to the increased use of PLD components. Very often the fuse bits are blown on PLDs and that makes them very difficult to copy or reverse engineer. I reverse engineer for educational purposes, not to save money. I look at it as paying myself for an education, if you have real work to do and not the time, it’s better to spend the money. The Fadal aftermarket has contracted quite a bit in recent years, but it still seems healthy enough to support the existing customer base.

No, the amplifiers are a 3rd party component. I’m not particularly interested in reverse engineering those.

I haven’t reverse engineered the memory board yet. AFAICT, there are jumpers/switches on the board for mapping each memory board into different memory segments. The memory boards hold 128KB of RAM, so that’s 2 segments. I think a max of 3 memory boards was supported, but it’s been a while since I looked at it. One of the challenges is that because the Fadal is S100 it uses special bus driver chips that are harder to find. It might make more sense to make a daughter board that goes under the CPU, like what memory add-ons for the PCjr did. You could get a single 512K SRAM chip and put it on an interposer that maps the maximum contiguous block of system memory to the CPU. Intel chips are designed with the reset vector at Fxxx so 0000 is where program memory/interrupt vectors start. I seem to recall the max for older Fadal controls was 422KB, but they count the video memory and memory on the axis controllers as part of the memory size (artificially inflated for marketing I’m guessing). The CPU has 32KB on board, plus 384KB in expansion, that’s 416KB, plus the 5 axis controllers, plus 1KB video memory makes 422KB. The video memory is mapped at 0xFBC00. The only other thing you’d need is a battery backup for the SRAM. Fadal used a simple transistor circuit so when the machine is powered on, RAM is powered by the mains and when turned off the RAM is powered by the lithium battery. I haven’t reverse engineered their power switching circuit, but you could probably find some more modern battery backup designs. A super capacitor would be worthwhile, but you have to isolate the RAM/capacitor from the main power supply when mains disappears. A super-cap combined with a Lithium cell would be belt and suspenders.

-Jordan

We had 1 single clustered table that contained all log records, when a record was processed it would update the table to indicate it was deleteable. We would try and perform batch deletes, but this caused us no end of locking havoc.

I think the ultimate solution ended up being to delete records in very short intervals. As a MySQL Support Engineer I recognize these problems today and there is a lot of best practices we didn’t know about. Today you could just delete the record immediately or do deletes in batches of 10,000 or less.

Ironically, you could sidestep these short lived types of tables and rename or use partitions, then drop partitions.

One of the interesting perspectives is that I work for Oracle, who bought Sun, who bought MySQL AB. When it comes to known bugs, if there is a patch, getting a hotfix is typically a fairly short turnaround. Our problems with Solaris didn’t end there, they culminated with a major faux pas when trying to deploy Netapp F740’s with Solaris 2.5.1 — you can’t.

We tried to get Solaris 2.5.1 working, but it was just buggy and Sun kept telling us to upgrade to 2.6, so we planned for this. What we didn’t account for is that the Veritas FS (vxfs) was a kernel module from a 3rd party, and we didn’t have that module for Solaris 2.6. We performed the upgrade and discovered we couldn’t access anything but the root filesystem AND the Netapps! We mirrored our root volume to a second disk for redundancy purposes, but by the time we decided we were just going to roll back, the cron job that mirrored the root volume had already run!

Faced with no turning back, we screamed at Sun until someone emailed us the .so file we needed to get our systems working, installed it, rebooted, and the system was back up. We were down for about 12 hours that night and that was also my first all-nighter. The biggest problem with doing these types of maintenance tasks after hours is that people lose their cognitive edge as the day wears on and mistakes are made at critical junctures.

It all seems trivial right now, with more industry experience under my belt than my age back then!

Chris wrote the initial draft of the schema and stored procedures for WebCom SMTP, enough to get the listener component working, I wrote all of the C code and revised a good portion of the schema and stored procedures to actually make them work as we neared production. We had quite a few hours of strategy sessions discussing the best way to store email messages inside the database server. Development of WSMTP was completed over several years in 3 major phases. Rightly or wrongly, the plan and choices worked well enough to outpace the message handling capacity of any competing solution available.

The main reason we could handle more message volume is that the discrete steps of processing messages were broken up into parallelizable processes, we effectively pipelined email processing. The part that came first was the WSMTP listener, because NMS was falling on its face and we needed a way to intercept and proxy messages around NMS so it wouldn’t handle outgoing customer email. In those days we put WSMTP on the e4000, ran NMS on the Sun Ultra1, and routed outgoing mail to the Solaris x86 box.

The Solaris x86 box became an impediment so we replaced it with Linux, which more than doubled its speed with no other changes. That was the first production Linux server we had. When we went to production with WSMTP we had a bunch of Windows NT 4 boxes running Sybase, they weren’t particularly robust. I happened to go to the Linuxworld Expo not long after that and Sybase was giving out free copies of Sybase 11.5 for Linux. There was no license agreement we had to sign, it was just a set of packages you could install. I think this was intended for evaluation purposes, but in their haste they forgot to define limitations. I built up another DB server and named it “Redeemer”, after the weapon in Unreal Tournament, but most people thought it had to do with Linux redeeming us 😀 I installed Sybase 11.5 on that box and figured out that the simultaneous connection limit was 412 connections, so we set it to 412. Somewhere around that time I also installed Sybase 11.5 on one of the Intel Apollo boxes and ported the WebCom Counter schema to that, so counter.webcom.com, aka abacus, was a standalone Linux box running Apache and Sybase.

Soon after Sybase 11.5 made its debut in WSMTP, all of the NT boxes were quickly transitioned over, this lessened the administrative overhead of those DB servers and picked up some speed. When Verio took over, Chris Merz got absorbed by Verio after the first RIF and went into their DBA group. He was responsible for bringing up Sybase 11.9.2 on Intel hardware for them, after which we were treated to some spanking fast Intel P3 XEON boxes. WebCom ended with 1 of those boxes taking most of the DB load (we could load balance connections at the client level) for WSMTP.

Bluntly, WebCom’s single host architecture was a product of the fact that I was a very raw and green sysadmin when we started, and it was easier for me to conceptualize and manage a single system, rather than dealing with distributed systems (the technologies for which were very primitive to non-existent in any form we could afford in early 1994). Another reason for the “big iron” approach was support… we could pay Sun Microsystems $20-30k a year for “Gold” level support, which I leaned on heavily. When something broke at an OS level, I had someone to scream at (and I did).

Two stories here:

a) at one point, the web server was locking up due to the filesystem not accepting writes; fixing it required rebooting the server, which in turn resulted in a multi-hour fsck of the non-journaled filesystem; it turns out that there was a bug in the filesystem (I recall something about an NMS “hole” or some such) and my screams got their third level support to issue us a pre-release patch

b) the Sybase database server was originally tied directly to the web server, every web page hit was logged directly to the database, and then we ran reports and resource usage billing against it; when we ran Sybase 9, this wasn’t a problem… but Sybase 10 was a disaster, and regularly hung under load (as I recall, that release ultimately kind of killed their company) and refused to accept further connections, which in turn hung the web server; after this happened a few times, Chris and I decided to disconnect the logging function from the database and have the web server write to a log processing tool that would handle that the task of uploading hit records to the database. Which solved that problem, but created another problem in that the database server (that Sparc 1000e) was running at 95% of capacity, and any time we wound up accumulating a backlog of pending transactions (such as Sybase crashing), the system simply could never catch up.