Thimbles & Care

Although webAgent was supposed to be a temporary solution, after a couple of years we still hadn’t found what we thought was an appropriate replacement, and some deficiencies in webAgent had become apparent, so we decided to do a second version. Some of the problems, real or perceived, that we saw in webAgent 1 were:

• Form variable handling. In webAgent 1, every form variable was used to create a local variable within the script, whether the script author had intended it or not. Adam Connor believed that a savvy attacker could leverage this to make a script act in an unexpected way.
• Only a single Broker service could be called in a given script.
• There were some bugs in webAgent 1 that would have required significant effort to fix. My favorite was string comparison of array occurrences. To make it easier to compare strings of different length, string contents were copied to a maximum-length buffer and padded before the comparison. But there was a comparison buffer for each variable name, so if you compared two occurrences of the same array, they always compared as equal: the first occurrence was copied to the buffer, and then the second occurrence of the array was copied to the same buffer, which was then compared to itself.
• By this time some commercial WYSIWYG HTML editors were available, but webAgent 1 didn’t play well with them. Many developers (John Wheat was a leader in this) wanted to be able to use Macromedia’s Dreamweaver to develop their scripts.
• The design of webAgent 1 didn’t really make adding new features easy.

To try to meet these needs, webAgent 2 was a completely new language, built from scratch. It was written in C++ rather than C, and using Terence Parr’s PCCTS compiler toolset instead of flex and bison. It supported multiple types of source files, in particular, a “.wbs” file type that was pure webAgent 2 code, and a “.wbt” file type (the ‘t’ was for “template”) that would be mostly HTML and, we hoped, could be maintained in Dreamweaver. (That didn’t really work, as things turned out.) Form variables were placed in a separate structure, *FORM, so they couldn’t accidentally override local variables. (The structure of “star-variables” like *FORM and *HEADERS proved to be a fruitful feature for enhancements to webAgent 2.) There was no limit on the number of Broker services that could be called within a single script. One webAgent 2 script could “invoke” another. Instead of being fully interpreted at run time, webAgent 2 scripts were compiled to an intermediate form (a “.WBX” file type) before being interpreted at runtime.

Unlike webAgent 1, which I wrote all by myself, webAgent 2 was more of a team effort, with Adam Connor contributing code for numeric processing and some utility routines. Despite this, while webAgent 1 was usable about three months after I started working on it, it took closer to 18 months before webAgent 2 reached a comparable stage. (It’s not an entirely fair comparison, because during webAgent 2 development I took off a couple of weeks when my son David was born, and then a couple of months later I had appendicitis which knocked me out for a month or so.)

Another aspect where webAgent 2 different from webAgent 1 was its cross-platform nature. When I wrote webAgent 1 I didn’t put any effort into making it run on any platform other than the HP-UX servers we had at the time. Since webAgent 2 would be partially compiled, we wanted to have the compiler run on the developer’s workstation. Later we decided that letting developers compile on their workstations and only upload the .WBX files could easily lead to lost source for critical applications, so the workstation versions we distributed only did syntax checking and couldn’t actually create a .WBX file; we only ran the full compiler on the servers. I did my primary development using CodeWarrior on my Mac (Classic—this was well before Mac OS X) and then had a Windows machine with Visual Studio to build and test the Windows version, and then I’d move the code to an HP-UX server and compile and test there. (After webAgent 2 was complete I converted the Windows machine to Linux and used it to develop the Green output service.) For Mac users I also wrote a BBEdit plugin so that you could perform a syntax check without leaving the editor.

After a decision was made to encourage the use of Java for web development, Adam Connor and Mike Godfrey rewrote the webAgent 2 runtime (the part that interprets the .WBX file during the actual web request) in Java. The compiler is still the original C++ code. Changes were made to webAgent to allow a webAgent 2 script to call a Java method, but we were never able to solve some of the security issues this raised, so webAgent 2 as a pathway to Java didn’t work out.