Rapid Prototyping with Tcl/Tk
Creating software is a complex process that embeds the programmer in rules and constraints. Customer needs fight against bugs in the program, and usability fights against production costs.
The current procedure to solve all these problems is object-oriented design and analysis, accomplished by methodically trying to split a problem into suitably small subdomains and providing a predefined path from analysis through design to implementation and testing.
Such a procedure may help for large programs and may be necessary if many programmers are involved. Most of the really good programs (such as those available for Linux, and even Linux itself) are written by only a few people, and more importantly, most of them don't start out knowing in which direction their software will evolve. Therefore, another paradigm is needed—one that is better suited to the needs of humans.
Here is a statement about rapid prototyping (RP) I found on the home page of Cycad Corporation at http://www.cycad.com/:
Rapid prototyping is acquainted with keywords like “Get to market faster, Beat the competition, Maximize profitability, Increase quality...” Not only does rapid prototyping provide all the benefits of getting to market faster, but for the first time it is possible to create Proof-of-Concepts to show customers before implementation and create proto-production units before committing to manufacturing. Both of these validation steps can save extensive amounts of resources and time.
While all of the above is certainly true, RP is particularly suited to the creation of software tailored to the customer's needs. One of the most important aspects in determining these needs is to find out what the customer really wants. Because the program takes its final shape while it is under development, not in advance as would happen with standard structured design, the programmer can respond to new information about customer needs. With RP this process is straightforward and allowed, while normally it indicates an analysis or design error in other methodologies.
Equally important is an easy way to test new ideas. The final product will be better if you have some sort of playground to find out before implementation whether or not your bright new concept is really as bright as you think. This is even more important in our fast changing world where we have to react quickly to new or changing demands.
Using rapid prototyping for software development does not necessarily depend on a language specifically designed for it. It is possible to use compiled languages like C/C++, but best results can be obtained with an easy to learn/easy to use interpreted language. To get the most benefit, turnaround times have to be short and modifications to existing code should be possible without much effort.
One area where rapid prototyping really shines is user-interface design (provided you have the right tools). You can create a dynamic visual model with minimal effort and provide users with a physical representation of key parts before system implementation. You can accommodate new or unexpected user requirements earlier, and modifications are much easier.
Another aspect is quality assurance and control. The same tools used to build the prototype can be adapted to build up a test harness that is useful up to the final implementation of the system. This results in fewer changes after delivery and therefore productivity improves.
One of the current trends in computer languages is the use of portable languages like Java, Perl or Tcl. While these are generally slower than a compiled counterpart they offer a lot of advantages for the developer (portability, simplicity, short turnaround times, etc.). If such a language can integrate compiled code (e.g., by using shared libraries), a developer can first concentrate on creating a working solution and later optimize the code at the critical spots.
Tcl/Tk is especially well-suited to rapid prototyping, as it has solutions to all of the requirements discussed above:
It's simple to use and to learn.
No compilation or development overhead is required—just write and run your code.
Graphical user interfaces can be built with very little effort.
It can be used for testing (through the use of its introspection facilities).
I do not wish to begin a language war here—the above statements are my own opinions. To make the best use of any language you need to know the language very well (Tcl is no exception) and you need the corresponding tools (syntax sensitive editor, debugger, performance analyzer, testing and documentation tools). Finally, you need a lot of good libraries and extensions which give you the power to create big applications with only a few lines of code.
The following example is meant to give you a feeling for what rapid prototyping is like. To keep it short and understandable it does not use any of the features of Tk at all. Although a graphical example would certainly be better suited to show the advantages of RP, the concepts presented can easily be extended to make use of Tk's extra features. It is one of my actual projects, and I will show it to you in the same way I developed it. As you may guess the customer and the programmer are in fact just one person—me.
Customer: “I need a nifty little library tool to work with values that are configurable by the end user. The values should come from a simple text file, and that's all there is to it.”
Programmer: “Okay. This one is really easy.”
I sit down and type some lines of Tcl code that give me the following interface:
proc Cfg <
This way I can use the configuration interface for a (hypothetical) text editor in this manner:
set Lines [Cfg "NumLines"]The format of the file should be kept simple, so I try the following to read values from it:
NumLines 20 WordWrap 1Reading the file into memory would need another function. As I don't want to pollute name spaces (true for both Tcl and C), I modify the signature of my procedure to be the following:
proc Cfg <which gives me the following:
Cfg read "myconfig.cfg" Cfg get NumLinesCode to implement all of this is written in 15 minutes. I add another 15 minutes to write some test cases so I can always verify the correctness of the whole implementation. Here is one test:
Test c1 "get simple value" {
Cfg read "test.cfg"
Cfg get NumLines
} 20
It reads my simple test configuration file and checks if the value
for NumLines is really the same as set in the configuration file.
It is all finished in 30 minutes. Sure, I have to rewrite it in C,
but first I'll cross check with my customer.
Customer: “Nice work, but you see, if an application gets bigger, we may have name clashes—consider two modules that use a configuration value named Width. One could be the width of a window and the other the width of the text in the editor. What we really need is a module dependent solution.”
Programmer: “Grrr. I should have known—simple things tend to get complicated with time.”
Modifying the interface is straightforward, but what about the file format? Far back in my mind is a little hint named win.ini. I cross check with a real windows installation on my DOS partition—yes, the format would be just right for my problem, and it will already be familiar to a lot of users.
So, I modify my configuration file to look like this:
; my config test file: 24/02/1997 [Editor] WordWrap=1 NumLines=20 Width=80 [Window] Width=320 Height=400 ; EOF
Reading such a file is far more complicated than the simple approach taken earlier. I pat myself on the shoulder that I have not yet written any C code.
Using some of the extra features of Tclx, an extension to Tcl, the file reader is ready and working in half an hour.
Now, I modify the interface as follows:
Cfg read "myconfig.cfg" Cfg get Editor NumLines Cfg get Editor Width Cfg get Window Width
and I modify all the test cases. Again, I present the whole thing to my customer.
Customer: “Yeah, that's just what I wanted. I have some real world examples, could you verify that they work correctly?”
Programmer: The customer gives me some real X Window System configuration files—did you think the above were real world examples? Anyway, I run some tests on them and poof, all my nice looking code breaks. A quick look at the files reveals the problems—backslashes, brackets, blanks and entries that span multiple lines. Parsing an X configuration file is apparently more complicated than I thought at first.
I add a lot of new test cases that show all the above failures. They help me to make sure that I have a working piece of code at last, without one change breaking something that worked previously.
Adapting the Tcl code is comparatively easy. It takes some time though, as I have to fiddle around a bit with regular expressions for the parser. As soon as all the tests complete without error, I show my work to the customer.
Customer: “While waiting, I have used your code in a real world program, and it has one big drawback. Consider the following example configuration:
lpr -#1 -Plj5 myfile.txt
This is a print command which should be executed with exec. Some parts are static but depend on the operating system (e.g., lpr versus lp), others (such as the selection of the printer) may be done by the user and still others (such as the file name) come directly from the program. What I need is a flexible substitution scheme for configuration values.”
Programmer: “Ha. Now that's a really big request. How do I create such a flexible substitution scheme?”
It takes me a day of thinking (actually this was handled by a background process in my brain) to find the solution—Tk uses substitution for its key binding mechanism, so why not use the same ideas here?
Again, I modify the interface, so it can process code such as the following:
Cfg get Printer Command\ "%c=1" "%f=myfile.txt"
and I add some lines to my configuration file:
[Printer] Command= lpr "#%c "Plj5 %fBefore returning the final value to the caller my routine substitutes all “%?” sequences with the corresponding values from the argument list. Using Tcl's regular expressions makes this quite simple, and the only thing that takes some time is writing the tests to check for all these crazy conditions like “%%” and “\%”.
The above example will return
lpr -#1 -Plj5 myfile.txt
just as expected by the customer.
Customer: “Great. This is what I have been looking for for years. There's just one sub-optimal item left. If the user accidentally deletes his configuration file, the program will no longer work. Isn't it possible to keep some default values for this case?”
Programmer: “I have already thought of this situation. The code would be more readable if an actual value were present, and it could be tested more easily as I wouldn't have to write configuration files all the time.”
I modify the interface once more by adding a new subcommand:
Cfg def <module> <name> \ <default> ?<sub>? ..."
The general syntax of the get and def sub-commands are the same (and both return the same configuration value when called with the same substitution values), but def also sets a default value when none is given.
If a configuration file has been read by Cfg, the values in this file take precedence, otherwise the default is used.
Here is an example:
Cfg def Printer Command \ "lpr -#%c -Plj5 %f" \ "%c=1" \ "%f=myfile.txt"
This command returns:
lpr -#1 -Plj5 myfile.txtCustomer: “I use your code in all of my Tcl applications, and it works like a charm. The only thing I noticed is that the programs are really slow on startup. I suspect this is related to your configuration code. Couldn't you do anything about that?”
Programmer: “Aaargh.”
Back in the configuration business. First, I check some of the customers applications. No wonder he has slow startup—he uses configurations quite excessively, even for language internationalization. I instrument the applications for profiling and create some snapshots for analysis. The resulting diagrams (see Figure 1 and Figure 2) clearly state the two highest CPU-cycle-eaters: reading configuration files and substitution of configuration values.
Figure 1. Tcl Profiler Bar Chart
The interface is comparatively stable now (the customer has already used it in all his programs, he won't change it that easily). I start rewriting some of the code in C. As I know the exact bottlenecks from the analysis, the first two C routines replace the configuration reader and the substitution engine.
Now all of the previously written tests come in handy—I can check if the new code works correctly with nearly no effort. Both of the routines are quite a bit of work as they do some very tricky operations and pointers always point to somewhere unexpected. Sure it's nice to have a debugger for Tcl (see Figure 3), but without a C debugger, the life of a C programmer would be truly difficult.
The new code is worth the price. Speed can be 5 to 10 times that of the original code, and this can mean the difference between a one second startup and a ten second startup.
There is a good deal more needed to make this little example into a full-fledged library, but it should be enough to see the general concepts. Here are the most important points to learn about the advantages of RP:
Work can begin with incomplete specifications.
Different implementations can be created in a straightforward and easy way using Tcl, until the customer is satisfied.
Working examples can be presented long before the application is complete.
If speed is a problem, the relevant parts can be rewritten in C.
Quality increases with added test procedures.
I did a really big project that integrated CAD (computer aided drafting) and PPC (production planning and control) software, and I successfully used the ideas presented above. (See Figure 4 and Figure 5.) The project took more than a year, and the specification changed more than once. The final solution shed another light on the versatility of Tcl: the user is able to define his own components (window frames, automatic actuators, etc.) using Tcl and its object-oriented extensions. This feature was just an offspring of our rapid prototyping philosophy as we simply provided the customer with our own tools.
Figure 5. A Second PPC Software Screen
Using Tcl for software development is not the ultimate bells and whistles solution. There are drawbacks such as the lack of real data types (anything is a string), the inefficiency of interpreted code (although a compiler is now available), and the fact that eval stays a mystery even to Tcl fanatics. Tcl shines when your program is mainly centered on a graphical user interface. Other tasks may be done better by Perl, C or even FORTRAN. As dynamically loaded, shared libraries are now quite common on several platforms, we can split problems into chunks and solve these chunks with the language that is most appropriate for it. Tcl/Tk may be the glue for the graphical interface in this scenario.
