DOS dBase to dB2k Automatic Conversion
How to fit a square peg into a round hole
- This article reviews the development
of an dB2k procedure to automatically convert DOS dBase code
to run under dB2k whilst retaining the "look and feel"
of the DOS application.
The challenge
Thousands of DOS based dBase program files, possibly a million lines of code - to be automatically converted to run under dB2k in a Windows 2000 environment. The DOS code had been continuously developed over the last two decades for very specific financial applications and had to be converted with 100% accuracy of its logical operation. It was however totally impractical to specify the application, its logic flow or to test all combinations of its use.
To add interest to the challenge, the application included third party DOS executables for specialised data analytical tasks, custom mail merge reports and the extensive use of ndx indexed tables, some with several million records distributed over networks with somewhat confusing paths.
Just in case this was all too easy, the resultant solution had to allow both the old DOS and new dB2k application to continue to work with the original data tables. The new dB2k application was to retain the same look and feel of the DOS code - no mouse, no expandable windows! Also, the original DOS code was to be further developed or modified and such changes were to be automatically converted to the dB2k environment.
The required time scale was three months - although "yesterday would be better".
Why me?
The key problem
DOS processes sequentially, sensibly completing one instruction before it starts the next. Windows runs on an event basis, it will ask for a process to start and then move on to the next instruction - regardless whether the previous instruction has finished or even started. To make it even worse thanks to the wonders of Microsoft the order in which the tasks are run, and the time slicing allocated to each task, is uncertain and beyond our control. This problem is common to all Windows applications, not just dB2k. However, the dB2k language further suffers from the absence of a command to return control to the Windows stack to "encourage" other tasks to complete - such as the DO EVENTS function in Microsoft's VBA.
The result is that the if one runs the DOS code in dB2k one can easily get ahead of oneself and nothing logical appears to run.
dB2k unique talents
dB2k has three major advantages over any other development language I know. Firstly, there is an amazing amount of the old dBaseIII or IV code able to run under dB2k. Sure the user interface is completely different, the default index is now mdx (as it has been for a long time) and there are many new ways of doing things - but the vast majority of data manipulation and processing code will still run.
Secondly, dB2k, as with dBase, maintains all program files, including its forms, in plain text ASCII files. Unlike many other packages, these files are not bound within other binary containers such as a database container nor are they buried within a project container. We can thus very simply write new program files, or edit existing program files, directly under our own program control.
Finally, and key to our solution, is dB2k's ability to modify at run time the attributes of the visual components within an open or closed form. Also when the form is "closed" it is not removed from memory and thus the form may be re-opened with the attributes as set when it was closed.
Microsoft could learn a lesson here!
Consistent code
The solution
Our solution was to retain the DOS based code with minor modification to call user defined functions designed to support the differences required by dB2k - for example to handle form based displays and to run external events in a timely manner. A library of user defined functions was developed and a master form provided to provide the main screen user interface. This form contained a number of text, entryfield and shape objects whose parameters were changed at run time to represent the @say/@get functions. Separate forms were used to represent any DOS based windows and wait functions.
A translation program was developed which tested at each line in the original DOS based prg file for code which required to be changed - e.g. for "?", "@", "set device" etc. A second library of functions was developed to convert these segments of code. In most cases the conversion program needed only to consider a single line of code, but some functions, such as the support of DOS based windows and set printer on/off instructions required the status set earlier in the program also to be considered. Similarly, the use of ":" line continuation requires several lines to be considered together.
The modal key
The key to the solution was to recognise where the program flow had to be halted, and where it could continue in its own good time. As discussed above, we retained the DOS code but modified the user interface code to refer to objects on a master form. The modified @say and @get commands were allowed to run in their own time but the program flow was halted at every @read command. The program was allowed to flow again once the last @get object had received a satisfactory entry.
The program flow control was achieved by opening the master form as a modeless form, modifying the objects on the open form as appropriate to display the @say/@get text, in the appropriate colour, style, position, picture, etc., and finally closing and re-opening the master form as a modal form. The master form had an additional entry field whose OnGotFocus event was set to close the form and thus allow the program flow again to once this object was selected.
The z-order of the objects on the form followed the order in which the objects were defined by the code - i.e. the order in which the @say and @get commands appeared in the original DOS code. The OnOpen event was set to setfocus() for the first entryfield object - i.e. for the first @get command. Other entryfield objects used to represent other @get commands had their status set to true, all other entryfield objects not thus mapped had their enabled status set to false. The user is to enter each item of data as if it the application was run under DOS, the entry is verified as before and the focus automatically moved to the next entry object. Eventually when all @get entry objects have been satisfied, the additional entry object gains focus, the form is closed and program flow returned to the prg code.
The master form
Control of objects on the master form
Unused objects are simply positioned off screen - as this seems to be the quickest way to control them. As a new object is required, the attribute of each object on the form of the required type (text, entryfield or shape) is tested and the first object not in use (i.e. positioned off screen) is selected, positioned, formatted and its datalink or text attribute set to represent the @get or @say function.
A screen clear or a part screen cleared is similarly achieved by simply moving off screen (and setting enabled to false for entryfields) those objects with a position within the area to be cleared.
Other forms
The dBase window form
The run event
Running external executables - such as third party mathematical routines or other data analysis gives additional timing and flow control problems. Once again, the dB2k routine will issue an instruction to start the "run" but unlike DOS dBase will not wait for its completion before it moves to process subsequent lines of code. Our solution was to set up a semaphore that indicated if the external procedure is still running, and stop the dB2k code until the semaphore is cleared.
This was achieved by translating the DOS run command to call at runtime a function to write and then run a batch file and place dB2k in a "sleep" mode until the batch file had completed. The batch file then runs the original DOS based executable and once the executable is completed writes a new check file to disk. In the mean time dB2k tested for the presence of the check file and then placed itself to "sleep" for a 100 mSec before reiterating the test.
The translation program
Before the code is run in dB2k, its is first translated to produce an new set of prg files. The translation process primarily undertakes string search and string manipulation. For example a @say command is converted to call, at run time, a user defined function which sets up the text object on the master form. The converted line includes arguments to be passed to the run time function to define the text, style, position, picture, etc. Each dBase command that requires conversion will be treated in this manner. This includes @, ?, *, &&, note, read, wait, clear, release, set, use, run, !, define/activate/deactivate window.
Great care had to be taken when manipulating the dBase code to cater for the variety of string formats encountered. For example when searching for "get" one had to check that the string is not part of another script such as occurs in a comment of "this is our target data". Similarly, as both "get" and "say" instructions can appear in the same line, it was necessary to convert such hybrid lines to two dB2k code lines. Likewise, as the @ instruction can relate to drawing a box rather then displaying text, this needed to be catered for and mapped on the form to a shape object rather then a text or entryfield.
Similarly, care had to be taken when handling data passed by the arguments - for example the @get xxx or the use xxx arguments may be values, text or a references requiring a macro substitution.
Reserved words and lost functions
The dB2k language supports a vastly expanded instruction set and thus words previously able to be used for variable names in the DOS code may no longer be used in dB2k. One example is "new" which in our case we automatically changed to "newzz".
Similarly, a few DOS based functions are no longer available, for example "row()" and "col()". In this case we added functions to our runtime procedure file of "row" and "col" which returned the value of the current cursor position - in terms of the equivalent character position, so that they could continued to be used as positional arguments with the @say/@get converted commands.
Avoiding confusion with runtime variables
A potential problem was that any variable name chosen for an object or variables used by the run time code could conflict with names used by the public variables within the original DOS code - of which there were many. To avoid this the "_app." prefix was added to all names of forms and any other public variable required by the runtime routines - such that they became attributes of the runtime application and could not possibly be confused with any DOS originated names. Also, any variables used by the runtime functions were declared as "private" so they could not be confused with the original code.
Furthermore, to avoid the original code from closing our runtime function and master form, etc., when the code performed instructions such as "close all" or "set procedure" the "persistent" property of the runtime elements were set.
Getting the correct length
Odd characters and index problems
A major cause of conversion difficulty was the presence of non-printable characters in the DOS code and data tables. Some of the original code had been developed or edited through the use of a third party editor which added its own extended ASCII characters to the code to provide tab type functions. These occasionally confused the string handling functions within the dB2k based translation process. To overcome this, every line of code had any invalid leading characters stripped off before any conversion was undertaken.
A similar problem existed with a few of the large tables where data was not able to be read for a few records or where the ndx index failed with tables of more then a million or so records. To overcome this problem, the BDE was set to level 5 and dB2k was used to re-create the tables and imported the data from the original tables to the new tables. The original DOS dBase and the new dB2k application could then use, re-index and maintain these tables.
The next move
Currently, my client is evaluating the conversion. My hope is that they will use this conversion facility to allow them to introduce development of new code using dB2k - and enhance the user interface and facilities through the use of dB2k.
As with all organisations, the solution should match their aspirations and we should not stereotype this. One great advantage of dBase, and even more so with dB2k, is that it provides the developer with great flexibility to achieve this.
Sample code - Windows Forms
The following presents extracts of code used for the translation and runtime functions. It is hoped that this demonstrates some of the techniques used.
The main and dBase windows forms are opened with specified handle names such they may be referred to directly without the need to find an instance of the form. This was achieved by added code before the header in the wfm file, for example in the form file "winplate.wfm" the following code was added:
parameter bModal,wname
winf = "_app."+wname
&winf = new winplateForm()
if (bModal)
&winf..mdi = false // ensure not MDI
&winf..readModal()
else
&winf..open()
endif
return
The form could then be directly defined, for example for dBase
window named "prompt", the run time function is:
*DOS dBase window support
function define_window(windowname,row1,col1,row2,col2)
private winf
do winplate.wfm with false,windowname
winf = "_app."+windowname
&winf..visible = false
&winf..top=row1*24
&winf..left=col1*10
&winf..width = (col2-col1)*10
&winf..height = (row2-row1)*24
return
The DOS code to define the window of:
DEFINE WINDOW "prompt" FROM 2,3 TO 10,70
Is the translated to dB2k code of:
define_window("prompt",2,3,10,70)
Hopefully, the translated code is easy for a dBase programmer to understand.
Sample Code - @ get
A more interesting examples of the translated code is the DOS dbase code of:
@row(),3 GET ans PICTURE "@I XXXXXX"
is translated to the dB2k code of:
atget(row(),3,ans, "@I XXXXXX")
In this example the variable ans defines a public variable set by the original DOS code which remains visible to the runtime function of atget(). The function row() which returned under DOS dBase the current row position is no longer applicable to dB2k but has been replaced by our own runtime function which returns the equivalent based on the last row position defined for other @say/@get functions. The picture clause may be further extended to handle colour, function, etc.
Sample Code- Form sizing
Referring back to the function define_window, we can see that as the form is defined with a specific handle name, and we can thus directly manipulate the form. In our example, the form was defined as "prompt". We can then set its width by the code :
_app.prompt.width = (col2-col1)*10
where col1 and col2 is the left and right columns values as used in DOS dbase and *10 is the width of a characters in terms of pixels.
To make the function reusable, the form name is passed as a parameter and evaluated at run time, thus the line is:
&winf..width = (col2-col1)*10
the first "." signifies the extent of the string to which the "&" macro substation is applied and the second "." associates the object, or in this case the parameter, with the form.
Sample Code - Mapping
When a new @get/@say function is to be mapped to the master form, the next object within the master form to be used is selected by testing the position attribute of each object as below:
function getnextobj(objtype)
private nobj, objname
Do case
case objtype = "ENTRYFIELD1"
for nobj=1 to 30
objname = "_app.mf.e" + ltrim(str(nobj,3,0)) //Object
names are e1, e2, ... to e30
If &objname..left=3000
exit
endif
next
_app.emax = max(nobj,_app.emax) //Last or highest in use entry
object
case ….
Endcase
return objname
Sample Code - atget() and Picture
The runtime function to support the @get function can then set the datalink property of the assigned object as below:
function atget()
parameters rowno,colno,saytext,atpicture
private objname,elink
objname = getnextobj("ENTRYFIELD1)
elink = objname+".datalink='"+ saytext +"'"
&elink
and similarly the picture clause is set by including within the
function:
if type("atpicture")="U"
.or. argcount()<>4
&objname..picture= ""
else
&objname..picture= atpicture
endif
Other attributes of the and other form object are similarly set.