Articles and Reviews
Although the articles below are in part at least a bit old
hat, they have been retained for the benefit of any new readers.
The GDI story has also been updated to additionally cover v7.01.
The reviews were originally published in the
EXE Magazine and are reproduced
here with the Editor's permission.
The GDI Story
7.01 is better but still not perfect.
How GDI leaks in 5.5
Minimising GDI loss
A 12 line text control!
Leak prevention
Some "front-burner"!
We all know that Windows uses lots of memory . It also uses two Resource heaps, GDI and User. Most software does not make heavy demands on these. Visual dBase (VdB) does, at least on the GDI heap. This heap contains information about graphical objects, such as pens, brushes, cursors, fonts, icons, buttons, scroll bars, and so on.
This was a particular problem in v5.x and initially it was thought it would not arise in v7.01. Sadly not true. Certainly the problem is far less acute but extensive use of the new Notebook class, especially if populated with many text, button and other objects, can still lead to screen corruption through inadequate free GDI resources. Allowance must also be made for printing as a typical report can grab as much as 10% of the GDI heap during the print operation. It should be released on completion of the print but any sort of glitch while printing may turn that 10% into a permanent loss. The notes below refer mainly to v5.5 but are retained as many developers are still using this version and in any case much is good sense in v7.01 too. For this latter version, there is another advantage in creating your own "Label" custom control from an Entryfield. Using this for most simple text will also make your system run appreciably quicker.
VdB presents two GDI problems. First, many of the controls, such as buttons and rectangles, seem to eat more GDI than they do in other packages. Second, VdB may not always release GDI after a Form or similar is closed. The GDI heap is 64k but usage is always shown as a %. The VdB About box reports the % currently available. When VdB is first loaded the available % is typically 70 in W3.1 and 88 in W95 but the figures will vary with your setup and what other packages are open. As available GDI drops VdB becomes unstable. It may keep running to as low as 20% but is more likely to crash at around 30%. This is W3.1, W95 seems to cope with rather less. With v7.01 in W95 I have seen a figure as low as 0% while printing without it causing a crash though by then the screen was looking a bit odd!
As an example, I noted the figures below on a small VdB app. On each run I selected one Form, moved to a different record, closed the Form and then Exited. Then checked free GDI before the next run. This is with v5.5
After 1st run, free GDI down by 13% . After a 2nd run by a further 7%. After a third run by a further 6% . After a fourth run by a further 7%. After a 5th run by a further 6%. After a 6th run (which throws an error message) by a further 8% . Opening all Forms on the app increases the rate of leakage. This is on a 486 with W3.1. It is much the same on two other machines with 3.1 and not much different with W95 though as it starts from a higher GDI figure two or three more runs are possible.
There are ways of minimising GDI usage within VdB but it pays to start with the highest possible % available. If your resources are running low, shut everything down and restart Windows. Look at the Program Manager menu item Help | About screen. It will show the available System Resources which is the lower of either GDI or User resource heaps. If you have less than around 75-80% after a fresh Windows restart, it may be too low to run large forms, particularly if they contain many objects, and more particularly if they contain multiple Images. A simple way to watch GDI usage is to place the Monitor (W95 Accessories, System Tools) in the W95/98 Toolbar tray.
If your resources are low after a fresh restart, take action to correct this by:
1. Remove rarely used/unused icons from the Program Manager
desktop.
2. Remove rarely used/unused fonts in Control Panel.
3. Remove desktop wallpaper/bitmaps.
4. Remove screen saver in Control Panel. Some screen savers eat GDI for lunch!
Other things you can do to resolve the problem:
1. Program Manager replacements/supplements such as Norton Desktop, DashBoard, etc, may intercept messages going to different windows. Sometimes, they don't reliably pass those messages on to the intended target window in which case, the window will not repaint at all or only be partially repainted. You might try removing the Program Manager replacement temporarily from your configuration to see if that has any effect.
2. If you have other applications running in parallel with VdB, particularly those with many buttons containing bitmaps, shut them down.
3. Simplify your forms. Remove background Rectangles, unnecessary bitmaps, etc.
4. Break your forms down into small ones and only open them when they are actually needed.
5. Move to Win95. In Win3.1, there is a single 64K GDI heap and a single 64K User heap. These 2 heaps are shared by all applications running under Windows. Win95 improves on this quite a lot in practise though the heap size itself is unchanged
Now for what you can do in VdB itself. It has been found that different controls eat GDI resources at very different rates. EntryFields are not too bad at 0.1% each. Just as well as we cannot do without these. Each text item uses twice as much, 0.2% but the worst seems to be Rectangles at 0.5% each. Now as the latter are normally used only to enhance appearance the moral is, delete all rectangles if you find GDI a problem. There is also a trick that allows one Text control to replace many individual ones. For example, where you have a column of 12 Text controls listing the 12 months. Try this instead:
DEFINE TEXT TEXT1 OF THIS;
PROPERTY;
FontBold .F.,;
Left 4.166,;
Top 1.0586,;
Width 39,;
Text "Months" + chr(13)
+ chr(13) +
"January"
+chr(13) + chr(13) +
"February",;
Border .T.,;
Alignment 9,;
FontName "Arial",;
Height 6.8232
Obviously add the remaining 10 months in the Text property. Two points to watch. You cannot control the vertical spacing except by the fontsize so you will have to position the relevant fields to match. Second, the Designer will ignore the whole Text line and not re-save it. My solution to this is to paste the whole text line (actually several lines with all twelve months of course) to below the ENDCLASS and pre-fix it with the "*" comment symbol. I can then just paste it back as necessary. It is too long to keep typing! Note also the Alignment property of 9, this forces word wrapping. In the above example, assuming title line and twelve months, I reduced GDI usage from 2.6% to 0.2%.
The other main VdB problem is GDI leakage. By this I mean the gradual erosion of GDI resources when some of those used are not released after the relevant form is closed. It has proved more difficult to pin the causes of this down precisely as examples of leakage have not been consistent. However several worthwhile things to do or avoid have been identified.
It has been found that just closing a form may not release all the GDI it had used. So if you have effectively finished with that form, besides closing it specifically clear it with:
TheForm.Release()
TheForm = ""
The last line "stubbs out" any reference to the form in memory. Release and clear modal forms in a similar way. These in fact may be more critical.
If you are using Custom Forms (as indeed you should be), any changes you make in a WFM form based on a CFM will be streamed by the Form Designer as a "Redefine". For example, if you re-position a button. Redefine seems to leak badly (though not always in an EXE for some unknown reason). The solution to this is to "override" the form open call and delete all the redefine lines. For those not familiar with this, here is an example:
Procedure Open
Form.Pushbutton1.Top = 2.5
Form.Pushbutton2.Top = 4.0
SUPER::Open()
Your own "Open" procedure takes precedence over the VdB Open() so you can specify any variations to the CFM properties and then call the VdB Open with the SUPER::Open(). This can be useful for other things and in most circumstances is better than having an OnOpen() event as the latter allows the user to see the screen being redrawn which of course is not ideal.
If you want to monitor GDI usage from within your application, you can modify and re-use an example "About" box that is amongst the supplied sample files. I find this useful and the technical information it shows does you no harm with clients! Or with W95/98 use the Monitor in System Tools, as mentioned above.
There are also various Windows utils around to monitor usage of all resources. I use a simple one called RESLIM.EXE a copy of which accompanies these notes. It reveals that some packages (such as Ami Pro) release all the GDI they have used on Exit but others (such as VdB and Word) do not, which seems to imply that it can be done - if you know how. You might then wonder why Lotus know how but not seemingly MS.
By now you might well be asking the question, why has Borland done nothing about this? Why indeed. I hit GDI problems within the first couple of days of using VdB in July 1995. If I hit it so quickly it is difficult to believe that none of the extensive testing revealed the problem. A review published by the EXE magazine in October 1995 made a clear reference to it and Borland had a copy of that issue. Yet when various users raised the problem on the VdB CompuServe Forum during April, May and June this year Borland claimed it was all news to them. However there were then assurances such as "Borland IS working on this. It's definitely front-burner stuff." and "we fully intend to resolve this issue in the short term". Yet it is now November and there is still no indication of whether they do have a solution, whether it is whole or partial and when any such solution will be made available. I dread to think how long any "back-burner" problems might take!
Bob Rimmington (c) Nov 1996
Visual dBASE 7
Visually, there is much to please
Report Designer
Component and Custom Controls
Inheritance
Language, Old and New.
SQL Objects and Data Modules
Out with DML, in with OODML
Compatibility and Deploying
Data Objects & the DBF Table
A Flawed Presentation?
Conclusions
[The review below is as originally written for the EXE Magazine early in 1978. A footnote covers some of the changes and corrections in v7.5 released in 2000]
A smarter face and a new vocabulary. But is it still dBASE?
It has taken Borland two and a half years to release a 32bit upgrade to Visual dBASE (VdB). The jump from v5.5 to 7.0 is no doubt partly to keep in step with the opposition but it also signifies that it is very much more a new product than a normal upgrade. In fact it is arguably a replacement for the core xBASE language itself, much of which dates back to the original dBASE II 15 years ago.
A description and assessment of the revised language, data
handling methods and other new concepts is likely to be of most
interest to programmers. We must also look at the many improvements
to the design tools and a number of new components.
Finally we must consider how well Borland have presented such
major changes in respect of both documentation and build quality.
The Desktop and Visual Designers are where most interface
development will be done. VdB writes full source code for EVERYTHING
that is built visually. In fact a masochist could write a complete
application in a DOS editor! The code is streamed to the Editor
window as components are placed on a form. It is a totally two-way
environment, arguably three way in fact as many properties can
be modified on the form, in the Inspector or in the Editor. Changes
to the last reflect back on the form once focus is moved to another
window. In most respects it must be a near ideal working environment.
Besides the Form, Inspector and Editor there are windows for
Component and Field Palettes. A treeview panel to the left of
the main Editor window proves a great aid to code navigation.
As first shipped Borland have chosen to make the Inspector, Component
and Field windows 'always on top' so that the Form itself is
'always at bottom'! An early correction to this is promised and
is certainly needed. [and was done in v7.01]
Even then, with so much on-screen real estate, a 14 inch screen
would prove very frustrating. For any serious development work
a 17 inch screen (at 1078 x 768) is really a minimum and a 19
inch would be preferable.
An integral Report designer completely replaces Crystal Reports as used in v5.5. Creating a report works much as for a form.Components and fields are dragged onto the design surface, positioned as required and text entered. A PageTemplate determines the general appearance of a report and would include an overall title. A Streamframe contains column headings and the data streamed from tables. There are the normal provisions for detail and group bands plus a footer. Navigational controls can be added either singly or in combination as a Custom Control. Reports default to view mode but can then be printed. Reports can be outputted as HTML files but the formatting is not quite right and a single HTM file for a long report is far from ideal, especially if over 2Mb as on one test! A labels designer sadly lists US label sizes only. As with forms, the Report Designer writes full source code that allows further fine tuning through the properties and methods of the report objects. Overall, the report tools are a big step forward but need rather more development. Such as smoother positioning of objects, better facilities for conditional selection, the ability to print an & and more templates than just one for a basic listing.
Other improvements include a new Project window with a tree view of all the relevant files. The Inspector also shows objects as a tree view in its drop down list and there is a new Navigator window. The previous Catalog feature has been replaced by the Project tool. The Code Editor is much improved. Besides the excellent TreeView panel it can now be customised in many ways, including a choice of Classic dBASE or Brief settings. No change though to the traditional Command Window, a facility possibly unique to dBASE and beloved by all programmers.
The long awaited Grid falls short of expectations. No facility
to set properties by cell (such as red for overdue accounts)
and setting some columns to read-only requires a work-around
such as making them calculated fields. Changing properties for
column
headings and content is so convoluted via the Inspector that
it is a relief to be able to do it manually in the code editor!
A Container control is an ideal tool for holding groups of buttons
and other components. The Notebook control is another very welcome
new feature and seems to work well. Other new controls include
Listview, Treeview, Slider, Progress and Report Viewer. This
last allows a report to be displayed while working on a form.
ActiveX controls are now supported too.
The Editor Control now understands and can optionally display basic HTML formatting tags via a popup context menu. A neat Format Toolbar is also available to set heading style, font typeface, HTML size and colour, text attributes and alignment.
VdB handles object inheritance through custom forms and controls. These are designed in the normal way but then saved as custom objects. Individual controls (such as a set of navigation buttons) can be saved as a custom container. It is best to sub-class everything, even entryfields, and build an application from such custom objects. VdB works very well this way. All custom properties can be changed in individual forms and built-in methods can be over-ridden. For example,
function form_open
.... do something
return BOOKINGSFORM::open()
will first run your own code and then call the built-in Open()
method to open the form.
As with v5.5, a collection of re-usable custom controls for v7.0 is already accumulating in websites and the CIS Forum libraries.
The traditional DOS interface commands, such as @ x,x GET have now gone completely but most of the rest remains with a very big proviso. From its earliest days dBASE introduced the concept of workareas to allow multiple tables to be open and accessible. These were used through what is now described as the Database Manipulation Language (DML) comprising 132 commands and functions. However VdB 7 has adopted SQL methods for handling the traditional DBF tables as well as all other types. With this has come a completely new OODML (Object Orientated DML) comprising 164 Classes, Objects, Properties and Methods. A DOS dBASE or Clipper programmer would now recognise very little indeed of the new VdB7 source code. At present both DML and OODML are available but Borland are recommending the latter and some new controls such as the Grid will only work with OODML. The implication seems clear, DML is retained at this stage to allow the immediate use of legacy code with minimal change but is unlikely to remain a permanent feature.
Other changes include a TRY ... CATCH ... FINALLY ... ENDTRY structure allows improved trapping and handling of exceptions and errors. Assignment (:= += etc.) and increment (++ ) operators are new to v7. The assignment operator can only be used with an existing variable or property so a useful way of minimising typos.
How then does the new SQL and OODML concept work? At its simplest, the icon for the relevant table is dragged onto a Form at design time. An SQL icon then appears on the Form and the code for an SQL Query Object is inserted into the source code for the Form. A Field Palette also fills with all the fields in the table and these can then be dragged onto the Form, each with the field name as a default text legend. Can anything be simpler? For a basic form, well yes, you could also let a wizard do it for you!
For less simple tasks, an SQL Designer allows queries to be created visually with options for selection, criteria, grouping, group criteria, sorting and joins. Normal SQL code is generated which may be viewed and optionally modified in an Editor Window. As with the other design tools, this one is entirely two-way, any changes reflecting back into the Designer. There is also a facility to save SQL statements to a Property Builder for re-use in other queries.
Although SQL Objects can be used directly, a Data Module Object will often be preferable. This is a container for other objects such as SQL queries, stored procedures and databases. It can also contain statements such as lookupSQL that will support a Combobox picklist showing fields in one table but datalinked to a field in the selected table. Data Module objects can then dragged onto forms to instantly set all the required links to tables. Done this way, the Data and Form objects will also automatically handle Sessions. These are the way VdB allow many users to work on the same tables and also enable each user to access a table for different purposes such as lookups while a report is running. Up to 2048 simultaneous sessions are supported.
This is clearly a big new concept for dBASE programmers and there is no doubt that many will blanch initially at the implications. Figures 1 and 2 give some examples of the now semi-redundant DML commands and a selection of the new objects and methods. In essence, forms and program code will now address rowsets derived from tables through SQL statements and not the tables themselves. The Rowset Class contains properties and methods that handle many of the tasks previously done though DML commands. Others are handled by further new classes such Field (each rowset has a fields property which points to an array). See Figures 3 and 4 for examples of old and new code.
Some rowset properties can supplement or replace part of a typical SQL SELECT.... expression. For a report an expression defining relationships, order and any filter will normally be fine but for a form anything as complex as that makes the rowset Read Only. Hardly suitable for adding or editing data. In these cases a simple SELECT .... statement can then be supplemented with rowset expressions to set (and change) table links, index order and any required filter. The rowset filter property is in itself an SQL expression that does not look immediately intuitive. Such as: form.rowset.filter:=[surname=="]+this.value+["]. Setting an index order is much easier, just form.rowset.indexname:-<TAG name>. Not all the old DML commands have new equivalents and ones such as SET CARRY and KEYMATCH() will now require more complex alternatives. See Figure 4 for fuller examples.
Typical of the changes that developers will now need to address is the effect of changing to a different index tag while in a form such as when a user is given a facility to view a grid in the sequence of each of the displayed columns. Until v7 the record pointer did not move, now it always reverts to the top of the rowset. [corrected in v7.01] It cannot be moved back by noting and setting the pointer back to a record number as a rowset has no record number. It does indeed have a Bookmark but this only applies to the current sequence. The Debugger looks as if it might prove useful but has some odd quirks and single steps only slowly, up to 10 seconds per step if many variables are displayed.
Unfortunately Borland are strangely reticent about it all. None of their pre-launch publicity mentioned the effect of the new data objects, there is no guidance on how to switch to the new approach while the definitions of the new methods and properties are woefully short of essential examples. In fact some important methods such as tabledbf(), dbfindex() and load() are not even mentioned at all. Developers will need to rethink their whole approach to how they construct their applications and will be looking for more help than they are getting.
No attempt has been made to maintain any compatibility with DOS programs and even 5.5 applications will not automatically run in v7. It is best to open and then re-save Forms in the Designer to stream out the new style code. This will achieve much of the necessary revision and there is a tool to assist in the conversion of reports. However other changes will normally be required and may not always prove easy. For example, the default ScaleFontSize and FontSize properties have increased from 8 to 10 points and this will certainly need correction. So many are likely to keep with 5.5 (soon to be 5.6 through an inline upgrade) for most existing applications while switching to v7 for systems still evolving and for anything new.
Another major change is that VdB7 is now aimed strictly at
Developers (there is no longer a Standard version, just Professional
and Client/server). It still has all the
interactive features of v5.5, in fact in various ways they are
better. Presumably, Borland have given up the unequal struggle
of trying to compete with an Access that comes free with MS Office.
Borland no doubt hope that this will end a common (but wrong)
assumption that VdB is an end-user rather than a serious tool.
It is now positioned as one of the range of Borland development
packages and targeted at both dBASE programmers and as the best
tool for extensive database projects.
The source code for forms and all other files is automatically compiled to object code when a form or application is run. A complete application can be made into an EXE file though options on a Build menu when the Project Explorer is open. InstallShield Express is bundled to allow creation of a set of discs for installation of a system on machines without a copy of VdB7. Besides the EXE itself the discs will include a large runtime file, some DLLs and much of the BDE. However program changes require only a new EXE to be copied onto the user machine. Also any additional applications can make use of the first installation of runtime and BDE files.
The data objects work through the BDE, the Borland Database Engine. The BDE is now better than ever. It will handle DBF, Paradox, MS Access and FoxPro CDX tables as native types (plus Oracle, SyBase, Informix, InterBase, MS SQL and DB2 in the Client/Server version). ODBC connections are fully supported and those tried worked very smoothly. One such that might interest many developers was with Sage 4 files through the Sterling 32 driver. The important point to note is that the new data objects allow all these varying types to be handled in just the same way. No doubt this was a major reason for the switch to SQL and OODML even for the traditional dBASE DBF tables
Significant changes have been made to the default DBF table structure. There are four new field types:
Long 4 digits Positive whole number
Double 8 digits Positive whole number
Autoincrement 4 digits Long integer auto incrementing
Timestamp 17 digits Date plus time as HH:MM:SS
Distinct indexes (no duplicates allowed) and primary distinct indexes are now available but index expressions including UDFs are no longer possible. Custom Field Properties now enable picture, colour and range criteria to be set at table level. A field is in fact an object that can even include properties such as a lookupSQL statement (ideal for Combobox pick lists) or methods such as beforeGetValue(). This enables what appears on the screen or a report to be a derived rather than an actual value. Such as a stock description rather than the stock code actually in the field. So the DBF table format can now be said to have the capability of a Data Dictionary. Finally, a total change to the default value of fields. In the past numeric fields defaulted to zero, logical to False and character to a nil length string. Now all default to NULL. This is merely mentioned in the Developer's Guide as a new feature for character fields only but there is no discussion in the documentation of the many implications. These include:
An Average of a numeric field is now calculated by dividing the total by the number of entered values, not the number of records. An accumulating total reverts to nil every time a record with a null value is encountered. A string concatenation (such as Title+Initials+Surname) will return nil if any of those fields are null. A logic check for True or False will fail if null. Programmers will need to find an answer to this and so must now either initialise every field at time of entry or include a check for field type in every calculation. It is a pity that Borland made no effort to explain this. Nor how to handle tables shared with other xBASE products as is otherwise now possible with the inclusion of the CDX index format for native connectivity. A simple method is to set the BDE level to 5 but this will exclude the new field types as above in the new level 7.
Borland had committed themselves to a release in the last
quarter of 1997. They achieved this, just, but there are various
indications that another month or two would have been well spent
on testing and refinement. There are various detail bugs of
course and some general fragility. With so much fresh to learn
many developers will nevertheless welcome the early release.
It is usable and is capable of building applications that can
be delivered. An in-line upgrade is promised 'soon'. However
Borland's track record on such upgrades is not good. For v5.5,
the first took a year and the second, just released, two and
a half years to address major issues such as the leakage of GDI
resources described in the EXE review of v5.5 in 1995.
Of more concern is the documentation. The Professional edition does not even include the Language Reference Guide (£20 extra plus postage in the UK). There are the omissions of some descriptions, far too few examples and nothing on the implications of NULL as described earlier. Worst of all, there is no guidance at all on how to tackle a new project using OODML instead of the traditional approach. Attempting to learn by the 'lets try this, now try that' approach just brings up brief and often misleading error messages. Which in turn reveals another major omission, no detailed list of error messages with suggestions for rectification. At this point, over two months after release, even a simple ASCII list of known bugs and documentation omissions is still awaited. It is said to be 'too early' for this. Too early for Borland perhaps but urgently needed by users.
The second difficulty is the overall effect of all the programming changes. They will require a largely new approach by xBASE programmers plus a familiarity with all the new OOP methods. Traditional xBASE is often so near to a sort of stilted English that many developers think through their routines in those terms. Such as SET FILTER TO SURNAME = "Smith" or TOTAL ON CUSTCODE TO CUSTTOTS. The new syntax is far less memorable and at first sight a lot more verbose. It is difficult to assess at this stage how far the apparent extra complexity arises from merely trying to replace old code with new on a line by line basis. Though form and report design might prove quicker these are not, as seems to be assumed, the principle part of all applications. For example, the current use of EDI between customers, suppliers and distributors is intended to minimise the entry of data into forms by users. Instead, to extract and process data received as files may require some thousands of lines of complex code working quite unseen to the user. discovering to what extent a completely fresh approach to program design might bring benefits of performance or simplicity is likely to require a long learning curve.
VdB 7 shows great promise, it may well prove to be the ideal tool for moving the many DOS database applications still running into the Windows environment. However Borland have done themselves no favours by supplying a version with rather too many initial blemishes and with quite inadequate guidance on how to use the new Data Classes and OODML. Nevertheless, although it would certainly benefit from a little more development work, it is usable as it is now. Providing of course it is accepted that it now has only a passing resemblance to the dBASE of old!
Footnote included with the above: Borland are reacting positively to bugs and other points raised by the writer and many others. An early version of an inline upgrade sighted while preparing the review confirmed that many of the flaws have already been removed. The inline to be released as a download will include further bug correction but a release date was not available as we went to press. Revised Help files are promised too. [The 7.01 release did indeed correct most bugs (not all) and included a few extra features that were not quite ready for the initial v7 release date]
Bob Rimmington March 1998
Version 7.5
The 7.01 release corrected the most glaring bugs but many others remained. The first priority of the new dBASE.Inc company has been to eliminate the most important of these and to make the product more stable. Some extra controls, much enhanced Web Wizards and a great deal of new documentation are also included - see the new new VdB 7.5 page for full details. The result is that VdB is now a little faster, a little smoother to use and very much easier to learn.
Updates to points raised in the original review include. The Report Designer is now far less quirky though still not ideal. However it is now clear that any weakness is only in the design tool, the underlying code has proved very stable. There is now a property that allows the printing of "&". The Grid though still lacks individual cell options, promised for a later release. OODML code has not proved so long winded as originally feared as it can be shortened for both convenience and slightly faster execution. For example:
b = Form.DATAMODREF1.ref.bookings1.rowset b.indexName = "BookingId" b.fields['BOOKDATE'].value = Date()
The new release also puts a lot of emphasis on the use of VdB in the construction and operation of web tasks such as interactive enquiries and shopping basket type sites. Overall, it is not a big step forward but a promising one. Under the new subscription scheme new releases are promised at three monthly intervals.