A frank exchange of views

Hints on upgrading OS X to Leopard

David Glassborow — Fri, 02 Nov 2007 10:41:00 +0000

Having just returned from two holidays: Colorado with a motorbike (lovely people, stunning scenery) and Firenze with a girlfriend (lovely people, stunning architecture and art) I decided to update my OS X partition to the new shiny leopard. Here are a couple of notes of how I got on, which is hopefully useful to anyone else doing the same.

Leopard

Initially I chucked the disk in and just did an install, this installed the OS over the previous one. Everything seemed to work fine, except I had installed Java 1.6 beta from Apple, and this didn’t work in the new OS (see google for rants about 1.6 JVM not being supported). I needed a working Java for programs such as FreeMind. I tried to sort it out by changing symlinks in the system folders, uninstalling the beta, but basically just broke everything. I decided to reinstall the whole thing, which requires you to notice the ‘customize’ button on the installer, allowing you to request it clear the disk first (but usefully also allows you to not install various things like printer device drivers which saves you gigs of space !).

Having a new install, things definitely felt faster. I also booted into bootcamp, put in the 10.5 disk which installed the latest windows drivers for everything.

I got most of my development software sorted in an hour or so. The new OS seems to have lots of development tools built in (I installed the XCode package from the install disk as well), like subversion, ruby and even ruby on rails (saves on things to install via MacPorts)

When you buy a Mac, it comes with the iLife software (iPhoto, iDVD and all that) but an OS upgrade doesn’t contain them, so I reinstalled iPhoto from my 10.4 disks which worked fine.

Parallels

To get Parallels working I had a look on google and people recommended running the newly released Beta 5540 (your license will work fine with this, no need for beta license) - find it here. I installed it, it detected my bootcamp partition, but trying to run it caused an immediate OS X crash. I restarted the machine any everything was fine. Or so I thought.

Next time I logged into windows I got an error immediately after entering my log in details: A problem is preventing Windows from accurately checking the license for this computer. Error code: 0x80070002, followed by an immediate logout. I tried both Parallels and Bootcamp and the same issue on both ! A Microsoft kb describes this as related to an issue in Windows paranoid licensing checks. Other similar issues seemed related the use of Volume Licensing keys (I use our companie’s VL keys for my windows machines) - I tried the suggested fixes but no help. After lots of messing around, I found a post on the interweb that suggested it was an issue with some files in system32, oembios.*. Looking at these 3 files, they had change dates in the last 24 hours. A copy of the files from another XP SP2 installation fixed the issue (via Safemode and a USB key) , returning change dates back to 2004, got everything working perfectly again.

No idea what broke these files, whether it was Parallels, Bootcamp drivers or something else ?

Another oddity, to fix windows to run in both Parallels and Bootcamp, I had to apply the fix once in each mode (which is really weird - somehow related to the use of different hardware configurations to allow parallels to do its magic - anybody understand how this works ??)

Misc

The only other issue I have come across was that once connected to a VPN (to a windows 2000 server), DNS lookups on my Mac were really slow (like 20s). Looking on the net, there were suggestions that the resolver stack had changed slightly in the new OS, and may be working badly with old routers / dns servers in some way. Adding my local DNS server to the VPN’s DNS settings (under Advanced in Network system preferences) seems to have done the trick though.

Now having everything working fine and dandy, there’s no excuse for putting off actual useful work, how disappointing ;-)

Remote Desktop for OS X

David Glassborow — Wed, 08 Aug 2007 12:06:21 +0100

If you need to do remote desktop / mstsc on a Mac, you can either use the flakey unreliable implementation from Microsoft, or use the brilliant open source CoRD

Accessing Mac Files from Windows

David Glassborow — Thu, 19 Jul 2007 12:58:00 +0100

If your booted into windows via bootcamp and still need access to a file on your Mac partition, you can use HFSExplorer to access them read only (via Mac OSX Hints )

Cheap Mac Software

David Glassborow — Mon, 16 Jul 2007 11:56:00 +0100

MacHeist are once again doing a great bundle of shareware software for $50.

With the Dollar being very weak compared to Sterling at the moment, this is a bargin for us in the UK.

Professional Development on a MacBook Pro

David Glassborow — Thu, 12 Jul 2007 16:26:00 +0100

Like a number of people in the Delphi Community ( Steve Trefethen, Dan Miser ), and the development community as a whole, I now do all my development on a MacBook Pro. I changed to an Apple machine six months ago, for a number of reasons:

I needed a new laptop, and the MacBook Pro hardware is so goddamn thin, light, and good looking.
I am becoming increasingly disillusioned by Windows, especially the new DRM stuff.
I fancied trying out OSX, a change is as good as a rest and all that.
Bootcamp and virtualisation allow me to use Windows XP on an Apple laptop.
The web is freeing us from being locked into particular Operating Systems.

As a professional developer, most of my clients run Windows, and most of my native development is done using Visual Studio 2005 or Delphi 2006, connecting to SQL Server or Oracle backends. I boot my laptop into Windows to do this work, or use Parallels to run it with Mac OSX (when I got my MacBook Pro I got it with 3G of RAM so Windows runs pretty fast inside OSX). Increasingly though I am doing web development, which means I can do much more work inside just OSX.

The more I have used OSX, the more I prefer it to Windows, so I have put together this list of tools and hints for doing professional development on OSX:

Using Bootcamp

Steve has got an excellent list of the keyboard mapping for Windows
Parallels allows runs Windows (from a bootcamp partition) in OSX
As does VMWare

Using OSX

One of the things there first confused me using OSX is the link between a visual Window and a running program. Having used Windows for the best part of 15 years, I had the stong mental link between having a open window and having a program running. OSX is much more like the systray in a way. In most instances, a program carries on running even if all its windows are shut. At first this seems odd, but now I am actually used to Mail and iCal running in the background, collecting mail and showing me reminders, without having windows cluttering up the place or being minimised. Also the dock changes icons depending on status of the program, so I can see that new mail is waiting, without requiring an open window to tell me.

I guess it is the unix heritage, but OSX seems much happier having loads of programs running a the same time, than Windows would (or maybe this laptop has just got too much memory).

Another change from the Windows way of thinking is about installation of software. In OSX an application is a directory with a name of BlahBlah.app. It contains all that is necessary to run. So drag and drop to install software, move things around as desired, and delete it when no longer wanted. Only software that needs to make system changes needs to have install scripts.

You need to install Quicksilver for launches programs, and more advanced voodoo. Have a look at some of the screen casts at The Apply Blog, a good beginner one is here. You won’t believe for a powerful bit of software it is.

Another thing worth getting your head around is this idea of OSX services. These are simple ‘services’ that are provided to all programs running, accessible of the programs menu. While there are some useful ones built in, there really come into the own where you start changing them using Service Scrubber and This Service. See some examples from John Gruber about writing your own scripts and making them available to all apps (although I’d recommend using Ruby over Perl ;-).

Web Development

It goes without saying that you should be using Firefox with Firebug for web development. Day to day browsing I use Camino which is the firefox engine inside a more OSX integrated and looking program.

Textmate is a great editor, as most people know, but if you are doing any JavaScript development make sure you get the JS Tools bundle. Having your script ‘lint’ed everything you save is a great productivity improver. Also get the GetBundle bundle to make it easy to manage extensions and find new ones. Blog writing is easy with the built in markdown support.
Eclipse (especially with the new CodeGear betas running inside Eclipse ) has plugins to do pretty much everything (although I still use Textmate most of the time).
Anybody know of a Fiddler equivalent for OSX ?

Subversion

Does anybody use anything else anymore ? I use the tigris plugin scplugin to have subversion integration with the finder. It is not quite as good as tortoise. There is also Svnx which is ok, but I don’t find the UI all that intuitive.

Database Access

For connecting to Oracle, I use the free SQL Developer which has a native OSX version. Most of the open source databases have native clients (e.g. Postgres), but otherwise I use Aqua Studio v4.7 which is free. One of the nice things is most of the tools are Java based, so if you can find a JDBC package for your database, chuck it in /Library/Java/Extensions and then all the Java apps on your machine have connectivity to it (I have found ones for SQL Server, Oracle, MySql and Postgres).

Other Languages / Database Engines

Anything available in the Open-source world, if not explicitly released for OSX, is normally available on DarwinPorts or Fink, projects to make linux/unix software compilable on OSX (which is a unix variant behind the scenes), and easily installable handling all the dependancies etc.

Are there any useful development tools or hints I have missed ? Let me know and I’ll try and keep a development bias list here or on this page.

And if your still using boring old Windows, come and join the revolution !

Rules of thumb to make a project work

David Glassborow — Thu, 12 Jul 2007 15:00:00 +0100

I am currently coming to the end of a large project for one of our clients, where I have been acting in the Business Analyst and Technical Lead roles. It has been a fun and successful project, and I wanted to pass along some of the lessons I’ve learnt (again) about how to make Technical Analysis (and the project as a whole) successful (and also make a checklist for next time around ;-).

It must be remembered that IT solutions (and really any business change) are all about two things: Incentives and Trade-offs.

Incentives

Incentives are important because ultimately people are lazy, and this is A Good Thing. We all have way too many things to do in our lives, both personal and professional, and therefore being lazy is a very successful evolutionary adaptation on the part of Homo Sapiens. The body only bothers to grow stronger muscles when we actually need them, rather than paying the cost of having them all the time. The same approach can be seen in our behaviour.

To make people want to use your new approach / program / business process, you need to make it worth their while. This seems obvious but I see it being forgotten time and again. Sometime the incentive is fear of being fired, but ideally you want to make your user’s lives easier and more efficient. If you can’t incentivise them to change their behaviour, they won’t.

Trade-offs

Any new system involves trade-offs. If it was a simple no brainer, it would have been done ages ago. Management often want better visibly of information (and normally control as well), but they need to be made to understand that this will have added costs, whether that is in increased paperwork, administration, or work enjoyment for the staff. Often people assume they get increased efficiencies (or whatever) for free, but there is always a downside. Not that that is a problem, but it needs to be understood and communicated (it goes into the areas of risk management). Communicating this as quickly as possibly, and ideally suggesting different cost/benefit scenarios, makes sponsors think more about why they are asking for things, and better controls their expectations.

From these two overriding principles, we then have a randomly ordered list of my current best practises.

Technical Analysis Best Practise

Always sit with the users
It’s all about incentive
Mastery of language is important
Its easier if you don’t care
Avoid the middle men
Flexibly and Power => Complex and slow
Easy to think != Easy to do. Details matter

The above points in more detail:

1 - Always sit with the users

Sounds obvious, but on every project I see it being ignored. 30 minutes of sitting with the actual users you are providing a solution for, will give you better and more accurate understanding than talking to management or reading analysis documentation for days. Business processes change, often the experts on that process are out of touch with what is actually going on. As well as getting a good detailed understand of what they do, and what their problems are, another benefit of watching users is how often there are tiny little changes you can make (trivial software changes like adding a shortcut, or making some non-modal) that can save each user huge amounts of time and stress everyday !

If you haven’t sat and watched your users recently, you’re probably not building a system that solves their real problems !

2 - It’s all about incentive

A reiteration of the point about, but really a reminder to think of the incentives of each distinct groups of users. Management will have different incentives than the day to day users. It is worth thinking how to incentivise each different group to use the system in the way you want it used.

3 - Mastery of language is important

The language you use, both in your application, but especially when communicating ideas and progress to project sponsors, is incredibly important. Firstly words are very powerful in the human mind, they are the key way we link thoughts, feelings, expectations and fears together. The project I have been working on was half way through when I came on board, and because it was late and not delivering, had a number of perceptions, worries and expectations, all hanging off the ‘name’ of the project. One of the first things the team did was change the project name. This gave us a blank sheet to work with, meant that people’s previous perceptions were ‘invalidated’, giving us breathing room to solve the problems in a different way.

Secondly, unless you use terms like Risk and Trade-off, management and sponsors won’t consider these things. Just by using the words, you can make them better consider that they are asking for, and incentivise them to make better decisions (I am labouring the theme enough yet ? ;-)

4 - Its easier if you don’t care

I am an outside supplier, so it is much easier for me to tell people the situation as is, rather than having to worry too much about internal politics. You obviously don’t go out of your way to cause problems, and need be sensitive to the people from the business that you work with who don’t have the same freedoms as you, but being truthful about situations is very important to the overall success of a project. Having said that, obviously always keep copies of all emails, and confirm verbal conversations via email, so you always have an audit trail to support your decisions.

5 - Avoid the middle men

In any remotely large business there will tend to be lots of layers of management (unless you work somewhere very groovy like Google Inc). Each layer will have there own agendas, desires and aims, and will wish to communicate information in ways that either make them look good or at least reduce how bad they look. Whilst this is understandable, and just human nature, projects are much more successful if you can talk honestly to the ultimate users at one end of the spectrum, and the key sponsors / decision makers at the other end. The more layers in the way, the more the problems, drivers, issues, etc. get obscured and less clear. Politically this can be hard, and take time, but it is worth it in the end.

6 - Flexibly and Power => Complex and slow

This is really a more concrete example of the trade-off principle. Flexibility and powerful is what everybody wants their computer systems to be. However these two requirements ALWAYS imply complexity, and complexity implies complex (and potentially) slow running software. The art of being a good designer or architect is coming up with abstractions that are both powerful and a simple as possible, but always remember that powerful and flexibly implies clever decision making which implies complexity. Bacteria are very simple, but don’t do a huge amount but breed. Look at the complexity, both in body and mind, to get something as flexible and powerful as a human being. Fight to keep things simple, and your are much more likely to get a working solution ready in time.

A good example of this is the control issue. Management quite often want to stop people using systems in certain ‘incorrect’ ways (not entering data in certain ways, formats, sequences, whatever) . Often their approach is to wrap these systems in further layers that prevent the ‘incorrect’ use from occurring. Two problems with this though: 1. It’s loads of work and 2. There are often certain exceptional situations where it is necessary to put in data ‘incorrectly’. Often the simpler approach is just to provide an exception report. Management can use this to make sure systems are being used correctly, and if there is a valid reason using it wrongly, its not a problem. Making it a management rather than IT issue.

7 - Easy to think != Easy to do. Details matter

This is obvious to developers, but sometime can be hard to communicate to project sponsors. It is something I still have problems with conveying successfully. Steve Yegge discusses the same point. For the same ‘lazy’ reasons as discussed at the start, people don’t think through what they are asking for, and whilst this is annoying, your job as a technical analyst is to communicate the consequences of what people are asking for. In a similar vein, any technology you have been asked to use should be ‘proof of concept’ed as quickly as possible. Things often work in principle, but in practise are too buggy, slow or unreliable to actually use in production systems. Ultimately this is why I don’t trust analysts who can’t program, they never understand the need to capture the exact details of a project, because they don’t actually have to build working systems, they don’t really realise its importance.

Hope this list is useful for somebody. I’m off to work out what the trade-offs and incentives for writing blog articles are ;-)

Feedburner and Apache redirects

David Glassborow — Tue, 05 Jun 2007 18:56:00 +0100

Looking at the load this blog is causing, virtually all of it’s requests are for the syndication feeds so I decided to off-load the network bandwidth to Feed Burner ( recently acquired by Google).

Feedburner basically checks your site every 30 minutes (you can also manually request an update) and caches the response. Rather than ask my subscribers to change the feeds source, I decided to use Apache to redirect all requests to my feeds to feedburner, unless it was actually feedburner requesting the information.

To get this to work took a bit of Apache shenanigans, so I thought I would document it here both for myself and for anyone else who needs to do the same. It is also a useful example of how powerful Apache is, particularly as a forward facing server than manages the virtual URL space and links it up to various webs server technologies and platforms behind the scenes (loose URL => implementation coupling). The blog is running on Ruby on Rails, Typo engine, running on port 4000 internally. Our firewall blocks outside access to this port, so we use Apache to proxy it for us (it rewrites any URL’s on the way out to be the correct external address). We also use it as an SSL gateway, so we setup all the certificates in just one place.

All configuration in Apache is done in the http.conf file. Here is the setup I am using for the blogs subdomain with notes of what is going on. There is also a similar VirtualHost setting for 443 (SSL Access), but doesn’t really add much by posting it here. I will give a quick summary of what is going on here, but full details can be found on the excellent Apache documentation

# Concept First blog
<VirtualHost *:80>
  ServerName blogs.conceptfirst.com
  DocumentRoot "e:/Blogs/www"
  ErrorLog "e:/blogs/logs/error.log"
  CustomLog e:/blogs/logs/access.log common
  DirectoryIndex index.html
  <Directory "e:/Blogs/www">
    Options none
    AllowOverride None
    Order allow,deny
    Allow from all
  </Directory>

Tell apache which virtual host name this is for, where static files are, where to put log files, etc.

  RewriteEngine on

RewriteEngine on tells apache to apply the following rules. Apache can handle redirects and proxying using Redirect and ProxyPass directives, but I had issues with the order things were being done in, so used rewrite rules for it all. Rewrite rules are more flexible and powerful than the individual Redirect or ProxyPass directives, so its worth understanding their capabilities in full.

  RewriteLogLevel 0
  # RewriteLog "e:/rewrite.log"

If having problems debugging rewrite rules, I’d recommend just setting up a log file and turning RewriteLogLevel to 9. Turn it off by putting back to 0 when everything is working.

  # Redirect feeds to feedburner unless actually feed burner. Only do main ones
  RewriteCond %{HTTP_USER_AGENT} !FeedBurner  
  RewriteRule /xml/rss20/feed.xml$ http://feeds.feedburner.com/ConceptFirst [R,L]
  RewriteCond %{HTTP_USER_AGENT} !FeedBurner  
  RewriteRule /xml/atom10/feed.xml$ http://feeds.feedburner.com/ConceptFirst [R,L]

This is the configuration that tells Apache to redirect all my traffic to feedburner unless its from Feedburner itself.

RewriteCond: A condition rule, applies to the next line. It is negated by !
RewriteRule: A rewriting rule, with a match part (regular expression) and a target
- [R]: Send a HTTP 302 redirect
- [P]: Do a internal proxy
- [L]: Stop applying rewrite rules after this one

So the first two lines tells Apache (in random syntax pseudo-code): IF (HTTP_USER_AGENT <> 'FeedBurner') AND (URL = '/xml/rss20/feed.xml') THEN SEND_REDIRECT_TO('http://feeds.feedburner.com/ConceptFirst') AND STOP_PROCESSING_RULES

Feedburner uses the HTTP header USER_AGENT set to FeedBurner so that is how we detect it and don’t redirect it to itself ! I am only redirecting my main feeds here, the categorised feeds and individual comment feeds are still handled by the blog engine.

  # Make admin secure
  RewriteRule /accounts(.*) https://blogs.conceptfirst.com/accounts$1 [R,L] 
  RewriteRule /admin(.*) https://blogs.conceptfirst.com/admin$1 [R,L]

These two lines use a similar rules to make sure the admin parts of the blog are handled through HTTPS so we don’t get any cleartext passwords floating around on the net. The $1 at the end of the redirect is the matched data from (.*) in the regular expression. So if the URL is /admin/login, $1 will be /login.

  # Proxy to mongrel for everything but the media directory
  RewriteCond $1 !^Media/(.*)
  RewriteRule /(.*) http://localhost:4000/$1 [P,L]

This is the rule that actually gets the blog pages from the internal ruby on rails app. The condition rule is checking the URL match to make sure it is not part of the Media subdirectory (this is a static directory that apache serves up, it contains images for blog entries, etc). Everything else gets passed to the server running on 4000.

  ExpiresActive On
  ExpiresByType text/html "now plus 1 day"
  ExpiresByType image/gif "now plus 1 week"
  ExpiresByType image/jpeg "now plus 1 week"
  ExpiresByType text/css "now plus 1 week"
  ExpiresByType image/png "now plus 1 week"
  ExpiresByType image/jpg "now plus 1 week"
</VirtualHost>

The Expires stuff is the Apache way of setting the HTTP expires headers when sending responses. I’ve set images to be cached on the client for a week to help reduce bandwidth and load on the server.

I greatly recommend Apache, its very easy to setup, and very powerful (although it does test your knowledge of regular expressions :-), we’ve used it to proxy Ruby On Rails, Cold Fusion, IIS and home grown web servers, and its great for rewriting URLs to make them technology agnostic and nice and RESTful.

SQL Server Spatial

David Glassborow — Tue, 05 Jun 2007 15:26:00 +0100

SQL Server 2000 has always been one of my favorite bits of software. I have written various software that runs on top of MSDE and always been really pleased with it.

Having spent a great deal of time recently working woth Oracle 10, I have really noticed how much easier administration is with SQL Server. I have also had lots of issues with Oracle drivers (in particular where CLOB fields are used).

I have to admit to being rather unwealmed by SQL Server 2005. I’m not a great fan of stored procedures (I belive the logic should be in a middle tier) and certainly not of running .net code inside a DB server.

One thing I was looking forward to was the addition of spatial algorithms to SQL Server.

Wikipedia notes that spatial data is planned for the next version, but for the mean time I will be using Oracle for my clients, and personally Postgres.

I think Microsoft really missed out here. More and more of my clients need to store spatial data, I would love to use SQL Server, but my money has to goes to Oracle or an open source database.

Pull your finger out Microsoft and make sure 2008 is spatial enabled !

Comments on my Weblog, Firebug

David Glassborow — Thu, 01 Feb 2007 20:28:00 +0000

I’ve been having a lot of problems with comments on this blog taking forever to post, being silently rejected, etc. After lots of messing around, I found it to be a combination of a) A theme that doesn’t handle problems with comments being locked and b) the built in anti spam protection making comments take 30 seconds rather than half a second to post.

I never want to calculate how much of my life I’ve lost to spam or spam filters, doesn’t bare thinking about.

Apologies to anybody who tried to post a comment.

Whilst doing this, I used a great new feature of the absolutely essential web development plug in for Firefox, Firebug. One of the new views, network activity monitoring, shows all the files loaded (the contents of the request and response) and how long overall the process took.

Absolutely essential for debugging AJAX code, but also brilliant to see where you can speed up your page load times. The following shows me debugging some time issues on one of a site using Virtual Earth. Watching some of the files load from Microsoft’s site, and take 10 seconds, suggested I make key files local. This massively reduces the load times, and makes the whole site feel much nicer.

I cannot recommend Firebug strongly enough !

Useful Delphi code

David Glassborow — Thu, 04 Jan 2007 11:12:00 +0000

Just realised I haven’t written a blog post for 2.5 months, and I’ve just got back from a lunchtime surfing session, so I’m not really in the mood for real work. I thought I’d post some of my basic Delphi library code I use time and again in the hope it is of some use to other Delphi developers. Do what ever you want with it, I don’t guarantee its without bugs, but most of it has been used in production code for quite a long time.

The full file with all the implementation can be found here: Source

The following are useful for conditional logic for development vs production code, debugging, etc.:

procedure BreakpointInIDE;
function CurrentExceptionMessage: string;
function IsRunningUnderIDE: boolean; inline;

The following function is useful for preventing the unsitely errors that can occur as a Delphi Application throws Exceptions as it shuts down. While ideally this should never happen, previous exceptions can leave forms in a have constructed state, and the TApplication’s attempt to clean up them then causes horrible dialogs frightening users. Even the IDE has been known to do this. The code hooks the various Exception raised pointed in the RTL to silently ignore the errors. If running inside the IDE, it will also post a message to the event log, and cause the IDE to breakpoint to highlight there is an issue.

procedure HookShutdownToHideExceptions;

Helper Classes for TList, TStream and TStrings. In particular I use the ID accessors of TStrings a lot when putting records into UI elements. Makes it very easy to look up the ID of the record in a combo box, for example. It uses the Object field in a list item to store the ID. Helpers require at least Delphi 2006.

TListHelper = class helper for TList
  procedure FreeSelfAndContainedObjects;
end;

TStringsHelper = class helper for TStrings
private
  function GetID(Index: Integer): ID;
  procedure SetID(Index: Integer; const Value: ID);
  function GetDescFromID(ID: ID): string;
  function GetIDFromDesc(const Desc: string): ID;
  function GetObjectByString(text: string): TObject;
public
  function AddPair(const Name, Value: string): integer;
  function AddID(const Desc: string; PrimaryKey: ID): Integer;
  function AddObj( const Name, Value: string; Obj: TObject ): Integer;
  function ContainsKey( const Key: string ): boolean;
  function ContainsString( const Text: string ): boolean;
  function ContainsValue( const Value: string ): boolean;
  procedure FreeSelfAndContainedObjects;
  function IndexOfID( ID: ID ): integer;
  property DescFromID[ID:ID]: string read GetDescFromID;
  property IDFromDesc[const Desc: string]: ID read GetIDFromDesc;
  property IDs[Index: Integer]: ID read GetID write SetID;
  property ObjectByString[ text: string ]: TObject read GetObjectByString;
end;

TStreamHelper = class helper for TStream
public
  function AsString: string;
  function RewindToStart: TStream;                                      // Returns self to allow easy chaining of calls
  procedure WriteString( const Text: string );
  procedure WriteStringAndBreak( const Text: string );
end;

The following two classes are 2 example simple container objects based on records rather than classes. The nice thing is they don’t have to be memory managed like classes do, the compiler will handle all of that for you. They are not as fully featured as string lists, but are useful for simple tasks. They are also reasonabilly efficient, as the dyanmic arrays they use are copy by reference. The only thing to remember is to use the Clone method if you actually want a whole new copy of the data. I have about 4 or 5 different variants of these to store ID => Name, Name => Variant, etc, but you get the idea.

TSimpleStringList = record
private
  FItems: array of string;
  function  GetCount: integer; inline;
  function  GetItem(index: integer): string;
  procedure SetItem(index: integer; const Value: string);
public
  function  Add(const line: string): integer;
  procedure Clear;
  function  Clone: TSimpleStringList;
  function  Contains(const s: string): boolean;
  function  IndexOf(const s: string): integer;
  function  IsMissing(const s: string): boolean;
  property  Count: integer read GetCount;
  property  Items[index: integer]: string read GetItem write SetItem; default;
end;

TSimpleStringListHelper = record helper for TSimpleStringList
public
  function AsNewStringList: TStringList;
  function StringsCommaSeperated: string;
end;

TSimpleDictionary = record
private
  FKeys: array of string;
  FValues: array of string;
  function GetCount: integer; inline;
  function GetKey(index: integer): string;
  function GetValue(const Key: string): string;
  function GetValueByIndex(index: integer): string;
  procedure SetKey(index: integer; const Value: string);
  procedure SetValue(const Key, Value: string);
  procedure SetValueByIndex(index: integer; const Value: string);
public
  function  Add(const Key, Value: string): integer;
  function AsNewStringList: TStringList;
  procedure AddStrings(Strings: TStrings);
  procedure Clear;
  function Clone: TSimpleDictionary;
  function ContainsKey(const Key: string): boolean;
  function IndexOfKey(const Key: string): integer;
  function IndexOfValue(const Value: string): integer;
  property Count: integer read GetCount;
  property Keys[index: integer]: string read GetKey write SetKey;
  property Values[const Name: string]: string read GetValue write SetValue; default;
  property ValueFromIndex[ index: integer ]: string read GetValueByIndex write SetValueByIndex;
end;

TSimpleDictionaryHelper = record helper for TSimpleDictionary
public
  function KeysCommaSeperated: string;
  function SemicolonDelimitedString: string;
  procedure LoadFromSemiColonDelimitedString( Text: string );
  procedure LoadFromStringList( StringList: TStringList );
  function Text: string;
  function ValueOrDefault( const Key: string; const Default: string ): string;
end;

Here is a simple subclass of Exception to allow Assertions on any Exception derived object. I tried to do this via a helper class, but the ‘self’ object in a class level helper called on a class does not appear to be a valid reference, so I couldn’t find a way to get it to work. Looking through the code to Indy inspired this approach to using Exceptions, I never use the raise statement anymore.

EBase = class( Exception )
  class procedure Assert( Test: boolean; const Msg: string );
  class procedure Throw( const Msg: string ); virtual;
end;

The last function is function ShowHourglass: IUnknown; This can be called in a UI function to make a form show a cursor, and when the function exits (normally or after an exception) be sure the cursor is returned to its previous state. The return value (IUnknwon) does not need to be stored, behind the scenes Delphi will store it anyway and make sure the compiler puts a hidden Try … Finally in at the end to make sure it is cleaned up. The cleaning up of the interface is what we use to reset the cursor.

Anyway hopefully this of some use to somebody. Happy New Year to everyone, here’s to a good year from CodeGear.

ATOM / RSS and Comments

David Glassborow — Tue, 17 Oct 2006 10:00:00 +0100

After all the effort of changing Weblogs, I’ve realised that Typo ‘out of the box’ does not support comments added to its syndication streams. I’ve hacked around in the code and managed to get something working. Rails XML helpers make this stuff pretty easy.

Weirdly there does not seem to be an agreed format for doing comments in ATOM at all, where as RSS has various different comment tags. I’ve used wfs:commentsRss and slash:comments nodes in both formats, with the comments just in Rss 2. The Feed Validator seems happy.

Anyway the following work well with my aggregator, Rss Bandit, as well as IE 7. If anyone has any problems with other aggregators, can you let me know. Thanks.

Local feeds:

Rss 2.0 (all)
Rss 2.0 (Just Delphi articles)
Atom 1.0 (all)
Atom 1.0 (Just Delphi articles)

Devco Wish List

David Glassborow — Mon, 16 Oct 2006 13:26:00 +0100

This is my wishlist for Delphi when the new company running it is planning the next version. My requests are not large things (except maybe the last one), and are in order of priority.

Summary:

Language construct: return
. operators on basic types
Include all source code (midas, dbExpress)
Ruby

Return

Readable concise code is something we all try to acheive when coding. I really miss the ‘return’ statement from C, because it makes code more concise. I’m bored of writing:

if blah then
begin
  Result := -1;
  exit;
end;

I want to be able to write just

if blah then Return(-1);

Shorter, and the intention of the statement is much more obvious. I would imagine it would be fairly easy to add to Delphi as a pseudo function (like ‘continue’ and ‘break’).

Dot Opeators

I’ve been doing a fair amount of Ruby recently, as well as some C#. Having the dot operator available on all basic types is lovely, and makes code much more readable. I’d much prefer

address.length

over

length(address)

Less visual clutter, much more OO. Even if behind the scenes is just syntactic sugar to call the length procedure, that would be a huge win for the readability of Object Pascal in these Object Orientated times.

Source Code

At Concept First we currently use the following third party libraries:

Devexpress: Quantim Grid, Quantum Bars
TChart charting component
CoreLabs dbExpress driver for SQL Server

We will only buy a component if it comes with full source code. Why ?

Fixing bugs when we find them
Adding new features
Tracking their changes under source control

We’ve made changes to all of the components we use, and I have all their source code under Subversion, so I can see what they have changed and see what possible impact it could have on my code.

One of the great strengths of Delphi has always been that the VCL comes with full sources. The two exceptions are the TClientDataset (midaslib.pas) and the dbExpress drivers. Its very annoying not to be able to debug into their code. I know Midas is in C++, but I have the full BDS with C++ support so that is not an issue. I use TClientDataset all over the place, and there are a few easy extensions that could make life much easier.

Ruby

As mentioned before, i’ve been playing around with Ruby and Ruby on Rails recently, and the first thing I had to do with is knock up a quick syntax highlighting editor using the SynEdit control. Having Ruby (and in an ideal world Javascript) in Delphi, with code complete, would be great, something I would be happy to pay for). Web development seems to be the way of the future, and having a nice IDE supporting Ruby could be a big win for Delphi / Devco.

Summary

I don’t know if my use of Delphi is similar to the majority of other users, but my priorities from it are:

Fast to develop in
Looks nice (generally easy thanks to DevExpress)
Source code so I don’t get held up by bugs
Reliable and fast IDE
Small self contained EXEs, without frameworks to install.

I use delphi for Win32 client applications. I don’t use it for web applications. I now use either Ruby on Rails ( my personal choice ) or Cold Fusion ( trusted in Enterprise environments ).

I’m not interested in:

Delphi.net
64 Bit support
Mobile development (be nice if Delphi supported, but such a small % of our work I’d be happy enough to use Visual Studio).

Anyway if I could have the requested changes made by the end of next week, that would be great DevCo ;-)

Pretty Please ?

Weblog moved to Typo

David Glassborow — Sat, 14 Oct 2006 11:37:00 +0100

What

I’ve moved my blog away from Blogger.com and on to a Typo installation hosted ourselves. Typo is a weblog application that runs on top of Ruby On Rails, the latest posterchild of the open source community. I chose a rails based blog because I’ve been doing work with rails recently, I understand it, and I really like it (I’m half way through writing up an entry about why I like it).

Why

The main reason is that blogger feed does not include comments people make. I personally read most blogs in an aggregator, and without comments being included, its only one half of the conversation.

For the last year we have been working on a product for a client that uses Queuing Theory to model how servers operate under load, an area known as ‘Capacity Management’. Performance agents log ongoing behaviour on the server, and then the results are plugged into various mathmatical formulae that show how the server will work if your email doubles, your websites have 10 times as much load, etc.

Here at Concept First we have a data centre hosted server that handles email, ftp, and various client’s websites. In the spirit of ‘dog-fooding’, I’ve been running monitoring software on our hosted server, and building models of its spare capacity.

The results are: we have stupid amounts of spare capacity. The server is a 1U rack mounted server from Dell, 1G of RAM, mirrored RAID disks, a single 2Ghz processor. A fairly low spec machine these days, we bought it about 3 years ago. And basically it never does anything. I’ve never seen the CPU loads rise about 5%, same for disk utilisation. Modern hardware is so fast, that a few web requests and a bit of email are not not giving it much of a workout.

For that reason, I’m happy to run an interpreted language framework for this blog, because we really do have CPU cycles to waste, and I don’t have time to waste. Having a Rails app running on my server full time will also allow me to understand if it’s stable under Windows.

How

Style

Typo is easy to install if you’ve worked with Rails, its just a load of files in a directory. You can choose from 2 built in themes or download lots of others. I’ve taken one of the built in ones and just changed it to make use of the full screen.

I hate fixed width websites. What is the point of my lovely 1920 wide screen laptop if the site is only 500 pixels wide !! Adjusting to the display has always been a key strength of HTML. Too many people are still designing for the web as though it was a piece of paper !

However the move away from fixed width has caused a small bug to display in IE 6, when using the search box in the top right hand corner. There is flickering as the AJAX call is done. However the issue does not occur in IE 7 or Firefox so it is no doubt one of the boatload of IE 6 rendering bugs, so i’m not going to pursuit it. Life is too short.

Database

Typo supports just about every database, but I’ve just left it running on a simple SQLite database. Typo caches everything so i don’t really need the hassle of putting it in a real RDBMS. The server is already running SQL Server, MySQL and Oracle XE. Maintaining what apps use what databases, where, is a bit of a headache. A file based database is just easier to manage if you can get away with it.

Webserver

Our server is running, or has run in the past, various different websites running on lots of different technologies: ASP, Websnap, Coldfusion, Zope, Php, Perl. To manage all this, we use Apache 2 running on Windows Server 2000. Apache has been good to us. Its very flexible, well documented, secure, and scalable. It is our Internet facing webserver. It handles SSL, compression (via mod_deflate) and virtual hosting.

To connect our hosted sites up to the Apache ‘gateway’ we’ve tried various things. We’ve tried Apache modules, ISAPI, FastCGI, SCGI, mod_php.

Apache modules / ISAPI: Nice and fast, but too tightly bound. When Apache versions change a recompile is required. Restart the server and you lose server state, like current sessions. Bugs can jeapodise the server stability.

FastCGI: Works well, we have various websnap sites running over FastCGI. However the protocol is complex, binary, and the whole FastCGI protocol is not actively developed any longer.

SCGI: A simpler version of FastCGI, I’ve run a couple of Rails sites on SCGI. The protocol is easier to understand, but its another set of config to learn, and not all technologies support SCGI.

With this in mind, we now use simple HTTP proxing. Each different application runs its own webserver on an internal port number, the firewall preventing any access to the outside world. Apache then uses its mod_proxy settings to pass the request to the applications webserver, before passing it back to the web browser.

Advantages:

Debugging is simple. Just fire up a copy of Fiddler.
All applications support SSL, Apache handles it.
All applications support compression, Apache handles it.
All application logged in same format, I can use the same tools.
Apache can use its redirect rules to enforce my rules (e.g. admin in Typo must be secure).
Upgrading any individual application has no effect on the others, everything is loosly coupled.
Most importantly: One set of syntax for me to remember.

It all works beautifully. It’s happily proxing to Rails, to web services exposed in Indy components. It’s fast, and it ‘Just Works’, not often you get to say that in the world of computing …

GIS

David Glassborow — Tue, 23 May 2006 16:09:00 +0100

I’m writing this posting for two reasons. Firstly my company is currently engaged in a project with a client about getting more of their data onto their intranet based mapping tool, so I’ve spent the last 3 days researching and getting back up to speed on the industry.

Secondly my brother has just got a new job working as sales manager for a company providing GIS data, so I thought an introduction and overview of the subject might be of use to him. Apologises if I get too technical Mike.

Geographic Information Systems

A GIS is a system the stores and displays maps electronically. Its a smart tool that helps us understand the world around us better, to get our work done more efficiently. I’ve been interested in GIS since one of my first jobs where I was in charge of writing a routing and optimising algorithm for solving a variant of the Travelling Sales Problem. Essentially our product was in charge of figuring out routes for workman to get their work down in the most efficient way possible, and send them routes on their mobile devices so they didn’t get lost. It was really interesting work, we worked with providers of GIS data about the road network, and with the UK goverment for access to electronic versions of their maps.

I think part of my interest also comes from travelling. When I first go to a new city, I’m always keen to find a map, to give me a feel of the layout. I always choose Lonely Planet over Rough Guide, because their maps are better. It was very strange when I was in Guyana in the Jungle, the only maps we had were from satelite images taken over 10 years ago, at a very low resolution. Most of the time we only have a vague idea of where we were at any time. Rivercourses flood, move, the tree cover prevents identification of key features. It made me realise how much we take accuracte maps for granted.

GIS is everywhere at the moment. Tom Tom are advertising their navigation products on TV. Google Local can show me a map of my local city in the time it takes to click a mouse, and Google Earth can let me zoom in from space to see my parents house, the roof of my dad’s landrover. NASA can let me look at the surface of Venus.

Raster and Vector

The most important division in GIS to understand is the different between Raster and Vector information. Raster information is pictures, its pixels that show us what our cities look like from space. Its the images that digital cameras give us. Its resolution dependant. Vector information is mathmatical in nature, its descriptions of things in terms of lines, circles, polygons.

Google earth is predominaly raster based, as you zoom in loads in images with more detail.

Google Local is vector based, it shows you your local highstreet as a series of straight lines, it shows you the nearby river as a polygon.

The real power comes when we use raster and vector together. When we turn on the country boundaries in NASA’s World Wind we seem them overlayed over the raster images of earth:

Raster information is an approximation to the real world, its pictures of the world. Vectors are mathmatical, they are ‘idealised’ versions of the world. The mathamtical nature of vectors allow us to use maths to answer questions like: “How many roads are there within 2 miles of the river ?”. Or this more relevant example “Show me all the pubs within 2 miles of where I live” in Mappoint 2004:

Raster examples:

Pictures of landscapes, from space or from aircraft
Pictures of clouds
Infrared pictures showing heat usage

Vector examples:

Countries, Regions, Postal boundaries
Streets, Motorways, Garages
Wind speeds, wave heights

Organisations

OpenGIS

The main organisation dealing with GIS standards appears to be the “Open Geospatial Consortium”, a group of 300 organisation worldwide. It has prepared various specifications for open standards:

GML: A geographical data format based on XML
Simple Features (SFSQL): A standard way to store data within standard SQL Databases
Various web interfaces for providing both vector and raster information in agreed formats (GML)

Companies

The leaders in the industry for GIS software seem to be ESRI, Mapinfo and Integraph. The two largest data provders are NAVTEQ, used by Google maps, and TeleAtlas.

Representation

An important thing to understand in GIS is that there is no universal ‘best’ representation for coordinates on Earth. The earth is not an exact sphere, its not flat, so various different schemes have been invented. Which is used is, as ever, is dependant on what your trying to achieve.

Longitude and Latitude

The standard way of representing points on the earth is obviously using long and lat. Longitude is the lines from the North Pole to the South Pole, with 0 degrees going through Greenwich, London. Latitude is the lines around the ‘waist’ of the earth, with 0 degrees at the equator, + 90 degrees being the north pole.

Longitude and Latitude are measured against a known datum. This is the reference model used, i.e. the mathematical figures used for calculating the best fit shape of the Earth. The most common currently in use is the WGS84 datum (WGS stands for ‘World Geodetic System’). It is possible to convert from one datum to another, but this can require some quite complex mathematics.

GPS

Global Positioning System is where alot of GIS data is coming from. It is a set of satelites flying through space, transmitting signals allowing the calculation of very accurate position. GPS figures are long/lat figures based on the WGS84 datum.

Grid References

Rather than capture coordinates as spherical coordinates (like long/lat), coordinates can be stored as a reference on a grid. As the Earth is obviously not flat, this is an approximation, but can be easier to deal with when mapping smaller areas. The UK uses a grid reference system for its ‘Ordance Survey’ (OS) maps. In this coordinate system, points are measured in Eastings and Northings. This grid is based on a different datum, OSGB36, based on a slightly different elipsoid representation of the Earth. The ‘Helmert’ datum transformation can be used to convert to WGS84, but with an approximate error of 7m (depending on location). The image below shows where the eastings and northings are measured from. The letters in the grids allow another way of specifying location, called National Grid References (NGRs), with further digits specifying easting and northing offsets within each 100km grid square. More digits give more accuracy or specify a smaller area. An example of an NGR is ‘NN 166 712’. This gives a 100m square area in Scotland. The good thing about NGS’s is that they are useful shorthand to give approximate position, or for capturing an ‘area’.

The UK goverment can provide digitial maps of the country in GML, with currently 440 million distinct features taking up 660 Gigabytes of storage. The data is called OS Mastermap

Formats

GML and SVG

GML and SVG are both XML based standards that are used within GIS. GML stands for Geography Markup Lanaguage, and is a format designed to hold geographical features, geometry (vector), raster information, as well as units, coordinate representation systems, and map styles.

SVG is actually just a generic format for holding vector graphic information, and stands for Scalable Vector Graphics. Its like an Open Standard version of Adobe’s Flash files, and supports both raster and vector information of any kind, not just geographic info. The customer we are doing the GIS work for currently uses SVG for their display of GIS data inside a webbrowser, using Adobe’s free SVG plugin for Internet Explorer (note: firefox supports a subset of SVG natively now). SVG has well defined Javascript bindings so its easy to create interactive displays.

To me GML seems a better storage format and SVG a better display format.

Other

There are various file formats in use within the industry. Although the OpenGIS is pushing GML, most data seems to be available in either ESRI’s shape format, or Mapinfo’s standard file format. Both are proprietary, but easy to parse formats that most tools support.

Databases

There seems to be a big push to moving ‘spatial’ capabilities into databases at the moment. This makes sense as really geometry is just another attribute of an entity. Moving the geometry into database tables allows this data to be kept alongside other attributes, as well as allowing databases to carry out geospatial queries.

The market leader in geospatial databases is Oracle. All versions of their database support what they call Locator features. This allows the storages of complex vectors into tables, as well as support for complex queries (e.g. find me all objects that lie within the boundary of this polygon, find me there nearest 5 customers to our store). The top end product also has spatial features that allow very complex geometric calculations to be done. The nice thing is that the free Oracle Express (10g at the moment) supports the locator features. IBM’s DB2 seems to have some sort of spatial extender but I couldn’t find any information on the web about it.

Some of the GIS software companies provide software to help with database storage, like ESRI’s ArcSDE, but these are just ‘layers’ that sit between software and the database, and are not open formats, tying you into using a particular vendor.

Two of the open source database engines have recently added spatial features to their engines. Postgres SQL has the addon PostGIS that supports the SFSQL specification mentioned above (as does Oracle). MySQL also has spatial extensions based on the SFSQL spec, but the current version (5.0) seems only to support planar (flat) and not spherical coordinate systems. The mathematics used for distances, areas, etc. are different for planar and spherical representations.

It was hoped that Microsoft SQL Server would add some spatial features to the new 2005 release, but this was not the case. There is however an interesting MSDN article showing how spatial features can be added to SQL Server, but this does not support open standards (surprise surprise).

Spatial Indexes

When reading about GIS formats, one of the key issues is how the formats are ‘indexed’ to make retrieval efficient. Oracle uses Helical Hyperspatial Codes (HHCodes), a data structure developed by the Canadian Hydrograph office. Dispite the fancy name, it seems to be a binary tree, dividing each dimension in order. Its designed to support as many dimensions as desired, to an arbitray level of detail. The MSDN article discusses the use of Hierarchical Triangular Meshes, which is a technique of covering a globe with a mesh of subdivided triangles. These seems work well for spherical mapping, but does not seem as general case as HHCodes. PostGIS and MySQL seems to use extensions of standard database indexes, rather than specific spatial indexing.

Summary

The GIS industry is becoming more and more important, as developers I can see us having to deal with spatial information more and more. If i get time I’ll try and write an article on how to access SFSQL spatial data from within Delphi. For more information see Wikipedia’s entry.

Comments (originally on blogger.com)

Paul Ramsey said…

PostGIS and MySQL use R-Tree indexes, which are rather specifically for spatial data (well, any data type which can be decomposed to a range rectangle). So does Oracle Spatial, for that matter. Space-filling curves, which map multi-dimensional data to something sortable into a B-Tree are ways of leveraging non-spatial database technology to be more useful for spatial data. Space-filling technicques tend not to be as balanced as R-Trees, however. The heirarchical grid is the final trick, and can be quite performant, but needs to be carefully tuned to the characteristics of the data, unlike R-Trees which automagically work for data of uneven scales.

Class RTTI

David Glassborow — Mon, 22 May 2006 16:09:00 +0100

This post follows up my previous one about RTTI in Delphi, inspired by Hallvard’s 2 posts here and here, and covers some advanced RTTI features in Delphi that I haven’t seen mentioned anywhere else.

$METHODINFO

While playing around with Websnap in Delphi, trying to extend some of the objects available for scripting, I came across the compiler directive METHODINFO.

The online documentation says:

The $METHODINFO switch directive is only effective when runtime type information (RTTI) has been turned on with the {$TYPEINFO ON} switch. In the {$TYPEINFO ON} state, the $METHODINFO directive controls the generation of more detailed method descriptors in the RTTI for methods in an interface. Though {$TYPEINFO ON} will cause some RTTI to be generated for published methods, the level of information is limited. The $METHODINFO directive generates much more detailed (and much larger) RTTI for methods, which describes how the parameters of the method should be passed on the stack and/or in registers. There is seldom, if ever, any need for an application to directly use the $METHODINFO compiler switch. The method information adds considerable size to the executable file, and is not recommended for general use.

My previous article showed this isn’t completely accurate, detailed RTTI is available for any Interface which has $TYPEINFO or $M around it. $METHODINFO seems to affect classes, in particular it will store detailed RTTI information for not only Published methods, but also Public ones.

Doing a search for this compiler directive in the delphi win32 source code gives us only 1 instance in WebSnapObjs.pas.

{$METHODINFO ON}
TScriptableObject = class(TObjectDispatch)
private
  FLookupList: TStringList;
  FLookupValues: TInterfaceList;
protected
  FPreferChild: Boolean;
  function DispatchOfName(const AName: string): IDispatch; virtual;
  function FindObject(const AName: string): TObject; virtual;
public
  constructor Create;
  destructor Destroy; override;
  class function DispatchOfObject(const AObject: TObject): IDispatch;
  function GetIDsOfNames(const IID: TGUID; Names: Pointer;
    NameCount: Integer; LocaleID: Integer; DispIDs: Pointer): HRESULT;
    override;
  function Invoke(DispID: Integer; const IID: TGUID; LocaleID: Integer;
    Flags: Word; var Params; VarResult: Pointer; ExcepInfo: Pointer;
    ArgErr: Pointer): HRESULT; override;
end;
{$METHODINFO OFF}

Websnap

Websnap is the poor cousin in the web framework world for delphi. Its never had much support, and seems now to be overshadow by ASP.net and Intraweb. I personally quite like it, although I code my own templates in VBScript or JavaScript rather than use any of the Design Time webpage design stuff.

Under the hood, websnap uses the ActiveScript engines provided in Windows. ActiveScript is a scripting host that can support many different COM based scripting languages, and Windows comes with VBScript and JScript (which is basically JavaScript). Other ActiveScript lanaguages are avaialbe including Python and Perl.

The original ASP by Microsoft uses the ActiveScripting engine to do its work. The asp template is turned into a vBScript or JScript program containing the HTML to output as well as the logic of the page. This is fed into the ActiveScripting engine and compiled ready for running. The ActiveScripting engine then has ‘objects’ added to it so the program can do useful work. The most obvious one is the Response object, but there are others like the Session object, etc. The program is then run and the page rendered.

Websnap pages, at least those using a TPageProducer, use this same process to produce HTML pages. The problem for the Delphi deisgners was how to link arbitary Delphi objects up to the ActiveScripting engine, which uses late bound IDispatch COM for communication. The IDispatch interface, one of the main underpining of the COM framework in Windows, uses a single call, Invoke for all method calls. This is where $METHODINFO comes it, the rich method RTTI is provided to allow a single procedure entry, Invoke, to call arbitary Delphi methods.

The VBScript or Javascript script running in the scripting of the websnap page needs to talk to Delphi objects (e.g. Page, Session), and it uses this Rich RTTI to acheive this. You can see the websnap objects that are exposed to the script, have a look in WebSnapObjs.pas, where TResponseObj, TProducerObj, etc.

The unit ObjAuto contains the code and header for retrieving the RTTI information using the following function:

function GetMethodInfo(Instance: TObject; const MethodName: ShortString): PMethodInfoHeader;

In turn, the base class of TScriptableObject (marked with $METHODINFO) uses the RTTI to find methods, and call them, at run time.

ObjAuto.pas

This contains the code to search for a method’s RTTI. Looking at GetMethodInfo, you can see it uses the system.pas vmtMethodTable offset to get hold the method table for the class. It then uses a search to find the correct entry. It also contains the code that allows an arbitary call to an object supporting RTTI to jump to the correct routine:

function ObjectInvoke(Instance: TObject; MethodHeader: PMethodInfoHeader;
  const ParamIndexes: array of Integer; const Params: array of Variant): Variant;

As you can see you just pass it parameters and variants, and it packages them into the correct types and does the call. The source code to this call shows all the complexity of packaging up all the parameters according different conventions, etc. This is ultimately how VBScript objects call methods on Delphi objects inside Websnap.

DetailedRTTI.pas

While playing with the metadata, I coded a few helper classes to aid exploration. You can download the code if you want to have somewhere to start. Just calling .RTTIMethodsAsString() on any object to get a list of its methods and their parameters. Its a bit rough and ready but you’re welcome to use it for whatever.

Summary

This article, and the previous one have shown that rich metadata for methods is available in Delphi, with supporting routines for accessing it. Interface metadata allows the VCL to support SOAP, multiple methods multiplexed to a single call. The rich class metadata allows the VCL to support a single function automatically being routed to other methods, allowing Websnap to expose objects to COM IDispatch automatically.

Comments (originally on blogger.com)

Hallvard Vassbotn said…

Great posts, David!

I reference them here

Interface RTTI

David Glassborow — Thu, 11 May 2006 16:07:00 +0100

Reading an article and its follow up by Hallvard about RTTI inspired me to put together a couple of posts about two related areas of RTTI in Delphi. In particular one of the comments on Hallvard’s blog about using this RTTI to call objects in some late bound fashion. This post and the next cover some of the advanced RTTI that I haven’t seen covered in other places. This post covers some of the possibilities for Interface metadata, and the next one will contain details about richer class RTTI for methods.

Interface Metadata

Delphi actually has richer metadata support for methods in an Interface that in a normal class. It looks like this was added to support the SOAP features of the VCL. I’m not sure which version of Delphi it was added so your mileage may vary if your not using 2006.

IInvokable

To use SOAP, you use a WDSL file to specify the method calls, parameters, etc. If you import a WSDL in Delphi, you will notice that all Interfaces in the generated file will be derived from IInvokable. A quick peak in the System unit will show that IInvokable is:

{$M+}
  IInvokable = interface(IInterface)
  end;
{$M-}

I.e. just a standard interface, but with RTTI metadata compiled in.

Looking at the help in BDS 2006 for {$TYPEINFO ON} mentions this:

Note: The IInvokable interface defined in the System unit is declared in the {$M+} state, so any interface derived from IInvokable will have RTTI generated. The routines in the IntfInfo unit can be used to retrieved the RTTI.

IntfInfo.pas

The main procedure of interest in IntIfnfo is:

procedure GetIntfMetaData(Info: PTypeInfo; var IntfMD: TIntfMetaData; IncludeAllAncMethods: Boolean = False);

This will give us a series of records describing the methods on the interface and the parameters needed for these interfaces, as well as the unit it was defined within, the ancestor Interface and the interface’s GUID. All the names are available. both function / procedures and the names of their parameters. Calling this procedure with an interface not having RTTI will raise an exception, calling it with a class’s typeinfo will just cause an a/v :-)

When doing SOAP calls, the developer just uses the defined interface like a normal interface. Behind the scenes, Delphi packages up the parameters and sends them via a SOAP envelope to the remote server. How Delphi does this shows us some of the potential of this RTTI in Delphi, and respect for the Voodoo that is TRIO.

RIO.pas

Located in the soap folder of Delphi’s source code, RIO.pas contains the class TRIO. TRIO is an object that represents a remote object, presumably it stands for Remote Interfaced Object.

When an application casts a TRIO descendant to a registered invokable interface, it dynamically generates an in-memory method table, providing an implementation to that invokable interface.

Looking at the source for TRIO, I’ve come to the conclusion that:

MyRioObject as IMyInvokableInterface

Will cause the TRio object to

Get the meta data for IMyInvokableInterface (from a registry InvRegistry object defined in InvokeRegistry.pas)
Allocate memory for a vtable for the interface
Allocate memory for ‘stub’ routines, marks it as containing executable code
Writes machine code stubs that takes the parameters and packages them up, then calls TRIO.Generic

This is a very crude representation I knocked up in Visio:

When you then make a call on the ‘generated’ interface, Delphi calls the vtable, the vtables holds the address of the generated machine code. The generated machine code pushes the parameters then calls the Generic function. This packages up the parameters, and then uses a SOAP call to call the remote service. The return is then packaged up and returned in a similar way, back through the generated stub. If you are interested in how the actual machine code is generated (taking into account the 5 different calling conventions, etc.) take a look at TRIO.GenVTable function.

I don’t know which of the Delphi team wrote this code, but its very very impressive.

Anwyay I hope this has given you a feel for some of the advanced metadata available with Interfaces. The RIO approach would allow you to write Interface proxies of any Interface with metadata, for security, logging and indeed other forms of RPC remoting. Let me know if anybody suceeds in such a thing !

My followup article on class RTTI.

Class Helpers - good or bad ?

David Glassborow — Mon, 08 May 2006 16:01:00 +0100

One of the things I love about the latest versions of Delphi is the compiler changes, originally made for .Net, that are being rolled back into Win32 code. I like finally having procedures in records, and I love class helpers.

Class Helpers

Helper classes were introduced in Delphi 8 as a way of binding the VCL to the .Net framework. To quote the Delphi help: “Class helpers are a way to extend a class without using inheritance. A class helper simply introduces a wider scope for the compiler to use when resolving identifiers.”

Very simply, they allow you to add your own code to existing objects without requiring the source code or recompiling. The new code only has public access to the original object, so it cannot access private or protected data.

A simple example from my library code:

TStreamHelper = class helper for TStream
public
  function AsString: string;
  procedure WriteString( const Text: string );
  procedure WriteStringAndLineBreak( const Text: string );
end;

I got bored of writing the same code for writing a string to a stream, so I put it into a class helper, and because TStream is a base class, it works for TFileStreams, TBlobStreams, whatever.

Really a class helper is a compiler trick, behind the scenes it is just functions operating on the class, but it doesn’t polute the name space, and works with intellisense. I find myself using them more and more. The other day I was writing some code to do a gradient fill for a custom background. Very quickly the code became a helper function on the TCanvas object. Some more examples from my company’s library code:

TCanvasHelper =  class helper for TCanvas
  procedure FillGradient( Bounds: TRect; StartColour, EndColour: TColor; IsHorizontal: boolean );
end;

TListHelper = class helper for TList
  procedure FreeSelfAndContainedObjects;
end;

TDatasetHelper = class helper for TDataSet
public
  procedure InsertIntoStrings( Strings: TStrings; NameField: string; IndexField: string = '' );
  function HaveFieldsChanged: boolean;
  procedure PostIfEditing;
end;

TFieldHelper = class helper for TField
public
  function HasChanged: boolean;
end;

TTreeNodesHelper = class helper for TTreeNodes
public
  procedure ExpandToLevel( Level: integer );
  function FindOrCreateNode( NodePath: string ): TTreeNode;
end;

TRectHelper = record helper for TRect
  function ContainsPoint( Point: TPoint ): boolean;
end;

Note: the last example shows that, at least in BDS 2006, you can also write helpers for records.

Bad Design

Reading the rest of the Borland help on class helpers, you come to this statement:

Class helpers provide a way to extend a class, but they should not be viewed as a design tool to be used when developing new code. They should be used solely for their intended purpose, which is language and platform RTL binding.

I don’t agree with this, they are far more useful than just a trick for “language and platform RTL binding”

Humane Interfaces

Recently I was reading some content on Martin Fowler’s website, a site that always has an interesting perspective on software development. One of his articles that struck a cord was on “Humane vs Minimalist” interfaces.

To briefly summarise his article, he talks about the different attitude between Ruby developers, who favour rich easy to use interfaces, compared to the Java crowd who tend to favour minimal interfaces. The example he gives is of a list class and how to get the last item:

Java:

aList.get(aList.size -1)

Ruby:

anArray.last

The Ruby interface has 78 methods, the Java one 25. However the Ruby one is clearly more readable, with less visual clutter.

At Concept First our 2 overriding rules for writing code are:

It should be as human readable as possible
It should be as concise as possible

For this reason I favour the humane interfaces. However writing minimalist interfaces I feel leads to more concise design and less easier to test. It seems to me that class helpers give us the best of both worlds. Objects can be designed with minimal interfaces, making them quicker to develop and easier to test. We can then use class helpers to make the interfaces more humane, more readable, easier to use. The fact that class helpers can only use public methods of the object also makes sure the minimalist interface is complete.

I started off using Class Helpers to add library code to VCL objects, for library code. I now find myself using them as a design decision, putting minimalist interfaces on my objects, and adding all the helper and nicety functions as helper objects. As Marco Cantu points out, it looks like the next version of C# will support a similar feature. Personally I feel its another great feature of Delphi giving us a competitive advantage when developing with Object Pascal.

So what does everybody else think, design feature or abomination ?

Comments (originally on blogger.com)

Anonymous said…

Well, Anders Hejlsberg would seem to agree with you that they, or at least the idea of being able to extend a class, is a good thing to expose as part of the language. See http://channel9.msdn.com/Showpost.aspx?postid=114680 and http://mtaulty.com/blog/(zu0cer45karubu55u0cqsa45)/archive/2006/03/20/9271.aspx

Specificl look at Extension methods. AKA Class Helpers in Delphi.

Anonymous said…

Brilliant article

Daniel said…

Hi, David! Great article! I personally use class helpers to hide implementation details of some code on the server side of my app, while still publishing classes to the client side. Specially in our SOAP application, when we want our clients to use the same class definition as in the server. Looking forward to seeing your article about REST x SOAP…

TOndrej said…

My gut feeling is that what the help says about class helpers is correct, ie. they shouldn’t be a part of new design. For various reasons, already written classes become sealed, ie. their interface cannot be changed anymore. An ideally written class would provide all the functionality specified in the design phase, and expose enough information for descendants to extend it as new requirements come. In the real world, however, classes often hide too much or, if you like, do not expose enough. That’s where I think class helpers are a big help, they’re a way out if you’ve locked yourself in. Your argument for writing minimal classes and, at the same time, more humane class helpers seems to be based on the fact that minimal classes are easier to test and debug. I agree, but don’t you have to test and debug the class helpers, too? Isn’t the effort the same or even greater after all? Why not write, test and debug the full class right from the beginning?

The above is just my 2c; I haven’t used class helpers yet at all. Perhaps I have to play with them a bit and get a better feeling about them first. P.S. A very nice blog so far, cheers!

David Glassborow said…

Tondrej I think my point is that testing a class helper is much much easier because it can only affect the public interface, not the internals of the class. If your written the minimalist class properly, its very unlikely the class helper can break it.

TOndrej said…

OK, that’s an interesting point. I guess I’ll take a closer look at those helper beasts ;-)

Anonymous said…

The limitations seem (no private field access) seem to prevent a lot of goodies. What probably happens is that if you use type X, all imported units in the interface are searched for helperforx, since this is the only way to avoid to scan all units in the entire project for helpers to compile a unit.

Introduction

David Glassborow — Sun, 07 May 2006 15:58:00 +0100

Why Blog

Because:

There are lots of technical details about delphi I’ve explored that may be useful to other developers. I’ve learnt loads from the Delphi community and I’d like to return the favour
There are lots of design ‘discussions’ (i.e. arguments) i’ve had with other developers, especially my busness partner, and I’m interested in the opinions of other people on the issues
To quote Paul Graham: “Expressing ideas helps to form them”

I’ve been meaning to write a blog, and host it, for a while, but haven’t got around to writing a web application to host it (although I’m still looking at doing it in RoR just to learn what all the fuss is about. So I’ve decided to host this on blogger for the moment, and maybe move it at some future point.

I’m going to write all the posts in markdown because thats the format we use internally at work, and its very easy to work with. If anybody knows any groovy tools that can markup delphi code into a web blogger friendly format, I’d really appreciate it.

In development, my prefered tool is Delphi, for lots of reasons. The greatest thing about it for me is that it comes with all the source code. My coding quality has improved more from reading the Delphi sources than anything else.

I live in the south west of England, in the UK. I live here because of the quality of life, especially being close tp the coast. In my spare time I love surfing, running, travelling, and going on expeditions to far away place like Guyana and West Papua.

Concept First

I work as a technical consultant for a small software company I started with my business partner, Dan Jones, called Concept First Ltd.

We are in our fifth year of trading, and do various types of work:

Business Process mapping and development
Web sites in ASP, Coldfusion, Websnap
Application development in Delphi (by choice), VB6, VB.Net, C# (by request)
Anything else that pays the bills !

A large number of our projects are integrating data in Enterprise environments (i.e. lots of random technologies chosen for reasons other than technical, fighting with vendors to get access to data, dispairing at money being wasted left right and centre …)

Technologies

At Concept First we like and use:

Delphi
DevExpress grids
Apache 2
Subversion
SQL Server (especially MSDE/Express editions)
Anything to do with geospatial data and mapping
Mind mapping software for writing specs
Bulleted lists ;-)

We dislike:

Analyists who can’t develop
Developers who can’t analyse
The new hoops you have to jump through to be a Microsoft Certified Partner company
Tying ourselves to Microsoft technologies
How bad most enterprise software is
How fickle the software industry is for the latest silver bullet
Fixed width webpages and tables for layout

Articles

I’ve got various ideas I want to write up into posts, to be done when I get time/motiviation:

Rich RTTI in delphi objects
Delphi’s class helpers, good or bad OO design ?
Websnap: good and bad points, how to extend it
Websnap and FastCGI
Null Object pattern in databases vs. explicit NULLs
Case sensitive languages and why they must be destroyed
Functionality vs Complexity - Time vs Money vs Quality
Business Logic in databases
REST vs SOAP
Tips to prevent premature optimisation

I also need to convince my business partner, Dan, to sort out a blog, coz I’m bored of him preaching to the converted.

Expect some rantings on ‘semanctic markup in html’ asap :-)