The Future of Database Application Development

I have been searching for a long time for an application development environment that was truly simple to use and helped me manage the application life cycle. In my opinion, nothing has come close to this yet.

Here is why I think we might be getting closer to Application Development Nirvana.

First off I should say that I build database applications, so I am mostly interested in something I call a Rich Data Driven Application. These are applications that are data centric but still have rich interactive interfaces. They are also almost always used by many people spread out all over the place. This discussion focuses on this type of application.

An application usually starts with a spark of an an idea, and it evolves. It is rarely designed. Large applications are probably going to be worked on by many different developers, over a long period of time. A GREAT application development environment must embrace change, both in requirements and in developers.

Here are some of features that I would like to see in my dream development environment.

1. Require little if any setup and configuration
2. Simple form creation
3. Rich fully interactive interface
4. Manage record locking and transactions
5. Easily accessible data store
6. Extensible with plugins
7. Code management tools, refactoring, version control.
8. Run on the web or the desktop. Lets face facts. The browser has won.

FileMaker is pretty darn good. You can do an awful lot very quickly. It is a great tool for whipping powerful data driven applications. It manages record locking and can even do transactions or batch processing. But its data store is proprietary, and non standard. Even though you can use ODBC almost nobody does because it doesn't really fit with the FileMaker model. And as for code management, forget it. It has none.

There are other tools out there but they either fall into the "easy to use" or "hard to use but very powerful" category. Take what Adobe is now calling the Flash Platform. It is very powerful, but it is not easy to use. Not even close. Replicating what FileMaker can do in a few hours would take months of time. It would also require very expensive server software or an expert to setup configure and maintain some of the open source projects that can help.

Projects and Rails and Django have proven that web applications can be built pretty easily. But they don't really offer record locking and rich two way interactivity is really tough to do. Despite the best efforts of many companies like Aptana, and Yahoo, using HTML/Javascript/AJAX to build interfaces remains a complicated mess.

There have been lots of different attempts create simple application tools but they often follow wildly different paths. This means that even though FileMaker solved the two way client server communication problem years ago, that solution could not be used with Flash on the front end, because they don't even speak the same language in this case a communication protocol. It works only for FileMaker Clients.

I think we haven't gotten then tools we want and need because not enough people have decided to solve common problems in a commonly accepted way. Once they do, we will be on our way to App Dev Nirvana.

I think there are three basic components to any rich data driven application.

1. transaction safe data store
2. A rich interface engines that allows the deployment of the interface to the platform of your choice
3. A two way connection between the two that supports all the features of both the client and data store.

Solutions have existed that provided some of these feature, but they are either expensive and proprietary, or open source and require serious expertise and lots of time to setup and configure. The problem was not that these problems haven't been solved, but rather that each attempt did it in a different way. This has the effect of creating islands of expertise that keeps things both expensive and complicated.

Another way to look at it is to think of it terms of concepts or patterns. Each attempt at solving these problems came up with a slightly different concept or pattern to apply to the problem. The actually technology used to implement the solution is important of course, but the concept is just as important.

So not only did we need time for the technology to evolve, we also needed the concepts or design patterns to evolve too. I think we are finally beginning to see a convergence on a set of concepts patterns and technologies that will provide abstracted solution to each of the three components of a Rich Data Driven application.

Lets look at each of three components I listed above.

First lets deal with the connection between the server and client. I know the least about this layer. But it seems that we have converged on HTTP as the protocol, and a few data formats. We have XML, JSON, and AMF as pretty good formats. The problem with HTTP is that it does not support server to client data push very well yet.

Today communication from server to browser requires either something like Comet or Flash or SilverLight in combination with server functionality. But the real solution is coming, HTML 5 and server side events.

Now on to the interface. The browser has won again. That isn't to say there won't be desktop applications for the foreseeable future, but more and more of them will simply be browser running in their own little runtime like Adobe AIR.

So the Browser and HTML have emerged as the dominant players in presenting your interface. Within the browser, interface engines like Flash and Sliverlight are already very powerful. They can even handle two way connectivity. My guess however is that the purely open HTML 5 / JavaScript solution will eventually win out, simply because it will not be vendor specific in any way.

On the server side things will probably remain pretty fragmented on the technology front. Different application servers will use different languages to provide the needed functionality. But the concepts and design patterns are beginning to coalesce. This is the most important thing. If each application server provides essentially the same features through common protocols and common design patterns then it really doesn't matter with the server uses Java, .Net, Python, Ruby or PHP.

Java and .Net are better suited right now, because they can more easily deliver the two way connectivity required by truly rich interfaces. But that won't last for too much longer. HTML 5 promises to change the game by essentially allowing the browser to respond to server events, making it much easier for servers based on Python, PHP, and Ruby to deliver this kind of two way connectivity.

But the key patterns emerging in the server side world have to do with abstracting the the data store into something that all of these server implementations and client implementations recognize. I the pattern known as ActiveRecord is beginning to win out. This pattern gained fame as fame as a key part of the Rails framework, and is now spreading like wildfire to other languages and frameworks. Just last week Aptana announce activerecord.js, a JavaScript Along the way it is picking up pieces of other patterns and frameworks that are making it even more powerful.

A great example of this convergence on ActiveRecord is being built by a company called MidenightCoders. Their WebORB presentation server supports SilverLight, Flash, or JavaScript on the client side and Java, .Net, PHP, Rails, and ColdFusion on the server side. At this stage, not all the server implementations support the complete feature set, but it is moving in that direction.

The WebORB Data Management Framework is based on ActiveRecord and a few other patterns. It provides full CRUD (Create, Retrieve Update, Delete) functionality for the data store. Depending on the server version you can also get cool stuff like changes from one client automatically propagated to all other connected clients and synchronization.

But a very significant thing about WebORB Data Management Framework (WDMF) is the abstraction level it provides. It is conceivable that you could start a project with .Net on the server, and Flash on the client, and then switch to Java on the server. It is so much that the switch would require minimal refactoring on the client, the key thing is that is using the SAME PATTERN to access the data store, so the overall design architecture would remain about the same. You are still dealing with ActiveRecord.

It looks to me like we are evolving towards commons solutions to the three components of an Rich Data Driven Application.

1. ActiveRecord = Data Store
2. Client = HTML 5 compliant browser, JavaScript ( with or without Flash/SilverLight)
3. Communication = HTTP (HTML 5) with XML/JSON/AMF

If these concepts become truly wide spread, then the tools will evolve that will make programming modern much easier then it is today. I am hopeful that is not too far off.