[vos-d] the state of VOS

[Peter Amstutz]

Good evening everyone.

I realize the mailing list has been extremely quite for several months, 
and combined with the approximately two weeks of downtime we experienced 
with interreality.org,one might be excused for thinking that VOS was 
dead.  I am happy to say that this is not the case.  However, a 
metaphore for the current project might be that it is currently 
cocooned, and undergoing a metamorphasis.

Specifically, I am currently engaged in a significant rewrite of the 
core runtime infrastructure, libvos.  I want to talk a little bit about 
the reasoning behind this, what there will be look forward to in the new 
core runtime, and what the project plans are for the medium term.

1. Although light years ahead of the VOS implementation it replaced 
("s3"), the current implementation (known as "s4") has started to 
encounter its own architechtural limitations and design flaws.  Some of 
the major issues we've encountered are:

 * confusing threading design whereby event callbacks are executed by 
arbitrary threads, placing a significant burden on the application 
developer

 * large memory footprint per vobject, sharply limiting the ability to 
create very large numbers of vobjects

 * no provisions for offline persistant caching

 * difficulty in designing scripting hooks that allow scripts to be on 
the same footing as native C++ code

 * identification of sites based on hostnames and/or IP addresses, 
leading to numerous problems with name aliasing and confusion


After giving these issues a great deal of thought, my conclusion was 
that these are ultimately fatal flaws in the design of the current 
implementation.  The best course of action was to start over from 
scratch, taking the knowledge gained from the previous attempts to craft 
a new core infrastructure that would be a giant leap forward in 
functionality.

It is very very important to note thatt none of these implementationn 
problems invalidate the core Vobject model!  It is still by firm belief 
that the basic, fundamental abstraction of message-sending actors with 
lists of links to other objects is the right one.  In that light, this 
redesign is not so much a radical change, as simply a refinement of the 
core ideas in light of further experience.


2. The goals of the new design are to provide robust solutions to the 
major problems outlined above, while maintaining the basic features of 
VOS that have made it successful.

Probably the greatest motivation for a new design was the desire to 
incorporate the ability to extend the system dynamically via scripting 
languages.  This presented a couple of problems: one problem was simply 
the inherent complexity of binding C++ interfaces into a target 
scripting languge.  The second was the a more philosophical problem.  
How do you decide what thread of control is responsible for handling a 
message?  When you start dealing with not just C++ code but other 
language runtime systems, it is no longer clear that simply executing a 
callback in whatever thread is convenient is desirable.

For the first problem, the solution was relatively straightforward.  The 
new libvos will offer a stripped down C API that is designed to be easy 
to bind to scripting languages.  Furthermore, the method interface to a 
Vobject type will now be specified in XML and wrapper/framework classes 
generated from the interface definition.  By writing a new module to the 
code generator, it will be straightforward to add support new languages.

For the second problem, the decision was to introduce a new restriction 
to the abstract Vobject model: an individual vobject now represents a 
single, independent thread of control.  A particular vobject is now only 
allowed to process a single message at a time.  Many vobjects may be 
executing simultaneously, but the model now states that a single 
specific vobject is only allowed a single thread of control.

Some of the other work includes ensuring that the actual vobject classes 
are designed with memory footprint in mind (something that was not part 
of the "just make it work" attitude of aspects of the s4 
implementation.)  The application is now no longer permitted to have 
direct pointers to the Vobject objects, but rather works through vobject 
id handles.  This makes it easier to shadow, swap and cache vobjects in 
the background, because it is no longer possible to invalidate a pointer 
held by an application.  Finally, sites will be uniquely identified 
using 128-bit public keys (using eliptic curve cryptography).  The 
connection process will include a challenge/response protocol to verify 
that a site owns the corresponding private key to go with its public 
identity.


3. This is going to take a while.  I don't have nearly as many hours in 
the week to dedicate to VOS as I would like, but I work on it when I 
can.  This is a major undertaking with a lot of pieces.  I have most of 
the design in mind already, it is mainly an issue of finding the time to 
grind through writing/testing/debugging the actual code.  (And of 
course, paper or mental designs have a way of running into all sorts of 
implementation issues when trying to express them in real code.)

Fortunately, one initiative I have taken with this new codebase is to 
build it from scratch using unit tests.  This will give a baseline 
assurance of code quality, which will hopefully reduce the time spent 
debugging in the long run.

My present plan is to make one more release based on the s4 codebase, 
this will be 0.24.  There are some important fixes and improvements in 
the development version that should be rolled out into an official 
release.  I don't yet have an estimated date for 0.24.  My goal is to 
first bring the s5 development branch to a state where it represents a 
minimal VOS implementation under the new design, then go ahead and 
switch gears to work on polishing up 0.24 for release.


4. It is probably fair to say that the history of VOS development has 
been characterized by the design, redesign and constant tinkering with 
the runtime, and as a result the full potential of the end-user 
applications (particularly the 3D world browser Ter'Angreal) has 
suffered for lack of attention.  On the other hand, nearly all of these 
work has been directly motivated by challenges that have come up in the 
design and implementation of the 3D client and 3D world server.  At the 
end of the day, fundamentally we are trying to solve a very complex 
problem.  Complex problems require complex solutions.

- Pete

[   Peter Amstutz  ][ tetron at interreality.org ][ peter.amstutz at gdit.com ]
[Lead Programmer][Interreality Project][Virtual Reality for the Internet]
[ VOS: Next Generation Internet Communication][ http://interreality.org ]
[ http://interreality.org/~tetron ][ pgpkey:  pgpkeys.mit.edu  18C21DF7 ]

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://www.interreality.org/pipermail/vos-d/attachments/20061003/9ad40661/attachment.pgp