Improve performance of DMI calls

Type: New Feature
Status: Closed
Priority: 2 Major
Resolution: Fixed
Component/s: DSO:L1
Labels:

Assignee: alex
Reporter: ssubbiah
Created: January 26, 2007
Votes: 1
Watchers: 1
Updated:
July 27, 2012
Resolved:
February 20, 2007

Description

Currently DMI internally uses DSO and creates a MAP of lists as a root and method invocation details are put into the lists. It uses 1 write lock to protect the data and also does some class resolution etc. This makes it slow.

I think we need to reimplement DMI to jsut use our internal transactions. This is how it works.

1) DMI is written to a transaction and sent to the server 2) Server retransmits it to all the clients (just like root creation or lock notification today) 3) Apply thread in the clients that receive it take it and adds it to a DMI local queue which then gets processed 4) Making DMI ride on transaction gives us restartability too

This is simple to implement as all the necessary parts are already built and as it involves no locking this will improve DMI performance greatly.

Comments

Eugene Kuleshov 2007-01-26

Saravanan, would it be hard to use thread pool instead of single apply thread? And perhaps make pool size configurable.

Saravanan Subbiah 2007-01-26

Are you talking about “receive_transaction_thread” or the DMI thread ?

receive_transaction_thread is single threaded to ensure transaction sequence (at least for now)

DMI thread can be multithreaded easily if we dont care about ordering.

Greg Wilkins 2007-01-26

HI - I’m very new to terracotta - so I am probably talking bollocks…. but I see a great need to extend the DMI mechanism.

I have recently used terracotta to cluster the cometd protocol, which as well as a mechanism for distributing data, it needs distributed events.

The specific use case is clients subscribed to communication channels in a publish/subscribe situation.

The object graph that is clustered represents the channels, the clients, the subscriptions and the queues of messages waiting to be delivered. Terracotta was perfect for that.

However, this is all done in an asynchronous environment without a thread allocated to each client and thus without any thread waiting to wake up when messages are added to a queue. I have thus used DMI to distribute a Client.resume() method to wake up the distributed client object . While this works it has several big problems.

Firstly it is slow and has latencies of about 300ms before a local call to resume (from within a synchronized block that already holds the Client lock) and 250ms to another node.

Secondly, the DMI call is distributed to ALL NODES!!! Ideally the client object will only exist in the node of the publisher and the node of the subscriber. Broadcasting DMI calls will result in the entire object graph being pulled into all nodes - thus breaking the scalability that I want to achieve with a cluster!

DMI should be able to deliver the event to only the nodes that have an instance of the object in memory. Also it would be great if the server worked out that the DMI method is going to immediately ask for a lock, and thus should dispatch the call only when the lock is available and in the same packet as a grant of that lock!

Without a good DMI mechanism I have two options to make this work and scale:

1) Implement some queues myself that can be distributed by tc. But to make these efficient, I will need to keep track of node ids and have queues per each node and track myself which nodes have which clients and blah blah blah oh my god I have just reimplemented the tc server! Beside… the moment I have a field called nodeId in my code… it has broken the promise of tc to deal transparently with the cluster. Plus I will want an API from tc so I don’t have to work out node IDs my self etc.

2) send the events by some other channel - perhaps broadcast or perhaps some point to point protocol or perhaps jgroups, JMS or similar. But then if I am using that, then why use tc for just the semi static data? I might as well send the whole message via this medium and not use tc at all.

For any use case that involves distributed asyncs events without waiting threads and efficient DMI mechanism is needed if tc is to be the clustering substrate.

cheers

Andy DePue 2007-01-26

I too would like to see DMI fleshed out some more, especially with a look toward events. Take a look at this thread for another issue related to firing distributed events: http://forums.terracotta.org/forums/posts/list/106.page - the issue seems to be about a DSO that fires events having both DSO and non-DSO listeners.

Steve Harris 2007-01-26

Just a heads up, the node id stuff is coming (and is actually already in trunk) for the next release. It is part of the cluster events stuff. The way we do it without an api is to rely on jmx interfaces. Of course we also have an spi which can be used by integraters and extenders of the dso platform. This is not drop in but it doesn’t add any com.tc classes and we feel it is quite useful.

DMI could be extended to support a mode similar to what you describe it certainly isn’t there now. It was originally designed for gui fireChanged events and not tuned at all. It is a fair question to ask whether we should grow dmi or add new concepts. I worry a little that what you are asking for would be a bit non-deterministic if we dynamically decided which node to execute the method on. If you used a pretty simple structure of your own to make those decisions I suspect the deterministic nature of it would lead to better results. I would Implement it as a single jvm multi threaded signaling and or eventing system and then extend it to the cluster.

To your point on transparency. I disagree that the stuff in my first point effects transparency. It certainly effects drop in but It still doesn’t require any com.tc imports if you do queue work. We try to keep an open mind about this stuff and recommend that people use dso in the way that will make the target product work as well as possible. I don’t get to hung up on drop in. Heck if one can leave an app 99 percent unchanged and make a few small changes to make it great I say make the changes. Especially if those changes are about performance.

anyway, we will definitely improve the perf on dmi because it’s easy and would be dumb not to. Let me know If I’ve swayed you at all on the mode where it doesn’t call the method if the target object isn’t loaded in a jvm. It’s actually not something that would be hard to do. Maybe we can walk you through an experimental version of it for you to play with and see how it feels.

Eugene Kuleshov 2007-01-26

Are you talking about “receive_transaction_thread” or the DMI thread ? receive_transaction_thread is single threaded to ensure transaction sequence (at least for now) DMI thread can be multithreaded easily if we dont care about ordering.

I meant dmi thread. But it seems like we’ll have a single thread for applying transactions. Not sure if that won’t be a bottle neck on high event volumes.

Also, if user want to process events in order he can use a single thread, else thread pool would give better performance. Ordering is a known “feature” of the most if not all message oriented systems, so user application can deal with it, because it has more specific information about the events.

Steve Harris 2007-01-26

If you have the code checked out I’m pretty sure if you want to experiment you can hack in to DistributedMethodCallManagerImpl class and do a lookupIfExistsLocal check on the object ID. if it isn’t local short circuit out of the run method for that call. That would allow you to experiment with the concept

Saravanan Subbiah 2007-01-26

Firstly it is slow and has latencies of about 300ms before a local call to resume (from within a synchronized block that already holds the Client lock) and 250ms to another node.

This is exactly what this solution targets. Riding it on the transaction and not aquiring locks will make it faster by decreasing the latency.

Secondly, the DMI call is distributed to ALL NODES!!! Ideally the client object will only exist in the node of the publisher and the node of the subscriber. Broadcasting DMI calls will result in the entire object graph being pulled into all nodes - thus breaking the scalability that I want to achieve with a cluster!

Like Steve mentioned above, DMI was originally designed to solve certain problem related to Swing apps. Sometime back we were toying around the idea of sending DMI calls to only the nodes that actually has the objects. (I am still not convinced one way or the other) It is certainly very easy to do, esp if we go with this new approach, but it is very hard for an user to deterministically know where the objects reside at runtime esp with the memory manager automagically faulting and flushing objects. May be the fact that we cant make a clear decision tells us that this should be user configurable ?

I think the two new features that are coming up in the next release should enable anyone to implement a fast event mechanism very easily without using DMI. The first feature is clustered events. The other feature is an optimization to linked blocking queue to not fault object into client heap until it is actually needed. With these two one can implement an easy fast event mechanism by having a queue per node and get optimal performance.

Greg Wilkins 2007-01-31

On the subject of transparancy…. I don’t mind changing the app somewhat to suite TC and I have already done so. The structure has been altered to separate the data and the event - so the data can be clustered normally and I just need a solution for the event.

However, I do think it is a stretch to have to modify the code to the point that it is talking node IDs and queuing events itself. I just don’t think that any application code should deal with that stuff for any reason - let alone the complexities of clustering. The chances that an arbitrary application developer can write code to correctly distributed event queues given just a node ID is rather small. I’d rather just use jgroups or JMS which hides all that from me.

For my particular use case, I don’t really care if the mechanism is DMI or something else (eg distributed events). However it does feel to me that DMI is the correct paradigm. I want to send an invocation to a particular object - that is what a method call does. Events are by nature broadcast and will end up going to all nodes etc. etc.

As for the timing and ordering issues - I don’t see that as a big issue. If I am running in a single node with multiple CPUs/threads then one invocation may “overtake” another if I don’t use synchronization. The only criteria I see as necessary is that

synchronize(myobj) { myobj.invokeA(); myobj.invokeB(); }

is well ordered.

Actually, I think a case could be made for special semanitcs for subset of DMI: synchronized methods with no return values These methods do not need to be distributed until the synchronized block is exited.

The only problem I see with that is that

synchronize(myobj) { myobj.field1=1; myobj.invokeA(); myobj.field2=2; myobj.invokeB(); }

would result on another node as:

synchronize(myobj) { myobj.field1=1; myobj.field2=2; myobj.invokeA(); myobj.invokeB(); }

So perhaps DMI is not the right mechanism?????

How about a callback mechanism that would allow an object to be notified when it has been updated by TC? Consider the following example (using make believe annotations (and I think annotation suck but at least they get this idea across)):

class Client { List messages = new ArrayList(); transient Object someWaitingObject;

public synchronized void addMessage(String message)
{
    messages.add(message);
    messagesUpdated();
}

@tc-update-callback messages
public synchronized void messagesUpdated(String message)
{
     synchronized(someWaitingObject)
    {
         someWaitingObject.notifyAll();
    }
} \}

So on the node that calls addMessage, the messagesUpdated method is called directly. On any other node that has an instance of the Client instantiated, the messagesUpdated method will be called when TC updates the messages field (at the end of the synchronize block).

cheers

Steve Harris 2007-01-31

That is an interesting idea. We currently have a callback mechanism for onload (when ever an object is loaded into a jvm) but having one for onchange seems like a nifty tool as well (not to mention incredibly easy for us to do since we can just do it inline with the change being applied).

I’m thinking about use cases. Can we think of any other ones in addition to a second way to notify gui’s instead of dmi and a mechanism for notifying an unshared lock. The example you gave can be done now without onChanged if that is a shared lock. You would just call notify all and it would effect all JVMs

Eugene Kuleshov 2007-01-31

Steve, it would be really neat to have onchange hook. There is also some similarity between dmi calls and the logically managed objects.

As for the use cases, I think that doing broadcast in the dso transaction gives user better control. I.e. he can do his own thread management to make broadcast asynchronous, which would be more natural then the current approach.

Fiona OShea 2007-02-07

Updating Due date to last day of Moraga dev iteration

Alex Voskoboynik 2007-02-20

done. the old 2.2.1 implementation clocked 33.22 DMI calls per second (36.60 on the originating node, 29.8 on the receiving node) the new implementation showed 684.43 DMI calls per second (1000 on the originating node, 368.86 on the receiving node)