Number of Qs..

Jul 22, 2007 at 9:38 PM
Hi, I just went through your whitepaper, very nice - and actually it's good to see Microsoft comming out with such projects. Alright, I have couple of questions:

1. Is this system actual being used by production system within microsoft, you hinted at it, but it wasn't quite clear? (WebPageEvent huh..?)
2. What kind of higher-level services do you think/suggest, we can build onto this system? I am intrested in creating a custom business event messaging system, would you consider this be a good infrastructure choice? Any memory limit issues there?
3. In the whitepaper you mentioned a number of improvements, any schedule to implement them? Especially, moving to a true mesh type topology?
4. I believe you have a in-memory queue, but I am not very clear if this can provide some sort of durable-messaging/eventing? And if it can, then syncronization becomes an issue, right.
5. Like you said, You need to flesh out (and also provide usage implications) for all the configuration section parameters.
6. Lastly, I am not very sure about this, how about implementing this as a transport channel for WCF??? I have to dust my knowledge about WCF channels but if it is possible, by using WCF we move to quite a high level of abstraction, without much investment!

Thanks,
Rishi
Coordinator
Jul 23, 2007 at 11:41 PM
1. One project in microsoft.com is under development which is using the event system. They are creating events for different application actions. The events are collected by a subscribing service and inserted into SQL for BI reporting. This should be in production within a couple of months.

I have built a real-time application monitoring dashboard for ASP.Net applications which uses the event system. It correlates the application metrics across the web farm and the dashboard is within ~3 seconds of what is happening on the web servers. This is to monitor web applications on microsoft.com and is just ready to test on a production server.

2. Above I named metric collection and monitoring as examples. I'm sure you could come up with lots. It depends upon the problem and solution as to if the event system is appropriate technology.

3. At the moment, I'm working on applications using the event system rather than adding features.

4. The in-memory queue is not durable. You could modify the queue class to be durable but I think this is an area of differentiation between a messaging system and an eventing system.

5. At some point.

6. I can't see any reason to use WCF for transport. It would be a significant perf hit and not provide value.



r5 wrote:
Hi, I just went through your whitepaper, very nice - and actually it's good to see Microsoft comming out with such projects. Alright, I have couple of questions:

1. Is this system actual being used by production system within microsoft, you hinted at it, but it wasn't quite clear? (WebPageEvent huh..?)
2. What kind of higher-level services do you think/suggest, we can build onto this system? I am intrested in creating a custom business event messaging system, would you consider this be a good infrastructure choice? Any memory limit issues there?
3. In the whitepaper you mentioned a number of improvements, any schedule to implement them? Especially, moving to a true mesh type topology?
4. I believe you have a in-memory queue, but I am not very clear if this can provide some sort of durable-messaging/eventing? And if it can, then syncronization becomes an issue, right.
5. Like you said, You need to flesh out (and also provide usage implications) for all the configuration section parameters.
6. Lastly, I am not very sure about this, how about implementing this as a transport channel for WCF??? I have to dust my knowledge about WCF channels but if it is possible, by using WCF we move to quite a high level of abstraction, without much investment!

Thanks,
Rishi

Jul 29, 2007 at 12:44 AM
Thanks for your reply. Alright, I was wondering why did you use GUID rather than a WS-Eventing style "topic" string to identity event types. Certainly, this is more flexible and can create presudo channels using wildcards.

Also, I don't know if you have heard of Advanced Message Queuing Protocol (AMQP)? Being an open-source standard (or future standard), it would have been a nice fit as a lightweight wire-level messaging infrastructure for eventing. Though AMQP is general purpose, would you consider using AMQP specifications or it is too heavy weight for your given deliverables?

Rishi
Coordinator
Jul 29, 2007 at 6:58 PM
A guid works nicely where there are many publishers who don't know about each other. A guid removes the risk of name collisions. Otherwise, someone needs to control naming and I didn't want to get into that issue. If you want to create many event types and want to include a "topic" property, you could create your own subclass from the event class, add your "topic" property, and then have everyone use your new subclass. If this property is something most people would want to use, it could be promoted to the event class. Adding another property would add additional overhead to the header. This would affect perf by adding more work to serialize/deserialze the events and this is the most expensive part of eventing.

I think AMQP is rather different from eventing. I think there are some fundamental differences between messaging and this type of eventing. Prior to building this eventing system, I built a reliable messaging system that shipped at the core of a now discontinuted product from Microsoft call Microsoft Business Network. It had more functionality than AMQP. AMQP is rather different from the eventing system and will never solve some of the problems the eventing system will solve and vice versa. One solution is not going to solve all problems and some problems well. So I think there needs to be multiple solutions for different types of problems.



r5 wrote:
Thanks for your reply. Alright, I was wondering why did you use GUID rather than a WS-Eventing style "topic" string to identity event types. Certainly, this is more flexible and can create presudo channels using wildcards.

Also, I don't know if you have heard of Advanced Message Queuing Protocol (AMQP)? Being an open-source standard (or future standard), it would have been a nice fit as a lightweight wire-level messaging infrastructure for eventing. Though AMQP is general purpose, would you consider using AMQP specifications or it is too heavy weight for your given deliverables?

Rishi

Jul 30, 2007 at 12:46 AM
I get your point, this not being a messaging system. One last question, because of the shared memory thing, should I be aware of or consider size limitations to an event message? Certaintly, one of the overriding goal is performance, and therefore is there any underlying (if not written) guidline on what should or should not be part of an event data structure?

I read about the "Microsoft Business Network", described as an "EDI on steriods" that uses "assisted peer-to-peer" network. It seemed intrestring, but is/was quite a big challange. Is there any chance that the underlying reliable messaging system be shared with the world - like you've done with this eventing system? And if supports durable messaging then that would just super, especially since the WS-* seem to have lost steam.

Thanks again.


keithh wrote:
A guid works nicely where there are many publishers who don't know about each other. A guid removes the risk of name collisions. Otherwise, someone needs to control naming and I didn't want to get into that issue. If you want to create many event types and want to include a "topic" property, you could create your own subclass from the event class, add your "topic" property, and then have everyone use your new subclass. If this property is something most people would want to use, it could be promoted to the event class. Adding another property would add additional overhead to the header. This would affect perf by adding more work to serialize/deserialze the events and this is the most expensive part of eventing.

I think AMQP is rather different from eventing. I think there are some fundamental differences between messaging and this type of eventing. Prior to building this eventing system, I built a reliable messaging system that shipped at the core of a now discontinuted product from Microsoft call Microsoft Business Network. It had more functionality than AMQP. AMQP is rather different from the eventing system and will never solve some of the problems the eventing system will solve and vice versa. One solution is not going to solve all problems and some problems well. So I think there needs to be multiple solutions for different types of problems.



r5 wrote:
Thanks for your reply. Alright, I was wondering why did you use GUID rather than a WS-Eventing style "topic" string to identity event types. Certainly, this is more flexible and can create presudo channels using wildcards.

Also, I don't know if you have heard of Advanced Message Queuing Protocol (AMQP)? Being an open-source standard (or future standard), it would have been a nice fit as a lightweight wire-level messaging infrastructure for eventing. Though AMQP is general purpose, would you consider using AMQP specifications or it is too heavy weight for your given deliverables?

Rishi


Coordinator
Aug 1, 2007 at 2:16 PM
There is a config setting for the windows service which specifies the size of shared memory. There are no restrictions for the size of events. Since the events are passed via shared memory, if you know you will be sending large events then you might want to increase the size of the shared memory allocation. You need to be aware that all the shared memory will be used since it is a ring buffer. This means as you increase its size, if the machine doesn't have sufficient real memory then the VMM will be paging parts of the shared memory out. This will reduce performance. So you want to keep the event size, shared memory size, and real memory size in balance.

The reliable messaging system (or ESB) in MBN was more similar to the pub/sub event system in routing of messages. Microsoft simply chose to market MBN as an assisted peer-to-peer service. In reality, the messaging system routed messages between nodes, be they across public or private networks. It used durable messaging, implemented the ISO standard for non-repudiation, was designed to work securely through a DMZ, and lots more. Though I would love to publish it to Codeplex, I don't think I could get permission.

Keith