Load Balancing Data Publishers and Sending Events to Multiple Receivers

WSO2 BAM/CEP has hight performance thrift based event receiving model, which basically receives the events via TCP. Even though thrift is high performance receiving protocol, load balancing the thrift events is problematic as you need tcp based load balancer rather http based load balancer. Therefore in WSO2 we have added support to have load balancing between Data bridge receivers (ie, WSO2 BAM servers, CEP servers) from the client side, by sending the events in a round robin manner to BAM servers, such that load of events will be balanced between them.

For this we have added a Wrapper class called LoadBalancingDataPublisher, which uses the AsyncDataPublisher in it. It not only load balances the events between the set of servers and also can send same events to some servers. All the capabilities of using the LoadBalancingDataPublisher is provided in BAM 2.2.0 documentation here, which explains the use cases of using the load balancing data publisher.

This provides more fail over handling also with load balancing, which can detect the node failure and stop further publishing for the dead node and it also recognozes the node startup and it starts load balancing the events from that instance.

I'll provide a more detailed description about using LoadDataPublisher to publish events to BAM/CEP in next article.

Non-blocking data publshing for BAM/CEP

You can publish the events to BAM/CEP by using the DataPublisher. AsyncDataPublisher is enhanced version of DataPublisher which incorporates all the constructors and API of of general DataPublisher and send events asynchronously. Thats is in general data publishers, making the connection is synchronous/blocking therefore the network latency might affect the connection time and publishing efficieny. The AsyncDataPublisher connects to the receiver asynchronously and cache and re-use the stream id in efficient manner.

There are mainly three steps involved in creating the AsyncDataPublisher.

1.Create a AsyncDataPublisher Instance with any of available constructors.

Eg: AsyncDataPublisher asyncDataPublisher = new AsyncDataPublisher("url","userName", "password");

2. Add the stream definition json string which you would like to publish via the AsyncDataPublisher. You can add any number of stream definitions and any time you can include that. But make sure before you publish the event for a specific stream definition, you should include it in the AsyncDataPublisher.

Eg: asyncDataPublisher.addStreamDefinition("stream def");

3. Start send events using the AsyncDataPublisher.

Eg: asyncDataPublisher.publish("stream name", "stream version", metaDataObjectArray, correlationDataObjectArray, payLoadDataObjectArray );


This will send events with additional advantage as mentioned above.

Whats is WSO2 BAM 2.0..

WSO2 BAM is a fully open source Business Activity Monitoring. Recently WSO2 BAM 2.0 has been released which a complete revamp of WSO2 BAM 1.x.

It's targeted to monitor not-only WSO2 servers but also other third party servers and applications and Key performance Indicators (KPI) and also providing high scalable storage of data via Cassandra and high business data analytics capability through Apache Hive & Hadoop. Also it has very high performance of data capture by using thrift. WSO2 BAM is very scalable, such as you can scale individual components of BAM according to your requirement. If you need high performance for data storage you can only scale the Cassandra storage as multi-node cluster, and if you target to do more analytics in your captured data then you can scale and have a hadoop cluster which makes your analytics faster. WSO2 BAM also has attractive pluggable dashboards, where some of them are in-built by default and some of them are released by separate toolboxes.WSO2 BAM enables you to not only use the default dashboards but also you can write your own gadgets or visualizing engines to plug and view the data.

The below diagram shows the architecture of WSO BAM.

1. Event/Data Receiving
   You can publish any data to BAM via data agents. Already there are data agents available for WSO2 Servers in order to publish the Service Statistics Monitoring, and WSO2 ESB for Mediation Statistics Monitoring and Activity Mediation Monitoring. Similar to this you write a simple java application to publish your third-party events through thrift as described in http://wso2.org/library/articles/2012/07/creating-custom-agents-publish-events-bamcep. And also you can publish the events via REST api to BAM as described in http://docs.wso2.org/wiki/display/BAM200/REST+API.
   
   
2. Data Dumping in Cassandra
   After receiving the events/data in BAM, it immediately flush the data Cassandra database. Cassandra data base provides high scalability and it's very suitable to store huge amount of data with hight performance.
   
   
3. Running business analytics
   Analyzer framework consists of Apache Hive and Hadoop. Hive is a data warehouse system for Hadoop that facilitates easy data summarization, ad-hoc queries, and the analysis of large datasets stored in Hadoop compatible file systems. Hive provides a mechanism to project structure onto this data and query the data using a SQL-like language called HiveQL. At the same time this language also allows traditional map/reduce programmers to plug in their custom mappers and reducers when it is inconvenient or inefficient to express this logic in HiveQL.

You can access the captured data which is stored in Cassandra and run the analytics on those data via writing the hive scripts. You can find sample hive scrips on WSO BAM samples in http://docs.wso2.org/wiki/display/BAM200/Samples. and more hive related queries can be found in https://cwiki.apache.org/confluence/display/Hive/Tutorial.

4. Storing the result in JDBC storage
   After running the business analytics in you captured big data, you can store the summarized results in JDBC storage such as MySQL, H2 Database, Oracle, etc. This facilitates to plug any external reporting engine or preferred dashboard gadgets. In WSO2 BAM samples we have written the summarized data into H2 database as it's the default database used. You can find more details of writing the results in the JDBC storage in http://kasunweranga.blogspot.com/2012/06/jdbc-storage-handler-for-hive.html.

4. Visualization
   Visualization of the data done by WSO2 Jaggery Framework . This data also can be visualized via writing google gadgets also. You can see the sample gadgets created in WSO2 KPI Monitoring sample in http://docs.wso2.org/wiki/display/BAM200/KPI+Monitoring+Sample. And also there are dashboards which includes specially designed gadgets for Service statistics Monitoring and WSO2 ESB montitoring. 

There is already available reporting using the jasper reporting available in http://docs.wso2.org/wiki/display/BAM200/Reports.

These all phases of WSO2 BAM provides to privilege of monitoring your own data and visualize your business aspects.

Try out WSO2 BAM now http://wso2.com/products/business-activity-monitor/

Enjoy WSO2 BAM!! :)

BAM Toolboxes - Quick Deployment for your Scenario

I have written BAM toolboxes, which is an installable archive in WSO2 BAM. This provides you a quick deployment of your scenario in WSO2 BAM in few seconds.

WSO2 BAM has importantly three components,

1. Receiving Data/Events and Storage
   - This is the main interface which BAM interacts with other servers. That is you can use WSO2 servers/ any third part servers/application and send the events/data to BAM. I'll include how you can send events to BAM in a separate article.
2. Analytics
   - This component analyzes the data that is received in part-1, and produces some useful information out of your data. Apache Hive and hadoop are used in this segment to analyze the big data and scale according to the requirement.
3. Presentation
   There are attractive dashboards, gadgets, reports can be produced in this segment out of the data you have summarized in step-2.

In reality during the testing stage, you might be writing your own customized artifacts in all three above components. The toolbox provides you the flexibility of combining all three as a single artifact and install in WSO2 BAM, such that all the components as mentioned above can come in to the BAM within few seconds, rather manually setting up the environment in production setup.

And also WSO2 BAM will be continuously releasing new toolboxes which includes different use cases such that you can just install that in WSO2 BAM and try out.

You can see how you can install the toolboxes in WSO2 BAM inhttp://docs.wso2.org/wiki/display/BAM200/Creating+a+Custom+Toolbox http://docs.wso2.org/wiki/display/BAM200/Introduction+to+BAM+Toolbox

And also you can find how you can create your own toolbox in,

http://docs.wso2.org/wiki/display/BAM200/Creating+a+Custom+Toolbox

WSO2 BAM 2.0 - New Begining ...

-->

WSO2 has released WSO2 BAM2.0 with WSO2 CARBON 4.0.1 release on September 8^th 2012.

The WSO2 Business Activity Monitor (WSO2 BAM) is an enterprise-ready, fully-open source, complete solution for aggregating, analyzing and presenting information about business activities. The aggregation refers to collection of data, analysis refers to manipulation of data in order to extract information, and presentation refers to representing this data visually or in other ways such as alerts. The WSO2 BAM architecture reflects this natural flow in its design.

Since all WSO2 products are based on the component-based WSO2 Carbon platform, WSO2 BAM is lean, lightweight and consists of only the required components for efficient functioning. It does not contain unnecessary bulk, unlike many over-bloated, proprietary solutions. WSO2 BAM comprises of only required modules to give the best of performance, scalability and customizability, allowing businesses to achieve time-effective results for their solutions without sacrificing performance or the ability to scale.

The product is available for download at: http://wso2.com/products/business-activity-monitor

The documentation is available at: http://docs.wso2.org/wiki/display/BAM200/WSO2+Business+Activity+Monitor+Documentation

BAM 2.0 has monitoring services, by using the Service Statistics Toolbox found in http://wso2.com/products/business-activity-monitor/2-0-0-toolboxes/. Via this you can monitor request, response, fault count, response time, etc for a hosted services.

WSO2 BAM 2.0.1 which includes attractive dashboards for WSO2 ESB monitoring will be released soon.