Broken pipe exception when connecting to cassandra

 Recently I required to WSO2 BAM receiver with high load. And during that I experienced the below exception.

[2013-03-04 15:56:43,010] ERROR {me.prettyprint.cassandra.connection.client.HThriftClient} -  Could not flush transport (to be expected if the pool is shutting down) in close for client: CassandraClient<cassandra03:9170-1581>
org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe
    at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:147)
    at org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:156)
    at me.prettyprint.cassandra.connection.client.HThriftClient.close(HThriftClient.java:98)
    at me.prettyprint.cassandra.connection.client.HThriftClient.close(HThriftClient.java:26)
    at me.prettyprint.cassandra.connection.HConnectionManager.closeClient(HConnectionManager.java:323)
    at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:272)
    at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
    at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
    at org.wso2.carbon.databridge.persistence.cassandra.datastore.CassandraConnector.commit(CassandraConnector.java:177)
    at org.wso2.carbon.databridge.persistence.cassandra.datastore.CassandraConnector.insertEventList(CassandraConnector.java:402)
    at org.wso2.carbon.databridge.datasink.cassandra.subscriber.BAMEventSubscriber.receive(BAMEventSubscriber.java:50)
    at org.wso2.carbon.databridge.core.internal.queue.QueueWorker.run(QueueWorker.java:80)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.SocketException: Broken pipe
    at java.net.SocketOutputStream.socketWrite0(Native Method)
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
    at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:145)
    ... 17 more
[2013-03-04 15:56:43,011] ERROR {me.prettyprint.cassandra.connection.HConnectionManager} -  MARK HOST AS DOWN TRIGGERED for host cassandra03(10.157.4.137):9170
[2013-03-04 15:56:43,011] ERROR {me.prettyprint.cassandra.connection.HConnectionManager} -  Pool state on shutdown: <ConcurrentCassandraClientPoolByHost>:


After a doing more research on the cassandra.yaml configuration, i found out changing the below propeties:

thrift_framed_transport_size_in_mb: 15
thrift_max_message_length_in_mb: 16


Increasing the above parameters will solve the problem, ie, you can increase 'thrift_max_message_length_in_mb' to 64 and 'thrift_framed_transport_size_in_mb' to 60, which will help to get rid of the mentioned exception.

How to configure MySQL server in linux and connect from remote server?

I recently required to connect to start the MySQL server and connect it from another client machine remotely. It wasn't too easy as I expected and I came across couple of issues during this, and I thought of blog it as it'll be useful to for someone else also. :)

These are steps I followed during this.

• Install the MySQL server.

                sudo apt-get install mysql-server

• Connect to MySQL server with 

                mysql --user=root --password=root

• Did ifconfig from my machine to find out my ip-address.
• And then used that ip address to connect to the MySQL server

               mysql --host=x.x.x.x --user=root --password=root

         Then I got the below error.
               ERROR 2003 (HY000): Can't connect to MySQL server on 'x.x.x.x' (111)

• The above error comes because of the bind-address of my sql server. In my.cnf file the bind-address of the my sql server has been mentioned and when the my sql server start up it'll bind to that address. By default the address in 127.0.0.1 which is the loop back address, and we can't connect via this address from another machine remotely.  The address you specify in bind tells mysql where to listen. 0.0.0.0 is a special address, which means "bind to every available network".

           Go to /etc/mysql/my.cnf  and change the bind address to 0.0.0.0.

• Then again when i tried to connect using the same command remotely I encountered the below error.

          "ERROR 1130 (HY000): Host 'hostname' is not allowed to connect to this MySQL server".

           

• After a bit of research, the fix in my case was to "GRANT" user root to connect to MySQL on any hosts. By default, user "root" was only allowed to connect to localhost and 127.0.0.1 hosts of MySQL. 

            #GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '<roots-password>' WITH GRANT OPTION;

   

After all the configuration above I could connect MySQL server remotely.:-)

Some useful tips:
- Start the my sql server ---> sudo service mysql start
- Stop the my sql server ---> sudo service mysql stop

Load Balancing Data Publishers and Sending Events to Multiple Receivers

WSO2 BAM/CEP has hight performance thrift based event receiving model, which basically receives the events via TCP. Even though thrift is high performance receiving protocol, load balancing the thrift events is problematic as you need tcp based load balancer rather http based load balancer. Therefore in WSO2 we have added support to have load balancing between Data bridge receivers (ie, WSO2 BAM servers, CEP servers) from the client side, by sending the events in a round robin manner to BAM servers, such that load of events will be balanced between them.

For this we have added a Wrapper class called LoadBalancingDataPublisher, which uses the AsyncDataPublisher in it. It not only load balances the events between the set of servers and also can send same events to some servers. All the capabilities of using the LoadBalancingDataPublisher is provided in BAM 2.2.0 documentation here, which explains the use cases of using the load balancing data publisher.

This provides more fail over handling also with load balancing, which can detect the node failure and stop further publishing for the dead node and it also recognozes the node startup and it starts load balancing the events from that instance.

I'll provide a more detailed description about using LoadDataPublisher to publish events to BAM/CEP in next article.