Monday, March 4, 2013

Broken pipe exception when connecting to cassandra


 Recently I required to WSO2 BAM receiver with high load. And during that I experienced the below exception.

[2013-03-04 15:56:43,010] ERROR {me.prettyprint.cassandra.connection.client.HThriftClient} -  Could not flush transport (to be expected if the pool is shutting down) in close for client: CassandraClient<cassandra03:9170-1581>
org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe
    at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:147)
    at org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:156)
    at me.prettyprint.cassandra.connection.client.HThriftClient.close(HThriftClient.java:98)
    at me.prettyprint.cassandra.connection.client.HThriftClient.close(HThriftClient.java:26)
    at me.prettyprint.cassandra.connection.HConnectionManager.closeClient(HConnectionManager.java:323)
    at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:272)
    at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
    at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
    at org.wso2.carbon.databridge.persistence.cassandra.datastore.CassandraConnector.commit(CassandraConnector.java:177)
    at org.wso2.carbon.databridge.persistence.cassandra.datastore.CassandraConnector.insertEventList(CassandraConnector.java:402)
    at org.wso2.carbon.databridge.datasink.cassandra.subscriber.BAMEventSubscriber.receive(BAMEventSubscriber.java:50)
    at org.wso2.carbon.databridge.core.internal.queue.QueueWorker.run(QueueWorker.java:80)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.SocketException: Broken pipe
    at java.net.SocketOutputStream.socketWrite0(Native Method)
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
    at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:145)
    ... 17 more
[2013-03-04 15:56:43,011] ERROR {me.prettyprint.cassandra.connection.HConnectionManager} -  MARK HOST AS DOWN TRIGGERED for host cassandra03(10.157.4.137):9170
[2013-03-04 15:56:43,011] ERROR {me.prettyprint.cassandra.connection.HConnectionManager} -  Pool state on shutdown: <ConcurrentCassandraClientPoolByHost>:


After a doing more research on the cassandra.yaml configuration, i found out changing the below propeties:

thrift_framed_transport_size_in_mb: 15
thrift_max_message_length_in_mb: 16


Increasing the above parameters will solve the problem, ie, you can increase 'thrift_max_message_length_in_mb' to 64 and 'thrift_framed_transport_size_in_mb' to 60, which will help to get rid of the mentioned exception.