Netty High CPU Issue
Today I debug the teiid code(teiid use netty 3.6.2) hit the Netty High CPU issue, The issue can be reproduced via the following code snippets:
public static void main(String[] args) throws InterruptedException {
ExecutorService nettyPool = Executors.newCachedThreadPool();
ChannelFactory factory = new NioServerSocketChannelFactory(nettyPool, nettyPool, 1);
ServerBootstrap bootstrap = new ServerBootstrap(factory);
bootstrap.setPipelineFactory(new ChannelPipelineFactory() {
public ChannelPipeline getPipeline() throws Exception {
return Channels.pipeline(new SimpleChannelHandler());
}});
bootstrap.setOption("keepAlive", Boolean.TRUE);
Channel serverChanel = bootstrap.bind(new InetSocketAddress(0));
serverChanel.close();
nettyPool.shutdownNow();
Thread.currentThread().sleep(Long.MAX_VALUE);
}
After above code ran, 2 threads(netty boss threads and netty worker threads) consume 100% cpu, the stacktrace as below:
"New I/O server boss #2" prio=10 tid=0x00007f7848194000 nid=0x3db7 runnable [0x00007f78448a2000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:317)
at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:193)
- locked <0x00000000d7b6bcd0> (a java.lang.Object)
at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:210)
at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:216)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:78)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000d7b6bcc0> (a sun.nio.ch.Util$2)
- locked <0x00000000d7b6bcb0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000d7b6bb98> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000000d7b6d058> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"New I/O worker #1" prio=10 tid=0x00007f784815c000 nid=0x3db6 runnable [0x00007f78449a3000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:317)
at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:193)
- locked <0x00000000d7b27f68> (a java.lang.Object)
at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:210)
at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:216)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:78)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000d7b27f58> (a sun.nio.ch.Util$2)
- locked <0x00000000d7b27ed8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000d7b27ce0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000000d7b3f488> (a java.util.concurrent.ThreadPoolExecutor$Worker)
Notes: complete code as HighCPUReproduce.
Resolution
After several times mail with Normal, this issue be fixed, pull request as https://github.com/netty/netty/pull/2868.
Base on above pull request, netty-3.6.10.Final is a fixed release.
If we change netty version to 3.6.10.Final
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty</artifactId>
<version>3.6.10.Final</version>
</dependency>
then run HighCPUReproduce again, the high cpu issed be fixed.