1 mycat接受客户端连接并发送握手报文

mycat1.6实现了AIO和NIO,本文只讲解NIO的流程。

在reactor模型，acceptor线程负责接受TCP连接请求。acceptor的实现类是io.mycat.net.NIOAcceptor。mycat将socket包装成前端连接FrontendConnection。

reactor线程负责处理IO请求，reactor的实现类是io.mycat.net.NIOReactor。reactor获取SelectionKey所绑定的连接AbstractConnection，调用AbstractConnection中的asynRead()和doNextWriteCheck()方法，来进行读写操作。

1.1 NIOAcceptor的实现

下面讲解上图1~3的过程。

1.1.1 NIOAcceptor的初始化

public final class NIOAcceptor extends Thread  implements SocketAcceptor{
    private static final Logger LOGGER = LoggerFactory.getLogger(NIOAcceptor.class);
    private static final AcceptIdGenerator ID_GENERATOR = new AcceptIdGenerator();

    private final int port;
    private final Selector selector;
    private final ServerSocketChannel serverChannel;
    private final FrontendConnectionFactory factory;
    private long acceptCount;
    private final NIOReactorPool reactorPool;

    public NIOAcceptor(String name, String bindIp,int port, 
            FrontendConnectionFactory factory, NIOReactorPool reactorPool)
            throws IOException {
        //设置acceptor线程名称
        super.setName(name);
        //设置端口号
        this.port = port;
        //给acceptor分配选择器
        this.selector = Selector.open();
        //打开一个server-socket channel
        this.serverChannel = ServerSocketChannel.open();
        //server-socket channel设置为非阻塞
        this.serverChannel.configureBlocking(false);
        /** 设置TCP属性 */
        //设置TCP属性,重用端口
        serverChannel.setOption(StandardSocketOptions.SO_REUSEADDR, true);
        //设置TCP属性,设置接收缓存
        serverChannel.setOption(StandardSocketOptions.SO_RCVBUF, 1024 * 16 * 2);
        // backlog=100 等待连接的最大数量
        serverChannel.bind(new InetSocketAddress(bindIp, port), 100);
        //注册OP_ACCEPT事件
        this.serverChannel.register(selector, SelectionKey.OP_ACCEPT);
        //设置连接工厂类，此处是FrontendConnectionFactory
        this.factory = factory;
        //设置reactorPool
        this.reactorPool = reactorPool;
    }

1.1.2 NIOAcceptor线程的run方法

run方法中for无限循环，selector.select(1000L)注册器设置1秒超时返回准备就绪的SelectionKey,如果SelectionKey有效且关注的是OP_ACCEPT事件，那么注册连接，否则取消SelectionKey，最后在finally块中清空所有key

@Override
    public void run() {
        final Selector tSelector = this.selector;
        for (;;) {
            ++acceptCount;
            try {
                tSelector.select(1000L);
                Set<SelectionKey> keys = tSelector.selectedKeys();
                try {
                    for (SelectionKey key : keys) {
                        //判断key是否有效，是否是acceptor事件
                        if (key.isValid() && key.isAcceptable()) {
                            //注册连接
                            accept();
                        } else {
                            //取消key
                            key.cancel();
                        }
                    }
                } finally {
                    //清空selectedKey
                    keys.clear();
                }
            } catch (Exception e) {
                LOGGER.warn(getName(), e);
            }
        }
    }

1.1.3 NIOAcceptor的accept方法

通过accept()获取SocketChannel，设置SocketChannel为非阻塞模式，包装成前段连接FrontendConnection，设置必要参数，获取一个reactor线程，并将前段连接FrontendConnection注册到该reactor线程中进行读写操作。此处调用reactor.postRegister(c)方法只是将连接放入注册队列，后面会详细讲解。

private void accept() {
        SocketChannel channel = null;
        try {
            //ServerSocketChannel设置成非阻塞模式。
            //在非阻塞模式下，accept() 方法会立刻返回，如果还没有新进来的连接,返回的将是null。
            channel = serverChannel.accept();
            //channel设置成非阻塞模式
            channel.configureBlocking(false);
            //channel包装成前段连接
            FrontendConnection c = factory.make(channel);
            c.setAccepted(true);
            //设置连接ID
            c.setId(ID_GENERATOR.getId());
            NIOProcessor processor = (NIOProcessor) MycatServer.getInstance()
                    .nextProcessor();
            c.setProcessor(processor);
            //获取一个reactor
            NIOReactor reactor = reactorPool.getNextReactor();
            //注册连接
            reactor.postRegister(c);
        } catch (Exception e) {
            LOGGER.warn(getName(), e);
            closeChannel(channel);
        }
    }

1.1.4 NIOAcceptor的closeChannel方法

先关闭socket，再关闭channel

private static void closeChannel(SocketChannel channel) {
        if (channel == null) {
            return;
        }
        Socket socket = channel.socket();
        if (socket != null) {
            try {
                socket.close();
            } catch (IOException e) {
               LOGGER.error("closeChannelError", e);
            }
        }
        try {
            channel.close();
        } catch (IOException e) {
            LOGGER.error("closeChannelError", e);
        }
    }

1.2 NIOReactor建立连接

下面讲解上图4~7的过程。

1.2.1 NIOReactor的初始化

NIOReactor初始化的时候设置一个读写线程RW,所有的IO操作实际上在这个线程中处理的。

public final class NIOReactor {
    private static final Logger LOGGER = LoggerFactory.getLogger(NIOReactor.class);
    private final String name;
    private final RW reactorR;

    public NIOReactor(String name) throws IOException {
        this.name = name;
        this.reactorR = new RW();
    }

1.2.2 NIOReactor的启动

NIOReactor的启动实际上是启动RW线程。

    final void startup() {
        new Thread(reactorR, name + "-RW").start();
    }

1.2.3 RW的初始化

RW的初始化时，会获取Selector实例，并初始化一个连接注册队列。

    private final class RW implements Runnable {
        private final Selector selector;
        private final ConcurrentLinkedQueue<AbstractConnection> registerQueue;
        private long reactCount;

        private RW() throws IOException {
            this.selector = Selector.open();
            this.registerQueue = new ConcurrentLinkedQueue<AbstractConnection>();
        }

1.2.4 RW的run方法

当selector.select(500L)方法选择一组键，其相应的通道已为 I/O 操作准备就绪，那么调用register(selector)方法（上图所示5）注册连接。如果有读写事件就绪，那么通过获取SelectionKey绑定的连接Connection，调用Connection的相应的读写方法。本文介绍首次建立连接，我们重点看register(selector)方法。

@Override
        public void run() {
            final Selector selector = this.selector;
            Set<SelectionKey> keys = null;
            for (;;) {
                ++reactCount;
                try {
                    selector.select(500L);
                    register(selector);
                    keys = selector.selectedKeys();
                    for (SelectionKey key : keys) {
                        AbstractConnection con = null;
                        try {
                            Object att = key.attachment();
                            if (att != null) {
                                con = (AbstractConnection) att;
                                if (key.isValid() && key.isReadable()) {
                                    try {
                                        con.asynRead();
                                    } catch (IOException e) {
                                        con.close("program err:" + e.toString());
                                        continue;
                                    } catch (Exception e) {
                                        LOGGER.warn("caught err:", e);
                                        con.close("program err:" + e.toString());
                                        continue;
                                    }
                                }
                                if (key.isValid() && key.isWritable()) {
                                    con.doNextWriteCheck();
                                }
                            } else {
                                key.cancel();
                            }
                        } catch (CancelledKeyException e) {
                            if (LOGGER.isDebugEnabled()) {
                                LOGGER.debug(con + " socket key canceled");
                            }
                        } catch (Exception e) {
                            LOGGER.warn(con + " " + e);
                        } catch (final Throwable e){
                            // Catch exceptions such as OOM and close connection if exists
                            //so that the reactor can keep running!
                            // @author Uncle-pan
                            // @since 2016-03-30
                            if(con != null){
                                con.close("Bad: "+e);
                            }
                            LOGGER.error("caught err: ", e);
                            continue;
                        }
                    }
                } catch (Exception e) {
                    LOGGER.warn(name, e);
                } catch (final Throwable e){
                    // Catch exceptions such as OOM so that the reactor can keep running!
                    // @author Uncle-pan
                    // @since 2016-03-30
                    LOGGER.error("caught err: ", e);
                } finally {
                    if (keys != null) {
                        keys.clear();
                    }

                }
            }
        }

1.2.4 RW的register(selector)方法

当selector.select()方法获取的就绪事件或者500毫秒超时之后，那么调用register(selector)方法（上图所示5）注册连接。在register方法(上图所示6)中，循环注册队列registerQueue中的连接。

private void register(Selector selector) {
            AbstractConnection c = null;
            if (registerQueue.isEmpty()) {
                return;
            }
            while ((c = registerQueue.poll()) != null) {
                try {
                    //注册读时间
                    ((NIOSocketWR) c.getSocketWR()).register(selector);
                    //向客户端发送握手报文
                    c.register();
                } catch (Exception e) {
                    c.close("register err" + e.toString());
                }
            }
        }

如上图7，NIOSocketWR的register(selector)方法给channel注册读事件

public void register(Selector selector) throws IOException {
        try {
            processKey = channel.register(selector, SelectionKey.OP_READ, con);
        } finally {
            if (con.isClosed.get()) {
                clearSelectionKey();
            }
        }
    }

为什么不直接在NIOAcceptor的accept()中调用NIOSocketWR的register(selector)注册读事件，而是通过注册队列的方式异步注册，主要的原因是acceptor线程直接channel.register(selector,int)方法和reactor线程的selector.select()方法会发生死锁。所以注册必须由reactor一个线程完成。

1.3 mycat向客户端发送握手报文

1.3.1 调用AbstractConnection的register方法

如上图7，调用AbstractConnection的register方法发送握手报文。

@Override
    public void register() throws IOException {
        if (!isClosed.get()) {

            // 生成认证数据
            byte[] rand1 = RandomUtil.randomBytes(8);
            byte[] rand2 = RandomUtil.randomBytes(12);

            // 保存认证数据
            byte[] seed = new byte[rand1.length + rand2.length];
            System.arraycopy(rand1, 0, seed, 0, rand1.length);
            System.arraycopy(rand2, 0, seed, rand1.length, rand2.length);
            this.seed = seed;

            // 发送握手数据包
            boolean useHandshakeV10 = MycatServer.getInstance().getConfig().getSystem().getUseHandshakeV10() == 1;
            if(useHandshakeV10) {
                HandshakeV10Packet hs = new HandshakeV10Packet();
                hs.packetId = 0;
                hs.protocolVersion = Versions.PROTOCOL_VERSION;
                hs.serverVersion = Versions.SERVER_VERSION;
                hs.threadId = id;
                hs.seed = rand1;
                hs.serverCapabilities = getServerCapabilities();
                hs.serverCharsetIndex = (byte) (charsetIndex & 0xff);
                hs.serverStatus = 2;
                hs.restOfScrambleBuff = rand2;
                hs.write(this);
            } else {
                HandshakePacket hs = new HandshakePacket();
                hs.packetId = 0;
                hs.protocolVersion = Versions.PROTOCOL_VERSION;
                hs.serverVersion = Versions.SERVER_VERSION;
                hs.threadId = id;
                hs.seed = rand1;
                hs.serverCapabilities = getServerCapabilities();
                hs.serverCharsetIndex = (byte) (charsetIndex & 0xff);
                hs.serverStatus = 2;
                hs.restOfScrambleBuff = rand2;
                hs.write(this);
            }

            // asynread response
            this.asynRead();
        }
    }

1.3.2 调用HandshakePacket的write(FrontendConnection c)方法

如上图8，调用HandshakePacket的write(FrontendConnection c)方法将拼装好的握手报文写入buffer,并发送到客户端。

public void write(FrontendConnection c) {
        ByteBuffer buffer = c.allocate();
        BufferUtil.writeUB3(buffer, calcPacketSize());
        buffer.put(packetId);
        buffer.put(protocolVersion);
        BufferUtil.writeWithNull(buffer, serverVersion);
        BufferUtil.writeUB4(buffer, threadId);
        BufferUtil.writeWithNull(buffer, seed);
        BufferUtil.writeUB2(buffer, serverCapabilities);
        buffer.put(serverCharsetIndex);
        BufferUtil.writeUB2(buffer, serverStatus);
        buffer.put(FILLER_13);
        //        buffer.position(buffer.position() + 13);
        BufferUtil.writeWithNull(buffer, restOfScrambleBuff);
        c.write(buffer);
    }

1.3.3 调用AbstractConnection的write(buffer)方法

如上图9，将buffer放入写队列并进行异步写操作。

@Override
    public final void write(ByteBuffer buffer) {
        if (isSupportCompress()) {
            ByteBuffer newBuffer = CompressUtil.compressMysqlPacket(buffer, this, compressUnfinishedDataQueue);
            writeQueue.offer(newBuffer);
        } else {
            writeQueue.offer(buffer);
        }

        // if ansyn write finishe event got lock before me ,then writing
        // flag is set false but not start a write request
        // so we check again
        try {
            this.socketWR.doNextWriteCheck();
        } catch (Exception e) {
            LOGGER.warn("write err:", e);
            this.close("write err:" + e);
        }
    }

1.3.4 调用NIOSocketWR的doNextWriteCheck()方法进行异步写操作

如上图10，调用NIOSocketWR的write0()方法，把写队列中的数据写出去，如果数据没有写完，那么继续关注写事件。

public void doNextWriteCheck() {

        if (!writing.compareAndSet(false, true)) {
            return;
        }

        try {
            boolean noMoreData = write0();
            writing.set(false);
            if (noMoreData && con.writeQueue.isEmpty()) {
                if ((processKey.isValid() && (processKey.interestOps() & SelectionKey.OP_WRITE) != 0)) {
                    disableWrite();
                }

            } else {
                if ((processKey.isValid() && (processKey.interestOps() & SelectionKey.OP_WRITE) == 0)) {
                    enableWrite(false);
                }
            }

        } catch (IOException e) {
            if (AbstractConnection.LOGGER.isDebugEnabled()) {
                AbstractConnection.LOGGER.debug("caught err:", e);
            }
            con.close("err:" + e);
        }

    }

如果AbstractConnection的writeBuffer有数据可写，先写writeBuffer。然后再写writeQueue写队列中的数据。

private boolean write0() throws IOException {

        int written = 0;
        ByteBuffer buffer = con.writeBuffer;
        if (buffer != null) {
            while (buffer.hasRemaining()) {
                written = channel.write(buffer);
                if (written > 0) {
                    con.netOutBytes += written;
                    con.processor.addNetOutBytes(written);
                    con.lastWriteTime = TimeUtil.currentTimeMillis();
                } else {
                    break;
                }
            }

            if (buffer.hasRemaining()) {
                con.writeAttempts++;
                return false;
            } else {
                con.writeBuffer = null;
                con.recycle(buffer);
            }
        }
        while ((buffer = con.writeQueue.poll()) != null) {
            if (buffer.limit() == 0) {
                con.recycle(buffer);
                con.close("quit send");
                return true;
            }

            buffer.flip();
            try {
                while (buffer.hasRemaining()) {
                    written = channel.write(buffer);// java.io.IOException:
                                    // Connection reset by peer
                    if (written > 0) {
                        con.lastWriteTime = TimeUtil.currentTimeMillis();
                        con.netOutBytes += written;
                        con.processor.addNetOutBytes(written);
                        con.lastWriteTime = TimeUtil.currentTimeMillis();
                    } else {
                        break;
                    }
                }
            } catch (IOException e) {
                con.recycle(buffer);
                throw e;
            }
            if (buffer.hasRemaining()) {
                con.writeBuffer = buffer;
                con.writeAttempts++;
                return false;
            } else {
                con.recycle(buffer);
            }
        }
        return true;
    }

数据写完了，取消对写事件的关注。

private void disableWrite() {
        try {
            SelectionKey key = this.processKey;
            key.interestOps(key.interestOps() & OP_NOT_WRITE);
        } catch (Exception e) {
            AbstractConnection.LOGGER.warn("can't disable write " + e + " con "
                    + con);
        }

    }

有数据需要写，关注写事件。

private void enableWrite(boolean wakeup) {
        boolean needWakeup = false;
        try {
            SelectionKey key = this.processKey;
            key.interestOps(key.interestOps() | SelectionKey.OP_WRITE);
            needWakeup = true;
        } catch (Exception e) {
            AbstractConnection.LOGGER.warn("can't enable write " + e);

        }
        if (needWakeup && wakeup) {
            processKey.selector().wakeup();
        }
    }

1.4 NIOReactor写事件处理

如果经过上面的流程数据还没有写完，那么会关注写事件，在NIOReactor类的RW线程类中在写事件就绪之后，会调用con.doNextWriteCheck()放写数据，实际上写的过程上我们是一样的，就不在重复。也是上图12~16的过程。

1.4.1 调用AbstractConnection的doNextWriteCheck()方法向客户端写数据

NIOReactor类中RW类的run方法，在写事件就绪后，调用con.doNextWriteCheck()方法写数据。

@Override
        public void run() {
            final Selector selector = this.selector;
            Set<SelectionKey> keys = null;
            for (;;) {
                ++reactCount;
                try {
                    selector.select(500L);
                    register(selector);
                    keys = selector.selectedKeys();
                    for (SelectionKey key : keys) {
                        AbstractConnection con = null;
                        try {
                            Object att = key.attachment();
                            if (att != null) {
                                con = (AbstractConnection) att;
                                if (key.isValid() && key.isReadable()) {
                                    try {
                                        con.asynRead();
                                    } catch (IOException e) {
                                        con.close("program err:" + e.toString());
                                        continue;
                                    } catch (Exception e) {
                                        LOGGER.warn("caught err:", e);
                                        con.close("program err:" + e.toString());
                                        continue;
                                    }
                                }
                                if (key.isValid() && key.isWritable()) {
                                    con.doNextWriteCheck();
                                }
                            } else {
                                key.cancel();
                            }
                        } catch (CancelledKeyException e) {
                            if (LOGGER.isDebugEnabled()) {
                                LOGGER.debug(con + " socket key canceled");
                            }
                        } catch (Exception e) {
                            LOGGER.warn(con + " " + e);
                        } catch (final Throwable e){
                            // Catch exceptions such as OOM and close connection if exists
                            //so that the reactor can keep running!
                            // @author Uncle-pan
                            // @since 2016-03-30
                            if(con != null){
                                con.close("Bad: "+e);
                            }
                            LOGGER.error("caught err: ", e);
                            continue;
                        }
                    }
                } catch (Exception e) {
                    LOGGER.warn(name, e);
                } catch (final Throwable e){
                    // Catch exceptions such as OOM so that the reactor can keep running!
                    // @author Uncle-pan
                    // @since 2016-03-30
                    LOGGER.error("caught err: ", e);
                } finally {
                    if (keys != null) {
                        keys.clear();
                    }

                }
            }
        }

AbstractConnection的doNextWriteCheck()放中实际上调用的NIOSocketWR类的doNextWriteCheck()方法，过程和上图的10~12的过程。如果经过13~17的流程数据还没有写完，则继续重复13~17的流程直到数据写完为止。