libiocp

Download
File Size Platforms Version Date
libiocp 1.00 3-Dec-2006.zip 16kb w2k/XP 1.00 03 Dec 06
libiocp 1.01 3-Dec-2006.zip 16kb w2k/XP 1.01 03 Dec 06
libiocp 1.02 4-Dec-2006.zip 17kb w2k/XP 1.02 04 Dec 06
libiocp 1.10 4-Mar-2007.zip 915kb w2k/XP 1.10 04 Mar 07
Codeview (latest version)

Table of Contents

  1. Introduction
  2. What's in the Zip File?
  3. Building the Software
  4. libiocp
  5. libiocp test
  6. list
  7. Essential OS Configuration
  8. Test Status
  9. Non-paged Pool Remarks
  10. License
  11. API Overview
  12. API Details
  13. Code Revision History

Introduction

libiocp is a minimalist implementation of a friendly, easy to use API over the I/O completion port API offered by Windows. I/O completion ports can be used with a range of I/O forms (disk, sockets, pipes, etc), but libiocp is only for use with sockets.

libiocp provides a solution to the C10K problem.

What's in the Zip File?

There are three directories;

I've included all the binaries produced by the build.

For the impatient, a direct link to the release build of the test binary - libiocp_test.exe.

Building the Software

I use a command line development environment, so each directory comes with a makefile. I use gnumake for makefiles. I'm not an expert makefile author, being pretty new at the art, so I'm afraid you have to enter the "rel" or "dbg" target - it won't work just to type "make".

The makefile for libiocp test does not contain dependency information on the libraries - you will need to manually remake them if necessary (and of course make them before you make libiocp test).

The software required to build is;

Probably the easiest thing for you GUI users to do is download the source files and make your own projects up.

libiocp

This is the library itself. The library files are in the "lib" directory, the public header file is in the "inc" directory.

libiocp test

This is the test client/server. The binary takes one argument, "c" or "s", which starts the binary up in client and server modes, respectively. The method of testing is first, in one dos window to run the server, then in another dos window, to run the client. The server instantiates a single libiocp object and puts a single listen socket into it, on port 2000. The client then connects as many sockets as possible. Once the client has connected as many sockets as he can, he then issues a single recv and a single send on each socket. The server, as soon as a connection is accepted, begins issuing send/recv calls on that socket. In the read and send callbacks for both the client and server, the next read and send call is made, thus continuing the constant flow of duplex data.

Each send is 4kb in length and contains π to 4,094 decimal places. The content is checked on receive. The checking code takes packet fragmentation into account.

list

This is a little, stripped down linked list library used by libiocp test. The libiocp test makefile knows where list is and so knows where the library and the public include file are.

Essential OS Configuration

Windows, by default, limits itself to creating outgoing sockets in the port range 1024 to 5000. A registry setting allows configuration of the upper value, and modification of this value is necessary or the largest number of supportable sockets is 3976.

The solution to given in this MSDN article.

Essentially, the registry key "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\MaxUserPort" has to be created and given a value. The key type is REG_DWORD and the highest value possible is 65534 (decimal), which is the recommended value.

I have saved a registry file, MaxUserPort65534.reg, containing just that key. You can download and merge that registry file with your registry to effect the change.

Note that a reboot is required after adjusting the registry setting.

Test Status

I've tested on my home 512mb W2K SP4 machine. Using a test client and server, my IP address and with MaxUserPort set to 20000, I was able to make just under 2,000 connections which are constantly sending and recieving data (e.g. just under 4,000 ports, since both ends of the connection were on my local machine).

I've run the test client/server for over an hour with no errors and no memory or handle leaks.

However, I've not yet tried manually simulating error conditions to check the error handling code.

I also need to do proper testing of graceful clean up (e.g. iocp_delete). That's probably the main outstanding issue right now.

Non-paged Pool Remarks

If the sending and receiving of data is not performed, the number of connections increases dramatically, up to about 16,000. This indicates it is true that an active socket uses more non-paged pool than an idle socket.

I strongly suspect that non-paged pool is the limiting factor, since when the test client/server is running with the largest number of sockets it can connect, I am then unable to perform any actions which require non-paged pool - e.g. I cannot play sound, enumerate services, etc.

The absolute maximum non-paged pool on a W2K machine is 256mb. This is specified by the registry key "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\NonPagedPoolSize". The default value 0 lets the system choose the maximum on startup, depending on a couple of factors, such as the total memory available. I believe my current value is 128mb, and with just under 2,000 active sockets, I'm using about 124mb of that memory.

I've not yet experimented setting this value to a higher number.

License

I'd like this software to do as much good as possible, which means being used as much as possible. So you are completely free to use this software in any and every way and specifically, if you can, to make money with it, because that means by my efforts, I have reduced the costs of developing software, which will increase the profit someone will make from the effort they have invested making their software, which in turn will increase the capital available to employ people.

API Overview

Essentially, you instantiate a libiocp object, give it a bunch of callbacks and then add listen or already connected sockets to it. When you add a socket, you get a libiocp state back for that socket. If a new connection comes in, the accept callback is called, and you get a libiocp state back for that new socket. If you want to send or receive, issue the libiocp send or read function calls. When they complete, the appropriate callback is called. If you want to continue reading and sending, issue another send or read call in the callback.

  1. int iocp_new( void **iocp_state, size_t read_buffer_size,
                  void (*accept_callback_function)(void *iocp_state, void *iocp_socket_state, struct sockaddr *local_address, struct sockaddr *remote_address, void *listen_socket_user_state, void **per_socket_user_state),
                  void (*recv_callback_function)(void *iocp_state, void *iocp_socket_state, void *read_buffer, DWORD byte_count, void *per_socket_user_state),
                  void (*send_callback_function)(void *iocp_state, void *iocp_socket_state, void *per_socket_user_state, void *per_write_user_state),
                  void (*error_callback_function)(void *iocp_state, void *iocp_socket_state, int operation_type, void *per_socket_user_state, void *per_write_user_state) );
  2. void iocp_delete( void *socket_iocp_state );
  3. int iocp_add_listen_socket( void *iocp_state, SOCKET socket, size_t accept_socket_backlog, void *listen_user_state );
  4. int iocp_add_connected_socket( void *iocp_state, void **iocp_socket_state, SOCKET socket, void *per_socket_user_state );
  5. int iocp_issue_recv( void *iocp_state, void *iocp_socket_state );
  6. int iocp_issue_sync_send( void *iocp_state, void *iocp_socket_state, void *write_buffer, DWORD number_bytes_to_write, void *per_write_user_state );
  7. int iocp_issue_async_send( void *iocp_state, void *iocp_socket_state, void *write_buffer, DWORD number_bytes_to_write, void *per_write_user_state );

API Details

int iocp_new( void **iocp_state, size_t read_buffer_size,
              void (*accept_callback_function)(void *iocp_state, void *iocp_socket_state, struct sockaddr *local_address, struct sockaddr *remote_address, void *listen_socket_user_state, void **per_socket_user_state),
              void (*recv_callback_function)(void *iocp_state, void *iocp_socket_state, void *read_buffer, DWORD byte_count, void *per_socket_user_state),
              void (*send_callback_function)(void *iocp_state, void *iocp_socket_state, void *per_socket_user_state, void *per_write_user_state),
              void (*error_callback_function)(void *iocp_state, void *iocp_socket_state, int operation_type, void *per_socket_user_state, void *per_write_user_state) );

This function instantiates a libiocp state object, which is the first argument to all the other libiocp functions. The read buffer size parameter is the size of the buffer, interally allocated, which is used with ReadFile(), and must be SOCKET_IOCP_MINIMUM_READ_BUFFER_SIZE bytes in size or greater. The four callback functions are discussed in their own sections. Note that the error callback is asserted; it must not be NULL, but the other callbacks can be. However, other functions will assert the callbacks they need.

void iocp_delete( void *socket_iocp_state );

Cleans up and deallocates a libiocp state.

This function must only be called after all sockets registered with the API have closed (e.g. had closesocket() called on them or failed in some way). If it is called prior to this point, an exception will occur when the next socket read or error occurs.

Note that when a socket closes (from any causes), the OS automatically removes it from the I/O completion port, socket list, which is why there is no iocp_delete_socket() function. The API internally knows when a socket has closed and cleans up its internal resource allocations for that socket.

int iocp_add_listen_socket( void *iocp_state, SOCKET socket, size_t accept_socket_backlog, void *listen_user_state );

To add a listen socket, call socket(), bind() and listen(), and then pass that socket into this function. Note that the accept socket backlog here is rather different to the normal backlog value used with listen. An I/O completion port has a number of already-allocated sockets (created with socket() and unbound) associated with each listen socket. This gives a pool of sockets immediately ready to be used for incoming connections. The accept socket backlog parameter here is the number of ready accept sockets to maintain for the listen socket. The API internally allocates replacement sockets as incoming connections occur.

int iocp_add_connected_socket( void *iocp_state, void **iocp_socket_state, SOCKET socket, void *per_socket_user_state );

Does as it says on the tin. The user can pass in a state pointer of his own for this socket, which comes back in the callback functions.

int iocp_issue_recv( void *iocp_state, void *iocp_socket_state );

Issues a read call on the socket. When the read completes, the read callback function is called.

int iocp_issue_sync_send( void *iocp_state, void *iocp_socket_state, void *write_buffer, DWORD number_bytes_to_write, void *per_write_user_state );

Issues a send on the socket. When the send completes, the send callback function is called. Using this function, only one send can be outstanding on a socket at any one time, and it is the user's responsibility to ensure this is so. (So if you call iocp_issue_sync_send() on a socket, you can't call it again until the send callback has been called. You can call it in the send callback, though; you don't have to let the callback return first.)

User state for this write can be provided, this, along with the per socket state, will come back to the user in the callback function.

int iocp_issue_async_send( void *iocp_state, void *iocp_socket_state, void *write_buffer, DWORD number_bytes_to_write, void *per_write_user_state );

Issues a send on the socket. When the send completes, the send callback function is called. Using this function, any number of sends can be outstanding on a single socket at any time. The overhead of this is one malloc per send. This overhead is not present in iocp_issue_sync_send().

User state for this write can be provided, this, along with the per socket state, will come back to the user in the callback function.

void accept_callback_function( void *iocp_state, void *iocp_socket_state, struct sockaddr *local_address, struct sockaddr *remote_address, void *listen_socket_user_state, void **per_socket_user_state );

This function is called when a connection occurs on a listen socket. local_address and remote_address contain the IP addresses of the local and remote ends of the socket. listen_socket_user_state is the user state given in int iocp_add_listen_socket. The user can provide per socket user state by setting the "void *" that per_socket_user_state points to. This value set by the user will be provided in the callbacks as the per socket state pointer.

void recv_callback_function( void *iocp_state, void *iocp_socket_state, void *read_buffer, DWORD byte_count, void *per_socket_user_state );

Called when a read completes.

void send_callback_function( void *iocp_state, void *iocp_socket_state, void *per_socket_user_state, void *per_write_user_state );

Called when a send completes. This callback is called for both sync and async writes.

void error_callback_function( void *iocp_state, void *iocp_socket_state, int operation_type, void *per_socket_user_state, void *per_write_user_state );

After any error on a socket, the API internally calls closesocket() on that socket. This is unfortunately necessary, due to the inability to disassociate a socket from an I/O completion port once association has occured.

Code Revision History

1.00 03-Dec-2006 Initial release.
1.01 03-Dec-2006 Changed library type from single threaded to multi-threaded. Changed code generation for debug variant from blend to Pentium Pro.
1.02 04-Dec-2006 Added comprehensive simple error checking, where the return value from the function indicates success or error. Added some comments to the source code.
1.10 04-Mar-2007 Added support for listen sockets and writes.