[Zlib-devel] zlib 1.2.4 released!

[John Bowler]

From: zlib-devel-bounces at madler.net [mailto:zlib-devel-bounces at madler.net] On Behalf Of William A. Rowe Jr.
>It might look something like;
>
>  ZEXTERN gzFile ZEXPORT gzwhopen OF((HANDLE file, const char *mode));
>
>  #define gzdopen(fd, mode) gzwhopen((HANDLE)_get_osfhandle(fd), mode)

Yes, that works, presumably surrounded by '#ifdef WIN32'.  Are there any other OSes where the POSIX layer is not native, so may suffer from the same issues?

> Of course gzdopen needs to be exported for a fair
> amount of time, at least as long as 1.2.x, but for a casual user this would
> avoid penalizing them for any mismatch of CRTs.

Right, but there's no significant overhead from continuing to export it (it's just a function call to _get_osfhandle then a tail call to gzwhopen.)

>There is the issue that CRT's fd buffering wouldn't correspond to the raw
>handle's buffering.  That is a serious problem that needs more consideration.

I would expect the CRT not to buffer anything.  Certainly on UNIX after a write() call all the user mode data has been copied to the kernel and before a read() call there is no pending data in user space; no data between where lseek() says the point is and where the pointer actually is in the kernel.

Indeed, surely the major change happened between 1.2.3 and 1.2.4?  In 1.2.3 fread() is called in several places and the data comes from the stdio buffer.  In 1.2.4 read() is called only in gz_load (gzread.c) but gz_load replaces fread() and GZ implements its own buffer (gzbuffer, added in 1.2.4).  Now, on UNIX, there is no other user-mode buffer and data goes directly from the kernel to the GZ buffer.

I've always avoided stdio whenever I have my own buffer.  GZ itself already has buffers on both the input and the output: for state->in and state->out buffers that are malloced within gzread.c and gzwrite.c.  In addition the caller provides a buffer (output for read, input for write) and internally inflate() (for write) supplies a simply enormous input buffer (the window.)

Recent tests for libpng suggest that reducing the size of the buffers in zlib can even increase inflate performance.  I ended up with an optimal output size for inflate() of 128 bytes when decompressing an optimally compressed stream (a 64Mbyte stream of 0).  Maybe a ridiculous test case but the message is that copying stuff around in the memory cache can create a severe performance hit.

John Bowler