[Zlib-devel] Re: [png-implement] Binary-incompatible change in the DLL interface of zlib

[Cosmin Truta]

On Mon, 28 Jul 2003, Gerard Juyn wrote:

> > - There are a couple that do not support STDCALL.
>
> That's quite amazing. What are those? How on earth could they access the
> Windows core API which is all STDCALL?

Let me put it this way:

There are compilable programming languages that offer source portability
across various operating systems. In this case, programmers are not
supposed to call local system API. Those languages offer interfacing
with other external languages via C bindings. Examples are Fortran and
Ada. Since VB and Delphi are intrinsically dependent on the Windows API,
they are not part of this set of portable languages. (Well, Delphi
solved it somehow in Kylix, but that's still different. A Delphi/Kylix
program is not "a portable program".)

Let's take an example.

An Ada program that has to be portable should NOT be polluted with
"pragma Convention(Stdcall, some_function)" constructs.

On the other hand, an Ada program that depends on the Windows API is not
portable, so using this pragma is not an issue.

But remember that people writing Ada programs using zlib have all the
rights to demand portability. Even more so, if a zlib program is
developed under Unix, it will most certainly not contain anything like
this, and that program will fail to work under Windows if a convention
different than CDECL is used.


> As you know, the purpose of a standard DLL is that you don't recompile
> it.  You need an import-library or import-unit (in the case of Delphi)
> which automatically links to the standard DLL, and loads it at
> run-time, or if you make some smart glue-code it can be loaded on
> demand. I wouldn't go that way with zlib unless it is used rather
> infrequently in several apps. But I do go that way with the standard
> libmng.dll and the Delphi interface. Works like a charm.

I'm sure it works, and I also have Borland-compiled programs that work
fine with MS libs. But this is because the compatibility problems are
avoided, luckily, not intentionally. It's described in Question 7 in
the DLL_FAQ:

  http://www.cs.toronto.edu/~cosmin/pngtech/zlib/DLL_FAQ.txt

I run into problems myself, I remember, when I tried to use the GLUT
DLL. My simplest OpenGL programs, as well as "official" OpenGL demos,
crashed pseudo-randomly when I compiled them with Borland C or with
Cygwin, but they run fine when I compiled them with MSC or with MinGW.

So if the internal state of the CRTL is kept intact, nothing wrong
happens and everything should work fine. Does that happen in zlib? I
would like to know the answer is "yes"; or if the answer is "no",
I would like to know how to avoid the clash caused by intermixing.
But until we know that, we cannot simply rely on the "It Works For
Me (TM)" paradigm.

The Cygwin people were nice and they started maintaining their own
CYGZ.DLL a long time ago.

One more thing:
The issue with Borland is moot. zlib is included in the standard VCL,
(and also in the standard VCL DLL, of course), and is compiled with
FASTCALL. I made a binary-compatible update, it's in
"zlib-1.2.0.f-cos1/contrib/delphi2/".



> As it is now I've hard-linked zlib into libmng.dll because I can't
> rely on the rest of the world (for historic reasons, which you are
> familiar with) to have the same copy of zlib.dll.

Good choice! ;)

Of course, security updates in zlib would imply the need to rebuild
libmng.dll, but that is much less of a problem.



> So far, I've run into 3 different kinds, but I'm sure
> there will be more floating around.

Me too.
*This* *is* *why* ZLIB.DLL *has* *to* *be* *discontinued*!!
The new ZLIB1.DLL *must* use a unique header, and leave no room for
modifications, other than the "official" ones.


> I would want to split libmng.dll into its constituant parts. But both
> zlib and the ijgsrc have a nasty design flaw. They depend on
> structures. Structures change. That's an age old problem we're all
> familiar with. lcms is a different story. It was designed more
> recently and flourishes as a standalone DLL.

In zlib, at least, the public z_stream structure didn't change since I
started to be familiar with zlib (that is, since zlib-1.0.4). It could
have been the same even earlier. There are additional structures used by
deflation and inflation, and those change. But, as in lcms, this is
opaque to the user, and binary compatibility is preserved.

The internal manipulation is done via the zstream.state field.

In other words, *very unofficially*, I can advise you that you can
safely rely on the stability of the z_stream structure, for a few more
versions to come. The final answer is up to Mark and Jean-loup, of
course. But given the experience, and the fact that everything went
smoothly even when the inflation code was changed significantly (from
zlib-1.1.4 to zlib-1.2.0), I think you shouldn't worry very much.



> > Not for the CRT DLLs, and not even for the MS demos that use the C API
> > (not the Windows API). See again Question 5.
>
> Do you want to consider zlib as giving core functionality (something that
> should have been in the core Windows API in the first place), or do you
> consider a useful addition?
> If the first than STDCALL should be your choice. If the second I'd
> still say go with STDCALL. I certainly wouldn't consider MS samples a
> way-of-life.

zlib does not offer "Windows" core functionality, but "C" core
functionality. A rule of thumb for the Windows core functionality is not
to use C API, except, perhaps, that provided in NTDLL.DLL (which btw.
is missing in Windows 9X). The reason originates in the intermixing
problems, explained in Question 7.

Look in the "include/" directory in your favorite Windows SDK, and see
how many headers include <stdio.h> and use WINAPI for the exported
functions. It shouldn't be more than 2 or 3.


> > If you search for "libpng.dll" over the png-implement archive in 2000,
> > you will see that Simon-Pierre Cadieux and I proposed to use STDCALL,
> > but then decided to use CDECL to avoid the (too many) problems that we
> > could have run into.
>
> Yes. I saw the discussion, but didn't feel like budding in. A standard
> libpng.dll is of little use to me. A standard zlib.dll is, but not
> with CDECL convention.

Aside from the VB problems, why isn't CDECL useful to you?

> As I remember some of the problems were name-mangling and even
> performance reasons.

Name mangling is yet another side effect of using a different calling
convention. Trust me. I remember this so well, mainly because
<shameless>it was me who eventually figured it out and solved the
problem</shameless>. The solution was "build the DLL by defining
ZLIB_DLL, and don't use the zlib-provided makefile". The outcome now
resides in "libpng/projects/msvc/zlib.dsp".

> The latter I really can't follow as we're talking about high-level API
> functions that are called a few hundred times at most, more likely a
> lot less. So you'd save a few msecs to decompress a 2MB zipfile, so
> what...

As a matter of fact, performance reasons were discussed in the favor of
STDCALL. At the end, we concluded that all the problems implied by
STDCALL don't worth the minor performance impact. We also concluded that
it's easier to have a VB-only libpng DLL, and another DLL for the rest
of the world.

In zlib it's even more irrelevant, because the zlib functions are
virtually never expected to be run a thousand times.


> > The nastiest problem was caused by having different
> > conventions for the static lib and for the DLL.
>
> (I think John Bowler just answered this one...)
>
> A DLL and static lib have nothing to do with each other. Perhaps the test or
> development environment wasn't suited for one or the other, or someone was
> trying to use both from a single test-app. Consider a DLL and static-lib
> separate items. Also a static-lib isn't a static-lib between different
> development platforms.

It does when the user compiles the static lib, and then compiles the
DLL, without knowing that something has to be changed. Obviously, the
zlib test programs are working well, too. Then ZLIB.DLL does't work with
LIBPNG.DLL (or with any other application that expects a different
convention), and everyone wonders why. Last time this happened, it
lasted a couple of months.


> > The problem could have been avoided by having STDCALL (or whatever) all
> > the way, but that was unacceptable when using zlib in other programming
> > languages. From the point of view of foreign languages (because there
> > are other issues as well), I think it's better to choose a single victim
> > (that is, the older Visual Basic), than many others. Besides, in VB,
> > some people would like to use COM interfaces and IDL/GUID/etc. junk, so
> > they would probably need a customized build of the DLL, anyway.
>
> zlib is used extensively in Windows as well. Perhaps you could ask
> them to provide a standard interface for all their own
> dev-platforms.... :)
>
> Just kidding of course.

You are right. Some of the things got fixed in Microsoft.NET.

The CDECL decision was based on the fact that it's better to sacrifice
and old VB by having a separate DLL for it (the new VB.NET works fine),
rather than sacrificing a few more other programming languages by not
allowing to write portable, platform-independent programs, when using
zlib.


> But if you and others in this project think
> you've made the right choice, than that's fine with me. It just won't
> be useful in my case.  Doesn't matter, I've got other fish to fry
> anyway. Just confirming my 2 eurocents....

The choice is not really final, and I wish I had posted this to libpng
earlier.

Counter-arguments can still be brought, and they will be incorporated
into the DLL_FAQ. This file essentially constitutes the specification
of the new DLL.

  http://www.cs.toronto.edu/~cosmin/pngtech/zlib/DLL_FAQ.txt

But the desire to use STDCALL everywhere in order to have a single DLL
is something that I consider less important than the desire to use CDECL
everywhere in order to allow portable programs in non-C programming
languages. I prefer to sacrifice the comfort of the old VB (whose
redistributable files, by the way, take a few megs anyway) instead of
crippling the rest of the non-C world.

But I am still open to counter-attacks :)


Cosmin