[Zlib-devel] pigz 2.1.4 output differs for -p 1 and -p 2

[Mark Adler]

On Nov 13, 2008, at 11:20 PM, Pilorz Wojciech wrote:
> It seems getting difference without -i is more difficult, but still  
> happens
>

Wojtek,

Thanks for the example.  I see what's going on there.

This note is in zlib.h for deflateSetDictionary():

     In addition, the current implementation of deflate will use at most
     the window size minus 262 bytes of the provided dictionary.

(Since it's documented, it must be a feature, not a bug.)

Parallel compression uses deflateSetDictionary() to provide the  
preceding 32K in order for that history to be available to compress  
the next block.  But per the note, not all 32K is actually used.  Non- 
parallel compression simply continues compressing, so all of the  
previous 32K is available.  Therefore the result ends up being  
different between the two, with non-parallel getting better  
compression when long-distance matches would be used.

Mark