[Zlib-devel] pigz 2.1.4 output differs for -p 1 and -p 2

[Mark Adler]

On Nov 15, 2008, at 1:40 PM, Mark Adler wrote:
> Now I'll look into remedies for the differences due to  
> deflateSetDictionary(), i.e. when not using -i.

Turns out it wasn't just deflateSetDictionary() using less than the  
provided 32K.  (Though the reason I thought that was important wasn't  
the real reason, but I digress.)  There is no good solution when  
compression level 3 or less is used.  In that case, one of the ways  
that deflate trades of compression for speed is to simply not insert  
all of the strings in the hash table used for match searching.  What  
it does or does not insert depends on the matches it gets.  As a  
result, deflateSetDictionary() cannot simulate that building of the  
hash table.

Therefore to get consistent pigz output for -p 1 and -p 2 without -i,  
you'd need the patch below for zlib, and you can't use compression  
level 3 or below.

Mark


diff -c zlib-1.2.3.3/deflate.c zlib-1.2.3.4/deflate.c
*** zlib-1.2.3.3/deflate.c	2006-09-03 23:57:22.000000000 -0700
--- zlib-1.2.3.4/deflate.c	2008-11-15 23:25:42.000000000 -0800
***************
*** 332,339 ****
           strm->adler = adler32(strm->adler, dictionary, dictLength);

       if (length < MIN_MATCH) return Z_OK;
!     if (length > MAX_DIST(s)) {
!         length = MAX_DIST(s);
           dictionary += dictLength - length; /* use the tail of the  
dictionary */
       }
       zmemcpy(s->window, dictionary, length);
--- 332,339 ----
           strm->adler = adler32(strm->adler, dictionary, dictLength);

       if (length < MIN_MATCH) return Z_OK;
!     if (length > s->w_size) {
!         length = s->w_size;
           dictionary += dictLength - length; /* use the tail of the  
dictionary */
       }
       zmemcpy(s->window, dictionary, length);