[Zlib-devel] Statically allocating memory for inflate/deflate

[Stewart Smith]

Hi!

I work on MySQL Cluster - a HA clustered database engine with some
realtime properties. One of the things I'm currently working on is
compressed backup and (local) check pointing for MySQL Cluster (NDB).
One of the limitations we have is that we can't malloc at runtime (as it
may either fail or take too long). All our file IO happens in separate
threads (as this is the only reliable way to get truly non-blocking IO..
including things like open(2)) and I'm doing the compression there.

So I need to statically allocate memory on startup of the daemon (not at
runtime, i.e. when the node is in the cluster).

We have a modified gzio called azio. The main change is to have the
buffers be statically allocated (and for NDB, 512byte aligned for
O_DIRECT). For the ARCHIVE engine, this gave a decent performance boost.
For NDB, we need to do direct IO to get around some linux memory manager
limitations on common setups (it behaves badly with lots of IO and lots
of pages locked in main memory).

But deflate and inflate need work... Although you can specify your own
memory allocation routines (which is useful), you can't find out (by
quizzing zlib) an upper bound on memory usage (for me to allocate at
startup).

My current "solution" is rather hackish...

Index: telco-6.2/zlib/inflate.h
===================================================================
--- telco-6.2.orig/zlib/inflate.h	2007-10-19 14:28:49.534071601 +1000
+++ telco-6.2/zlib/inflate.h	2007-10-19 14:29:25.535837366 +1000
@@ -16,6 +16,9 @@
 #  define GUNZIP
 #endif
 
+#ifndef __ZLIB_INFLATE_H__
+#define __ZLIB_INFLATE_H__
+
 /* Possible inflate modes between inflate() calls */
 typedef enum {
     HEAD,       /* i: waiting for magic header */
@@ -113,3 +116,5 @@ struct inflate_state {
     unsigned short work[288];   /* work area for code table building */
     code codes[ENOUGH];         /* space for code tables */
 };
+
+#endif


Which then lets me do something like this in azio:


#include "../../../../zlib/zutil.h"
#include "../../../../zlib/zconf.h"
#include "../../../../zlib/inftrees.h"
#include "../../../../zlib/inflate.h"
#include "../../../../zlib/deflate.h"

#define AZ_MEMLEVEL 8

struct az_alloc_rec {
  size_t size;
  size_t mfree;
  char *mem;
};


size_t az_inflate_mem_size()
{
  return sizeof(struct inflate_state)
    + ((1U << MAX_WBITS)*sizeof(unsigned char));
}

size_t az_deflate_mem_size()
{
  return sizeof(deflate_state)
    + ((1U << MAX_WBITS)*(2*sizeof(Byte)))
    + ((1U << MAX_WBITS)*sizeof(Pos))
    + ((1U << (AZ_MEMLEVEL+7))*sizeof(Pos))
    + ((1U << (AZ_MEMLEVEL+6))*(sizeof(ush)+2));
}

voidpf az_alloc(voidpf opaque, uInt items, uInt size)
{
  struct az_alloc_rec *r = (struct az_alloc_rec*)opaque;

  if((items * size) > r->mfree)
    abort();

  r->mfree -= items*size;

  return (r->mem + r->size) - r->mfree;
}
void az_free(voidpf opaque, voidpf address)
{
  // Oh how we hack.
  struct az_alloc_rec *r = (struct az_alloc_rec*)opaque;
  r->mfree = r->size;
}


And then, in startup of the file thread:

az_mempool.size = az_mempool.mfree = 
		az_inflate_mem_size()+az_deflate_mem_size();
ndbout_c("NDBFS/AsyncFile: Allocating %d for In/Deflate"
	"buffer",az_mempool.size);
az_mempool.mem = (char*) ndbd_malloc(az_mempool.size);
azf.stream.opaque= &az_mempool;

Which gets me 310256 bytes allocated (and things seem to be working
okay.... apart from my other bugs :)

Of course, the nice change would be having az_deflate_mem_size and
az_inflate_mem_size inside zlib so that they always return the correct
values if any of the internal structures change (instead of what would
happen with my code here which is to possibly abort() at runtime).

Thoughts very much appreciated.

cheers,
stewart
-- 
Stewart Smith, Senior Software Engineer
MySQL AB, www.mysql.com
Office: +14082136540 Ext: 6616
VoIP: 6616 at sip.us.mysql.com
Mobile: +61 4 3 8844 332

Jumpstart your cluster:
http://www.mysql.com/consulting/packaged/cluster.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 827 bytes
Desc: This is a digitally signed message part
URL: <http://madler.net/pipermail/zlib-devel_madler.net/attachments/20071019/aacf9785/attachment.sig>