looking for a compression library

P

pisky

Hi,
I have written a soap client for excel using WinHttp. The API I am calling
uses big attachments, hence I would like to send and receive compressed
messages. I just need to find a library that I can use such as gzip or
deflate. I am new to using MS products and haven't yet found a way of quickly
finding the libraries I need. Incase it makes a difference, I am using excel
2007.

tanks
p
 
K

Karl E. Peterson

pisky said:
I have written a soap client for excel using WinHttp. The API I am calling
uses big attachments, hence I would like to send and receive compressed
messages. I just need to find a library that I can use such as gzip or
deflate. I am new to using MS products and haven't yet found a way of quickly
finding the libraries I need. Incase it makes a difference, I am using excel
2007.
Click to expand...

I've found ZLib very useful.

Main home page: http://www.zlib.net/
Windows build: http://www.winimage.com/zLibDll/
 
P

Patrick Schmid [MVP]

P

pisky

Thanks Karl. This library is what I am looking for. I downloaded it but
cannot seem to get a reference to it in Excel. I tried running RegSvr32 but
get the message "zlibwapi.dll was loaded, but the DllRegisterServer entry
point was not found". What do I need to do to use the dll?

many thanks
P
 
K

Karl E. Peterson

pisky said:
Thanks Karl. This library is what I am looking for. I downloaded it but
cannot seem to get a reference to it in Excel. I tried running RegSvr32 but
get the message "zlibwapi.dll was loaded, but the DllRegisterServer entry
point was not found". What do I need to do to use the dll?
Click to expand...

It's not based on COM. You just call it directly. Details are very C-oriented, and
can be found in the zlib.h file. Here are the VB(A) declarations for the two main
functions I've used:

Private Declare Function compress Lib "zlibwapi.dll" (dest As Any, destLen As
Any, src As Any, ByVal srcLen As Long) As Long

Private Declare Function uncompress Lib "zlibwapi.dll" (dest As Any, destLen As
Any, src As Any, ByVal srcLen As Long) As Long

Then I wrote these two functions to handle the housekeeping:

Private Function DecompressData(Data() As Byte, ByVal OriginalSize As Long) As
Long
Dim BufferSize As Long
Dim Buffer() As Byte
Dim nResult As Long

' Allocate the smallest allowed compression buffer
' (1% larger than the uncompressed data plus 12 bytes).
BufferSize = (OriginalSize * 1.01) + 12
ReDim Buffer(0 To BufferSize - 1) As Byte

' Decompress data
nResult = uncompress(Buffer(0), BufferSize, Data(LBound(Data)), UBound(Data) -
LBound(Data) + 1)

' Truncate buffer to compressed size
ReDim Data(0 To BufferSize - 1) As Byte
CopyMemory Data(0), Buffer(0), BufferSize

' Return error code (if any)
DecompressData = nResult
End Function

Public Function CompressData(Data() As Byte) As Long
Dim OriginalSize As Long
Dim BufferSize As Long
Dim Buffer() As Byte
Dim nResult As Long

' Build working buffer for ZLib must be at
' least 101% of original plus 12 bytes.
OriginalSize = (UBound(Data) - LBound(Data) + 1)
BufferSize = OriginalSize * 1.01 + 12
ReDim Buffer(0 To BufferSize - 1) As Byte

' Compress byte array (data).
nResult = compress(Buffer(0), BufferSize, Data(LBound(Data)), OriginalSize)

' Truncate original array to compressed size.
ReDim Data(0 To BufferSize - 1) As Byte
CopyMemory Data(0), Buffer(0), BufferSize

' Return error code (if any).
CompressData = nResult
End Function

It's a bit involved. As you can see, you must insure that your container's file
format is self-descriptive enough to reveal the original size of the uncompressed
data.
 
S

Sinna

Karl said:
It's not based on COM. You just call it directly. Details are very C-oriented, and
can be found in the zlib.h file. Here are the VB(A) declarations for the two main
functions I've used:

Private Declare Function compress Lib "zlibwapi.dll" (dest As Any, destLen As
Any, src As Any, ByVal srcLen As Long) As Long

Private Declare Function uncompress Lib "zlibwapi.dll" (dest As Any, destLen As
Any, src As Any, ByVal srcLen As Long) As Long

Then I wrote these two functions to handle the housekeeping:

Private Function DecompressData(Data() As Byte, ByVal OriginalSize As Long) As
Long
Dim BufferSize As Long
Dim Buffer() As Byte
Dim nResult As Long

' Allocate the smallest allowed compression buffer
' (1% larger than the uncompressed data plus 12 bytes).
BufferSize = (OriginalSize * 1.01) + 12
ReDim Buffer(0 To BufferSize - 1) As Byte

' Decompress data
nResult = uncompress(Buffer(0), BufferSize, Data(LBound(Data)), UBound(Data) -
LBound(Data) + 1)

' Truncate buffer to compressed size
ReDim Data(0 To BufferSize - 1) As Byte
CopyMemory Data(0), Buffer(0), BufferSize

' Return error code (if any)
DecompressData = nResult
End Function

Public Function CompressData(Data() As Byte) As Long
Dim OriginalSize As Long
Dim BufferSize As Long
Dim Buffer() As Byte
Dim nResult As Long

' Build working buffer for ZLib must be at
' least 101% of original plus 12 bytes.
OriginalSize = (UBound(Data) - LBound(Data) + 1)
BufferSize = OriginalSize * 1.01 + 12
ReDim Buffer(0 To BufferSize - 1) As Byte

' Compress byte array (data).
nResult = compress(Buffer(0), BufferSize, Data(LBound(Data)), OriginalSize)

' Truncate original array to compressed size.
ReDim Data(0 To BufferSize - 1) As Byte
CopyMemory Data(0), Buffer(0), BufferSize

' Return error code (if any).
CompressData = nResult
End Function

It's a bit involved. As you can see, you must insure that your container's file
format is self-descriptive enough to reveal the original size of the uncompressed
data.
Click to expand...
This is not really an issue if you can allocate a buffer that's large
enough to decompress all data. In the application I've implemented I
make a suggestion of the expected uncompressed size, with a minimum
default size.


Perhaps off-topic, but...

In the code snippet above the comments mention that the buffer size
should be at least 101% of the original plus 12 bytes.
Where does that come from? I've been using the zlibwapi library for a
long time and when I know the uncompressed size on forehand, I allocate
the buffer with the uncompressed size (not 101% + 12).

Can you give me a direction to some document stating this requirement?
Googling around doesn't give me a decent result.

TIA,
Sinna
 
S

Sinna

Karl said:
It's not based on COM. You just call it directly. Details are very C-oriented, and
can be found in the zlib.h file. Here are the VB(A) declarations for the two main
functions I've used:

Private Declare Function compress Lib "zlibwapi.dll" (dest As Any, destLen As
Any, src As Any, ByVal srcLen As Long) As Long

Private Declare Function uncompress Lib "zlibwapi.dll" (dest As Any, destLen As
Any, src As Any, ByVal srcLen As Long) As Long

Then I wrote these two functions to handle the housekeeping:

Private Function DecompressData(Data() As Byte, ByVal OriginalSize As Long) As
Long
Dim BufferSize As Long
Dim Buffer() As Byte
Dim nResult As Long

' Allocate the smallest allowed compression buffer
' (1% larger than the uncompressed data plus 12 bytes).
BufferSize = (OriginalSize * 1.01) + 12
ReDim Buffer(0 To BufferSize - 1) As Byte

' Decompress data
nResult = uncompress(Buffer(0), BufferSize, Data(LBound(Data)), UBound(Data) -
LBound(Data) + 1)

' Truncate buffer to compressed size
ReDim Data(0 To BufferSize - 1) As Byte
CopyMemory Data(0), Buffer(0), BufferSize

' Return error code (if any)
DecompressData = nResult
End Function

Public Function CompressData(Data() As Byte) As Long
Dim OriginalSize As Long
Dim BufferSize As Long
Dim Buffer() As Byte
Dim nResult As Long

' Build working buffer for ZLib must be at
' least 101% of original plus 12 bytes.
OriginalSize = (UBound(Data) - LBound(Data) + 1)
BufferSize = OriginalSize * 1.01 + 12
ReDim Buffer(0 To BufferSize - 1) As Byte

' Compress byte array (data).
nResult = compress(Buffer(0), BufferSize, Data(LBound(Data)), OriginalSize)

' Truncate original array to compressed size.
ReDim Data(0 To BufferSize - 1) As Byte
CopyMemory Data(0), Buffer(0), BufferSize

' Return error code (if any).
CompressData = nResult
End Function

It's a bit involved. As you can see, you must insure that your container's file
format is self-descriptive enough to reveal the original size of the uncompressed
data.
Click to expand...

Another comment on the code snippet you posted (again about the working
buffer size):

Following
http://edais.mvps.org/Tutorials/ZLib/Files/EDais - Using ZLib in VB.pdf
there's a function available to determine the maximum size the
compressed data can be given the uncompressed size:

Private Declare Function compressBound Lib "ZLibWAPI.dll" ( _
ByVal srcLen As Long) As Long

So the following line in the CompressData call can be changed from:
BufferSize = OriginalSize * 1.01 + 12 into
BufferSize = compressBound(OriginalSize)
Click to expand...

Sinna
 
K

Karl E. Peterson

Sinna said:
This is not really an issue if you can allocate a buffer that's large
enough to decompress all data. In the application I've implemented I
make a suggestion of the expected uncompressed size, with a minimum
default size.
Click to expand...

I "worked around" it by knowing how to compute the expected size of the compressed
data based on values stored in my data file's header.
Perhaps off-topic, but...

In the code snippet above the comments mention that the buffer size
should be at least 101% of the original plus 12 bytes.
Where does that come from? I've been using the zlibwapi library for a
long time and when I know the uncompressed size on forehand, I allocate
the buffer with the uncompressed size (not 101% + 12).

Can you give me a direction to some document stating this requirement?
Googling around doesn't give me a decent result.
Click to expand...

Here: http://www.zlib.net/manual.html

From the function description for compress:

=======================
int compress (Bytef *dest, uLongf *destLen, const Bytef *source, uLong sourceLen);
Compresses the source buffer into the destination buffer. sourceLen is the byte
length of the source buffer. Upon entry, destLen is the total size of the
destination buffer, which must be at least 0.1% larger than sourceLen plus 12 bytes.
Upon exit, destLen is the actual size of the compressed buffer.
=======================

I see that compressBound (the function you mention in a subsequent post) wasn't part
of the library at the time this documentation was produced. Re-reading that though,
assuming one wanted to take it at face value, I wonder if I did misinterpret it --
should I have used 1.0001 rather than 1.01 above?

I can't find official docs on v1.23 -- is the zlib.h file it?
 
K

Karl E. Peterson

Sinna said:
Following
http://edais.mvps.org/Tutorials/ZLib/Files/EDais - Using ZLib in VB.pdf
there's a function available to determine the maximum size the
compressed data can be given the uncompressed size:

Private Declare Function compressBound Lib "ZLibWAPI.dll" ( _
ByVal srcLen As Long) As Long

So the following line in the CompressData call can be changed from:
Click to expand...

Very cool! Excellent resource, there, too. Archived. Thanks much... :)
 
S

Sinna

Karl E. Peterson wrote:
Here: http://www.zlib.net/manual.html

From the function description for compress:

=======================
int compress (Bytef *dest, uLongf *destLen, const Bytef *source, uLong sourceLen);
Compresses the source buffer into the destination buffer. sourceLen is the byte
length of the source buffer. Upon entry, destLen is the total size of the
destination buffer, which must be at least 0.1% larger than sourceLen plus 12 bytes.
Upon exit, destLen is the actual size of the compressed buffer.
=======================
Click to expand...

Thanks for the information (however I'm not using it any longer, see below).
I see that compressBound (the function you mention in a subsequent post) wasn't part
of the library at the time this documentation was produced. Re-reading that though,
assuming one wanted to take it at face value, I wonder if I did misinterpret it --
should I have used 1.0001 rather than 1.01 above?
Click to expand...
I think I get you right. When I first read abound compressBound I
thought it would be something like compressing the first x bytes...
As you triggered my attention with the 101%, I took the time to take a
closer look at the doc with the known result...
I can't find official docs on v1.23 -- is the zlib.h file it?
Click to expand...

Can't find them either. Google is not always your friend...

Sinna
 
Top