[Bitpim-devel] Reducing wallpaper file size

[Stephen Wood]

Since the Sanyo phones seem to only want to accept images less than 16K
in size, I am looking for ways to reduce the byte size of PNG images.
When I import an image from a file, BitPim, in resizing the image, often
makes a file larger than 16K. If I use xv to convert this resized file
to an 8 bit format like BMP or GIF, and then import that file, I usually
get well under 16K in file size.

I have tried copying the image to a bitmap and back to png with wxBitmap
(with an 8 bit color depth) and wxImage, but I don't really know what I
am doing am and looking for suggestions. There are some routines that
seem like they ought to take a depth=8 argument, but they don't.

Stephen

[Roger Binns]

Post by Stephen Wood
Since the Sanyo phones seem to only want to accept images less than 16K
in size, I am looking for ways to reduce the byte size of PNG images.

Would reducing the resolution help? We could do a binary search
to find the largest resolution that meets the size requirements.
(Yes, very ugly but it would work!)

Post by Stephen Wood
I have tried copying the image to a bitmap and back to png with wxBitmap
(with an 8 bit color depth) and wxImage, but I don't really know what I
am doing am and looking for suggestions. There are some routines that
seem like they ought to take a depth=8 argument, but they don't.

I was trying to do this a while back because BCI only has a palette
with 256 entries. Converting to gif first would do the trick since
it has the same limitation.

I suspect that wxPython 2.5 may have some new features to help.

Worst case we would have to resort to PIL
http://www.pythonware.com/products/pil/

PIL can definitely do it, although I really don't want to add yet
another dependency for such small functionality.

Roger

[Stephen Wood]

I have experimented with getting wxPython to reduce image file size, but
have not been successful. I write the image out in various formats to a
temp file and then read it back in as png. But nothing reduces file
size, probably because all the formats that can be written out support
24 bit pixels. Unfortunately GIF writes are not supported. (Hopefully
this will change with the GIF patents expiring).

The word that describes what I am trying to do is "quantize", namely
reducing the number of colors that an image uses.

One possibility would be to use the netpbm tools to reduce the color
map. This could be done with "pngtopnm | ppmquant 256 | pnmtopng".
These programs are on most Linux systems and perhaps on MacOSX. Windows
versions of these exist, so I supposed we could include these binaries
in the package, but this is not paletable, just to reduce the palette a
bit.

wxWindows has a class wxQuantize that looks like it does exactly what I
need. But it does not seem to be mapped/wrapped into wxPython. I'll
see if I can add a wrapper for wxQuantize in my own copy of wxPython.
Is it possible to add wrappers to wx things without having to build our
distribution of wxPython? How easy is it to get the wxPython developers
to add things like this?

Stephen

Post by Roger Binns

Post by Stephen Wood
Since the Sanyo phones seem to only want to accept images less than 16K
in size, I am looking for ways to reduce the byte size of PNG images.

Would reducing the resolution help? We could do a binary search
to find the largest resolution that meets the size requirements.
(Yes, very ugly but it would work!)

Post by Stephen Wood
I have tried copying the image to a bitmap and back to png with wxBitmap
(with an 8 bit color depth) and wxImage, but I don't really know what I
am doing am and looking for suggestions. There are some routines that
seem like they ought to take a depth=8 argument, but they don't.

I was trying to do this a while back because BCI only has a palette
with 256 entries. Converting to gif first would do the trick since
it has the same limitation.
I suspect that wxPython 2.5 may have some new features to help.
Worst case we would have to resort to PIL
http://www.pythonware.com/products/pil/
PIL can definitely do it, although I really don't want to add yet
another dependency for such small functionality.

[Roger Binns]

written out support 24 bit pixels. Unfortunately GIF writes are not
supported. (Hopefully this will change with the GIF patents
expiring).

Have you tried wxPython 2.5.x? I think they eventually re-enabled
GIF saving in that.

One possibility would be to use the netpbm tools to reduce the color
map. This could be done with "pngtopnm | ppmquant 256 | pnmtopng".
These programs are on most Linux systems and perhaps on MacOSX.
Windows versions of these exist, so I supposed we could include these
binaries in the package, but this is not paletable, just to reduce
the palette a bit.

wxPython supports pnm directly so the only external binary we need
is ppmquant. We need external binaries for audio processing anyway
and there is already code to help. Look in bpaudio to see stuff
I wrote there (it is somewhat Windows specific) but the meat of it
isn't.

The other thing I would like to do is find the maximum number of
colours that still fit within the image size. This can be done
by doing a binary search of the number of colours to quantize
to that results in a file of the right size. (Yes it sounds
gross, but it will result in the best image quality which
I think is more important than what happens behind the scenes.)

wxWindows has a class wxQuantize that looks like it does exactly what
I need. But it does not seem to be mapped/wrapped into wxPython.

I checked the latest daily build of wxPython and it wasn't wrapped
there either.

I'll see if I can add a wrapper for wxQuantize in my own copy of
wxPython. Is it possible to add wrappers to wx things without having
to build our distribution of wxPython?

It will be a right royal pain given the number of platforms. Additionally
I would like to move to the Unicode version of wxPython at some point
as well.

How easy is it to get the wxPython developers to add things like this?

I have had success in the past, although the next release can often
be quite a wait. I have sent a message to the group which Robin
usually responds to so we'll see what he comes back with.

Roger

[Stephen Wood]

Post by Roger Binns

written out support 24 bit pixels. Unfortunately GIF writes are not
supported. (Hopefully this will change with the GIF patents
expiring).

Have you tried wxPython 2.5.x? I think they eventually re-enabled
GIF saving in that.

I have the wxWidgets CVS. common/imaggif.cpp has

bool wxGIFHandler::SaveFile(... )
{
if (verbose)
wxLogDebug(wxT("GIF: the handler is read-only!!"));

return FALSE;
}

Post by Roger Binns

One possibility would be to use the netpbm tools to reduce the color
map. This could be done with "pngtopnm | ppmquant 256 | pnmtopng".
These programs are on most Linux systems and perhaps on MacOSX.
Windows versions of these exist, so I supposed we could include these
binaries in the package, but this is not paletable, just to reduce
the palette a bit.

wxPython supports pnm directly so the only external binary we need
is ppmquant. We need external binaries for audio processing anyway
and there is already code to help. Look in bpaudio to see stuff
I wrote there (it is somewhat Windows specific) but the meat of it
isn't.

Good point that only ppmquant is needed. Probably need a statically
linked version of it too.

Post by Roger Binns
The other thing I would like to do is find the maximum number of
colours that still fit within the image size. This can be done
by doing a binary search of the number of colours to quantize
to that results in a file of the right size. (Yes it sounds
gross, but it will result in the best image quality which
I think is more important than what happens behind the scenes.)

For the image I am playing with, 325 colors is the threshold below which
I end up with an 8 bit png. As I reduce the number of colors above
that, the file increases in size. What I think I'll do is first test if
the file is small enough, and if not, quantize with 256. If that
doesn't make the file small enough, then I'll do a binary search.

I do have to make investigate if it is really a 16KB file size limit and
not 8bit PNG vs 24 bit png that determines when the phone rejects the
image.

Post by Roger Binns

... wxQuantize ...
How easy is it to get the wxPython developers to add things like this?

I have had success in the past, although the next release can often
be quite a wait. I have sent a message to the group which Robin
usually responds to so we'll see what he comes back with.

Thanks!

Stephen

[Roger Binns]

Post by Stephen Wood
I have the wxWidgets CVS. common/imaggif.cpp has
bool wxGIFHandler::SaveFile(... )
{
if (verbose)
wxLogDebug(wxT("GIF: the handler is read-only!!"));
return FALSE;
}

I guess you could supply that code :-) Actually I think the
patent may still be in force in some parts of the world.

Post by Stephen Wood
Good point that only ppmquant is needed. Probably need a statically
linked version of it too.

Yup. At some point I want to put in place the infrastructure
to run Windows command line programs in the Linux BitPim.
I think it can be done by having our own copy of the main
wine executable and two of wine's shared libraries.
This has to be done at a minimum to use Qualcomm's PureVoice
to Wav converter which is a Windows only binary. However
there is no reason why we can't use a static Windows binary
of ffmpeg and ppmquant as well :-) [I can't hear your
screams from here :-]

For the Mac I am hoping that qemu based projects will eventually
do the right thing, although there is also a PureVoice codec
included in QuickTime so maybe that can be used instead.

Post by Stephen Wood
For the image I am playing with, 325 colors is the threshold below
which I end up with an 8 bit png.

That doesn't make sense. Surely 8 bit png's are 255 colours or
less?

Post by Stephen Wood
What I think I'll do is
first test if the file is small enough, and if not, quantize with
256. If that doesn't make the file small enough, then I'll do a
binary search.

If the phone does support PNG with more than 256 colours then I
would rather we try to give the user the maximum number supported.

For the VX4400 I did discover that the phone only takes the first
256 entries out of the palette anyway for a PNG so the images looked
really wierd. The BCI format is also very similar to 8 bit PNG.
It is basically an 8 bit palette with zlib encoded pixel data.

Roger

[Stephen Wood]

Post by Roger Binns
...

Post by Stephen Wood
wxLogDebug(wxT("GIF: the handler is read-only!!"));

...

Post by Roger Binns
I guess you could supply that code :-) Actually I think the
patent may still be in force in some parts of the world.

Media upload capability on the Sanyo phones is pretty pathetic. People
should really download their wallpaper and ringers over the air. I
don't want to put too much effort into this.

Post by Roger Binns
...
This has to be done at a minimum to use Qualcomm's PureVoice
to Wav converter which is a Windows only binary.

See: http://www.cdmatech.com/solutions/products/purevoice_download.jsp

Post by Roger Binns

Post by Stephen Wood
For the image I am playing with, 325 colors is the threshold below
which I end up with an 8 bit png.

That doesn't make sense. Surely 8 bit png's are 255 colours or
less?

The resultant png doesn't have 324 colors of course. That just seems to
be the threshold at which ppmquant n | pnmtopng makes 8 bit pngs.

Stephen

[Roger Binns]

Post by Stephen Wood

Post by Roger Binns
This has to be done at a minimum to use Qualcomm's PureVoice
to Wav converter which is a Windows only binary.

See: http://www.cdmatech.com/solutions/products/purevoice_download.jsp

Wow, they have finally updated it. My preference would actually
be to build a module using their SDK. However the licensing
terms prevent that. See bottom half of
http://www.cdmatech.com/solutions/downloads/pvsdk_readme.txt

I did try twice in the past to get permission just to redistribute
the pvconv binary, but they lost interest when it was apparent
no money was involved, and would never actually answer my
question as to whether I could redistribute it.

So the plan is tell users to download the binary themselves.
The SDK can't be used since the license isn't remotely GPL
compatible.

Post by Stephen Wood

Post by Roger Binns
That doesn't make sense. Surely 8 bit png's are 255 colours or
less?

The resultant png doesn't have 324 colors of course. That just seems
to be the threshold at which ppmquant n | pnmtopng makes 8 bit pngs.

I presume it also looks at how close the colours are to
each other as well.

Roger

[Roger Binns]

Stephen,

Did you ever get to the bottom of the wallpaper size/number of
colours issue?

Roger

[Stephen Wood]

Roger:

I did determine that I could do what was needed with either external
netpbm programs, but I never got around to assembling a set of binaries
for DOS, Mac and Linux. and coding it. Also, wx.Quantize looked like it
could do it, but that wasn't available in the wxpython we were using.

I just looked at the wxpython cvs and it appears that SWIG definitions
have been added for wx.Quantize (On Aug 2, probably in response to the
request that you made). I'll check tonight to see if Quantize is
actually available in the build we use, and if so, try to get something
into the next BitPim release.

Stephen

Post by Roger Binns
Stephen,
Did you ever get to the bottom of the wallpaper size/number of
colours issue?
Roger

[Stephen Wood]

The wxPython we use has wx.Quantize, but so far I have not been able to
get it to reduce image byte size. It actually increases the size. I
think I still need to figure out how to force the PNG to be an 8 bit
image rather than 24 bit.

Steve

Post by Roger Binns
Stephen,
Did you ever get to the bottom of the wallpaper size/number of
colours issue?
Roger

[Stephen Wood]

I had no luck reducing image size or converting PNGs to 8bit using
wx.Quantize. I can reduce the image size using the following routine
which I call on the fly just before writing an image to the phone. (Is
that OK, or should the reduction be done wallpaper.py?)

def convertto8bitpng(self, pngdata):
"Convert a PNG file to 8bit color map"
print "In convertto8bitpng"
pnm=common.gettempfilename("pnm")
f=os.popen('pngtopnm >'+pnm,'w')
f.write(pngdata)
f.close()
f = os.popen('ppmquant 256 '+pnm+ '| pnmtopng','r')
pngquantdata=f.read()
f.close
os.remove(pnm)
return pngquantdata

I have only tried this with the netpbm commands built into my version of
Linux. I have found prebuilt binaries (gallery @
netpbm.sourceforge.net) for Linux, Windows, and MacOS. The Linux
binaries are static, and I have not tried the Windows binaries yet.

Where should these binaries go in the cvs tree, and how do we actually
use them in a way that will work within builds (and from cvs)?

Stephen

Post by Stephen Wood
The wxPython we use has wx.Quantize, but so far I have not been able to
get it to reduce image byte size. It actually increases the size. I
think I still need to figure out how to force the PNG to be an 8 bit
image rather than 24 bit.
Steve

Post by Roger Binns
Stephen,
Did you ever get to the bottom of the wallpaper size/number of
colours issue?
Roger

[Roger Binns]

Post by Stephen Wood
I had no luck reducing image size or converting PNGs to 8bit using
wx.Quantize. I can reduce the image size using the following routine
which I call on the fly just before writing an image to the phone.
(Is that OK, or should the reduction be done wallpaper.py?)

Any idea what the problem is with wx.Quantize? I would very much prefer
this is all dealt with in wxPython itself.

Make a new file named bpvideo.py to put this kind of thing in. The
intention is to mirror bpaudio.py.

At some point I will want to add code to convert the video from these
phones into motion jpeg.

Post by Stephen Wood
I have only tried this with the netpbm commands built into my version
netpbm.sourceforge.net) for Linux, Windows, and MacOS. The Linux
binaries are static, and I have not tried the Windows binaries yet.

Windows generally doesn't have such issues. (Note how the distributed
BitPim will work on Win98 and up and I believe it will actually work
on Win95).

Post by Stephen Wood
Where should these binaries go in the cvs tree, and how do we actually
use them in a way that will work within builds (and from cvs)?

The binaries were initially in the resources directory. I have since
made a directory named helpers which is where they should go. (You
will see ffmpeg in there already).

Don't forget to use -kb when adding them (I always forget and CVS makes
it real hard to fix)

I propose the following naming convention:

.exe - Windows binaries
.lbin - Linux binaries
.mbin - Mac binaries

That will make it easy for the code to find the binaries and make it
easy for the packaging tools to know which files to pick up.

It is probably also worthwhile making another file named helpers.py
to put some of the logic into. In particular I would move the run
and getexecutable methods from bpaudio into there. getexecutable
should also be fixed to look in the helpers directory first and
if not found scan the path. It will also eventually require some
error handling type stuff.

We will be needing the pvconv (PureVoice convertor) program available,
but won't be able to ship it with BitPim. Consequently the user will
have to install it in the helper directory, or on the path somewhere.
getexecutable will have to whine if it isn't found.

Roger

[Stephen Wood]

Post by Roger Binns

Post by Stephen Wood
I had no luck reducing image size or converting PNGs to 8bit using
wx.Quantize. I can reduce the image size using the following routine
which I call on the fly just before writing an image to the phone.
(Is that OK, or should the reduction be done wallpaper.py?)

Any idea what the problem is with wx.Quantize? I would very much prefer
this is all dealt with in wxPython itself.

So would I. Below is what looks to be the relevant lines that added
quantize to wxPython from the file wxWidgets/wxPython/src/_image.i in
the wxWidgets CVS tree. It might have something to do
wxQUANTIZE_RETURN_8BIT_DATA being commented out in the enums and the 5
argument being forced to zero. I don't know if that is just a
simplification chosen by whoever added quantize to _image.i, or if doing
so would be incompatible with the rest of wxPython.

Stephen


%{
#include <wx/quantize.h>
%}

enum {
wxQUANTIZE_INCLUDE_WINDOWS_COLOURS,
// wxQUANTIZE_RETURN_8BIT_DATA,
wxQUANTIZE_FILL_DESTINATION_IMAGE
};


DocStr(wxQuantize,
"Performs quantization, or colour reduction, on a wxImage.", "");

class wxQuantize /*: public wxObject */
{
public:

%extend {
DocStr(
Quantize,
"Reduce the colours in the source image and put the result
into the
destination image, setting the palette in the destination if
needed. Both images may be the same, to overwrite the source image.", "
:todo: Create a version that returns the wx.Palette used.");

static bool Quantize(const wxImage& src, wxImage& dest, int
desiredNoColours = 236,
int flags =
wxQUANTIZE_INCLUDE_WINDOWS_COLOURS|wxQUANTIZE_FILL_DESTINATION_IMAGE)
{
return wxQuantize::Quantize(src, dest,
//NULL, // palette
desiredNoColours,
NULL, // eightBitData
flags);
}
}
};

[Roger Binns]

Post by Stephen Wood
So would I. Below is what looks to be the relevant lines that added
quantize to wxPython from the file wxWidgets/wxPython/src/_image.i in
the wxWidgets CVS tree. It might have something to do
wxQUANTIZE_RETURN_8BIT_DATA being commented out in the enums and the 5
argument being forced to zero. I don't know if that is just a
simplification chosen by whoever added quantize to _image.i, or if
doing so would be incompatible with the rest of wxPython.

This is where I asked for it to be wrapped:

http://aspn.activestate.com/ASPN/Mail/Message/wxPython-users/2132265

Apparently it can only reduce to less than 256 colours anyway:

http://aspn.activestate.com/ASPN/Mail/Message/wxPython-users/2133322

So I assume the problem is that Quantize is reducing the image
to 256 colour, but returning an Image with 24 bit colour data
(only using 256 distinct colours). The PNG code doesn't count
the colours and always returns a PNG with 24 bit colour.

If you can verify that, I can whine at Robin some more.

Roger

[Stephen Wood]

Post by Roger Binns
...
http://aspn.activestate.com/ASPN/Mail/Message/wxPython-users/2133322
So I assume the problem is that Quantize is reducing the image
to 256 colour, but returning an Image with 24 bit colour data
(only using 256 distinct colours). The PNG code doesn't count
the colours and always returns a PNG with 24 bit colour.
If you can verify that, I can whine at Robin some more.

The xv file viewer does call what Quantize puts out to be 24 bit. I

Post by Roger Binns

Should be doable, although I probably won't be able to allow the 8-bit
data to be returned like the C++ version does without extra overhead
that probably wouldn't be worth it. Check the docstring for details of
how the wrapper gets implemented.

Stephen

[Roger Binns]

Post by Stephen Wood
netpbm.sourceforge.net) for Linux, Windows, and MacOS. The Linux
binaries are static, and I have not tried the Windows binaries yet.

BTW did you find out how the binaries are built. netpbm lets you
build a single monolithic binary which switches on argv[0] to
figure out what to actually do. You just hard/soft link the
other commands to the monolithic binary.

Roger

[Vic Heintz]

BrewFileLockedException: File is locked

I get the above exception when I try to overwrite a file on my Samsung
a670 phone using the filesystem view in BitPim. ( I picked a file
called "memo" which contains the text of the phone's memo pad
accessory.) I was attempting to replace the file with a copy of itself
just to see if it would work.

I know it is possible to write files to the phone because I have been
successful in creating a brew/ringer directory and adding ringtones to
it which indeed work. Should BitPim be able to overwrite arbitrary
files in Filesystem view? How is it done with "Send Phone Data" for
supported phones?

I am starting to learn the structure of the various files that make up
the Contacts list, memos, schedules, etc. on my phone, but I am not
going to waste my time if I will never be able to overwrite them.

Vic

[Roger Binns]

Post by Vic Heintz
I get the above exception when I try to overwrite a file on my Samsung
a670 phone using the filesystem view in BitPim. ( I picked a file called
"memo" which contains the text of the phone's memo pad accessory.) I was
attempting to replace the file with a copy of itself just to see if it
would work.

http://www.bitpim.org/testhelp/faq-brewfilelockedexception.htm

I do believe that there is a command that changes file attributes but
haven't had the time to play with it.

Roger