[BitPim-devel] Problem with UNICODE filenames still not fixed :(

[Yura]

Hi!

I wrote to this mail list some months before and described an issue
that doesn't allow bitpim to run on non-English versions of Win XP.
Other people also wrote about that.
Problem still exists in the last build.

As far as I understand, any file operation fails, when file path
contains non-ascii characters.
BitPim stores some information in user home directory (\Documents
and Settings\Username\xxxx\xxx...), and in Russian versions of XP
this path is renamed and contains Russian characters. So BitPim
even fails to start :(

After I installed it on English XP version, everything seems to be ok,
BUT when I try to open any file, or select a directory with Russian
characters, it silently dies.

There are many people outside US that want to use BitPim, among them
people in Russia, Ukraine but they can't :(

This unicode issue may lie deep in standard pythons libraries, because
as far as I remember, I had similar problems with other python sw.

BUT THERE'S A SIMPLE WORKAROUND: have an option (maybe, a checkbox in
installer) to avoid use \Documents and Settings\ subfolders and store
all information in \Program Files\BitPim\ subdirectories. It would
allow at least to run the software. Then accessing only ascii-named
files wouldn't be a great problem, because most people here don't
use Russian characters in file names for compatibility.

I am not familiar with Python, and don't have time to learn this
language now, but I have can help with testing. If someone is willing
to solve this, you can contact me by ICQ 39861555 or MSN:
***@ukr.net

Here is debug info copied form bitfling log:

---------------------------------------------------------------------
22:57:16.608: Traceback (most recent call last):
File "wx\_core.pyo", line 13266, in <lambda>
File "bitfling\bitfling.pyo", line 642, in StartIfICan
File "bitfling\bitfling.pyo", line 635, in GetCertificateFilename
UnicodeEncodeError: 'ascii' codec can't encode characters in position 26-28: ordinal not in range(128)

Variables by last 8 frames, innermost last

Frame ? in bp.py at line 73
_donowt = <class __main__._donowt at 0x009F66C0>
ignorer = <function ignorer at 0x009FD270>
__name__ = '__main__'
__doc__ = 'Main entry point to Bitpim\n\nIt invokes BitPim in gui or commandline mode as appropriate\n\***@Note: Only gui mode is supported at the moment\n'

Frame run in bitfling\bitfling.pyo at line 785
menu = <wx._core.Menu; proxy of C++ wxMenu instance at _a0750801_p_wxMenu>
mw = <bitfling.bitfling.MainWindow; proxy of C++ wxFrame instance at _50720801_p_wxFrame>
args = ['C:\\Program Files\\BitPim\\bitpim.exe', 'bitfling']
theApp = <wx._core.PySimpleApp; proxy of C++ wxPyApp instance at _b0be0701_p_wxPyApp>
taskwin = <bitfling.bitfling.MyTaskBarIcon; proxy of C++ wxPyTaskBarIcon instance at _b02d0901_p_wxPyTaskBarIcon>

Frame MainLoop in wx\_core.pyo at line 7493
self = <wx._core.PySimpleApp; proxy of C++ wxPyApp instance at _b0be0701_p_wxPyApp>

Frame MainLoop in wx\_core.pyo at line 6926
args = (<wx._core.PySimpleApp; proxy of C++ wxPyApp instance at _b0be0701_p_wxPyApp>,)
kwargs = {}

Frame <lambda> in wx\_core.pyo at line 13266
event = <wx._core.PyEvent; proxy of C++ wxPyEvent instance at _f02e0901_p_wxPyEvent>

Frame StartIfICan in bitfling\bitfling.pyo at line 642
self = <bitfling.bitfling.MainWindow; proxy of C++ wxFrame instance at _50720801_p_wxFrame>

Frame GetCertificateFilename in bitfling\bitfling.pyo at line 635
path = u'C:\\Documents and Settings\\\u0414\u043e\u043c\\\u041c\u043e\u0438 \u0434\u043e\u043a\u0443\u043c\u0435\u043d\u0442\u044b'
self = <bitfling.bitfling.MainWindow; proxy of C++ wxFrame instance at _50720801_p_wxFrame>

-------------------------------------------------------------------------------------

Here is the error:
UnicodeEncodeError: 'ascii' codec can't encode characters in position 26-28: ordinal not in range(128)

And this is the unicode path attempted to access:
path = u'C:\\Documents and Settings\\\u0414\u043e\u043c\\\u041c\u043e\u0438 \u0434\u043e\u043a\u0443\u043c\u0435\u043d\u0442\u044b'

[Joe Pham]

Post by Yura
BUT THERE'S A SIMPLE WORKAROUND: have an option (maybe, a checkbox in
installer) to avoid use \Documents and Settings\ subfolders and store
all information in \Program Files\BitPim\ subdirectories.

Can you live with a command line option specifying the initial data dir (instead of the default one)? Once BitPim is up & running, you can specify where to store your data: Menu Edit -> Settings -> Disk Storage.

Another approach is to change the the BitPim storage path by changing its value in the Registry (with the ussual warnings of potentially screwing up your registry apply).

-Joe Pham




______________________________________________________________________
Revolutionize the way you surf the Web!
Try NetZero HiSpeed 3G.
Visit http://www.netzero.com to sign up today!

[Roger Binns]

Post by Joe Pham
Another approach

The best fix is for someone to provide a patch. We use
Unicode for the wxPython side of things. Python is dual
mode being in Unicode mode if you give unicode strings
else text mode.

The function SHGetFolderPath is supposed to return a
unicode string if appropriate according to the doc.

This should only be a line or two of code to fix. But it
requires someone with an affected system to work on.

Roger

[Roger Binns]

Post by Yura
Frame GetCertificateFilename in bitfling\bitfling.pyo at line 635
path = u'C:\\Documents and Settings\\\u0414\u043e\u043c\\\u041c\u043e\u0438
\u0434\u043e\u043a\u0443\u043c\u0435\u043d\u0442\u044b'

I have committed a fix for this specific case. For some reason the
string was converted to plain str.

Roger

[Simon C]

Post by Roger Binns
The best fix is for someone to provide a patch. We use
Unicode for the wxPython side of things. Python is dual
mode being in Unicode mode if you give unicode strings
else text mode.
The function SHGetFolderPath is supposed to return a
unicode string if appropriate according to the doc.
This should only be a line or two of code to fix. But it
requires someone with an affected system to work on.

I am able to reproduce by using a path with non-ascii characters.

There are three problems that prevent bitpim from running from the command
line.
1) com_brew.py. line 935.
print 'setting path to',bitpim_path
needs to be deleted or made unicode safe as print does not work with unicode
strings. A quick hack is to change it to
print 'setting path to',`bitpim_path`
This results in the unicode part of the string being printed as raw hex in
the console output.


2) A bug in sqlite3 (bug #1206) that prevents the database from being
opened. It is fixed in 3.2.7. I downloaded the new version of aspw
(3.2.7-r1) which I assume uses this new sqlite version and this fixed the
problem.


3) In guiwidgets.py, the line:
if not os.access(item.filename, os.F_OK):
and in common.py (207)
execfile(filename, dict, dict)
both fail with:

An unexpected exception has occurred.
Please see the help for details on what to do.

Traceback (most recent call last):
File "C:\Python23\Lib\site-packages\wx-2.6-msw-unicode\wx\_core.py", line
13266, in <lambda>
lambda event: event.callable(*event.args, **event.kw) )
File "C:\projects\bitpim\gui.py", line 1299, in
OnPopulateEverythingFromDisk
self.wallpaperwidget.getfromfs(results)
File "C:\projects\bitpim\wallpaper.py", line 448, in getfromfs
return self.genericgetfromfs(result, None, 'wallpaper-index',
self.CURRENTFILEVERSION)
File "C:\projects\bitpim\guiwidgets.py", line 1345, in genericgetfromfs
common.readversionedindexfile(os.path.join(self.thedir, file), d,
self.versionupgrade, currentversion)
File "C:\projects\bitpim\common.py", line 207, in readversionedindexfile
execfile(filename, dict, dict)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 54-61:
ordinal not in range(128)

I tried filename.encode(sys.getfilesystemencoding()) but this inserted '?'s
in the string which did not work.
All I can think of to fix this is to avoid using these functions by finding
an alternative which is not too hard.
Joe, I can work on a fix if you want some help.

Simon

[Joe Pham]

Post by Simon C
Joe, I can work on a fix if you want some help.

You're certainly welcome to it. The only thing is that I'd like to hear a response from the original poster. Another approach is to display the Config Dialog if (1) Configuration is not set, or (2) by command line option (something like bp.py -c).

-Joe Pham




______________________________________________________________________
Revolutionize the way you surf the Web!
Try NetZero HiSpeed 3G.
Visit http://www.netzero.com to sign up today!

[Simon C]

Joe,

There was more work than I thought before. Functions str(), access() and
execfile() will not take filenames with non-ascii characters. I fixed it so
that the path to any file used by bitpim can contain non-ascii characters,
if the actual filename contains non-ascii there could still be problems so
if you tried to add an image with non-ascii characters in the filename it
would not work, I don't think this is a big limitation as I think the phones
only work with ascii anyway.

11 files were touched, I've attached diffs as well. This is part 1 of 3, as
it exceeds the max sourceforge e-mail size.

If the person who raised this in the first place wants to try it out send me
an e-mail.


Simon