[Bitpim-devel] Calendar Database storage observations

[d***@netzero.com]

I made some progress with the calendar stuff and was able to store the calendar dict using the database, and made the following observations:

- The data seem all to be there with all the necessary details.
- All the indirect tables are named with prefix 'phonebook__', which may have been intentional, but was confusing at first.
- As a result, the 'serials' and 'categories' lists share the same tables with the phonebook contacts data, which again may be part of the design. Separately, the global table/list 'categories' does not seem to be used.

Overall, I thought that the databas stuff works pretty well and is fairly easy to integrate. The only minor suggestion I have is to internalize the 2 calls: 'ensurerecordtype' and 'extractbitpimserials' into database if possible.

-Joe Pham

________________________________________________________________
NetZero Gift Certificates
Give the gift of Internet access this holiday season.
http://www.netzero.com/give

[Roger Binns]

Post by d***@netzero.com
- All the indirect tables are named with prefix 'phonebook__', which
may have been intentional, but was confusing at first.

That is a good old fashioned bug! I just fixed it. Of course I never noticed
since all my testing was with phonebook :-) The good news is that the
indirect entries all mention which table they indirect to, so the data
will always remain consistent.

Post by d***@netzero.com
Separately, the global table/list 'categories' does not seem to be used.

Lines 851 and 877 of phonebook.py disagree with you. The categories are
managed by the phonebook module and accessed using pubsub.

Post by d***@netzero.com
Overall, I thought that the databas stuff works pretty well and is fairly easy
'ensurerecordtype' and 'extractbitpimserials' into database if possible.

ensurerecordtype is only needed if you randomly add data to your internal
structure without making it be of the type you want. The phonebook code
uses it after getting from the import code and when reading index.idx
from disk. I would imagine that long term the function will end up unused.

extractbitpimserials can't be done automatically since you need to know
how you are managing your data and the storage of serials. It is a
*really* bad idea to have serials in two places for the same record as
that lead to them conflicting. The phonebook module only stores the
serials as the 'serials' dict member and the overall dict key is opaque.
Consequently it needs the extractserials function before handing data
to be saved. Other data sources will know better how to deal with their
own serials.

BTW I will breaking basedataobject into two classes at some point.
The serials stuff will move into a subclass since not all dataobjects
will use serials in the same way.

Roger

[d***@netzero.com]

Separately, the global table/list 'categories' does not seem to be >>used.

Lines 851 and 877 of phonebook.py disagree with you. The categories >are managed by the phonebook module and accessed using pubsub.

I meant from within the 'calendar' table. Intuitively, the calendar 'categories' list should be using the 'categories' table, instead it's using the 'phonebook__categories' table (or 'calendar__categories' table). The 2 categories tables appear to be equivalent but not identical.

Consequently it needs the extractserials function before handing data
to be saved. Other data sources will know better how to deal with >their own serials.

Based on my own experience, it appears that I must call 'extractbitpimserials' on my dict for the database stuff to work, irregardless of how the calendar serials are handled internally. If that were the case, 'savemajordict' may as well call it automatically. It's not really a big deal in either case anyway.

BTW I will breaking basedataobject into two classes at some point.
The serials stuff will move into a subclass since not all dataobjects
will use serials in the same way.

I guess that means I'd have to subclass from the new subclass then.

-Joe Pham



________________________________________________________________
NetZero Gift Certificates
Give the gift of Internet access this holiday season.
http://www.netzero.com/give

[Roger Binns]

Post by d***@netzero.com
I meant from within the 'calendar' table. Intuitively, the calendar
'categories' list should be using the 'categories' table,

The category values stored by phonebook/dict etc are just strings.
They aren't linked to anything else.

The categories stored in the categories table are just strings and
are mainly there to give the user interface a way of showing the
user a selection. It is automatically updated by imported phonebook
entries. This makes its function largely cosmetic.

So the reason there is no linkage between the two is that there is
no linkage between the two :-) Although they could be linked, that
raises all sorts of issues such as what happens if you delete a
category in the ui or what happens if you remove the last phonebook
entry that uses a particular category.

Post by d***@netzero.com
If that were the case, 'savemajordict' may as well call it automatically.

If savemajordict always called it then we could never remove the call.
It requires some work to execute so I'd rather only make it a permanent
part of savemajor dict if all the callers do that anyway. We'll
have a better idea once more data types are moved over to using the
database.

Roger

[Roger Binns]

Joe,

I have just committed the code that works correctly in my tests for
doing indirect dicts (eg for repeat).

There is currently one issue caused by code you have committed which
causes an exception. You need to ensure that fields are only set
if they have a value. Do not set fields to None and/or zero length
strings. This is so that we can tell if a field was set to a blank
value vs not set to anything at all. (In some cases we may then
want to synthesize a field value based on other fields).

Anyway the database code falls over if you do something like set
fields to None and every field has that value (eg I get that with
priority). Since none of the fields have a non-None value it
doesn't know what type to create the field as in the database
and falls over. The good news is that the rollback code works
correctly as well :-)

I have attached my current index.idx, so you can reproduce the
issue.

I suggest copying the phonebook code and renaming the index.idx
out of the way after reading and using the database only.

You'll also need to make the code stable by around Monday, but don't
worry too much if it takes longer. I'll be racing you by doing
the new wallpaper and ringtone view code and I'm not too sure
which of us will finish first :-)

Roger

[d***@netzero.com]

Roger, thanks for the info. I just committed my changes, so if you have some time, please test them out. Based on my own testing, as far as I could tell, the calendar stuff seems to work properly with your database. Even on the earlier version which has none and empty dict values, everything was save and restored properly. But I followed your advice and cleaned out the dict of none and empty values.

-Joe Pham



________________________________________________________________
NetZero Gift Certificates
Give the gift of Internet access this holiday season.
http://www.netzero.com/give

[Roger Binns]

Post by d***@netzero.com
Even on the earlier version which has none and empty dict values, everything was save
and restored properly.

I got exceptions because all of the priority fields had None, and so the database
code didn't create any form of column since it never saw an indirect form or a
value form. Then at insertion time it fell over because the column didn't exist
which raises an SQL error. I was real pleased that the transaction wrapper
caught that and did a rollback. Of course the problem then was that the
database had no sign of the table or values so I couldn't tell what it looked
like at the time of the exception :-)

Post by d***@netzero.com
But I followed your advice and cleaned out the dict of none and empty values.

Cool. You also don't need a version 4. Do what the phonebook code does.
If an index.idx exists it is read, saved to the database and renamed to
index-is-now-in-database.bak. If it doesn't exist then the database
is read.

Users can get back the state before the upgrade by renaming the .bak
file to index.idx.

My one tree where I kept getting the error is under open heart surgery at
the moment for the wallpaper/ringtone code. On my other tree I keep getting
the following exception. The index.idx is attached.

File "/home/space/bitpim/bpcalendar.py", line 733, in getfromfs
self.CURRENTFILEVERSION)
File "/home/space/bitpim/common.py", line 187, in readversionedindexfile
versionhandlerfunc(dict, version)
File "/home/space/bitpim/bpcalendar.py", line 794, in versionupgrade
dict['result']['calendar']=self.convert_dict(dict['result'].get('calendar', {}), 3, 4)
File "/home/space/bitpim/bpcalendar.py", line 810, in convert_dict
return self.__convert3to4(dict, ringtone_index)
File "/home/space/bitpim/bpcalendar.py", line 822, in __convert3to4
r[k]['start']='%04d%02d%02dT%02d%02d'%\
TypeError: unsubscriptable object
Frame getfromfs in /home/space/bitpim/bpcalendar.py at line 733
dct = Keys ['__builtins__', 'result']
{'__builtins__': {'help': Type help() for interactive help, or help(object) for
dict = Keys ['categories', 'phonebook', 'ringtone-index', 'wallpaper-index']
{'phonebook': {}, 'wallpaper-index': {1: {'origin': 'builtin', 'name': 'Balloons
self = <bpcalendar.Calendar; proxy of C++ wxPanel instance at _28946208_p_wxPanel>

Frame readversionedindexfile in /home/space/bitpim/common.py at line 187
currentversion = 4
versionhandlerfunc = <bound method Calendar.versionupgrade of <bpcalendar.Calendar; proxy of C++ wxPa
version = 2
dict = Keys ['__builtins__', 'result']
{'__builtins__': {'help': Type help() for interactive help, or help(object) for
filename = u'/home/rogerb/.bitpim-files/calendar/index.idx'

Frame versionupgrade in /home/space/bitpim/bpcalendar.py at line 794
self = <bpcalendar.Calendar; proxy of C++ wxPanel instance at _28946208_p_wxPanel>
version = 4
dict = Keys ['__builtins__', 'result']
{'__builtins__': {'help': Type help() for interactive help, or help(object) for

Frame convert_dict in /home/space/bitpim/bpcalendar.py at line 810
to_version = 4
ringtone_index = Keys []
{}
self = <bpcalendar.Calendar; proxy of C++ wxPanel instance at _28946208_p_wxPanel>
dict = Keys ['098173935ee8310d598f143d07be37a76020ff57', '0a663b0d820ca81455de3fed5f028a0948651e5a',
'0b6c71565dcef4069904c368d10419da01a9624a', '120048c86ad42a6c1aa3d8655dfb6385024f8a4e', '1ec389918218e588dd36654279a0374de01a9dab',
'25bc6b258da652e83126a88b8d374b11510f32c8', '4375de2acd14490539ca6b892c4e68c5813eccff', '570f0909be00c56594bf50aa471e8ceaf95aa14b',
'5be39c0199ad222d3c8fa2a5338a2b031c42ed5f', '8174e03f732096c68ae1a283f0b6727c91032d35', '9c9b3bcb7ff3082589abcf25abe80ed790e408ec',
'b939b479822d9fdfbf290d5d521029cf1a915d6d', 'ce6449851918c9906c3b05e1d327534181769b28', 'dd7d64e8f21ac48cc4c53249aa6a715b2288f418',
'ec7ba2d7d687ebde8af4ddea75d209429f083645', 'fc4e9e1b6a4203d1237f4b33ac6b01a6cebeecd8']
{'fc4e9e1b6a4203d1237f4b33ac6b01a6cebeecd8': <bpcalendar.CalendarEntry object at
from_version = 3

Frame __convert3to4 in /home/space/bitpim/bpcalendar.py at line 822
ringtone_index = Keys []
{}
e = <bpcalendar.CalendarEntry object at 0x423e4aec>
k = 'fc4e9e1b6a4203d1237f4b33ac6b01a6cebeecd8'
r = Keys ['fc4e9e1b6a4203d1237f4b33ac6b01a6cebeecd8']
{'fc4e9e1b6a4203d1237f4b33ac6b01a6cebeecd8': <bpcalendar.CalendarEntry object at
dict = Keys ['098173935ee8310d598f143d07be37a76020ff57', '0a663b0d820ca81455de3fed5f028a0948651e5a',
'0b6c71565dcef4069904c368d10419da01a9624a', '120048c86ad42a6c1aa3d8655dfb6385024f8a4e', '1ec389918218e588dd36654279a0374de01a9dab',
'25bc6b258da652e83126a88b8d374b11510f32c8', '4375de2acd14490539ca6b892c4e68c5813eccff', '570f0909be00c56594bf50aa471e8ceaf95aa14b',
'5be39c0199ad222d3c8fa2a5338a2b031c42ed5f', '8174e03f732096c68ae1a283f0b6727c91032d35', '9c9b3bcb7ff3082589abcf25abe80ed790e408ec',
'b939b479822d9fdfbf290d5d521029cf1a915d6d', 'ce6449851918c9906c3b05e1d327534181769b28', 'dd7d64e8f21ac48cc4c53249aa6a715b2288f418',
'ec7ba2d7d687ebde8af4ddea75d209429f083645', 'fc4e9e1b6a4203d1237f4b33ac6b01a6cebeecd8']
{'fc4e9e1b6a4203d1237f4b33ac6b01a6cebeecd8': <bpcalendar.CalendarEntry object at
self = <bpcalendar.Calendar; proxy of C++ wxPanel instance at _28946208_p_wxPanel>

Roger

[d***@netzero.com]

was real pleased that the transaction wrapper caught that and did a rollback

Is that what it was. I saw sucessive entries of the same item in the table and couldn't firgure out what it was, now I know what to look for.

You also don't need a version 4.

Good advice! The conversion routine was screwing me up. What it means now is that version 3 index.idx never existed: you either have a v.2 index.idx or you're reading off the database!

Tested the changes with your v2 index.idx, seemed to be working ok. Committed those changes. btw, you seem to have your hands full with stuff for next week, let me know if you'd want me to look at the 16-bit bmp issue. Both the A650 and the A670 can use bmp, so I'm also intersted in it.

-Joe Pham



________________________________________________________________
NetZero Gift Certificates
Give the gift of Internet access this holiday season.
http://www.netzero.com/give

[Roger Binns]

Post by d***@netzero.com

was real pleased that the transaction wrapper caught that and did a rollback

Is that what it was. I saw sucessive entries of the same item in the table and
couldn't firgure out what it was, now I know what to look for.

The transaction wrapper is what happens at the SQL level. SQLite doesn't support
nested transactions, so there is some magic that issues BEGIN EXCLUSIVE TRANSACTION
and END EXCLUSIVE TRANSACTION and deals with the nesting. If an exception occurs,
then it issues a ROLLBACK at the end instead. You can see those being printed
out. In this case all sorts of stuff was happening such as tables being created
and changed, values inserted etc and then an exception happened. The ROLLBACK
successfully (and correctly) threw all that work away.

What you are seeing is something different. The table is a journal. In the
simplest case, say you did this:

savedict( {"56565": {'name': "foobar"} } )

The table would then look like

rowid uid deleted name
0 56565 0 foobar

And then you did this:

savedict( {"1235": { 'name': "bambi" } } )

The table needs to show the previous active entries as deleted and add bambi

rowid uid deleted name
0 56565 0 foobar
1 56565 1 foobar
2 1235 0 bambi

There will *always* be a difference between rows for the same entry.

Post by d***@netzero.com
Tested the changes with your v2 index.idx, seemed to be working ok.
Committed those changes.

I still get various exceptions in the UI when creating and editing
calendar entries. I don't know which of them have always been
there and which are caused by the rejigged code.

Post by d***@netzero.com
btw, you seem to have your hands full with stuff for next week, let me
know if you'd want me to look at the 16-bit bmp issue. Both the A650
and the A670 can use bmp, so I'm also intersted in it.

The priority is definitely on the calendar being rock solid. The code
already does 24 bit bmp and does it fine. That is the format for the
VX6000. It is just the VX4500 that is broken in requiring 16 bit bmp.

I've done enough work to see that we are SOL with wxWdigets, PIL or
netpbm. It shouldn't be that hard to write something to save, but
I do need example C code. Doing the save code in Python would be
painful. I did pure Python code for reading BCI and it is really slow.

The most important thing I need to know is whether the file should
be written out using truncated colour information (of which there
are 3 variants) or using a palette.

Roger

[d***@netzero.com]

Post by Roger Binns
I still get various exceptions in the UI when creating and editing
calendar entries

Would appreciate any specifics you can provide.

Post by Roger Binns
The most important thing I need to know is whether the file should
be written out using truncated colour information (of which there
are 3 variants) or using a palette.

[Roger Binns]

[d***@netzero.com]

Attached is one of them.

Thanks for the info. I'll look at them tonight. I don't have access to my code right now.

0RRRRRGG GGGBBBBB (X1 R5 G5 B5)
RRRRRGGG GGGBBBBB (R5 G6 B5)
0000RRRR GGGGBBBB (X4 R4 G4 B4)

I believe it's B5 G6 R5, but I'll dig through my old stuff and let you know for sure.

-Joe Pham



________________________________________________________________
NetZero Gift Certificates
Give the gift of Internet access this holiday season.
http://www.netzero.com/give

[d***@netzero.com]

Post by d***@netzero.com
Would appreciate any specifics you can provide.

Attached is one of them.

Changes committed. I also added a dialog displaying an 'Invalid date/time entry' message. I can take it out if you think it's unecessary.

-Joe Pham



________________________________________________________________
NetZero Gift Certificates
Give the gift of Internet access this holiday season.
http://www.netzero.com/give

[Roger Binns]

Post by d***@netzero.com
Changes committed. I also added a dialog displaying an 'Invalid date/time
entry' message. I can take it out if you think it's unecessary.

It depends when that gets triggered.

By far the best way is to change the background of items to a colour
such as light red. You can also give a tooltip if the user hits
Ok.

My biggest preference would be for a control for entering date/time
that doesn't let you enter invalid values in the first place. Unfortunately
wxPython doesn't have such a thing.

I looked around at Qtopia, Outlook, Evolution and Windows Control Panel.
They all have quirky text type controls usually with drop downs for
the calendar. And they all let me enter obviously wrong values and would
whine in an unfriendly manner on pressing Ok. Qtopia wins the prize for
not even whining despite 234323:78am not being remotely valid by anyone's
standards.

Roger

[d***@netzero.com]

Post by Roger Binns
By far the best way is to change the background of items to a colour
such as light red.

I'll give that a try.

-Joe Pham



________________________________________________________________
NetZero Gift Certificates
Give the gift of Internet access this holiday season.
http://www.netzero.com/give