[Bitpim-devel] Phonebook/Calendar sync w/external programs?

[Steven Palm]

I'm just trying to get a feel for the direction things are going to
sync the phonebook and calendar to external programs. I know Roger has
been doing some work on getting into the Windows Address Book stuff,
but on a quick once-over, I haven't seen any hints of beginnings of
code in bitpim to deal with this.

I have wrapped methods in Python to allow the ability to interact with
both AddressBook and iCal on MacOS X. I can retrieve/set information in
both applications, but I'm not sure the best way to slip this into
bitpim. (Actually, I have code that works, but to make it user friendly
I should wrap it into modules that abstract the details to a simpler
higher level API. My current approach is to use Python to wrap
AppleScript to actually talk to the respective applications. Works
pretty well in my initial testing.)

One thought I had on the phonebook was that you may not want to have
your entire computer address book in the phone. Perhaps there should be
a setting somewhere for a particular group name that gets synchronized?
Just a thought... Same thing with calendars... You can have multiple
calendars in iCal, I suspect only one would synchronize to your phone.

-. ----. -.-- - -.--
Steve Palm - ***@n9yty.com
-. ----. -.-- - -.--

[Chris Cleeland]

Post by Steven Palm
I have wrapped methods in Python to allow the ability to interact with
both AddressBook and iCal on MacOS X. I can retrieve/set information in
both applications, but I'm not sure the best way to slip this into
bitpim. (Actually, I have code that works, but to make it user friendly
I should wrap it into modules that abstract the details to a simpler
higher level API. My current approach is to use Python to wrap
AppleScript to actually talk to the respective applications. Works
pretty well in my initial testing.)

Cool. Those higher level APIs should be designed such that they could be
useful for accessing any external calendar, including things like Yahoo!
calendar, etc. Basically, the existing modules you designed would be
replicated for other calendar backends, and the higher level API (that
doesn't yet exist) would be the generic API.

Post by Steven Palm
One thought I had on the phonebook was that you may not want to have
your entire computer address book in the phone. Perhaps there should be
a setting somewhere for a particular group name that gets synchronized?
Just a thought... Same thing with calendars... You can have multiple
calendars in iCal, I suspect only one would synchronize to your phone.

This is definitely the same way I was thinking. I would also like to be able
to import schedule information from somebody else's calendar, e.g., my
wife's, and have each entry auto-prefixed with some character or string so
that I can easily discern my events from hers.

As far as pushing schedules onto the phone, I can think of several ways in
which one might want to constrain/filter a calendar, e.g., by date range, by
time slot, etc. Maybe the best way to allow this is to allow arbitrary
python code to be written? However, that could be a giant security hole...

[Roger Binns]

Post by Steven Palm
I haven't seen any hints of beginnings of
code in bitpim to deal with this.

Look harder :-) There is a module named importexport almost
500 lines long and a whole bunch of code in phonebook.py!

The intention is for 0.7 to have the ability to import. (This isn't
the same thing as synchronization, but is an important first step).

If anyone actually wants to design synchronization then that would
be a good idea. The actual process would have to recognising changes
in various data sources and applying them to others. Overall trivial
big picture but really difficult when you get down to nuts and bolts.
It is also an absolute prerequisite that it isn't stupid. I would
rather not have the feature at all, than have one that magically
duplicates data.

Post by Steven Palm
but I'm not sure the best way to slip this into bitpim.

Start with importing address information. There are two stages,
both of which are present in importexport and phonebook
respectively.

Try importing a CSV file to get a feel for the process.

The first stage is handled in importexport.py and allows the user
to choose what they want to import. It is currently missing a
category checklistbox which I will be adding.

In the case of CSV files, the user also gets to define the columns.
For other data sources (vcard, WAB, etc) we already know what the
columns are and the column header stuff won't be present.

The user will be able to filter what gets imported (eg requiring
at least a phone number, or for the entry to be in a particular category).

The importexport code then converts the entry into the bitpim python
dictionary format and supplies it to importdata() in phonebook.py.
That then brings up another dialog that controls the merging of
the imported entries with what you already have.

The process is very similar to reconciling transactions in programs
like Quicken or Microsoft Money. An imported entry could match
an existing entry and be 'merged' or it could be a new entry.
The code takes its best guess and then lets the user override what
happens. I have also tried to put a numerical 'certainty' value
on each match and merge, but will probably remove that as it isn't
user friendly. (The code should just get it right).

That code mostly works at the moment. However there is an edit control
for each line missing (to allow you select a different match or to
say an entry should be new or deleted). I also needed a entryeditor.
So I went off and wrote one. But that needed the publish/subscribe
system in order to deal with wallpapers and ringtones correctly.
And the VX4400/6000 code wasn't in any fit shape to cooperate so
I went in and fixed that. I am in the final throes of that
and then will be able to go back to what I was working on in the
first place before the several week detour :-)

So there are several things you can do:

1 - Get various data sources working but not hooked into the bitpim
code (ie they are ready for 0.8)
2 - Complete the code for the current import phonebook code (gui and
mechanism)
3 - Write new code for importing calendar (gui and mechanism)
4 - Write code for generic phonebook export (gui and mechanism)
5 - Write code for generic calendar export (gui and mechanism)
6 - Design and completely think through a synchronisation algorithm.
It should be able to cope with multiple phones (eg people wanting
it to work with both their and their partners phones), and not
be stupid. There should also be some sort of design for an
undo/redo system in there as well, as I suspect this would
all be based on some sort of transactioning and differencing
engine. I would also expect it to need some sort of real
database behind the scenes, and have already selected sqlite
(and pysqlite as the wrapper) for those purposes

I'll be completing the gui bit of #2 so don't bother working on that.
However the mechanism bit is what needs the most work, I would be
very happy for someone else to do it. Start at the MergeEntries
function in phonebook.py.

Anyway, let us know what tickles your fancy. #2 (both gui and mechanism)
is what needs to be completed for the 0.7 release. Everything else
can be later.

Roger

[Steven Palm]

Post by Roger Binns

Post by Steven Palm
I haven't seen any hints of beginnings of
code in bitpim to deal with this.

Look harder :-) There is a module named importexport almost
500 lines long and a whole bunch of code in phonebook.py!

I did find that, sorry for not referencing it. I was looking for
beginnings of integration with your WAB code into bitpim, which I
didn't find. That's easy, it's not there. ;-) Given the roadmap for the
project, it makes sense, I guess I'm just jumping the gun a little bit
trying to get too much in too quickly. Roll over, Crawl, Walk, Run...
That's it. :-)

Post by Roger Binns
The intention is for 0.7 to have the ability to import. (This isn't
the same thing as synchronization, but is an important first step).

Yes, I agree. For now, then, I will focus on a convenient way that
enables users to pull the AddressBook/iCal information into bitpim.

Post by Roger Binns
If anyone actually wants to design synchronization then that would
be a good idea. The actual process would have to recognising changes
in various data sources and applying them to others. Overall trivial
big picture but really difficult when you get down to nuts and bolts.
It is also an absolute prerequisite that it isn't stupid. I would
rather not have the feature at all, than have one that magically
duplicates data.

Indeed. The MacOS AddressBook and iCal program have (as I suspect do
most others) UID's attached to the main entries. In the case of the
address book, for a given name you can have, of course, multiple
addresses, and bits of contact information. Each address has it's own
UID, as does each piece of contact information (phone number, email,
Instant Messenger handle, URL, etc...). Each iCal event also has a UID.

Obviously, this is a careful line to walk... I see the bits to store
serial numbers and source in bitpim. It would take careful work,
though, to deal with entries created on the phone v computer, as well
as created on one device, modified on the other, etc... And that is
only assuming one device and one source of each type of data on the
computer.

This is where I often tend to go for a certain base level of
functionality and call it "good enough" for me. That's the difference
between programming something becuase YOU want it versus trying to meet
the (real or perceived) needs of users.

Post by Roger Binns
Anyway, let us know what tickles your fancy. #2 (both gui and
mechanism)
is what needs to be completed for the 0.7 release. Everything else
can be later.

I'll ponder the list and soak up the code a bit. Since my Python
skills are still being formed, I'm leary of doing a whole lot just
yet... No reason to put a bunch of shoddy code in there until I grok
the "Pythonic way" to do things. I'm much more comfortable at this
point with making the Mac version well supported and integrated in it's
own right. I enjoyed the libusb stuff and packaging up a distribution,
and my current work on getting data out of the other programs on the
Mac.

Thanks again, Roger, for a great program. It's nice to be able to do
even a little bit to help.

-. ----. -.-- - -.--
Steve Palm - ***@n9yty.com
-. ----. -.-- - -.--

[Roger Binns]

Post by Steven Palm
Indeed. The MacOS AddressBook and iCal program have (as I suspect do
most others) UID's attached to the main entries. In the case of the
address book, for a given name you can have, of course, multiple
addresses, and bits of contact information. Each address has it's own
UID, as does each piece of contact information (phone number, email,
Instant Messenger handle, URL, etc...).

BitPim only stores UIDs for the entry as a whole, not for each piece
of data. I would assume a really strong synchronisation algorithm
would like it for every piece of data.

Post by Steven Palm
Each iCal event also has a UID.

On the 4400/6000 there are no ids at all for calendar stuff.

Post by Steven Palm
And that is
only assuming one device and one source of each type of data on the
computer.

Yup. It is where other programs annoy me. They generally place
too much value on the uid and not enough looking at what they are
actually doing. They can get very confused if you have multiple
devices (eg a Palm, Outlook and a Cellphone) and make changes
on all of them.

Post by Steven Palm
Since my Python
skills are still being formed, I'm leary of doing a whole lot just
yet...

Parsing that usb.ids file I mentioned yesterday would be a good
self contained start then.

Post by Steven Palm
No reason to put a bunch of shoddy code in there until I grok
the "Pythonic way" to do things.

You can write shoddy code and email it to me for integration.
It is often quicker for me to beautify the code and merge
it, than to write it all from scratch.

Roger

[Chris Cleeland]

Post by Roger Binns

Post by Steven Palm
I haven't seen any hints of beginnings of
code in bitpim to deal with this.

Look harder :-) There is a module named importexport almost
500 lines long and a whole bunch of code in phonebook.py!
The intention is for 0.7 to have the ability to import. (This isn't
the same thing as synchronization, but is an important first step).
If anyone actually wants to design synchronization then that would
be a good idea. The actual process would have to recognising changes
in various data sources and applying them to others. Overall trivial
big picture but really difficult when you get down to nuts and bolts.
It is also an absolute prerequisite that it isn't stupid. I would
rather not have the feature at all, than have one that magically
duplicates data.

Absolutely. There are lots of existing sync programs "out there"; would it
make sense to try to leverage some of their algorithms, or structure the
storage format that bitpim uses such that they could be more easily
leveraged? I'm thinking particularly of something like 'rsync'...

[Roger Binns]

Post by Chris Cleeland
I'm thinking particularly of something like 'rsync'...

rsync is for synchronising files, and is one way only (you
can't make changes on both ends). It isn't even remotely
applicable.

The only program I found when I researched this a while back
is http://multisync.sf.net and it is Linux only.

The canonical open standard for this stuff is SyncML. However
there aren't actually any appopriate implementations, and I
would rather my phone etc worked now than wait for or actually
implement the standard.

Roger