[Bitpim-devel] Outlook Calendar Import Design

[d***@netzero.com]

I don't know if anyone else is working on this, but I'm planning to implement the feature of importing Outlook Calendar into BITPIM. Here are some of my initial requirements, design ideas on how to do that. As always, suggestions and comments are welcome.

Outlook-to-BITPIM fields mapping:

Outlook BITPIM
Subject & Location description as 'Subject(Location)'
Start start
End end
IsRecurring repeat (more on this later)
ReminderSet alarm
ReminderMinutesBeforeStart alarm
Categories <None> used for filtering only

Alarm setting: if ReminderSet: alarm=ReminderMinutesBeforeStart else: alarm=None

Recurring/Repeat events:
Outlook BITPIM
IsRecurring=F repeat=None
Daily+Every 1 day repeat=daily
Daily+Every weekday repeat=monfri
Weekly+Every 1 week repeat=weekly
Monthly+Every 1 month repeat=monthly
Yearly+every same m/d repeat=yearly

Any other Recurring combination may have to be manually created in BITPIM, for example: Daily+Every 3 days=multiple events 3 days apart each. Question: is this worth the effort, or should these events just be ignored and not imported?

Question: How should BITPIM handle recurring events with 'No End Date', if at all?

Filtering: users can filter their data by: (start date, end date, categories)
Acceptant criteria:
start date>=filter start date and
end date<=filter end date and
one or more categories is in the filter categories set.

Question: Any other filtering parameters?

User Interface:
Preview panel similar to Contacts Import preview panel:
users would be able to select another Outlook folder, setting filter parameters, and 'OK' to import the data into calendar (no other editing capabilities at this point).

I'm planning to do just "Replace" import (wipe out all BITPIM calendar events and import new Outlook events). Should "Add" import also be considered?

That's all I have so far.

Thanks,

-Joe Pham


________________________________________________________________
NetZero Platinum $9.95.
NetZero HiSpeed $14.95.
Sign up Today at http://www.netzero.net or
Visit Sam Goody, Suncoast or MediaPlay Stores.

[Roger Binns]

[Apologies for the HTML mail - it is needed for a table lower down]
Having a list of mappings is good. You should also look at vcal and
ical and do mappings for them. Add in some phones for fun :-)
Remember that the mappings need to be bi-directional (ie import and
export).

Post by d***@netzero.com
Outlook BITPIM
Subject & Location description as 'Subject(Location)'

That is *REALLY* bad. It loses information and makes things ambiguous
when going back to outlook again. There should be seperate subject
and location keys. (Note that you can combine them when going to
the phone providing you can unambiguously uncombine them coming
back. See * below)

The main notes field is also missing.

Post by d***@netzero.com
Start start
End end

You should also state what the convention is for an all day event. Some
calendar systems mark this specially, others recognise it as events starting
at 0:00 and ending at 23:59 (that technically is an all day minus one minute
event).

Post by d***@netzero.com
IsRecurring repeat (more on this later)
ReminderSet alarm
ReminderMinutesBeforeStart alarm
Categories <None> used for filtering only

We need categories in BitPim.

Post by d***@netzero.com
Alarm setting: if ReminderSet: alarm=ReminderMinutesBeforeStart else: alarm=None
Outlook BITPIM
IsRecurring=F repeat=None

Better is that the key is not present.

Post by d***@netzero.com
Daily+Every 1 day repeat=daily
Daily+Every weekday repeat=monfri
Weekly+Every 1 week repeat=weekly
Monthly+Every 1 month repeat=monthly
Yearly+every same m/d repeat=yearly

There are two types of daily repeat. One is a list of days of the week (eg
every tuesday and thursday) and the other was the every so many days. My
suggestions is this for by day of week (define 0 as Sunday or something like
that)

repeat="dailydow:0,2,5"

and this for when it is every so many days (eg 5):

repeat="daily:5"

This is how Outlook exposes repeats at the API level:


Value RecurrenceType Properties Example
0 olRecursDaily Interval Every N days
DayOfWeekMask Every Tuesday, Wednesday, and Thursday
2 olRecursMonthly Interval Every N months
DayOfMonth The Nth day of the month
3 olRecursMonthNth Interval Every N months
Instance The Nth Tuesday
DayOfWeekMask Every Tuesday and Wednesday
1 olRecursWeekly Interval Every N weeks
DayOfWeekMask Every Tuesday, Wednesday, and Thursday
5 olRecursYearly DayOfMonth The Nth day of the month
MonthOfYear February
6 olRecursYearNth Instance The Nth Tuesday
DayOfWeekMask Tuesday, Wednesday, Thursday
MonthOfYear February

Post by d***@netzero.com
Question: How should BITPIM handle recurring events with 'No End Date', if at all?

It already does! No end date should correspond to the end key being missing.

See getentrydata method in bpcalendar.py. To display what is happening
on any particular day, it looks explicitly for events on that day. (It
is arguably a bug that it doesn't look for events from the previous day
that have a duration of more than 24 hours ...)

The for each repeating event, it checks to see if the date being examined
is after the start and before the end.

Of the existing fields defined in bpcalendar.py, the following should
be removed:

- changeserial
- snoozedelay
- pos

The following should be added:

- serials (same format as for phonebook)
- wallpaper

Both wallpaper and ringtone should be names not numbers.

Post by d***@netzero.com
Filtering: users can filter their data by: (start date, end date, categories)
start date>=filter start date and
end date<=filter end date and
one or more categories is in the filter categories set.

For the UI, I wouldn't show the actual dates, just a drop down saying
things like "this year", "in the last month", "for the next two months".

Post by d***@netzero.com
Should "Add" import also be considered?

Yes. You'll regret it later if you don't at least design with it in mind
at the begining. (Yes that is bitter experience talking :-)

The other big thing to consider is how to deal with overridden repeats.
Say you had a repeat every Thursday, but then editted only the one for
tomorrow to have an earlier start time, changed the description etc.

On the VX4400 and how BitPim does it at the moment, a suppression is put
in for the particular date in the master event. A new event is generated
on that date with the new information. The new event however does not
have any information linking it back to the original master.

Roger

[d***@netzero.com]

Roger, thanks for the feedbacks: they're very helpful. I have to digest this and get back to you later. One thing that caught my eyes was your suggestion of changing the bpcalendar dict: I didn't look at all the details of your suggestion but wouldn't that potentially break existing com_phone* code? I thought that was one of the constraints going in.

-Joe Pham

Please note: message attached

[Roger Binns]

Post by d***@netzero.com
One thing that caught my eyes was your suggestion of changing
the bpcalendar dict: I didn't look at all the details of your suggestion
but wouldn't that potentially break existing com_phone* code? I thought
that was one of the constraints going in.

It most certainly isn't a constraint. Yes, all the com_phone code
will have to be updated. That is why I want to make sure it is
well thought out and future proof so we only do such a big change
once.

The current calendar format is basically exactly what the VX4400
does. It is broken in many many ways. I wanted to fix it after
completing the phonebook (which is in a very good state now).

BitPim also supports automatically updating what the user has
on disk. Look in the index.idx files and see the FILEVERSION
variable.

Roger

[d***@netzero.com]

Based on Roger's feedbacks and suggestions, I've updated the Outlook-BITPIM-mapping as follows:

Outlook BITPIM
Subject description:'string'
Location location:'string'
Start start:time.timetuple
End end:time.timetuple # absent for recurring events w/ no-end-date
IsRecurring repeat: # more on this later
ReminderSet alarm:int # key present if T, absent if F
ReminderMinutesBeforeStart alarm:int # minutes before start time
Categories categories:['string']
AllDayEvent allday: # key present if T, absent if F
EntryID serials: { serial1:'string' }
FolderID serials: { serial2:'string' }
Attachements notes:'string' # when applicable
CreationTime creationtime:time.timetuple
LastModificationTime modtime:time.timetuple
synctime:time.timetuple # time of last import/sync
ringtone:'string' # name of ringtone
wallpaper:'string' # name of wallpaper
origin:'outlook' # data from Outlook

'repeat': { # only one type (key) should be present for each event
'daily': { 'interval': int, 'dow': ['sun', ..., 'sat'] }
'monthly': { 'interval': int, 'dom': int }
'monthlynth': { 'interval': int, 'instance': int, 'dow': ['sun'..'sat'] }
'weekly': { 'interval': int, 'dow': ['sun'..'sat']
'yearly': { 'dom': int, 'moy': int }
'yearlynth': { 'instance': int, 'dow': ['sun'..'sat'], 'moy': int }
# optional
'suppressed': [(y,m,d)] # days excluded from this recurring event
}

Filtering paramters: (startdate, enddate, categories)
Users can either 'Replace' or 'Add' outlook events into BITPIM.
Preview pane will be available for users to select Outlook calendar foler, view events pending for import, set filtering parameters, and OK or CANCEL.

This design reflects Roger's intention of revamping the BITPIM Calendar
data structure & processing, which would affect individual phone calendar implementations. As an alternative, I'd suggest a 2-phase approach to do Outlook calendar import:

1. Implement a quick, reusable, and expandable Outlook import that would work with current calendar structure. Concurrently but separately, re-work BITPIM calendar with the new features.
2. Update individual phone implementations and import to the new calendar structure once it's available.

Please let me know what you think.

Thanks,

-Joe Pham


________________________________________________________________
NetZero Platinum $9.95.
NetZero HiSpeed $14.95.
Sign up Today at http://www.netzero.net or
Visit Sam Goody, Suncoast or MediaPlay Stores.

[Roger Binns]

Post by d***@netzero.com
AllDayEvent allday: # key present if T, absent if F

There are two ways to deal with this. One is the explicit key you
have above. The other is to require the start time as 0:00 and the
end time as 23:59. I lean slightly towards the latter as it removes
what it ultimately duplicated data. However it is worth checking
vcal etc first.

Post by d***@netzero.com
Attachements notes:'string' # when applicable

The attachments are seperate and we don't need them. We should at
least get the main description/notes bit.

Post by d***@netzero.com
CreationTime creationtime:time.timetuple

What do we need this for?

Post by d***@netzero.com
LastModificationTime modtime:time.timetuple
synctime:time.timetuple # time of last import/sync

Those aren't needed and aren't values we can track anyway.

Post by d***@netzero.com
origin:'outlook' # data from Outlook

This is done with serials since the same entry could be present in
multiple devices and data sources.

Post by d***@netzero.com
1. Implement a quick, reusable, and expandable Outlook import that
would work with current calendar structure. Concurrently but separately,
re-work BITPIM calendar with the new features.

Actually the existing schema is very broken (eg changeserial)
My preference is to update the schema to what we want as the
final version which should require not too many changes. (It
should even be possible to have a piece of shim code that takes
the new format and produces records in the old format to feed
to the individual phone modules).

Roger

[Vic Heintz]

Post by d***@netzero.com
This design reflects Roger's intention of revamping the BITPIM Calendar
data structure & processing, which would affect individual phone
calendar implementations.

Any changes you make to BitPim's calendar for Outlook should be
compatible with vCal and iCal. (see this link http://www.imc.org/pdi/ )
Some of us use iCal and not Outlook.

Vic

[d***@netzero.com]

Vic, thanks for the info. Roger also mentioned the same thing.

-Joe Pham

Post by d***@netzero.com
This design reflects Roger's intention of revamping the BITPIM Calendar
data structure & processing, which would affect individual phone
calendar implementations.

Any changes you make to BitPim's calendar for Outlook should be
compatible with vCal and iCal. (see this link http://www.imc.org/pdi/ )
Some of us use iCal and not Outlook.

Vic



-------------------------------------------------------
This SF.Net email is sponsored by: InterSystems CACHE
FREE OODBMS DOWNLOAD - A multidimensional database that combines
robust object and relational technologies, making it a perfect match
for Java, C++,COM, XML, ODBC and JDBC. www.intersystems.com/match8
_______________________________________________
Bitpim-devel mailing list
Bitpim-***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitpim-devel


________________________________________________________________
NetZero Platinum $9.95.
NetZero HiSpeed $14.95.
Sign up Today at http://www.netzero.net or
Visit Sam Goody, Suncoast or MediaPlay Stores.

[Vic Heintz]

Post by Roger Binns

Post by d***@netzero.com
AllDayEvent allday: # key present if T, absent if F

There are two ways to deal with this. One is the explicit key you
have above. The other is to require the start time as 0:00 and the
end time as 23:59. I lean slightly towards the latter as it removes
what it ultimately duplicated data. However it is worth checking
vcal etc first.

Here is how vCal stores an all day event:

BEGIN:VEVENT
SEQUENCE:3
DTSTART;VALUE=DATE:20041116
DTSTAMP:20041029T182930Z
SUMMARY:All-day event repeating every year and never ending
UID:18375D10-3A2A-11D9-9FA5-003065D7577C
DTEND;VALUE=DATE:20041117
RRULE:FREQ=YEARLY;INTERVAL=1
END:VEVENT

Vic

[Roger Binns]

Did you get that from the spec, or from what Apple does?
The latter is not necessarily the same as the former :-)

Post by Vic Heintz
DTSTART;VALUE=DATE:20041116

They did the simplest thing which rather obvious. Leave
off the time.

Roger

[d***@netzero.com]

Post by Roger Binns

AllDayEvent allday: # key present if T, absent if F

There are two ways to deal with this. One is the explicit key you
have above. The other is to require the start time as 0:00 and the
end time as 23:59. I lean slightly towards the latter as it removes
what it ultimately duplicated data. However it is worth checking
vcal etc first.

Outlook uses both the flag and 0:00 to 0:00 scheme. I think either the 0:00-23:59 or 0:00-0:00 format would work.

Post by Roger Binns

Attachements notes:'string' # when applicable

The attachments are seperate and we don't need them. We should at
least get the main description/notes bit.

The main description would be the Subject field. Outlook has no 'Notes' field per se, but other formats might.

Post by Roger Binns

CreationTime creationtime:time.timetuple

What do we need this for?

LastModificationTime modtime:time.timetuple
synctime:time.timetuple # time of last import/sync

Those aren't needed and aren't values we can track anyway.

We'd need this if we ever want to sync, if not then they're not needed.

Post by Roger Binns

origin:'outlook' # data from Outlook

This is done with serials since the same entry could be present in

multiple devices and data sources.

The field's gone.
One field that I notice vCard has that Outlook does not have is 'Priority': 0 for none, 1-5 with 1 being the highest priority. Is this worth adding to the dict?

Post by Roger Binns

1. Implement a quick, reusable, and expandable Outlook import that
would work with current calendar structure. Concurrently but separately,
re-work BITPIM calendar with the new features.

Actually the existing schema is very broken (eg changeserial)
My preference is to update the schema to what we want as the
final version which should require not too many changes. (It
should even be possible to have a piece of shim code that takes
the new format and produces records in the old format to feed
to the individual phone modules).

Sounds like we have an initial agreed upon new calendar dict. For the next step, how would you want to coordinate to work to actually get this done?

-Joe Pham


________________________________________________________________
NetZero Platinum $9.95.
NetZero HiSpeed $14.95.
Sign up Today at http://www.netzero.net or
Visit Sam Goody, Suncoast or MediaPlay Stores.

[Roger Binns]

Post by d***@netzero.com
Outlook uses both the flag and 0:00 to 0:00 scheme. I think
either the 0:00-23:59 or 0:00-0:00 format would work.

Based on what Vic posted, I think even easier is to leave off
the time in the start and end dates.

Post by d***@netzero.com
The main description would be the Subject field. Outlook has no
'Notes' field per se, but other formats might.

I beg to differ. See screenshot! The vast majority of phones
don't have the field.

Post by d***@netzero.com

Post by Roger Binns

Post by d***@netzero.com
CreationTime creationtime:time.timetuple

What do we need this for?

Post by d***@netzero.com
LastModificationTime modtime:time.timetuple
synctime:time.timetuple # time of last import/sync

Those aren't needed and aren't values we can track anyway.

We'd need this if we ever want to sync, if not then they're not needed.

So we have an item that is in Outlook, on two phones and in Palm desktop
and an ical server. What is the correct value for those fields?

Post by d***@netzero.com
One field that I notice vCard has that Outlook does not have is
'Priority': 0 for none, 1-5 with 1 being the highest priority.
Is this worth adding to the dict?

Yes. Look for the red exclamation mark in the toolbar in Outlook.

The field should be absent if not set in the data source. I would
be inclined to use a numeric range of 1 thru 10 or thru 100. For
current data sources, we will always be able to map to for from that
range.

Post by d***@netzero.com
Sounds like we have an initial agreed upon new calendar dict.
For the next step, how would you want to coordinate to work to
actually get this done?

I recommend writing the shim code to convert from the new format
back to the old first. Then we'll work out what is next.

I am working on the database side of things at the moment, and
think I have something workable. I am going to start with
moving the phonebook into the database first.

Roger

[John O'Shaughnessy]

How rough do you want code?

I can check-in my guiwidgets.py, my p_lgvx6100.p and my com_lgvx6100.py if
you would like. They are still pretty rough.

Or we can wait until after this weekend when I'll have more time to make it
less rough.

Thanks,

John

[Roger Binns]

Post by John O'Shaughnessy
I can check-in my guiwidgets.py, my p_lgvx6100.p and my com_lgvx6100.py if
you would like. They are still pretty rough.

Mail them to this list first so we can all take a quick look.

Roger

[Vic Heintz]

Post by Roger Binns
Did you get that from the spec, or from what Apple does?
The latter is not necessarily the same as the former :-)

I must confess it is what Apple iCal exports as a vCal file.

Post by Roger Binns

Post by Vic Heintz
DTSTART;VALUE=DATE:20041116

They did the simplest thing which rather obvious. Leave
off the time.

Note that a one-day all day event has an end date of the following day,
however, I have confirmed that iCal will import a vCal file where the
DTEND is omitted and interpret it as a one-day event. A two-day event
would look like this for Nov 16-17:
DTSTART;VALUE=DATE:20041116
DTEND;VALUE=DATE:20041118

ie the date indicated in DTEND is not part of the event (i guess the
implied time is 00:00:00 which is midnight at the end of the prior day,
right?)

Vic