Saturday, September 26, 2009

Migrating from Gmail to Google Apps

Notes taken while I migrated from Gmail+Calendar+Contacts to Google Apps with equivalent features.

Migrate Emails

I used imapsync on Fedora 11 to do the dirty work of transferring emails from Gmail to Google Apps.  I used the script as defined by this blog.  I made some customizations to ease switching between multiple users that I transferred to Google Apps.

If you have access to a semi-recent version of Fedora you can install imapsync with "yum install imapsync".

Save the following script text to a file and "chmod 755".  Update the USER1 and USER2 lines with your account names.  You'll need to create two files called passfile1 and passfile2 that contain the passwords for each account.

Note: Compared to script from above log, I added --allowsizemismatch to get past some emails that somehow had wrong file sizes stored.   You may wish to try without that option first.

--- begin script ----

USER1E=`echo ${USER1} | sed 's/@/\\\\@/'`
USER2E=`echo ${USER2} | sed 's/@/\\\\@/'`

echo "***** Starting *****" > $LOGFILE

echo "" >> $LOGFILE

time imapsync --host1 \
  --port1 993 --user1 ${USER1}  \
  --passfile1 $PASSFILE1 --ssl1 \
  --host2 \
  --port2 993 --user2 ${USER2} \
  --passfile2 $PASSFILE2 --ssl2 \
  --syncinternaldates --split1 100 --split2 100 \
  --authmech1 LOGIN --authmech2 LOGIN \
  --useheader "Message-ID" \
 --useheader "Date" --skipsize --allowsizemismatch \
  --regexmess "s/Delivered-To: ${USER1E}/Delivered-To: ${USER2E}/gi" \
  --regexmess 's/Subject:(\s*)\n/Subject: (no--subject)$1\n/ig' \
  --regexmess 's/Subject: ([Rr][Ee]):(\s*)\n/Subject: $1: (no--subject)$2\n/gi' 2>&1 | tee -a $LOGFILE

echo "" >> $LOGFILE
echo "***** COMPLETE *****" >> $LOGFILE
echo "" >> $LOGFILE
--- end script ---

Migrate Calendars

Transferring your Primary calendar is hardest part.  If you have multiple calendars, the non-primary calendars can simply be Shared with the new Google Apps account and you will see no differences.  For your primary calendar, I found it easiest to save an ical version of events and then import it.

Use these steps to save save an ical version of your events from Gmail:  Under "My Calendars" list, click the Settings.  Under Settings, click on your primary calendar.  Towards bottom of new screen lists you "Private Address".  On the green ICAL icon, right click and click your Save Link As option (using Firefox anyways).

On your Google Apps calendar, click the Settings on your "My Calendars" list.  On next screen, there is an option to Import Calendar.  Select it and when prompted, give the filename of your ICAL saved in previous step.

Migrate Contacts

The follow info on contacts was taken from a blog also on migrating.

To preserve contact groups, each group needs to be migrated individually.
  1. On Google Apps mail, pre-create all your Groups that existed under Gmail.
  2. On GMail contacts, use the Export option to export each group in Google’s CSV format. Then export one more CSV of “Everyone (All Contacts)”
  3. On Google Apps contacts:

    1. Use the Import option to import the CSV of Everyone, leaving the  “Add these imported contacts to: ” option unchecked.
    2. Then use the Import option to import each group in turn, checking the “Add these imported contacts to:" and making sure to select the correct group name that was created in step #1.

Migrate iPhone

If you were using Google's Sync option for iPhone using Active Sync you were already pointing at the correct servers.  Just need to change Username from to new and also the Password.

To be safe, I toggled OFF and then back ON the settings for Mail, Contacts, and Calendar and also selected "Delete from iPhone" when prompted.  This did a force resync of these items although calendar ended up empty.

You will also need to use Safari and browse to  Look for instructions for configuring "Google Apps" and follow them.  You will need to do this to switch google sync to use your Google Apps calendars.

Sunday, March 8, 2009

Backing up Windows Vista with Free Software

I’ve been attempting to deal with the sad state of backup affairs that Windows Vista is in.  I have one computer with a spare hard drive that I could easily dump files to.  This computer is running Vista Basic which contains a backup program that can only be manually invoked and which only does partial backups.  I have another computer running Vista Home Premium which can automate backups but has no spare hard drive and I’m suffering from well documented issues with Vista and sharing remote disks.
I looked for free or cheap programs to help me automate this task but most would have been difficult to recover from; as they used ZIP archives.   I preferred a “rsync” based solution as thats what I use under Linux with great success.  So here are the steps I used to get it working.  This is a collection of bits of information I found all over the web.

Cygwin Setup

The first step was to install Cygwin.  Using the Cygwin SETUP.EXE, make sure and install the “rsnapshot” program.  This program automates using rsync to backup the hard drive (using hard links to save space) and also doing the housework of rotating directories over time.
It is *very* important that after installing cygwin that you tell it to not use NT-style security.  You will run into unsolvable file permission issues with your backed up files unless you do this next step.  Navagate to My Computer->Properties->Advanced->Environment Variables->System Variables.   Either edit or add “CYGWIN = nontsec tty".
Another option would be to investigate cwrsync.  It basically automates the above steps for you; installing a mini-cygwin environment within its own directory tree.

Shadow Copy Support

All versions of Vista include support for creating Shadow versions of drives.  This allows you to see a file system in a consistent state at some exaxt point in time.  I believe it also allows you to open/read files that applications already have opened.  Window’s Backup and most commercial backup programs make us of this feature.
Microsoft has a command line program that allows you to hook into this feature.  You can request to create a shadow copy of a drive and mount it to a drive letter.  I’ve found that backing up a shadow drive is infinitely more stable then backing up an live drive.
The steps later require installing this command line program.  Install the Volume Shadow Copy Service SDK.

rsnapshot setup

Edit /etc/rsnapshot.conf to your liking.  Here are some options that I used to backup my C: drive (shadowed as a J: drive) to my F: drive .
snapshot_root /cygdrive/f
exclude_file /etc/rsnapshot.exclude
logfile /cygdrive/f/snapshot.log
no_create_root 1
backup /cygdrive/j/ localhost/ 
Next, you’ll need to create the file /etc/rsnapshot.exclude file that is referenced above.  It contains a list of Vista’s default junction points and will prevent copying the same file multiple times and prevent infinit recursion error messages from being printed.

#Junction points or symlinks
- /cygdrive/j/Users/All Users
- /cygdrive/j/Users/Default User
- /cygdrive/j/Users/All Users/Application Data
- /cygdrive/j/Users/All Users/Desktop
- /cygdrive/j/Users/All Users/Documents
- /cygdrive/j/Users/All Users/Favorites
- /cygdrive/j/Users/All Users/Start Menu
- /cygdrive/j/Users/All Users/Templates
- /cygdrive/j/Users/Public/Documents/My Music
- /cygdrive/j/Users/Public/Documents/My Pictures
- /cygdrive/j/Users/Public/Documents/My Videos
- /cygdrive/j/ProgramData/Application Data
- /cygdrive/j/ProgramData/Desktop
- /cygdrive/j/ProgramData/Documents
- /cygdrive/j/ProgramData/Favorites
- /cygdrive/j/ProgramData/Start Menu
- /cygdrive/j/ProgramData/Templates
- /cygdrive/j/Documents and Settings

#Excludes these junction points common to every user profile
- /cygdrive/j/Users/*/Application Data
- /cygdrive/j/Users/*/Cookies
- /cygdrive/j/Users/*/Local Settings
- /cygdrive/j/Users/*/My Documents
- /cygdrive/j/Users/*/NetHood
- /cygdrive/j/Users/*/PrintHood
- /cygdrive/j/Users/*/Recent
- /cygdrive/j/Users/*/SendTo
- /cygdrive/j/Users/*/Start Menu
- /cygdrive/j/Users/*/Templates
- /cygdrive/j/Users/*/AppData/Local/Application Data
- /cygdrive/j/Users/*/AppData/Local/History
- /cygdrive/j/Users/*/AppData/Local/Temporary Internet Files
- /cygdrive/j/Users/*/Documents/My Music
- /cygdrive/j/Users/*/Documents/My Pictures
- /cygdrive/j/Users/*/Documents/My Videos

#Temporary and in-user user data
- /cygdrive/j/Users/*/AppData/Local/Microsoft/Windows/Temporary Internet Files
- /cygdrive/j/Users/*/AppData/Local/Temp
- /cygdrive/j/Users/*/NTUSER.DAT
- /cygdrive/j/Users/*/ntuser.dat.LOG1
- /cygdrive/j/Users/*/ntuser.dat.LOG2
- /cygdrive/j/Users/*/AppData/Local/Microsoft/Windows/UsrClass.dat
- /cygdrive/j/Users/*/AppData/Local/Microsoft/Windows/UsrClass.dat.LOG1
- /cygdrive/j/Users/*/AppData/Local/Microsoft/Windows/UsrClass.dat.LOG2
- /cygdrive/j/Users/*/AppData/Local/Microsoft/Windows Defender/FileTracker
- /cygdrive/j/Users/*/AppData/Local/Mozilla/Firefox/Profiles/*/Cache
- /cygdrive/j/Users/*/AppData/Roaming/Microsoft/Windows/Recent
- *.lock

#Installation folders and system data
/cygdrive/j/Program Files
/cygdrive/j/System Volume Information

Scripts to perform copy

The following script should be used as a template.  You will need a script for each “interval” line in your /etc/rsnapshot.conf and it needs to pass in “hourly”, “daily”, “weekly”, or “monthly” to the rsnapshot program.
The script will create a shadow version of the C: drive and map it to drive J:.  It will then invoke rsnapshot to backup the J: drive and afterwards delete the showdow drive.
### Change to temporary directory so that we have a known location
### of temporary files created in local directory.
cd /tmp
### Create shadow drive of C: and store off its ID
/cygdrive/c/Program\ Files/Microsoft/VSSSDK72/TestApps/vshadow/bin/release-serve
r/vshadow.exe -p -nw -script=vars.bat c:
SHADOW_ID_1=`grep SHADOW_ID_1 vars.bat | sed "s/.*=//"`
### Map to drive J:
/cygdrive/c/Program\ Files/Microsoft/VSSSDK72/TestApps/vshadow/bin/release-serve
r/vshadow.exe -el=$SHADOW_ID_1,J:
### Run backup
/usr/bin/rsnapshot daily >> /cygdrive/f/rsnapshot.stdout 2>&1
### Delete shadown drive
/cygdrive/c/Program\ Files/Microsoft/VSSSDK72/TestApps/vshadow/bin/release-serve
r/vshadow.exe -ds=$SHADOW_ID_1
### Go back to temporary directory and clean up files.
cd /tmp
rm vars.bat

Scheduling Backups

You can use Window’s Scheduled Tasks tool to run the above script at the intervals of your chosing.  Start->Accessories->System Tools->Scheduled Tasks.
The command line to use is basic form shown below.  Replace with the filename you created using above script template.
c:\cygwin\bin\bash.exe /path/to/
Note: Just because you run the “weekly” interval on a given day each week, you still need to run the “daily” interval that same day.  Same thing applies for all intervals.

Automating emails to cancel iCalendar events

In a follow up to my post about emailing iCalendar requests,you may wish to send out an automated email to cancel that request and have the user automatically be prompted to remove it as well.  That can be handled with minor modification to the previous script and .ics files.

First, find the following piece of code:
icspart = MIMEBase('text', 'calendar', **{'method' : 'REQUEST', 'name' : 'meeting.ics'})
and change that to a CANCEL method:
icspart = MIMEBase('text', 'calendar', **{'method' : 'CANCEL', 'name' : 'meeting.ics'})
Next, in the original .ics file, change any line with METHOD: to read METHOD:CANCEL and any line with STATUS: to read STATUS:CANCELLED.

Thursday, January 22, 2009

Automating emails of iCalendar (.ics) files

I was surprised after a period of googling that I couldn’t find a python script to email iCalendar files. More specifically, I was wanting to email a *.ics file as an attachment and if it was received by an Outlook client I would like for Outlook to automatically add the buttons to Accept/Deny/etc just like when sent from Outlook itself.
I found a site that did most the research on this task already but they used Perl. Also, their script creates the email but doesn’t send it.
I wanted to use python since it had all included modules already to both create the message and send it using SMTP. For your pleasure, here is my version of the script.
#!/usr/bin/python import smtplib import os from email.MIMEMultipart import MIMEMultipart from email.MIMEBase import MIMEBase from email.MIMEText import MIMEText from email.Utils import COMMASPACE, formatdate from email import Encoders send_from = "" send_to = "" subject = 'Meeting Invite' msg = MIMEMultipart('alternative') msg['From'] = send_from msg['To'] = send_to msg['Date'] = formatdate(localtime=True) msg['Subject'] = subject msg.add_header('Content-class', 'urn:content-classes:calendarmessage') msg.attach(MIMEText("See attachement for Meeting Invite.")) icspart = MIMEBase('text', 'calendar', **{'method' : 'REQUEST', 'name' : 'meeting.ics'}) icspart.set_payload( open("meeting.ics","rb").read() ) icspart.add_header('Content-Transfer-Encoding', '8bit') icspart.add_header('Content-class', 'urn:content-classes:calendarmessage') msg.attach(icspart) #print msg.as_string() smtp = smtplib.SMTP('localhost'); smtp.sendmail(send_from, send_to, msg.as_string()) smtp.close()
This requires an iCalendar file called meeting.ics to already exist in current directory.
This script does the minimum work possible and leaves plenty of room for improvements. For example, it would be nice to have text body of the email be the same as DESCRIPTION in the meeting.ics file so that user can see what the calendar item is about even if their application doesn’t support *.ics files. And it could base Subject off of SUMMARY.
From other web site, it sounds like the email sent isn’t compatible with Mac OSX iCal because they want PUBLISH instead of REQUEST. I guess we can’t please everyone right now.