How-to: OTA Photo sync an N900 with iPhoto

First up, if you’ve not seen the original post and video, checkit. Secondly, this is a long detailed post. Fetch a drink and a snack.

Right, here it is, the how-to for syncing the N900 over the air with iPhoto. First, before I go any further a strong warning:

This software setup is *extremely* experimental. It’s really just a proof of concept. Don’t even attempt it unless you understand all the steps here, and the risks. I’ve only had an N900 3 days, it’s barely been tested. If you do try it, backup all your data etc. I make no promises these steps will work for you, and assume no responsibility if they delete all your data, or worse. It may also use a LOT of data over the network, so beware!

Blunt, but I felt it had to be said. I don’t want someone complaining to me if they loose all their photos! A second, lesser warning – this how-to is not for the faint of heart. You need to be comfortable with the terminal etc. It’s not written for novice linux/mac users, it’s written for someone like myself that knows what they’re doing. If it seems over-complex, well hopefully there’ll be a nice easy to use application version of all this some time soon, check the last section for my thoughts on that. Having said that, if you do know what you’re about and some part is unclear – fire away in the comments and I’ll clarify it.

How does it work then?

First of all, lets get a basic overview of what’s going on:

There are 3 machines involved here. The N900 and the mac, and a server to enable the syncing when either the mac or the N900 are turned off. This enables the syncing to happen easily wherever I am, as the N900 will always be able to sync. This solution suits me, as I already have an always on server.

The N900 regularly synchronises (2-way, with Unison) with a folder on my server. The mac does the same, so changes will eventually propgate both directions. This means new photos taken end up in your iPhoto library, and any edits you do will end up on the N900 to show to friends and family. Now, in terms of the syncing rules, you can basically ignore the server in the middle, it’s just an implementation detail. In iPhoto, we sync with a specific (smart) album, not the entire library. So syncing works as follows:

At the N900 end

  • Photos on the N900 not in the iPhoto album are pushed to iPhoto if they’re new, or deleted from the N900 if they’ve already been uploaded (i.e. they’re in iphoto but have been explicitly removed from the sync-album, and so the N900. But that’s ok, as they’re still in the main iPhoto library)
  • Photos in iPhoto but not on the N900 are sync’d into the N900 (so you can push other pics from your library to your device).
  • Photos edited in iPhoto will replace originals on the device. (but iPhoto keeps the unedited versions, so that’s ok)
  • If a photo is edited both ends, the newer version takes preference.

At the iPhoto end

As per the N900 except:

  • Edited photos from the N900 (that have been previously sync’d) will appear in iPhoto as a new version alongside the original.
  • Photos removed on the N900 will not be removed from the sync album (and so will eventually sync back to the N900 unless removed in iPhoto).

In general, it’s been chosen to avoid losing photos if at all possible, erring on the side of caution. It could work in different ways – you can edit the scripts below. Additionally, the N900 backs up all its photos before every sync, so you should never loose a picture (though edits will still replace originals, but again iPhoto should have the original still). As there’s a server in the middle, it takes a pair of syncs for a photo to make it from one end to the other.

When does it sync?

OK, a small confession. In the video it looks like it’s instant. In fact for the purposes of a short video I’m manually triggering the required sync actions. For automated use, cron jobs handle the syncing. They can be as frequent or infrequent as you like – in this example the mac syncs every half hour, the N900 hourly. In addition a manual sync can be triggered by a desktop shortcut on the N900, and the same way (or via spotlight etc) on the mac. Finally, syncing can be turned off, by placing a “skip.txt” file in a specified place. No syncing will happen till it’s removed again.

Where does it sync?

Everywhere. Wi-fi, 3G, EDGE, GPRS, the scripts don’t care. It just uses whatever net connection the phone/mac happen to have. It would be trivial to change the scripts to only sync over wifi, or only certain networks etc. E.g. maybe you don’t have a server, you could set it to sync only on your home wifi when your mac’s turned on (or indeed manual triggered sync). I’ll not go into all the permutations here, if you’re capable of understanding this how-to, you can make the modifiations yourself. I just ask that you post them in the comments for others to see too.

I understand you sync with Unison, but how do you make that work with iPhoto?

Good question, glad you’re keeping up with me. This was the trickiest bit of the whole process. I wanted the sync to work all the time, and not require iPhoto to be running either, and yet not mess with the iPhoto database or risk corrupting it. To that end, to get files out of iPhoto I use iPhotoFS, to get them in I use the poorly-documented auto-import folder. Any photos in there will be auto-imported the next time iPhoto is launched, or brought to the foreground (if running). Then there’s a smart album that ensures that all photos autoimported from the N900 end up in the sync album. So then we just need to mash all that together so we can unison it. To do that we use UnionFsFuse (we’re using MacFUSE for both these filesystems). This isn’t the place to explain UnionFS, but roughly it unions several filesystems into one folder, writing changes to the highest layer it can that’s read-write. We use a temporary sync dir on the mac, and layer like this:

So what unison sees is the union of any files waiting to import to iPhoto, any files already in iPhoto. Then it will write any new files to the sync dir. Once the sync is done, the files are moved into the auto-import dir, this way iPhoto won’t trigger a sync untill it has all the photos.

Technical Details

First up, you will need:

A *nix server with:

  • SSH server
  • Unision
  • A username to use for syncing, here we use “n900user”
  • A sync directory, we’ll use “~/picsync”

A mac with:

  • iPhoto, with a smart album “N900”
  • MacFUSE
  • iPhotoFS (make a note of where you put the app)
  • UnionFsFuse (to compile for MacFUSE note this, you may also need “CFLAGS -D__FreeBSD__=10” Also “make install” won’t work, DIY it like “sudo cp src/unionfs /usr/local/bin/”)
  • Unison (same version as the server)

An N900 with

  • SSH client (in Extras repo)
  • Unison. Needs to be the same version as the server. Either compile it yourself, or get a precompiled one from this thread here. The thread has some notes on compiling it too.
  • Fcron – following the info here, and my note below.

I installed Fcron with the following steps (slightly different from the thread!):

get the deb onto the n900 from: https://repo.codemages.net/opt/ (or add the repo and do it that way)

dpkg -i fcron_3.0.1-2_armel_opt.deb
useradd fcron
chown fcron:fcron /etc/fcron.*
chown root:fcron /etc/fcron.conf
 chmod 644 /etc/fcron.conf
chown -R fcron:fcron /var/spool/fcron

Then edit /etc/event.d/rcS-late
And after initctl emit MOUNTS_OK
put:

/etc/init.d/fcron start

I’ll assume you can get all the above going yourself.

Step 1

Set up public key authentication so you can login over SSH to your server without a password from both the N900 and your mac. BE AWARE OF THE SECURITY IMPLICATIONS OF THIS. Someone with your N900 would not need your password to get into your server. Excellent instructions can be found here but note that “authorized_keys2” on the server needs the second key appended to it (so the key for both the N900 and the Mac are in there). The instrucitons there would have you overwrite it.

Step 2 – the Mac

First up, create a smart album in iPhoto. We’ll assume it’s called “N900” and the conditions should be set to “camera model is N900”. I also match “keyword contains Movies” to capture videos too, and a rule to add any photos I put in another album too, to sync non-n900 pictures.

Save the below script as “iPhotoSync.sh” and make it executable. I’ll assume you save it to “/Users/USERNAME/n900testing/iPhotoSync.sh”. Then edit the variables at the top of the file to match your setup:

#!/bin/bash
#CONFIGURE THESE BITS!
######################
#path to iPhoto library. NOTE the quotes to protect spaces!
PHOTOLIB="/Users/USERNAME/Pictures/iPhoto Library"
#path to iPhoto FS app
PHOTOFS="/Users/USERNAME/apps/iPhotoFS.app"
#a directory used for temporary file syncing
SYNC="/Users/USERNAME/n900testing/sync"
#album in iphoto to sync with. NOTE you should ideally ensure this is a smart album
#and will contain any new N900 imports! if not then photos will repeatedly sync in
#or delete from the N900 (TEST THIS!)
ALBUM=N900
#The remote share to union with
REMOTE=ssh://USER@SERVER/picsync
#If this file exists, syncing will be turned off
SKIPSYNC="/Users/USERNAME/n900testing/sync/skip.txt"

#######################
#Leave alone the stuff below here!

IMPORT=$PHOTOLIB/Auto\ Import
FS=$PHOTOFS/Contents/MacOS/iPhotoFS
ALB=/Volumes/iphotofs/Albums/$ALBUM
FILES=$SYNC/files
UNION=$SYNC/union

if [ -f "$SKIPSYNC" ]; then
 echo "skipping"
 exit
fi

export UNISONLOCALHOSTNAME=mac

if [ ! -d "$FILES" ]; then
 echo "No FILES dir, creating.."
 /bin/mkdir "$FILES"
fi
if [ ! -d "$UNION" ]; then
 echo "No UNION dir, creating.."
 /bin/mkdir "$UNION"
fi

if [ ! -d "/Volumes/iphotofs" ]; then
 echo "Mounting iphoto"
 $FS & #TODO better if this didn't pop a finder window
fi

/usr/local/bin/unionfs -o cow "$FILES"=RW:"$IMPORT"=RO:"$ALB"=RO "$UNION"
/usr/bin/unison -batch -ui text -confirmbigdeletes -times -prefer newer -perms 0 -ignore 'Name .*' "$UNION" "$REMOTE"
/sbin/umount "$UNION"
/bin/mv "$FILES/"* "$IMPORT"
/usr/bin/touch $SYNC/lastrun

A few points to note about the above:

  • If the $SKIPSYNC file exists, it won’t sync.
  • The file “lastrun” in $SYNC will have a modification date equal to the last run.
  • iPhotoFS won’t be unmounted, to stop it popping a finder window each time it runs. This is something iPhoto

Run the script from the command line, test it with a few files that it’s working right. You can make an automator .app to allow you to trigger it manually from a double-click or spotlight. I assume you can do that yourself if you’re reading this, if not post in the comments and I’ll add it to the how-to.

Now we need to automate syncing with cron. There’s a great guide here. The crontab I use is as follows:

MAILTO=""
# min  hr mday month wday command  
 */30 *  *    *     *    bash /Users/USERNAME/n900testing/iPhotoSync.sh

Without the blank MAILTO output from the script would be added to /var/mail/USER

Step 2 – the N900

Save this script somewhere, make it executable, and setup the variables. I assume it’s in “/home/user/n900Sync.sh”. Note it’s not a typo at the top, we use the “ash” shell.

#!/bin/ash
#CONFIGURE THESE BITS!
######################
#path to photos. NOTE the quotes to protect spaces!
PHOTOS="/home/user/MyDocs/DCIM"
#path to dir to backup photos
BACKUP="/home/user/MyDocs/.photoBackups"
#The remote share to union with
REMOTE=ssh://USERNAME@SERVER/picsync
#If this file exists, syncing will be turned off
SKIPSYNC="/home/user/MyDocs/skip.txt"

#######################

if [ -f "$SKIPSYNC" ]; then
 echo "skipping"
 exit
fi
export UNISONLOCALHOSTNAME=n900

if [ ! -d "$BACKUP" ]; then
 echo "No BACKUP dir, creating.."
 mkdir "$BACKUP"
fi

cp "$PHOTOS"/* "$BACKUP"/.
/usr/bin/unison -batch -ui text -confirmbigdeletes -times -prefer newer -perms 0 -ignore 'Name .*' "$PHOTOS" "$REMOTE"

Again, you want to check this works before we setup the cron job.

We setup fcron first by setting an editor e.g. “export EDITOR=/usr/bin/vim” then as root “fcrontab -e”. I use:

MAILTO=""
# min  hr mday month wday command  
 */59 *  *    *     *    /bin/su - user -c "/home/user/n900Sync.sh"

So then every hour it runs the sync script as “user”. To create a manual sync shortcut in your applications menu/desktop, then on the n900 as root create: “/usr/share/applications/hildon/picsync.desktop”, same permissions as the other files in that dir, contents are:

[Desktop Entry]
Version=1.0
Encoding=UTF-8
Name=PicSync
Icon=tasklaunch_file_manager
Exec=/home/user/n900Sync.sh
Type=Application

Of course you can change the icon if you want. And we’re done, enjoy!

Where now?

I’ve only got this N900 as a trial for a month. I’ve no incentive to develop this further as I’ll have it give the device back :-( . It’s a proof of concept. I’ll hand this over to the Maemo community to develop further. I’d like to see this as an app that can be installed from the repository, with a nice GUI to configure the various options. Other features that would be good to add would be:

  • Choose what networks to sync on, or what type of network.
  • Maybe sync over 3G only if there aren’t many new pics since last sync etc. Limit data.
  • Option to sync when the camera shutter is closed.
  • Software to configure the OSX side.
  • More testing! What happens on failed syncs etc.
  • GUI

If anyone does take this on and extend the idea, please credit this blog, or me by name – Iain Wallace.

Finally, thanks for reading all this, hope it’s of use!

Advertisements

4 Responses to “How-to: OTA Photo sync an N900 with iPhoto”

  1. […] yourselves: How-to: OTA Photo sync an N900 with iPhoto How to do something differently… 2000 words+ but I wanted it well documented, so someone else can take my proof of concept and run […]

  2. The world of Maemo need people like you. 3 days and you have a brilliant idea that works.

    Thanks for taking the time to write out this post and explaining how you did!

  3. […] Si queréis ver el tutorial para configurar la sincronización podéis visitar la página de doitdifferent. […]

  4. Nice writeup. I just got an n900 yesterday.
    I’d like to start where you left off with this project.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: