I’ve been a loyal pidgin user for a long time, and for the last couple of years it’s sat somewhat uncomfortably on the Ubuntu desktop. Obviously, Empathy became the default IM client a while back, but the more troublesome part, for me, has been Unity dropping support for the well established system tray icon specification in favour of their own Application Indicators. Ignoring the relative merits of the standards, and the questionable claim that you can unilaterally deprecate a widely used standard, Ubuntu has been good at providing indicator replacements for all the tray icons I care about, with one notable exception – Pidgin. Rather than providing a pidgin icon, they instead provided integration with the central messaging indicator. While this is a fine aspiration, I find the messaging indicator a very poor replacement – it doesn’t offer reasonable behaviour for showing and hiding pidgin and has problems dismissing new message notifications.

Initially, it was possible to turn on the system tray compatibility function in Unity with a simple dconf setting, but in 13.04, this was changed so that it would only work for java, and nothing else. In turn, this led to the creation of ppas for 13.04 and 13.10 to provide patched versions of Unity with the general purpose system tray restored (a one line change, apparently). I’ve been running with that for a while, but I didn’t want to swim upstream on this issue forever, so I decided to write a pidgin plugin that provides a proper indicator, with the same menu and behaviour as the tray icon.

It turned out to be an interesting exercise – creating indicators is extremely simple – all credit where it’s due – with the main challenge being building the menu without reinventing a wheel that’s already present inside pidgin. The pidgin tray icon (docklet is the internal name) is not a plugin, although there is a partial concept of different providers. Unfortunately, the interface can’t be used to drive an indicator as it assumes it can show the menu itself, while indicators require the menu be shown by the indicator. Ultimately, I had to copy the docklet code into my plugin to make the necessary modifications.

It would be possible to modify the docklet interface in pidgin to allow for an indicator provider with minimal impact on the existing providers, but I wanted to offer a working solution without requiring a newer version of pidgin, never mind the complexities of feeding changes upstream, etc. But, there’s an aspirational project there.

So, without further ado, I’d like to offer to all the stubborn people who want pidgin to work they way it always used to in Unity.

At some point I’ll get around to producing a deb for it, but just source for now.

Enjoy!

A couple of weeks ago, Han-Wen Nienhuys, the author of go-mtpfs, pointed out to me that Android’s MTP implementation includes a set of methods that allow you to do normal read and write operations on files without having to do the whole download/upload dance. With these extensions, you can expose files in the way that most people expect – you can just open a text file, picture, video etc, make changes and save it back. As a bonus, this functionality also allows you to do very useful operations like copy or move a file on the device.

I’ve now had a chance to put together an initial implementation of support for these extensions, and my PPA is in the process of rebuilding packages, so people can try them out easily. I’ve not started the upstreaming process on the GVFS changes as I still need to get the libmtp changes approved and upstreamed, but the libmtp maintainer has been AWOL for a few weeks now.

Obviously, it’s important to remember that these extensions are Android specific and won’t help you if you have a non-Android device, nor if your Android device doesn’t use Google’s MTP implementation (which, unfortunately, includes most Samsung devices).

You can grab Ubuntu packages from my ppa and the source is available on my github page.

Hi everyone,

Happy New Year! And the new year brings new updates on the gvfs MTP front. I received a bunch of useful feedback from the gvfs maintainers last month, which I was finally able to get time to sit down and address over the last few days. Accordingly, I’ve made a series of updates that fix a variety of things, from small memory leaks all the way to, finally, implementing the right way to tell Nautilus to do directory downloads/uploads in the right way (You’ve got to return a specific error code) – which fixes the one remaining functional gap in the code. Uploading a directory is still not fully working as I need to handle the way Nautilus ends up referring to uploaded directories by name when trying to upload their contents. Right now there’s no logic to remap the name to the MTP entity ID so the file uploads fail, but I know what has to be done.

I think we’re nearing the finish line, with respect to getting this merged upstream. *phew*

As always, the easiest way to try the code out is to install the packages from my PPA. I have also put a build of libmtp 1.1.5 in there so that unlock events and thumbnails work out of the box too.

Enjoy!

Hi again! It’s been a while since I’ve had something to write about, and it’s filesystems again, but with less April Fool’s.

What is MTP?

MTP is a standardised protocol that was originally designed to allow a PC to effectively manage the contents of a media player device – specifically, audio, video and image files. It is, in turn, based on an older specification called PTP (Picture Transfer Protocol) that was designed for use with Cameras. Note that neither of these uses cases has to do with managing the contents of an arbitrary filesystem. Of course, you can read more about MTP on wikipedia.

Why should I care?

Well, most people didn’t need to care about MTP for a long time – the chances were pretty good that their media player device didn’t use MTP (it either used USB Mass Storage or was an Apple device with its own crazy protocol), and their camera had a reasonable chance of using USB Mass Storage, and in the worst case you could always eject the memory card and use a reader with it.

However, since Android 3.0 (Honeycomb), Android devices have stopped using USB Mass Storage for PC connectivity, and switched to MTP. Now wait, you say, why would you use MTP to manage the contents of an arbitrary filesystem – a very good question. The primary reason is that USB Mass Storage is a block level protocol, and consequently operates below the filesystem layer. This means that it can’t be used to share a filesystem between the phone/tablet and the PC – only one device can read/write at a time. In older Android devices, this meant having a separate partition or memory card that was inaccessible to the phone while the PC was using it. But, from Honeycomb onward, Google wanted to have a more unified filesystem on the device, and not have to worry about ensuring there was a storage area that could be unmounted at random times. MTP may be an ill-fitting choice, but it’s the only standardised protocol which offers the key required feature – that being that you can have both the phone and PC use the filesystem at the same time.

This is posible because MTP treats files as the atomic data unit, rather than blocks. So you read and write whole files, and nothing smaller. At that point, the PC interacts with the filesystem pretty much like any other application running on the phone.

Ok, so I have to care about MTP to access files on my phone. Why’s that worth talking about?

Well, here’s the kicker. Your shiny new Android phone uses MTP, and there are a plethora of applications and components for Linux that notionally can manage MTP devices. Unfortunately, they are all limited in various ways, that make them pretty much unusable. The most common flaw is that the tools use an MTP library call that attempts to enumerate the entire device filesystem in one go. This causes the initial connection to be very slow, and on the newest Android devices, it flat out doesn’t work, as the phone end will timeout the operation before it completes. This ends up taking out every single tool on the market (including: mtpfs, gmtp, banshee, rhythmbox, gvfs gphoto2 backend) except for one: go-mtpfs.

go-mtpfs?

go-mtpfs is a recent creation of a Google employee, who was aware of the timeout behaviour of android devices, and so they wrote a FUSE filesystem (much like the original mtpfs) but which did on-demand file enumeration, as each directory is loaded and queried (which is how the Windows MTP implementation works). The end result is quick connections, and a tool that can talk to Android devices well.

So we’re done?

Well, no, not quite. Although go-mtpfs uses MTP the right way, it can’t avoid the horrible impedence mismatch between MTP’s file-atomic model and a traditionally filesystem where you can do random I/O within a file (ie: open, seek, read, write, close, etc). MTP only allows you to download a complete file, or upload one – you can’t even move a file between locations on the device through MTP (you have to download it, delete it, then upload it to the new location). Of course, FUSE doesn’t care that you can’t provide normal filesystem semantics, so you have to improvise. In this case, that means that any read/write operations requires go-mtpfs to do an elaborate and fragile dance to download a file, modify it, and upload it again, and so on. This causes simple file operations to behave strangely, and things can go very bad if you use a tmpfs for /tmp and try to access a very large file.

And so: gvfsd-mtp

And finally we reach the meat of this post. gvfs is the virtual filesystem layer that’s used by Gtk+ based desktop environments (GNOME 3, Unity, XFCE, etc). It happens to allow backends to implement a much higher level API than FUSE, and this API happens to explicitly offer ‘pull’ and ‘push’ operations (download and upload respectively). As such, it’s possible to meaningfully map functionality without jumping through crazy hoops.

So, I’ve been working for the last few weeks on a native mtp backend for gvfs. It’s heavily based on the existing gphoto2 backend, but only attempts to implement the operations that MTP can cleanly accomplish.

The end result is a filesystem like view that you can effectively browse through nautilus, and that you can download files from and upload files to with a reasonable hope of success. It’s not seamless as gvfs/nautilus do not do anything very special when you attempt an unsupported operation – so trying to just open a file will probably fail (although some GNOME apps like evince or file-roller seem to be able to detect the behaviour and will automatically download the file and open the temporary copy), but it’s functional and reliable.

Another thing I did was to avoid caching any metadata that I could avoid caching. All other MTP implementations tend to save directory listings after getting them and don’t go back to the device. But the device can be adding/removing files all the time, so it’s important to be able to get a fresh listing when you need it. To achieve that, I took advantage of the fact that gvfs lets you define ‘display names’ for files – which can be completely different from the reported filename. So I directly mapped the MTP object IDs (numbers) to the reported filename and made the real filename appear as the display name – so the view in Nautilus looks completely normal but the gvfs URI might be something like mtp://[usb:001,015]/65537/1/128/2445

This general behaviour is actually very similar to how Windows presents MTP devices – they appear as normal looking drives and folders and files in Windows Explorer but you can only really download and upload files – it doesn’t pretend to offer normal read/write operations on the files.

Ok, where do I get it?

Here!. I’m continuing to work on it – I should be able to provide thumbnails from the MTP metadata, and I’d like to implement a way to copy directories back and forth (gvfs doesn’t implement recursive push/pull for you – you have to do it yourself), but it’s otherwise very usable – it was proper plug/unplug detection and I modified the gphoto2 backend to not grab mtp devices like it used to. Eventually I’d really like to get it upstream (especially as you can’t build gvfs backends outside of the gvfs source tree) but there’s a fair way to go yet.

Enjoy!