Rotation confuses face recognition

I have around fifty thousand digital photos in my library, and it has become impractical to browse through them looking for some event, or some place, or some person. So other than archiving, there really is no value in collecting photos if you can’t go back and find the ones you are looking for.

A few years ago I switched to using Image Ingester Pro and Adobe Photoshop Lightroom to import and catalog my photos. Now one of the first things I do after importing photos, is to add keywords describing the event, the place, and the people. But, I still have tens of thousands of photos that have no keywords, and there is no easy or automated way to add keywords to these photos. One thing that can be automated is adding people keywords to photos based on automated face recognition.

Unlike free Windows Live Photo Gallery, or free Google Picasa, or free Apple iPhoto, none of Adobe’s products offer face recognition, something often discussed and complained about in Adobe forums. Professional photographers may argue that face recognition is a gimmick, and I agree that for professional workflows it may not be required, but at less than $300, Photoshop Lightroom is an ideal tool for use by amateur and family photographers, and I think face recognition is a must have feature.

Although Lightroom does not directly offer face recognition, there are convoluted ways to add people keywords to Lightroom using Google Picasa and Jeffrey Friedl’s Picasa Face-Recognition Import plugin. The process requires you to add all your photos to Adobe Lightroom and Google Picasa, then use Picasa to detect faces, and assign them to contacts, then use Jeffrey’s import plugin to add the Picasa people as keywords to photos in Lightroom.

Picasa is not the most friendly app to work with, it may alter photo metadata without your intent, adding support for new RAW formats takes a long time, there are lots of bugs related to managing people and duplicate contacts happens all the time. The Picasa to Lightroom conversion experience is not something I am prepared to deal with on an ongoing basis, unfortunately I am also not aware of any other ways of automatically tagging people in pictures in Lightroom.

The next best thing would be to use an application that does support face recognition in addition to Lightroom, even if the two tools do not integrate or share metadata. Since I was already quite familiar with Picasa, and I have a Mac but don’t use it as a primary machine, that left me with Windows Live Photo Gallery (WLPG).

Unlike Picasa that uses its own built in image rendering technology, WLPG uses Windows Imaging Component (WIC) technology to render and interpret image metadata. The downside is that you need to install RAW image WIC codecs in order for WLPG to display RAW images, the upside is that you can install a WIC codec instead of waiting for the app to natively support the RAW format. As an example, the Panasonic DMC-LX5 was released July 2010, Picasa added support for DMC-LX5 RW2 files in Picasa 3.9, released December 2011, 17 months after the camera’s release.

WIC does have its drawbacks, some camera manufacturers do not release WIC codecs at all, and some big name manufacturers, like Canon and Nikon, are stuck in the dark ages with no x64 codec support. At this time I am aware of two alternate suppliers of RAW WIC codecs, Axel Rietschin’s FastPictureViewer Codec Pack, and Microsoft’s Camera Codec Pack. Microsoft’s Camera Codec Pack is free, but offers limited camera support, and as we’ll see later, limited Explorer and WLPG integration support. FastPictureViewer Codec Pack (FPVCP) costs $14.95, is frequently updated, supports almost all camera’s and formats under the sun, integrates with Explorer and WLPG, and is what I use.

With FPVCP installed, WLPG was easy to use, the contact and names feature integrated nicely with Windows Live contacts, without any of the weirdness of the equivalent functionality I found with Picasa. Once faces were tagged, a semi automated process requiring manual verification, it was easy to find photos I was looking for, e.g. I could say show me all photos in December 2010, with me, my wife, and our daughter in the picture.

So this finally brings me to the actual problem I wanted to write about. I noticed that WLPG would get confused when tagging faces in some portrait rotated pictures. When you view the image standalone, the faces are correctly recognized, and the bounding rectangles are correctly drawn over the image. But, the thumbnails are completely wrong, taken from a different part of the image. It seems like the thumbnails are taken from the landscape coordinates view of the image, not portrait coordinates of the image.
See the following pictures as an example, note how the thumbnails in the portrait view image are taken from the wrong part of the image:
WLPG.Face.JPG.LandscapeWLPG.FPVCodecPack.Face.JPG.Portrait

I reported the problem in the WLPG support forum, and after some back and forth, I provided sample pictures where I could reproduce the problem, but I was told that they were unable to reproduce the problem using the same pictures. As I was not crazy, and I had seen this behavior on two different machines, I wanted to create steps to reproduce the problem.

The sample images were taken of a magazine page taped to a door, using a Canon 5D Mark II, a Panasonic DMC-LX5, a Panasonic DMC-ZS7, and an iPhone 4. I took 5 pictures with each camera in each mode; face centered, face top-left, face top-right, face bottom-left, and face bottom-right. I did this in landscape mode, portrait mode, JPG mode, and RAW mode.

I fired up a clean Windows 7 Ultimate x64 SP1 VM, installed WLPG v15.4.3538.513, and Picasa v3.9, and I dropped my collection of sample images on the machine.

As I viewed the images in Explorer, I immediately noticed a difference between my machine and the test machine, the test machine Explorer view did not display the JPG images using the correct rotation, while my machine did. This is when I remembered that I have FastPictureViewer Codec Pack installed on all my machines, and that this may have something to do with the face rotation problem.

See the following pictures of the Explorer view with and without FPVCP, note how the FPVCP version displays the CR2 thumbnails and displays the JPG files in the right rotation:
Explorer.5DMk2Explorer.5DMk2.FPVCodecPack

Testing Windows Live Photo Gallery showed that as with Explorer, it also does not display the images using the correct rotation. This was a real big disappointment for I can’t believe that a photo application with all the bells and whistles of WLPG neglects to correctly rotate images.

See the following pictures of WLPG with and without the FPVCP, note how the FPVCP version displays the CR2 thumbnails and displays the JPG files in the right rotation:
WLPGWLPG.FPVCodecPack

Since WLPG did not correct the rotation on portrait images, it was unable to recognize any faces in these images. So when Microsoft said they can’t reproduce the problem, they neglected to mention that the portrait images did not render correctly nor detect any pictures at all.

See the following pictures of WLPG with and without the FPVCP, note how the FPVCP version displays the JPG files in the right rotation, but WLPG uses the wrong image coordinates:
WLPG.Face.JPG.PortraitWLPG.FPVCodecPack.Face.JPG.Portrait

Interestingly enough, RAW images in both landscape and portrait detected the faces correctly:
WLPG.FPVCodecPack.Face.RAW.LandscapeWLPG.FPVCodecPack.Face.RAW.Portrait

I repeated the tests using Microsoft’s Camera Codec Pack (MCCP).

Notice how MCCP does not correct the rotation of JPG images in Explorer, nor does it render the CR2 file thumbnails in Explorer, vs. FPVCP that does both:
Explorer.5DMk2.MSFTCodecPackExplorer.5DMk2.FPVCodecPack

Notice how MCCP does not correct the rotation of JPG images in WLPG, nor does it render the CR2 file thumbnails in WLPG, vs. FPVCP that does both:
WLPG.MSFTCodecPackWLPG.FPVCodecPack

I will reply to the Windows Live Photo Gallery support thread with this information, and I will also open a support ticket with FastPictureViewer, let’s see what happens.

[Update : 28 December 2011]
I received an email from from Ardfry Imaging, informing me that they were shipping a x64 DNG codec before FPV existed, and they they are still offering a variety of codecs.

Advertisements

DxO Optics Pro and Adobe Lightroom on Vista x64

I bought my first SLR camera, a Canon EOS 20D, and my first wide angle lens, a Canon EF-S 10-22, several years ago. Amongst the many articles and reviews I read, online and in magazines, I came across a product called DxO Optics Pro, a product that would supposedly correct, amongst other things, lens distortion. This seemed like the perfect tool to help me with my wide angle pictures that looked very “fishy eyed”. And since DxO, then version 4, was receiving good reviews, I decided to trial the product. In those days the DxO licensing model was to pay per camera body and per lens module, not a cheap investment, but I really liked the results and so I purchased the product.
Not too long after I purchased DxO, they changed their licensing model, and instead of paying per module, they came up with a Standard and an Elite product, with the distinction being the camera bodies that are supported. The Elite version typically supports more high end cameras, I am sure there is no technical difference in the calibration techniques, this just seemed like a way to get more money from professionals that own high end cameras.
In the mean time I switched to using Adobe Lightroom version 1, and DxO fell in disuse. Every now and again I did come across an article or forum post with less than flattering reports of the poor product quality in DxO version 4.5, and very poor quality in version 5. The other thing I read, and confirmed with DxO tech support, is that DxO did not support x64 editions of Vista, and since I had switched to Vista Ultimate x64 edition, this would be a problem.
I switched from taking mostly JPG images to mostly RAW images, and manual correction in LR became very time consuming, so I was again interested in DxO. By now I was using LR 2.2 x64 edition, and by now DxO 5.3.2 did support Vista x64 editions. I purchased an upgrade for DxO, I bought the more expensive Elite edition, since I have dreams of one day owning a 5D Mark II.
In standalone mode DxO 5.3.2 worked great, but the integration with LR was not so great. The problem is that DxO does not support integration with the x64 edition of LR, when calling DxO from x64 LR nothing happens. The DxO installer is also not fully x64 compatible, it creates a start menu shortcut to the “DxO Download Manager v5”, but it sets the path to “C:\Program Files\DxO Labs\DxO Download Manager v5\DxODownloadManager.exe” instead of “C:\Program Files (x86)\DxO Labs\DxO Download Manager v5\DxODownloadManager.exe”, obviously a hardcoded path instead of dynamically discovered path.
Before I get to DxO, I think Adobe should have supported in-process DLL based plugins in LR, just like they do with Photoshop. For whatever reason this type of suggestion seems to make Adobe very uncomfortable, and they go to great lengths to defend their position, reads more like marketing firefighting, and say why the LUA based scripting and external process launching is so much better than plugins, at least when compared to Apple Aperture. I am a software developer, there is no way that anybody can convince me that a scripting language or launching an external process is better, or as good, as a native DLL based plugin with a rich integration API.
So how does DxO integrate with LR in light of the lacking plugin model? When you edit an image using an external editor, LR launches the editor with the image path specified on the commandline. LR provides three options for what file is passed to the external editor; the original file, a copy of the original file, or a copy of the LR processed version of the file. The problem is that when the file is a RAW file, LR only allows a copy of the LR processed version of the file to be passed, and this processed file is useless to a RAW editor.
DxO, being a RAW file processor, obviously needs access to the original RAW file, and they go about it in a clever, but cumbersome way. LR allows for configuration of the file type and file name of the new file it generates, and DxO exploits this file naming scheme to infer the original filename. E.g. the original RAW file is called Test.CR2, LR creates a copy of this file and names it Test-Edit.TIF, DxO infers the original filename is Test.CR2, opens Test.CR2 as the input file, and uses Test-Edit.TIF as the output file.
DxO can run in one of two modes; “plugin-in” mode (their spelling, not mine) or normal mode. When launched by LR, DxO runs in plugin mode. In plugin mode DxO only allows processing of the images passed in on the commandline, and the output processing automatically replaces those same input files. 
Since DxO does not have a way of explicitly launching in plugin mode, they infer the mode by interpreting how they are being launched. DxO looks at their parent process, and if the parent process is called Lightroom.exe, and the Lightroom.exe has version information that looks like LR, then DxO automatically enters plugin mode.
So why does DxO not work when launched from the x64 edition of LR? When launching DxO from x64 LR the DxO process starts and immediately terminates without showing any UI. It appears that DxO fails to determine information about the parent process when the parent process is a x64 process. I am speculating, but this is probably related to the technique that DxO is using to determine information about the parent process. Typically one would use the PS API or ToolHelp API’s, along with NtQueryInformationProcess(), but when mixing x86 and x64 process types, they do not provide consistent information, and extra handling is required by using IsWow64Process(). A more portable, but less reliable, alternative is to use WMI. DxO is not a native application, it is written in .NET, presumable C#, but it apparently still suffers from some x64 related problem.
So how do we solve the problem? You can install both the x64 and the x86 editions of LR on your x64 system, and when you want to work with DxO you can use the x86 edition of LR. This however is not very convenient. My solution was to create a x86 proxy process that looks like LR, is discoverable by DxO, and convinces DxO to enter plugin mode. LR is then configured to launch the proxy process instead of DxO, the proxy process launches DxO and passes the same commandline parameters as passed by LR to DxO, waits for DxO to terminate, and returns the exit code to LR.
Instead of inferring the mode of operation by looking at attributes of the parent process, it would have been much simpler, and it would have avoided the x64 problem, if DxO had a stub executable that always launched in plugin mode. And, it would have been even better if LR supported DLL based plugins with programmatic access to image information instead of inferring it. But, DxO would then have had to write a native x64 DLL, and if they can’t even write WoW64 compatible x86 code, I don’t know if that would have been a solution.
So how did I figure this out? 
At this point I have to remind you that I have absolutely no connection with DxO, nor insight into the coding of DxO, nor did I reverse engineer or disassemble DxO, my analysis and solution is purely based on observing LR and DxO behaviors by using Microsoft Process Monitor.
I compared the behavior of DxO on a x86 system with the behavior on a x64 system, and I noticed that on the x86 system the DxO process opens the LR executable file on disk. Looking at the call stack I noticed that DxO calls the.NET runtime calls the GetFileVersionInfoW() API. On the x64 system DxO never attempts to open the LR executable. I installed the x86 version of LR on the x64 system, and noticed a similar behavior to the x86 system. 
I created a C++ project in Visual Studio 2008, and wrote code to take the input commandline parameters and call the DxO executable with the same parameters as passed in by LR. I built a x86 binary, renamed the output file Lightroom.exe, and when I run “Lightroom.exe c:\test\test-edit.tif”, DxO would open in plugin mode, but would not load the original test.cr2 file. I added version information to the proxy project, and matched the version information contents to that of the real LR binary. Repeating the test, DxO now believes that it was called by LR and it opens in plugin mode and it finds the correct RAW file.
The source and binary file is available here
Please remember that I provide no warranty at all, I did minimal testing, so use at your own risk.
To install simply extract, rename DxOProxy.exe to Lightroom.exe, and change the LR configuration to point to the proxy Lightroom.exe instead of the DxO binary.
In closing, and in light of my previous post related to my trouble with the Google Email Uploader on x64; with all the available programming documentation, and free compilers, and free virtualization testing environments, why is it so difficult for some ISV’s to produce quality and compatible software?

Microsoft Expression Media / iView Media Pro to Adobe Photoshop Lightroom converter

This article was originally posted here.

Introduction:

Because of severe performance problems I experienced with both iView Media Pro and Microsoft Expression Media, I decided to switch to Adobe Lightroom.

Unfortunately Lightroom does not understand catalog sets nor does it understand people tags, so the transition is not that simple.

After I could not find a simple, or even complicated solution, I decided to write my own conversion application.

This application will convert hierarchical catalog sets into keywords, and will add people tags to keywords.

I considered directly converting to Lightroom’s native SQLite database format, but the reverse engineering effort did not seem justified.

Disclaimer:

I am providing this utility and the source code as is, no warranties are provided, use at your own risk.

Backup all your data, validate the conversion results, do not discard your backups.

Source:

I wrote the utility in C++ using Visual Studio 2005 SP1.

I tested on Vista Ultimate x86.

I tested with Microsoft Expression Media 1.0.8104.0, and iView Media Pro 3.1.3.42E6.

The code utilizes the COM API’s exposed by Expression Media and Media Pro.

Unfortunately neither application’s COM API’s work reliably, I can only hope that the quality of future versions will improve.

See the source code comments for problems I encountered.

Usage:

  1. Download the archive and extract the contents to your folder of choice.

    The archive contains the source, and two binaries, one for Expression Media
    and one for Media Pro.

    The binaries are compiled to use either the Expression Media or the Media Pro COM type libraries.

  2. Run Expression Media or Media Pro, and open the catalog you want to convert.

    Make sure that only one catalog is open.

    Make sure you have a backup of your catalog and of all your media files.

  3. Open a command prompt and change the directory to where you extracted the files.

    Run: ExpressionMediaExport.exe [Full path to the catalog file you opened in
    Expression Media or Media Pro enclosed in double quotes]

    E.g. [ExpressionMediaExport.exe “c:\media\catalog.ivc”]

    The utility will convert catalog sets to keywords, and will convert people to
    keywords.

    You can close the command prompt.

  4. The catalog sets will be converted so that each catalog node is a keyword and
    the keywords are assigned to all items in that catalog.

    E.g. if you have “Travel” and “Travel\Someplace”, the keywords will be
    “{Root}{Travel}”, and {Root}{Travel}{Someplace}”.

  5. People are added to keywords.

    E.g. if you have “John Doe” as a person tag, then “John Doe” will be added to
    keywords.

  6. Sync the updated information to the media files.

    In Expression Media or Media Pro [Edit][Select All], [Action Sync
    Annotations][Export annotations to original files].

  7. Import the images into Adobe Lightroom.

    Use the category keywords to create new collections.

    Select all images containing a category keyword e.g. “{Root}{Travel}”, create a
    new collection called “Travel”, and add all selected images to the collection, do
    the same for child collections.

Known Problems:

  • If you are using Expression Media, you first have to fix the COM registration
    that is not correctly set during installation.

    On Vista [Start][All Programs][Accessories][right click the Command Prompt icon
    and select Run as administrator].

    On XP [Start][Programs][Accessories][Command Prompt]

    Change directory to the Expression Media folder: [cd “C:\Program
    Files\Microsoft Expression\Media 1.0″]

    Register the Expression Media COM interfaces: [media.exe /regserver]

    You can close the command prompt.

  • If you encounter failures writing keywords to media items, try adding
    keywords to all items that fail using the main application.
  • If you get an error “Active catalog does not match requested catalog”, it means the catalog you specified on the commandline did not match the default catalog. The problem is that the Open() API always fails, and I have to use the Attach() API to connect the active catalog, and for integrity reasons I make sure the catalog on the commandline is the same as the active catalog that will be modified.
  • Complain to Microsoft that the COM API’s are not fully functional.

Download:

Version 1.0.1.1: ExpressionMediaExport.1.0.1.1.zip

  • Added people to keyword export.

Version 1.0.0.2: ExpressionMediaExport.1.0.0.2.zip

  • First release.

Links: