Early in my career working in a Digitisation environment at some point I would have to become familiar with the process of handling images. Being attached to the Quality Assurance (QA) team early on in the job, it was made clear that I would end up close enough to those images and perform some kind of checks on them. The last thing I expected was orientation issues between operating systems.

The Poblem

The main issue was that the images were rotated to point east (90 degrees) when they should be pointing north at (0/360 degrees) when being uploaded to the OCR server. Thinking it was just a small bug, a quick and simple fix was to simply re-upload the images and call it a day but the issue persisted as we tried again and again to get those images in the correct orientation pointing north.

From what I was told, the images that were received from a third party were not in the correct readable orientation. I was told that one of the system admins had rotated them (images) on MacOs. Naturally we did the same thing to confirm it for ourselves, rotate the images on MacOS then attempt another upload. It gave the same result, images pointing east. We even tried rotating the images in Ubuntu's image viewer, same result with the images pointing east.

The Invesitgation

Digging deeper into the issue I discovered that the thumbnails were not reflecting what was shown when selecting and opening the images, prompting a deep investigation into the matter. 

First I checked the metadata of the image on Mac and found a lead as to what was going on. I was inexperienced with Python at the time and cautious of any software I downloaded as it was the company's laptop. My options were limited but I never gave up. Eventually I stumbled upon the mdls tool embedded in Mac and found a lead on the issue. kMDItem Orientation = 1 in the image's metadata which should mean the image appears upright in Ubuntu but it does not.

The original proposed theory is that the images stored/captured in the state of a 6 got automatically "rotated" to a state of a 1 on Mac only to then be displayed in the state of an 8 in Ubuntu. Why they were in a state of 6 I don't know and it's above me so who knows.

https://blog.praveen.science/crazy-image-exif-orientation-bug-or-feature/

The Switch Up

Next we tested the issue on Windows to see if the images were being automatically rotated but they were not and hand to be manually rotated to the correct orientation of north (0/360 degrees) and thought that had fixed the problem but switching back to Ubuntu gave the same result, thumbnail was oriented north but upon opening the image it was again point east.

We ran further experiments with both Mac and Windows systems. Mac proved effortless while Windows showed promise. I struck gold with Windows when I discovered that the thumbnails on windows were pointing east (90 degrees) for a split second before rotation north. That was the small yet noticeable breakthrough I needed to keep pursuing a fix. I had a stupid thought in that moment, pulling on my knowledge of math concepts and bearings, what if we rotate the images on Windows one less than we should and it worked. I rotated a single image to east, opened it on Ubuntu and it was pointing south. I found my breakthrough. I took that same image and rotated it west(270 degrees), clockwise and not counter-clockwise to maintain consistency, and when I got back to Ubuntu the image was finally pointing North. Further experiments proved that both systems were entangled and either was one rotation behind or one rotation ahead of the other.

The Fix

So problem solved right? Well that is only one image in a special case, what if you get 100 images. You can 100% rotate each image manually but no one in their right mind is going to do that. After this discovery a Python script was made to properly account for the image's metadata to reflect the assumed orientation needed for Ubuntu to read them upright. The images were rotated clockwise to point west (270 degrees) on MacOS which would then rotate them to point North (0/360 degrees). Why Mac despite the issues? Python was already installed there so we kept things nice and tidy

# The script opens each image, applies a 360 degree toration
# which chnages nothing visually but foreces Pillow to physically
# rewrite the piexl data to match the EXIF orientation tag
# and saves it back. The reulst is a clean file where the pixels and 
# metadata both say 1, which Ubuntu reads correctly


from PIL import Image
import os

path = "folder_path"
file_list = os.listdir(path)

for file in file_list:
    file_path = os.path.join(path, file)

    try:
        with Image.open(file_path) as image:
            rot = image.rotate(360, expand=True, fillcolor="white")
            rot.save(file_path)
            print("Image roateted and saved")
    except Exception as e:
        print(f"fail {file}:{e}")
print("done!")

The proposed theory was that the images that were rotated on Mac only reflected the preview/thumbnail being rotated and not the actual internal metadata of the image, effectively producing the results seen in Ubuntu. Mac was not applying a rotation but rather it was one rotation behind Ubuntu. The images appeared upright on Mac but because the rotation is applied to the thumbnail and not the metadata on Mac and not Ubuntu, Ubuntu only recognises the rotation encoded the thumbnail on Mac as the image rotation metadata does not appear until rotation. It then checks and corrects the rotation appearing to be one step ahead. 

The 6-1-8 Theory

The proposed theory is that the images were encoded with a 6, when they should be encoded with a 1. Ubuntu reads the metadata and finds a 6 which results in our image being horizontal. MAC on it automatically changes the 6 to 1, however, this is only visual and not properly encoded into the image's metadata. Both Ubuntu and Mac were reading the metadata correctly, however Mac was only updating the thumbnail rather than the pixel data, so when the images were passed to Ubuntu, it read the unchanged pixel data and displayed it sideways as a 6 rather than a 1. The metadata says 1 but the pixel data says 6 which produces the pivot to 8 in Ubuntu. To us humans it's wrong but to Ubuntu it's right. If the pixels are still encoded as a 6 but the display encodes them as 1 then the pixels are still stuck at a 6 and then when the rotation is applied, it rotates to 8 so that the pixels can be oriented to a 1.

While the experiment did help to build the script to rotate the images, we still do not know why Ubuntu rotated the images another 90 degrees.

What I learned

Computers are weird and if you think you know them you don't. In all seriousness, image metadata goes deeper than just the visual pixel we see. Its a dance of 1s and 0s, reds, greens and blues that all need to line up the correct way for us humans to make sense of it all.