Comparing photo-services • Part 2

This is a continuation from my previous post. I’ve had some photos printed at four photo-printing facilities and made a small experiment to see which was best. The results were very surprising!

This project was supposed to be a small joke, but instead blew up a bit. I decided to take that ‘blowing up’ to the next level and write the research up as a real science paper 🔬

Sorry for the seriousness, but it was fun to write this way again! 🤓

Methodology 🔗

Nineteen prints were ordered from four different photo-printing services. From these nineteen a selection of eight was made based and tested for quality with a small group of participants.

When ordering the photos, Albelli and Profotonet had the option of optimizing the photos. Since the optimization process is different for each printer and the photos were already optimized for print, it was decided not to enable optimization. Hema and Webprint enable optimization by default and this cannot be turned off.

All photos were printed on matte paper (instead of glossy). Profotonet uses a different type of paper from the others (Fujicolor Professional Paper versus Fujicolor Crystal Archive Paper Supreme), which seems to be a bit less shiny.

Lastly, the printing services differ in the way non-standard photo sizes are handled. Albelli and Hema let you choose how to cut the photo off, Profotonet and Webprint print the whole photo (where Profotonet chooses to size ‘down’ and Webprint ‘up’).

Experiment setup 🔗

There were six participants, of which the first was a test-run that was discarded (n=5). All experiments were performed in an informal setting.

All participants looked at eight different photos. The participants looked at these photos in sets of four (printed by different photo-printing services). The sets were randomized for each trial. The participants were asked to make four ‘piles’, ranging from ‘I like this photo-quality the most’ to ’this is the worst of the four’. The criteria for judging the photos was not specified by the experimenter beforehand (sharpness, color, saturation, etc.), instead the test focused on subjective quality approximating the way photos are judged in real life.

The participants were not told which printing service belonged to which photo until the end of the test.

Measurement 🔗

Three different methods of judgement were used. During the experiment, photos were ordered in four columns ranging from best to worst by the participants. For the explanation below the amount of photos in each column was summed up and labeled C₁ ... C₄.

1. Score per quality 🔗

The highest quality photos got 4 ‘points’, the worst only 1 ‘point’ (in between 2 and 3 points respectively). The score for each photo-printing service was thus: C₁*4 + C₂*3 + C₃*2 + C₄*1.

2. Best - worst 🔗

Only two columns were used for this measurement. The amount of photos in the ‘worst quality’ column was subtracted from the amount of photos in the ‘best quality’ column (C₁ - C₄)

3. 2xBest - 2xWorst 🔗

All columns were used: the best and worst column counted twice, the others once. The ‘worst quality’ columns were used as penalty for the ‘best-quality’ columns (C₁*2 + C₂* - C₃ - C₄*2).

4. Average score 🔗

For each print a value was attached ({10,7,4,1} – ranging best to worst) based on the column the print ended up in for each participant. From this, the average and standard-deviation were calculated.

Results 🔗

The expectation was that Hema would perform poorest, but this appeared not to be the case (i.e. the null-hypothesis is "Hema performs very well" was not completely rejected).

Ik weet natuurlijk niet hoe het in het echt was

Of course, I don’t know how it was in real life

Deze is minder scherp, maar wel mooi helder

It’s less sharp, but it is nicely clear

(Over Profotonet): Oh jee, deze is wel heel erg donker

(About Profotonet): Oh boy, this one is very dark

(Over Hema): die is wat minder scherp

(About Hema): this one is a bit less sharp

Consistently the best scoring printing service is Webprint. The worst scoring is Profotonet. This was completely unexpected, since Profotonet is supposed to be the best printing service in the Netherlands. Albelli and Hema scored equally well.

The scoring-methods are described above. Higher is better for all methods

Subjectively, Profotonet prints the sharpest and Hema least sharp. In some Hema-prints a moire-pattern can be found (probably due to additional JPEG-compression somewhere in the ordering-process). Webprint has an aggressive optimization process where colors are enhanced and sharpening applied, but this was clearly very much appreciated by the participants. For both Albelli and Profotonet the ‘optimize photos’ was turned off, and many subjects noted that these photos were too dark (especially for Profotonet). Prints from Profotonet also seemed to have less contrast.

Some example-photos 🔗

In this first photo the optimization of Webprint is very obvious. Profotonet is sharper and more faithful to the original.

Again the main difference comes from the optimization process. Less clear is the fact that the Hema print has a lot less detail.

Conclusion 🔗

In the future we will order photos from Webprint, they are cheap and offer the best all-round quality. The way they print ‘differently-sized’ photos is better than the others.

Discussion 🔗

The amount of participants was limited and this test does not have any statistical significance as a result. Future work should account for this. The lighting conditions were different for each participant. Obviously this should be kept consistent in future experiments.

The option for optimization was turned off for Albelli and Profotonet. Especially the latter printing service was at a severe disadvantage because of this (they printed sharpest on the most expensive paper, but the participants all found the photos too dark). Apparently the experimenter is not as good at optimizing photos on his computer as he thought… For new experiments these settings should be turned on.

Due to time and budgetary constraints, the research in photo-quality is now over 🤓

the raw results can be found in this Apple-Numbers file