Compression

[Davidhs]

Although I have used BBPro for a few years, I have not before worried too much about compression.
Now I am involved in a project to create a church record (part of a wider project run by NADFAS - Nat Assoc Dramatic & Fine Art Societies). The number and size of pictures brings the compression issue into higher focus.
NADFAS specify TIFF, but do not specify what sub-format of TIFF or any compression parameters.

What compression parameters does BBPro use when converting RAW to TIFF?

The BBPro manual says it "... does not output compressed TIFF files because LZW compression is covered by a patent owned by Unisys." I had thought, and have often read on apparently authoritative web sites, that that patent had now expired. So what is the position?

I have reservations about the NADFAS choice of TIFF. It is an old and rather loose specification and in many ways an unsatisfactory choice for archiving images starting now in 2007. PNG seems a much better choice. And it is supposed to have better compression. But when I run tests, the PNG file sizes out of BBPro are not that much smaller than TIFF.

File sizes: Raw from Nikon D80: 10.8 MB (mixed shot of garden shrubs, walls and cloudy sky)
Tiff 8 bit: 29.4 MB
Tiff 16 bit: 58.8 MB
PNG 8 bit: 17.4 MB
PNG 16 bit: 52.1 MB

I am a little disappointed.
Or is this what i should expect?

David H-S
North Yorkshire

[DavidB]

David,

There is an interesting discussion to be had about compression and file sizes, and I hope others more qualified than I will help you there. My reason for replying is to wonder whether your options are quite as you described them.

As a member both of a NADFAS group and of a church which was documented under the scheme some years ago, I have some understanding of your requirements. I also know that NADFAS is a deeply conservative organisation which has yet to come to terms with the digital age, so it is interesting to speculate how informed their choice of the TIFF format actually is.

This may well sound like heresy, but high quality JPEG is the obvious candidate for distributing and archiving the final product. The principal disadvantage of JPEG - its lossiness - is not really an issue with first generation copies that have not been over-compressed. So if all the processing stages except this final one are done with RAW, DNG, TIFF or PSD files (thereby keeping processing loss to a minimum), you are unlikely to see any difference if you save the final version as good-quality JPEGs.

The advantages of JPEG are

it is so widely used that it is more likely to be future-proof than any other format,
it is probably the best format in which to record metadata (which, I imagine, would be of some importance to your project),
it offers probably the best quality for a given file size.


For backup purposes, it would be sensible to keep the original RAW files (plus any associated files such as XMP sidecars) and working copies of any Photoshop (or whatever) files. If you plan to retain no more than, say, 1000 images, I should have thought that the whole project would fit on to a DVD.

I hope that these thoughts are useful,

David

[Davidhs]

Firstly, Thank you to DavidB:
I quite agree that sensibly used JPEG has many virtues. And indeed with JPEG 2000 there is a lossless option.
NADFAS finally published updated guidance to recorders last November. I only received a copy last week and am drafting a note to them saying please reconsider on TIFF. It maybe OK for scanned images but not for photographs.

Recognising that one of their problems is that they have no control over the photographers who do the work, I have not taken issue on JPG; rather I have proposed PNG because it is well defined standard, because there is less scope for people to mess up the images by ill considered compression, it provides lossless compression and is supposed to compress well, is well implemented, etc.
They have specified CD storage, using "gold CDs specified by NMR" (I have not yet researched that), which means that file size is more of an issue. I have not researched whether DVD is yet considered to have an adequate shelf life.

I am relieved to hear you expressing similar reservations to mine. I intend to take the matter further with NADFAS and am glad of the support.

Secondly,
I have done a little more investigation of file sizes and am still puzzled.
I used the same picture as before and always ran each test starting from the same original RAW NEF.

File sizes: Raw from Nikon D80: 10.8 MB (mixed shot of garden shrubs, walls and cloudy sky)
Saved with Breezebrowser 1.7
Tiff 8 bit: 29.4 MB
Tiff 16 bit: 58.8 MB
PNG 8 bit: 17.4 MB
PNG 16 bit: 52.1 MB
I then tried with two image manipulation utilities that I have used for some years.
Irfan View, the wonderful free utility with many plugins by Irfan Skiljan: Still converting from the same NEF:
Tiff LZW: 33 MB - bit depth unspecified in the options.
Tiff packbits: 29.5 MB
PNG default compression 17.3 MB
PNG highest compression 15 MB - this uses an external plugin to achieve max compression

I also tried using PMView 3.50: shareware (originally many years ago an OS/2 utility). This cannot process NEF so i started with the 58 MB TIFF.
PNG: 18.4 MB - bit level made no difference

The Nikon creates 12 bit/channel images. How IRFANView handles that is not clear.

I also have the feeling that these two, IrfanView in particular, run the conversions much faster than BB.

I should appreciate guidance, please, on what to expect. And if I have some setting wrong, please

I have used, enjoyed and benefited from BreezeBrowser over several years and hate to raise concerns.
The others don't have facilities comparable to BB's facilities.

David H-S

[Chris Breeze]

BBPro writes TIFF files uncompressed and PNG files using lossless compression.
Most apps that write compressed TIFF files use LZW compression which can result in the compressed file being larger than the equivalent uncompressed file. This isn't quite a strange as it seems because photos, particularly 16-bit images, contain noise which is essentially random data which doesn't compress well.

A compressed NEF file from a D80 contains the the raw sensor data from the 10 million red, green and blue pixels at 12-bits i.e. about 15Mb uncompressed and around 10Mb compressed. When the raw file is converted to a 16-bit image you end up with 10 million RGB pixels at 16-bits which is about 60Mb (10 million x 3 x 2). This is basically the size of the uncompressed TIFF BBPro saves - the reason for the slight discrepency is camera manufacturers define a mega-pixel as 1,000,000 pixels whereas a megabyte is 1,048,576 bytes (1024 x 1024).

There is probably little to be gained by using 16-bit TIFFs if they have been edited and optimised. For archiving purposes it might make sense to also store the original NEF file or perhaps convert it to DNG or even archive both.

[Davidhs]

Sorry to resurrect a thread I started in July. My attention has been elsewhere, but the problem is still with me.

NADFAS, following guidance from the National Archives, now require TIFF. But they don't specify any more than that. The people with whom I have exchanged emails do not seem to appreciate there is more to it than that.

Debating TIFF versus alternatives that seems a bit like the Betamax vs VHS debates of the '70s. The world of archivists has chosen TIFF, so that's it.

BTW, out of interest, does anyone know what other countries' archivists require?

I accept Chris' point that there would be no value in compressing 16 bit TIFF.
But what about compressing 8 bit TIFF? This is not provided by BB for understandable historical reasons, but the Unisys patent issue has gone away.

Chris: Is there a case for providing LZW for 8 bit TIFF in BB?
Or alternatively would you consider supporting saving as JPG within a TIFF wrapper, an option that the TIFF standard allows!

Or do you take the view that practical people, who don't have to deal with NADFAS or the National Archives people, would never use it, preferring 100% JPG or JP2? I appreciate you must focus on the needs of the majority.

Thanks, David H-S

[Chris Breeze]

It's really a question of priorities and the development time involved. There have been very few requests for the ability to write LZW TIFFs and so I can't really give this a high priority.
I don't think adding support for JPEG compression for TIFFs would be a good idea because it is bound to cause problems with older apps which can read ordinary TIFFs but not JPEG compressed ones.

[DavidB]

David

XnView, which (I think) is the nearest you can get to BB Pro in free software, has an Export function in which you can convert multiple files to TIFFs, with no compression or any of the available compression options. XnView is a bit wacky, and it is appallingly documented, but it is a useful adjunct to BB Pro for some purposes.

I think that we have to assume that NADFAS have no idea at all about digital archiving. I was recently given a copy of their Church Recorders' Handbook, which has a great deal of information about using transparencies but refers you to the NADFAS website for guidance on digital recording. That information is held behind logons which are not freely available. NADFAS have clearly put the digital age into their 'too difficult' box. I don't know about your local group, but mine uses only transparencies at its lectures.

For a while, TIFF was certainly the de facto standard for images submitted to agencies and image banks. I think that some agencies now accept large, high quality JPEGs. This is likely to be a changing situation as the technology develops, though I am sure that the 'librarian' culture will do all it can to hold it back.

David

[Davidhs]

David,

Thank you for mentioning XnView. I have briefly tried it and will explore it further.

I think it is not really fair for us to blame NADFAS for the problem; they take guidance via English Heritage who in turn are advised by, I think, the National Archives. I have yet to locate a UK based image file formats expert who provides guidance to the latter.

If you are interested - offline from this forum - I could let you have PDF copies of the NADFAS photography papers and a bit more background to my dealings and email exchanges with some of the players.

I can be reached via djhs and the node is bcs org uk.

Yes, TIFF has been a de facto standard, but the reasons that was sensible in the 90s really don't hold up in 2007 with full colour digital photography. What was good for faxes, bi-level images or files with just a few constant colours is not good for today's needs. Chris's explanation of how inappropriate LZW is for colour photographs is a strong illustration of that.

I have by email tried to explain that to NADFAS / English Heritage, but so far I have not found anyone who is in a position technically to discuss any alternative.

I recently found a report from the Harvard libraries about their attempts to convert old image repositories to JP2:
http://www.dlib.org/dlib/december05/abrams/12abrams.html

David

[DavidB]

David

Many thanks for an interesting post.


I think it is not really fair for us to blame NADFAS for the problem; they take guidance via English Heritage who in turn are advised by, I think, the National Archives. I have yet to locate a UK based image file formats expert who provides guidance to the latter.
I don't do blame, but I was being critical of NADFAS, not for seeking and taking advice, but (1) for showing no sign that they understood the background to that advice and therefore the implications of their decision and (2) for seemingly failing to understand the technological (and therefore cultural) environment in which, as self-designated archivists, they are now operating.


If you are interested - offline from this forum - I could let you have PDF copies of the NADFAS photography papers and a bit more background to my dealings and email exchanges with some of the players.
Thank you: I'd be very interested to see them. I've given you my email address in a private message.


Yes, TIFF has been a de facto standard, but the reasons that was sensible in the 90s really don't hold up in 2007 with full colour digital photography. What was good for faxes, bi-level images or files with just a few constant colours is not good for today's needs. Chris's explanation of how inappropriate LZW is for colour photographs is a strong illustration of that.

I have by email tried to explain that to NADFAS / English Heritage, but so far I have not found anyone who is in a position technically to discuss any alternative.
Fully agree, and this reinforces the criticisms I have made.


I recently found a report from the Harvard libraries about their attempts to convert old image repositories to JP2:
http://www.dlib.org/dlib/december05/abrams/12abrams.html
I am really grateful for your link to this fascinating document. To see people in the archiving business taking a proactive and intelligent approach to the business of data management in image archiving (something that concerns every BB Pro user) is a real eye-opener. It is particularly interesting to see how metadata seems to have created as many, if not more, problems for the Harvard team as actual image data. Plenty of BB Pro users will relate to that finding as well!

[Davidhs]

David, you wrote:

I've given you my email address in a private message.
Nothing received so far at 1350 Wed ... djhs on the node bcs dot org dot uk - Br Computer Society.

[DavidB]

David, you wrote:

Nothing received so far at 1350 Wed ... djhs on the node bcs dot org dot uk - Br Computer Society.

Hopefully, you have my message now.

David