Last Thursday I decided I would do some photographic tests to see just what Fuji's LMO or Lens Modulation Optimizer does compared to not using it. I did some shots of the trunk of a paper bark tree around the neighborhood. This was probably not the best choice of test subject, since only a narrow band would be in the area of focus. I have plenty of brick wall's to shot but the proverbial joke about folks taking pictures of brick wall's instead of 'real' photographs is in my mind whenever I see one and am tempted to take some 'demo' shots.
So, I found the area of focus and created a table of 500 x 500 pixel extracts from each aperture with LMO turned on. To create the non-LMO comparison I decided I would use a raw converter since LMO is only applied to Fuji's in camera raw to jpeg engine. Of course the obvious solution, which I realized only last night, was to use the Fuji's in camera raw engine to develop the exact same way but with LMO off. That would be the KISS way to do it. And no, I do not mean the rock band KISS. So perhaps it was for the best that I got caught up in a protracted search for more information. Here's what happened and why this comparison is still unpublished.
First distraction was using the raw converter, I have Photo Ninja and that was my first choice. The color's are not the same as Fuji's jpegs and I can't really get them the same in post processing as I am a little color challenged. While I was doing the post processing I realized that I also have a deconvolution sharpening plugin from Focus Magic and I could use that and compare it to the Fuji LMO results. It has been claimed, but not by Fuji, that I can find, the LMO is a deconvolution operation. Fuji does claim that it works selectively between the edge's to the center of the image. Straight deconvolution, like used by Focus Magic, would effect the entire image equally.
So off I went to do some research and see if I could find more sophisticated deconvolution software to compare against the Fuji. It was then that I found some old code created by John Costella over a 2 year period and released in 2001. http://johncostella.com/unblur/ I then got distracted from there because he had developed this code to process the infamous Zapruder film of the JFK assassination. The results of this sharpening and reconstruction of the film frame by frame was claimed to prove that the film was significantly altered if not faked in it's entirety. It's not my intention to weigh in on that argument, but I did find it interesting and it in turn lead me back into the seemingly endless material on the assassination. I spent a few hours on that before I realized I dropping down the rabbit hole again.
Then preparations for Christmas began and I needed to take some holiday photo's and do other things.
So that's how that happens with me, I start out to do one thing, then a small amount of second guessing sends me to the internet to make sure I am not making a fool of myself, which seems nearly impossible to prove on the internet :) I feel like I am an internet James Burke with a new series of 'Connections' episodes. Before I know it, day's if not weeks have passed, I have pursued and dropped several other lines of research and have lost my zeal for posting anything at all. Then I have a break through as described above and am now thinking of going out with my tripod and using a brick wall.....
But wait a minute, have I actually examined the evidence I do have in hand? Not closely it turns out, so I spend some time with that and it looks like I can see differences in detail both between lmo on and off and between lmo and focus magic. Better yet, the differences are greater at higher apertures where diffraction blurring is most pronounced.
Tuesday, December 24, 2013
Wednesday, December 11, 2013
Generational change in photography
Two recent blog posts by photographic bloggers that I follow, Kirk Tuck here: http://visualsciencelab.blogspot.it/2013/10/the-graying-of-traditional-photography.html and David Taylor-Hughes here: http://soundimageplus.blogspot.co.uk/2013/12/unapologetically-old-school-continuing.html discuss the notion that generational change is changing photography (or at least the photography market).
I don't agree about the generation gap but then it's not black and white, 100% one cause or the other.
This discourse is much more complicated than any one of us will be able to explicate. It's complicated because several orthogonal issues are involved. One of those is our attitudes towards technology which is coupled with our attitudes towards change. Another issue is the nature of market driven change in technology dominated industries, of which photography is firmly planted. Finally there is the issue of ageism or age discrimination, is it the same as saying the old guard doesn't "get it"? I have a 30+ years of professional experience in IT and 30+ years of amateur experience in photography, which means I am a member of the cohort who is clinging on to the past in the face of tumultuous change.
Change and lack of it. A contradiction. Change is the one constant in IT, yet old stuff that works hangs around seemingly forever. The change comes with newly installed systems. I once did a survey of web servers on our intra-network and discovered that the release level of the software was much pretty much stuck at the level it was when installed, no matter how long the server had been running. Some people are reluctant to change expertise while others are eager to move on to something new. In my experience, neither one is associated with age except that it's widely assumed that the older you are the less you know about new stuff. That assumption is ageism. I think a lot of it has do with the individual, did you have to struggle to acquire expertise or understanding? If so, you are more likely to resist change. Do you like to be challenged or prefer a relaxed stability? This leads us to another issue:
Do you revel in the means and methods of photography? This is not orthogonal to having what I will call the artistic eye, but you can do one without the other, or both. Back in the film/chemistry days both Edward Weston and Ansel Adams did darkroom work with chemicals. But there was a vast gap between the 'geekiness' between the two. Adams was a technical nerd while Weston was not. Both had great artistic vision.
I spent of lot time 'calibrating' my photography workflow from exposure to printing, based on Adam's writings, it was very technical and right up my alley, but it did little to nothing to make me a better photographer. Then, as today, my best shots are opportunistic that just happen to work out. My planning for shots is rather sterile, technically good (or not :) but not artistic.
The old farts are missing the market, they don't get it. Established technology companies can loose their way not because they are run by a bunch of 50 yr old + executives. It's far too easy to say that the 50 year old cohort is stuck in the past. What they are really stuck in is capitalism. To be a successful large company means you meet your sales figures every quarter and to do that you ask your existing clients what they want from you. And they pretty much tell you that they want more of the same, incrementally improved and made cheaper. That great disruptive invention in the lab your team came up with, well, the sales projections show that it will not make a measurable impact on your sales goals, you have to let it slide for now. Maybe next year your clients will ask for it. The problem is, that by the time your clients ask for it, it's being provided by a start-up company whose sales are great for their size but inconsequential for yours until they start selling into your client base.
Not all large companies and senior executives behave this way, Steve Job's started his turn-around of Apple when he was 42 and kept it up until he was 55. Notably, what makes Applelarge today is not what the company did for most of it's life, i.e. make PC's.
To all the claims that photography workflow will be dominated by ease of transfer to social media, all I can say is that cell phones do the entire workflow quite well, or good enough for the majority, right now. That disruptive train has left the station and won't be coming back in until the next disruption. So where does that leave Nikon, Canon and Sony, not to leave out Fujifilm, Pentax, Panasonic, Olympus, Sigma and Ricoh? I am not sure. I don't think providing ease of publishing to social media will turn things around, it won't hurt but it won't replace the cell phone camera. They all are iterating the past, some with more panache than others. Are any of them trying to disrupt the market? Still not clear. Is micro four-thirds or Nikon/Sony CX or Pentax Q disruptive? Doesn't seem that way, it could be argued that all those products are just incremental improvements or variations on the same configuration of technologies. Could Lytro be disruptive? Could be, time will tell, but I doubt it because of how photographs are used.
Here is what I think drives photography disruption: How are the photographs used?
Professionally via prints or electronic distribution, i.e. fine art, advertising, event recording.
Personally via prints or electronic distribution, i.e 4x6 mass market printing and the internet. (maybe photo frames?)
I think cell phones handle most of the personal use. For those situations where they don't I see all kinds of people form all kinds of ages using DSLR's and mirrorless. I see the existing cameras handling the professional needs as well. Those needs will have change in a substantial way to drive disruption.
Disruption in the existing use means simpler and cheaper, it's hard for me to imagine anything else coming along other that what we already have or have seen. But then that's the nature of the future, largely unknowable.
Disruption could also be in a new use for photography. I can't tell you what that might be, once again, it's large unknowable until it's revealed. Maybe someone has seen a small niche that can't be served by the existing technology and a new kind of product is serving that market?
I don't agree about the generation gap but then it's not black and white, 100% one cause or the other.
This discourse is much more complicated than any one of us will be able to explicate. It's complicated because several orthogonal issues are involved. One of those is our attitudes towards technology which is coupled with our attitudes towards change. Another issue is the nature of market driven change in technology dominated industries, of which photography is firmly planted. Finally there is the issue of ageism or age discrimination, is it the same as saying the old guard doesn't "get it"? I have a 30+ years of professional experience in IT and 30+ years of amateur experience in photography, which means I am a member of the cohort who is clinging on to the past in the face of tumultuous change.
Change and lack of it. A contradiction. Change is the one constant in IT, yet old stuff that works hangs around seemingly forever. The change comes with newly installed systems. I once did a survey of web servers on our intra-network and discovered that the release level of the software was much pretty much stuck at the level it was when installed, no matter how long the server had been running. Some people are reluctant to change expertise while others are eager to move on to something new. In my experience, neither one is associated with age except that it's widely assumed that the older you are the less you know about new stuff. That assumption is ageism. I think a lot of it has do with the individual, did you have to struggle to acquire expertise or understanding? If so, you are more likely to resist change. Do you like to be challenged or prefer a relaxed stability? This leads us to another issue:
Do you revel in the means and methods of photography? This is not orthogonal to having what I will call the artistic eye, but you can do one without the other, or both. Back in the film/chemistry days both Edward Weston and Ansel Adams did darkroom work with chemicals. But there was a vast gap between the 'geekiness' between the two. Adams was a technical nerd while Weston was not. Both had great artistic vision.
I spent of lot time 'calibrating' my photography workflow from exposure to printing, based on Adam's writings, it was very technical and right up my alley, but it did little to nothing to make me a better photographer. Then, as today, my best shots are opportunistic that just happen to work out. My planning for shots is rather sterile, technically good (or not :) but not artistic.
The old farts are missing the market, they don't get it. Established technology companies can loose their way not because they are run by a bunch of 50 yr old + executives. It's far too easy to say that the 50 year old cohort is stuck in the past. What they are really stuck in is capitalism. To be a successful large company means you meet your sales figures every quarter and to do that you ask your existing clients what they want from you. And they pretty much tell you that they want more of the same, incrementally improved and made cheaper. That great disruptive invention in the lab your team came up with, well, the sales projections show that it will not make a measurable impact on your sales goals, you have to let it slide for now. Maybe next year your clients will ask for it. The problem is, that by the time your clients ask for it, it's being provided by a start-up company whose sales are great for their size but inconsequential for yours until they start selling into your client base.
Not all large companies and senior executives behave this way, Steve Job's started his turn-around of Apple when he was 42 and kept it up until he was 55. Notably, what makes Applelarge today is not what the company did for most of it's life, i.e. make PC's.
To all the claims that photography workflow will be dominated by ease of transfer to social media, all I can say is that cell phones do the entire workflow quite well, or good enough for the majority, right now. That disruptive train has left the station and won't be coming back in until the next disruption. So where does that leave Nikon, Canon and Sony, not to leave out Fujifilm, Pentax, Panasonic, Olympus, Sigma and Ricoh? I am not sure. I don't think providing ease of publishing to social media will turn things around, it won't hurt but it won't replace the cell phone camera. They all are iterating the past, some with more panache than others. Are any of them trying to disrupt the market? Still not clear. Is micro four-thirds or Nikon/Sony CX or Pentax Q disruptive? Doesn't seem that way, it could be argued that all those products are just incremental improvements or variations on the same configuration of technologies. Could Lytro be disruptive? Could be, time will tell, but I doubt it because of how photographs are used.
Here is what I think drives photography disruption: How are the photographs used?
Professionally via prints or electronic distribution, i.e. fine art, advertising, event recording.
Personally via prints or electronic distribution, i.e 4x6 mass market printing and the internet. (maybe photo frames?)
I think cell phones handle most of the personal use. For those situations where they don't I see all kinds of people form all kinds of ages using DSLR's and mirrorless. I see the existing cameras handling the professional needs as well. Those needs will have change in a substantial way to drive disruption.
Disruption in the existing use means simpler and cheaper, it's hard for me to imagine anything else coming along other that what we already have or have seen. But then that's the nature of the future, largely unknowable.
Disruption could also be in a new use for photography. I can't tell you what that might be, once again, it's large unknowable until it's revealed. Maybe someone has seen a small niche that can't be served by the existing technology and a new kind of product is serving that market?
Sunday, August 11, 2013
Old vs New
My intended followup to my last entry is postponed, pun intended. Looking at my history of photographs got me interested in checking out some 'legacy' lenses. This has been far more complicated than I thought. I did not want to use a static indoor test scene with resolution chart embedded because I wanted to see if this phenomena of 'rendering an image' would also impact results. Probably the most repeated tests, simply because I wanted the results to be different, were between a Zeiss Contax T* Distagon 28MM F2.8 made for the Contax SLR, and a Leica Panasonic Summilux 25mm F1.4 made for a micro four thirds camera. I used an Olympus E-PM2 since my EM-5 was stolen. This is an outdoor shoot, in late afternoon, bright sun from a tripod and with the PanaLeica set to F2.8. Here those shots are:
The first is from the PanaLeica and three things are apparent even at a small size, it's sharper, more detailed and better contrast. (These are OOC JPEG's with the same settings). These were manually focused and the focus point was the same in each one, a knot in the wood.
One could argue that the Zeiss should be stopped down as much as the Leica, but it won't change the results. The Zeiss get's better up to F5.6 but it's still not as good as the Leica at F2.8. One could also argue that since both these lens were made in Japan, they are not true Zeiss nor Leica, yet both companies allowed their names to be used singly and prominently branded Zeiss and Leica by Contax and Panasonic.
My explanation is two fold, but I will likely never test it out. First off, the Zeiss is made to cover twice as large an area, it could be used on a full frame camera so using it on the PM2 was never in it's design parameters, secondly the Leica is newer technology made for the camera it was used with. So I believe that it's off to eBay for the Zeiss. As much as the name and build lends some prestige the results surely do not.
If I am right in my explanations of the differences, we have an example of how engineering in the past cannot anticipate the needs of the future. But this is just one sample and it's possible that either this Zeiss lens or this Olympus camera are just not compatible with each other....I have another Zeiss Contax, the 50MM F1.7 and I can put that to the test, what I don't have is a comparable lens for the Olympus, just standard 40-150mm zoom... More later
Friday, July 12, 2013
What get stored is getting larger
My last entry tried to describe how historical data in the electronic age can get lost and degraded over time. I intimated that the advent of purely digital storage systems would counter that scenario due to several properties of digital storage.
- Digital storage is content format neutral.
- Copying from one generation to the next generation of storage systems is a one step process and the same user cost no matter how large the content.
- For the same or lower operational cost, the next generation has larger capacity.
- You (or someone) is willing to expend the on going operational cost.
I used to think that continuous and sometimes rapid improvement of technological would have to come to an end. I don't think that end will be in sight as long as we maintain a commercially viable economy. ( I might be able to explain that assumption in a future post). The real bug-a-boo is the assumption that someone is willing to expend the costs. Even with the costs declining it's still a cost and has to be measured against any other use or value that can be had from expending that cost on something else. Which is a convoluted way of saying that maybe I, or you, or someone will get bored or tired of keeping the content around. Or find something else to store of higher value in that space.
I believe that content format will also play a role in that lack of interest in continuing to maintain the content, this is really a degradation of value. And it plays out in decades and sheer quantity. If you come to believe that the content has little to no value, say for example, because the content was acquired in a way no longer considered viable, why would you expend any effort to maintain it?
I think that statement is true no matter what the content, whether it be digital medical records or digital photographs. The variability in format is part of the picture, but in a subtle way. Let's take digital photographs. In the beginning of my storage I used jpeg and slowly added raw when it became available. I still shoot jpeg for a variety of reasons. First the summary:
A flower sample from each, a same sized crop from what a print would look like viewed from the same distance. At the smaller size all the cameras are pretty good (minus the bad exposures in several of them). At the larger size, I have never printed this large, to my eye's things don't start looking good until 12 MP. Too bad I didn't have the foresight to photograph exactly the same thing with each camera over the last 12 years :)
Here is what I realize. With flower pictures or any scene that can be easily found again in the real world, the superiority of camera's 2 to 3 generations ahead is so great that I am unlikely to keep the earlier photo's if I have to make a choice. I am still copying them forward, simply because it has little to no cost, since between 2001 and 2013, my storage had increased into the Terabytes on the server side and 8 to 32 GB memory cards in the camera allow me to shoot 2 week vacations in both raw and jpeg.
One more thing, while one can argue that for my purposes of on screen viewing and at max 8 x 10 prints, 12MP is as good 16MP. I do not think the MP increase is over and there are several reasons for that. There are advantages in reducing noise and increasing apparent acuity but perhaps the most important is that it allows a true digital zoom. The example here is to compare the Canon XT image to the Olympus EM5 image, one could crop out a 8 MP or so image from the Olympus and achieve acceptable digital zoom. First quality digital zoom wont happen until we have 24 to 36 MP to play with. You can already see that advantage in the smallish Nokia Lumia phone cameras that use 48MP to reduce noise and allow digital zoom.
| Date | Camera | MegaPixels | JPEG sizes | Raw Sizes |
|---|---|---|---|---|
| June 15, 2001 | Casio QV3000 | 3 | 1 | N/A |
| June 12, 2004 | Panasonic FZ10 | 4 | 3-4 | N/A |
| March 26, 2005 | Canon Digital Rebel XT | 8 | 3-4 | 10 |
| Feb 6, 2008 | Olympus E510 | 10 | 7 | 9 |
| April 5, 2009 | Nikon D300 | 12 | 5-9 | 11-12 |
| April 22,2009 | Panasonic G1 | 12 | 3-5 | 14 |
| March 26, 2012 | Olympus E-M5 | 16 | 8-10 | 12-13 |
A flower sample from each, a same sized crop from what a print would look like viewed from the same distance. At the smaller size all the cameras are pretty good (minus the bad exposures in several of them). At the larger size, I have never printed this large, to my eye's things don't start looking good until 12 MP. Too bad I didn't have the foresight to photograph exactly the same thing with each camera over the last 12 years :)
|
Camera |
11 x 15 print |
4 x 7 print |
|---|---|---|
|
Casio QV3000 |
 |
 |
|
Panasonic FZ10
|
 |
 |
|
Canon Digital Rebel XT
|
 |
 |
|
Olympus E510
|
 |
 |
|
Nikon D300
|
 |
 |
|
Olympus EM5
|
 |
 |
|
|
|
|
Here is what I realize. With flower pictures or any scene that can be easily found again in the real world, the superiority of camera's 2 to 3 generations ahead is so great that I am unlikely to keep the earlier photo's if I have to make a choice. I am still copying them forward, simply because it has little to no cost, since between 2001 and 2013, my storage had increased into the Terabytes on the server side and 8 to 32 GB memory cards in the camera allow me to shoot 2 week vacations in both raw and jpeg.
One more thing, while one can argue that for my purposes of on screen viewing and at max 8 x 10 prints, 12MP is as good 16MP. I do not think the MP increase is over and there are several reasons for that. There are advantages in reducing noise and increasing apparent acuity but perhaps the most important is that it allows a true digital zoom. The example here is to compare the Canon XT image to the Olympus EM5 image, one could crop out a 8 MP or so image from the Olympus and achieve acceptable digital zoom. First quality digital zoom wont happen until we have 24 to 36 MP to play with. You can already see that advantage in the smallish Nokia Lumia phone cameras that use 48MP to reduce noise and allow digital zoom.
Tuesday, July 9, 2013
Yet another new start.
Yeah, I have heard that before. Blogging is something that happens inside my head more than on the keyboard. I blog when I am driving or walking the dog. When I sit down to type, my intentions change and drift off into irrelevance.
I retired from the University of Michigan Medical School last September and moved away from my hometown of some 40 years, Ann Arbor, Michigan. We moved to be nearer Kathie's family and that has been a blessing and good move. My love/hate relationship with technology continues into retirement, sometimes it seems as if I have not retired at all since I still build, run and fix computers. My perspective has changed.
I now take preserving my digital artifacts (mainly photo's) with a high degree of paranoia. I used to speculate and wonder how we, that is the IT departments of medical centers, were going to be able to keep over a 100 years of medical records. Think of it, computers are at most 65 years old and most digital originated medical records less than a decade. We have no precedent. The same thing is true for our personal digital lives, our photo's, documents, movies and music. In my relatively long career in computers, just a little over 40 years and my nearly 45 years with home entertainment electronics I have seen so many fundamental format changes take place that I loose track.
My conclusion has been that the only way forward is to continuously migrate digital records to new technology. It's important when doing this migration to not loose data and that includes resolution data as well. Let me recount my history of audio format's as a parable of what mostly likely is happening in the entire digital era.
Well, I open of course in the analog era. My first source of music was 33 1/3 rpm and 45 rpm plastic records. These were spirals of a continuous groove dug into the plastic and reproduced with diamond or ruby needles tracing the modulations of an audio signal in the plastic. This was in the sixties, by which time this technology, first deployed by Thomas Edison in 1877 was close to a hundred years old. Think of it, a hundred years of technological development had reached it's pinnacle of perfection. In an early replay of the format wars which came to characterize recording and playback systems, the records disc, initially inferior to the records cylinder, but cheaper to make and mass produce, had become standardized as the 78 rpm record by 1925.
There was no home recording technology until the reel to reel tape drive appeared, copyright and patent battles settled and Sony started mass producing them. I owned several of them, a Sony, a Tandberg and a Teac. At their best, they could reproduce the entire range of a 331/3 rpm vinyl record (LP's or Long Play). In a comprize of data integrity (as we would call it now) mainly a degradation in signal to noise and frequency response, I used slower and slower recording speeds since that extended the capacity of my tapes. (hey, does this sound familiar to anyone?) I started recording my LPs on first play to minimize the noise degradation that always seems to accompany the playback of LP's. Due to time and costs, only a small portion of my LP collection was ever transcribed, and of those none are with me today.
Into the 70's, portable access to music was coveted, We wanted to listen to our music in our car's and then everywhere we were. The first format I used for that was the 8 track tape cartridge. This was a continuous loop of tape ingeniously placed into a plastic box which was open on one end to expose the tape for reproduction purposes. Of course, this was a highly compromised format, but very popular for a while. IT fit easily into a car, but was still too big for true portability. Home recording was very limited and never really took off. Then came the cassette tape. This was a miniaturized reel to reel tape and very portable. Technological advances attempted to overcome the compromises of the format and home recording became easy and cheap. Hence the appearance of that new social phenomena, the mix tape. Yet another social (or anti-social as it were) development, the 'walkman' appeared. A tape player you could wear on your belt and listen to in private with headphones. I eventually abandoned reel to reel and started recording my LP's onto cassette (it was good enough) and making mix tapes. Only a small part of my small reel to reel collection was migrated onto cassette. My main music collection was still LP's and I could re-record them onto cassette.
Indeed, I started a massive project to transcribe all my LP's onto cassette. Then came the Compact Disc, the first mass produced digital recording formation. Within a decade the cassette was dead. I, along with everyone else, stopped buying LP's and started buying CD's. I still transcribed them onto cassette, mostly out of habit. One day, I realized I was no longer using any of my cassette's, I had a portable CD player and a CD player in my car. Cassette tape was dead, CD's were audibly superior, but nor very recordable. Almost none of either my LP nor my cassette collection made the transition into CD's. The music that did re-appear, was because I purchased a CD copy.
CD's are a weird combination of analog waveform and digital recording. Reasonably lossless. But the next portable format was all digital and to make it small enough to be mass marketed, it involved compromising with data integrity. This was the MP3 format. Today, my home recorded CD's have MP3's on them, enabled with new technology since the basic CD format was digital in the first place, one could replace the analog waveform recording with a digital recording. There are loss less digital formats and indeed I once again embarked on a migration project, this time trans-coding all my CD's into loss less digital FLAC (free lossless audio codec) and storing them on a home media server. It's been years underway but I still have not had the time or inclination to migrate the entire CD collection.
So let's recap this long story:
LP's to reel to reel tape - less than 10% of the LP collection.
reel to reel to cassette - only a few titles made it because I had the 'original' LP's.
LP's to cassette's - maybe 33% were recorded and some were re-purchased.
cassette to CD - zero percent, a total replacement technology.
LP to CD - zero percent, no point to it.
CD (replaced LP's as the 'original' collection and now very much larger)
CD to MP3 - only a small amount of 'mix tapes' (ha, an old term continues).
CD to FLAC - so far less than 50% has made it. Once again, time and effort come into play.
So, the history of this analog to digital transcoding is not very impressive as far as any statistic is concerned, Data is lost (LP's and CD's not transcribed) and Data integrity is lost (LP to analog tape and LP/CD to MP3).
That brings us to the server story and pretty much back to my current interest in digital storage technology.
My first server was basically just my home PC. My next server was an old home PC turned into a Linux server. Following that was a dedicated Linux server. And after my most recent move, I purchased a small, low powered, linux based media server from Iomega (then bought by EMC and now bought by Lenovo).
So, I have already had 3 server technology transitions. But because each one of those was simply a way to record a digital format, there was no loss (of the source format) and the entire collections were able to be transferred. The reason the entire collections were transferred is that it took one operation for the entire collection. Basically, Copy *.* and go to bed..... Digital offers a great advantage over analog in that entire digital formats can be preserved easily, as to whether they will be understandable in 100 years, that's another story. Photography is perhaps a better way to tell that story however.
What is my history with photography like? Strangley enough, it's very similar to audio. Analog formats ruled the day for about a 100 years and copy and preserving was an act of reproduction using the newer cameras and films. Then came digital. My digital collection is now close to 13 years old. The only universal storage format is another compromise with data integrity, the jpeg format (very similar to the MP3 format in result). The lossless formats exist, but each camera manufacturer uses their own. Since we have owned five different brands, I now have five different lossless formats; CR2 (Canon), NEF (Nikon), ORF (Olympus), ARW (Sony) and RW2(Panasonic). Each one of them has made the server transition intact. The question is whether knowledge of those formats will last for the next 100 years?
Yes, there is a universal, lossless digital image format from Adobe called DNG. But I don't use that yet and along with the software ability to understand digital formats, that's another blog entry,.
I retired from the University of Michigan Medical School last September and moved away from my hometown of some 40 years, Ann Arbor, Michigan. We moved to be nearer Kathie's family and that has been a blessing and good move. My love/hate relationship with technology continues into retirement, sometimes it seems as if I have not retired at all since I still build, run and fix computers. My perspective has changed.
I now take preserving my digital artifacts (mainly photo's) with a high degree of paranoia. I used to speculate and wonder how we, that is the IT departments of medical centers, were going to be able to keep over a 100 years of medical records. Think of it, computers are at most 65 years old and most digital originated medical records less than a decade. We have no precedent. The same thing is true for our personal digital lives, our photo's, documents, movies and music. In my relatively long career in computers, just a little over 40 years and my nearly 45 years with home entertainment electronics I have seen so many fundamental format changes take place that I loose track.
My conclusion has been that the only way forward is to continuously migrate digital records to new technology. It's important when doing this migration to not loose data and that includes resolution data as well. Let me recount my history of audio format's as a parable of what mostly likely is happening in the entire digital era.
Well, I open of course in the analog era. My first source of music was 33 1/3 rpm and 45 rpm plastic records. These were spirals of a continuous groove dug into the plastic and reproduced with diamond or ruby needles tracing the modulations of an audio signal in the plastic. This was in the sixties, by which time this technology, first deployed by Thomas Edison in 1877 was close to a hundred years old. Think of it, a hundred years of technological development had reached it's pinnacle of perfection. In an early replay of the format wars which came to characterize recording and playback systems, the records disc, initially inferior to the records cylinder, but cheaper to make and mass produce, had become standardized as the 78 rpm record by 1925.
There was no home recording technology until the reel to reel tape drive appeared, copyright and patent battles settled and Sony started mass producing them. I owned several of them, a Sony, a Tandberg and a Teac. At their best, they could reproduce the entire range of a 331/3 rpm vinyl record (LP's or Long Play). In a comprize of data integrity (as we would call it now) mainly a degradation in signal to noise and frequency response, I used slower and slower recording speeds since that extended the capacity of my tapes. (hey, does this sound familiar to anyone?) I started recording my LPs on first play to minimize the noise degradation that always seems to accompany the playback of LP's. Due to time and costs, only a small portion of my LP collection was ever transcribed, and of those none are with me today.
Into the 70's, portable access to music was coveted, We wanted to listen to our music in our car's and then everywhere we were. The first format I used for that was the 8 track tape cartridge. This was a continuous loop of tape ingeniously placed into a plastic box which was open on one end to expose the tape for reproduction purposes. Of course, this was a highly compromised format, but very popular for a while. IT fit easily into a car, but was still too big for true portability. Home recording was very limited and never really took off. Then came the cassette tape. This was a miniaturized reel to reel tape and very portable. Technological advances attempted to overcome the compromises of the format and home recording became easy and cheap. Hence the appearance of that new social phenomena, the mix tape. Yet another social (or anti-social as it were) development, the 'walkman' appeared. A tape player you could wear on your belt and listen to in private with headphones. I eventually abandoned reel to reel and started recording my LP's onto cassette (it was good enough) and making mix tapes. Only a small part of my small reel to reel collection was migrated onto cassette. My main music collection was still LP's and I could re-record them onto cassette.
Indeed, I started a massive project to transcribe all my LP's onto cassette. Then came the Compact Disc, the first mass produced digital recording formation. Within a decade the cassette was dead. I, along with everyone else, stopped buying LP's and started buying CD's. I still transcribed them onto cassette, mostly out of habit. One day, I realized I was no longer using any of my cassette's, I had a portable CD player and a CD player in my car. Cassette tape was dead, CD's were audibly superior, but nor very recordable. Almost none of either my LP nor my cassette collection made the transition into CD's. The music that did re-appear, was because I purchased a CD copy.
CD's are a weird combination of analog waveform and digital recording. Reasonably lossless. But the next portable format was all digital and to make it small enough to be mass marketed, it involved compromising with data integrity. This was the MP3 format. Today, my home recorded CD's have MP3's on them, enabled with new technology since the basic CD format was digital in the first place, one could replace the analog waveform recording with a digital recording. There are loss less digital formats and indeed I once again embarked on a migration project, this time trans-coding all my CD's into loss less digital FLAC (free lossless audio codec) and storing them on a home media server. It's been years underway but I still have not had the time or inclination to migrate the entire CD collection.
So let's recap this long story:
LP's to reel to reel tape - less than 10% of the LP collection.
reel to reel to cassette - only a few titles made it because I had the 'original' LP's.
LP's to cassette's - maybe 33% were recorded and some were re-purchased.
cassette to CD - zero percent, a total replacement technology.
LP to CD - zero percent, no point to it.
CD (replaced LP's as the 'original' collection and now very much larger)
CD to MP3 - only a small amount of 'mix tapes' (ha, an old term continues).
CD to FLAC - so far less than 50% has made it. Once again, time and effort come into play.
So, the history of this analog to digital transcoding is not very impressive as far as any statistic is concerned, Data is lost (LP's and CD's not transcribed) and Data integrity is lost (LP to analog tape and LP/CD to MP3).
That brings us to the server story and pretty much back to my current interest in digital storage technology.
My first server was basically just my home PC. My next server was an old home PC turned into a Linux server. Following that was a dedicated Linux server. And after my most recent move, I purchased a small, low powered, linux based media server from Iomega (then bought by EMC and now bought by Lenovo).
So, I have already had 3 server technology transitions. But because each one of those was simply a way to record a digital format, there was no loss (of the source format) and the entire collections were able to be transferred. The reason the entire collections were transferred is that it took one operation for the entire collection. Basically, Copy *.* and go to bed..... Digital offers a great advantage over analog in that entire digital formats can be preserved easily, as to whether they will be understandable in 100 years, that's another story. Photography is perhaps a better way to tell that story however.
What is my history with photography like? Strangley enough, it's very similar to audio. Analog formats ruled the day for about a 100 years and copy and preserving was an act of reproduction using the newer cameras and films. Then came digital. My digital collection is now close to 13 years old. The only universal storage format is another compromise with data integrity, the jpeg format (very similar to the MP3 format in result). The lossless formats exist, but each camera manufacturer uses their own. Since we have owned five different brands, I now have five different lossless formats; CR2 (Canon), NEF (Nikon), ORF (Olympus), ARW (Sony) and RW2(Panasonic). Each one of them has made the server transition intact. The question is whether knowledge of those formats will last for the next 100 years?
Yes, there is a universal, lossless digital image format from Adobe called DNG. But I don't use that yet and along with the software ability to understand digital formats, that's another blog entry,.
Subscribe to:
Posts (Atom)