What get stored is getting larger

My last entry tried to describe how historical data in the electronic age can get lost and degraded over time.  I intimated that the advent of purely digital storage systems would counter that scenario due to several properties of digital storage.

• Digital storage is content format neutral.
• Copying from one generation to the next generation of storage systems is a one step process and the same user cost no matter how large the content.

I realize that there are some unspoken assumptions behind this claim as well.

• For the same or lower operational cost, the next generation has larger capacity.
• You (or someone) is willing to expend the on going operational cost.

I used to think that continuous and sometimes rapid improvement of technological  would have to come to an end.  I don't think that end will be in sight as long as we maintain a commercially viable  economy. ( I might be able to explain that assumption in a future post).  The real bug-a-boo is the assumption that someone is willing to expend the costs.  Even with the costs declining it's still a cost and has to be measured against any other use or value that can be had from expending that cost on something else.  Which is a convoluted way of saying that maybe I, or you, or someone will get bored or tired of keeping the content around.  Or find something else to store of higher value in that space.

I believe that content format will also play a role in that lack of interest in continuing to maintain the content, this is really a degradation of value.  And it plays out in decades and sheer quantity.  If you come to believe that the content has little to no value, say for example, because the content was acquired in a way no longer considered viable, why would you expend any effort to maintain it?

I think that statement is true no matter what the content, whether it be digital medical records or digital photographs.  The variability in format is part of the picture, but in a subtle way.  Let's take digital photographs.  In the beginning of my storage I used jpeg and slowly added raw when it became available.  I still shoot jpeg for a variety of reasons.  First the summary:

Date	Camera	MegaPixels	JPEG sizes	Raw Sizes
June 15, 2001	Casio QV3000	3	1	N/A
June 12, 2004	Panasonic FZ10	4	3-4	N/A
March 26, 2005	Canon Digital Rebel XT	8	3-4	10
Feb 6, 2008	Olympus E510	10	7	9
April 5, 2009	Nikon D300	12	5-9	11-12
April 22,2009	Panasonic G1	12	3-5	14
March 26, 2012	Olympus E-M5	16	8-10	12-13

A flower sample from each, a same sized crop from what a print would look like viewed from the same distance.  At the smaller size all the cameras are pretty good (minus the bad exposures in several of them).  At the larger size, I have never printed this large, to my eye's things don't start looking good until 12 MP.  Too bad I didn't have the foresight to photograph exactly the same thing with each camera over the last 12 years :)

Camera	11 x 15 print	4 x 7 print
Casio QV3000
Panasonic FZ10
Canon Digital Rebel XT
Olympus E510
Nikon D300
Olympus EM5

Here is what I realize.  With flower pictures or any scene that can be easily found again in the real world, the superiority of camera's 2 to 3 generations ahead is so great that I am unlikely to keep the earlier photo's if I have to make a choice.  I am still copying them forward, simply because it has little to no cost, since between 2001 and 2013, my storage had increased into the Terabytes on the server side and 8 to 32 GB memory cards in the camera allow me to shoot 2 week vacations in both raw and jpeg.

One more thing, while one can argue that for my purposes of on screen viewing and at max 8 x 10 prints, 12MP is as good 16MP.  I do not think the MP increase is over and there are several reasons for that.  There are advantages in reducing noise and increasing apparent acuity but perhaps the most important is that it allows a true digital zoom.  The example here is to compare the Canon XT image to the Olympus EM5 image, one could crop out a 8 MP or so image from the Olympus and achieve acceptable digital zoom.  First quality digital zoom wont happen until we have 24 to 36 MP to play with.  You can already see that advantage in the smallish Nokia Lumia  phone cameras that use 48MP to reduce noise and allow digital zoom.

Yet another new start.

Yeah, I have heard that before.  Blogging is something that happens inside my head more than on the keyboard.  I blog when I am driving or walking the dog.  When I sit down to type, my intentions change and drift off into irrelevance.

I retired from the University of Michigan Medical School last September and moved away from my hometown of some 40 years, Ann Arbor, Michigan.  We moved to be nearer Kathie's family and that has been a blessing and good move.  My love/hate relationship with technology continues into retirement, sometimes it seems as if I have not retired at all since I still build, run and fix computers.  My perspective has changed.

I now take preserving my digital artifacts (mainly photo's) with a high degree of paranoia.  I used to speculate and wonder how we, that is the IT departments of medical centers, were going to be able to keep over a 100 years of medical records.  Think of it, computers are at most 65 years old and most digital originated medical records less than a decade.  We have no precedent. The same thing is true for our personal digital lives, our photo's, documents, movies and music.  In my relatively long career in computers, just a little over 40 years and my nearly 45 years with home entertainment electronics I have seen so many fundamental format changes take place that I loose track.

My conclusion has been that the only way forward is to continuously migrate digital records to new technology.  It's important when doing this migration to not loose data and that includes resolution data as well.  Let me recount my history of audio format's as a parable of what mostly likely is happening in the entire digital era.

Well, I open of course in the analog era.  My first source of music was 33 1/3 rpm and 45 rpm plastic records.  These were spirals of a continuous groove dug into the plastic and reproduced with diamond or ruby needles tracing the modulations of an audio signal in the plastic.  This was in the sixties, by which time this technology, first deployed by Thomas Edison in 1877 was close to a hundred years old.  Think of it, a hundred years of technological development had reached it's pinnacle of perfection.  In an early replay of the format wars which came to characterize recording and playback systems, the records disc, initially inferior to the records cylinder, but cheaper to make and mass produce, had become standardized as the 78 rpm record by 1925.

There was no home recording technology until the reel to reel tape drive appeared, copyright and patent battles settled and Sony started mass producing them.  I owned several of them, a Sony, a Tandberg and a Teac.  At their best, they could reproduce the entire range of a 331/3 rpm vinyl record (LP's or Long Play). In a comprize of data integrity (as we would call it now) mainly a degradation in signal to noise and frequency response, I used slower and slower recording speeds since that extended the capacity of my tapes.  (hey, does this sound familiar to anyone?) I started recording my LPs on first play to minimize the noise degradation that always seems to accompany the playback of LP's.  Due to time and costs, only a small portion of my LP collection was ever transcribed, and of those none are with me today.

Into the 70's, portable access to music was coveted,  We wanted to listen to our music in our car's and then everywhere we were.  The first format I used for that was the 8 track tape cartridge.  This was a continuous loop of tape ingeniously placed into a plastic box which was open on one end to expose the tape for reproduction purposes.  Of course, this was a highly compromised format, but very popular for a while.  IT fit easily into a car, but was still too big for true portability.  Home recording was very limited and never really took off.  Then came the cassette tape.  This was a miniaturized reel to reel tape and very portable.  Technological advances attempted to overcome the compromises of the format and home recording became easy and cheap.  Hence the appearance of that new social phenomena, the mix tape. Yet another social (or anti-social as it were) development, the 'walkman' appeared.  A tape player you could wear on your belt and listen to in private with headphones.    I eventually abandoned reel to reel and started recording my LP's onto cassette (it was good enough) and making mix tapes.  Only a small part of my small reel to reel collection was migrated onto cassette.  My main music collection was still LP's and I could re-record them onto cassette.

Indeed, I started a massive project to transcribe all my LP's onto cassette.  Then came the Compact Disc, the first mass produced digital recording formation.  Within a decade the cassette was dead.  I, along with everyone else, stopped buying LP's and started buying CD's.  I still transcribed them onto cassette, mostly out of habit.  One day, I realized I was no longer using any of my cassette's, I had a portable CD player and a CD player in my car.  Cassette tape was dead, CD's were audibly superior, but nor very recordable.  Almost none of either my LP nor my cassette collection made the transition into CD's.  The music that did re-appear, was because I purchased a CD copy.

CD's are a weird combination of analog waveform and digital recording.  Reasonably lossless.  But the next portable format was all digital and to make it small enough to be mass marketed, it involved compromising with data integrity.  This was the MP3 format.  Today, my home recorded CD's have MP3's on them, enabled with new technology since the basic CD format was digital in the first place, one could replace the analog waveform recording with a digital recording.  There are loss less digital formats and indeed I once again embarked on a migration project, this time trans-coding all my CD's into loss less digital FLAC (free lossless audio codec) and storing them on a home media server.  It's been years underway but I still have not had the time or inclination to migrate the entire CD collection.

So let's recap this long story:

LP's to reel to reel tape - less than 10% of the LP collection.
reel to reel to cassette - only a few titles made it because I had the 'original' LP's.
LP's to cassette's - maybe 33% were recorded and some were re-purchased.
cassette to CD - zero percent, a total replacement technology.
LP to CD - zero percent, no  point to it.
CD (replaced LP's as the 'original' collection and now very much larger)
CD to MP3 - only a small amount of 'mix tapes' (ha, an old term continues).
CD to FLAC - so far less than 50% has made it.  Once again, time and effort come into play.

So, the history of this analog to digital transcoding is not very impressive as far as any statistic is concerned,  Data is lost (LP's and CD's not transcribed) and Data integrity is lost (LP to analog tape and LP/CD to MP3).

That brings us to the server story and pretty much back to my current interest in digital  storage technology.
My first server was basically just my home PC.  My next server was an old home PC turned into a Linux server.  Following that was a dedicated Linux server.  And after my most recent move, I purchased a small, low powered, linux based media server from Iomega (then bought by EMC and now bought by Lenovo).

So, I have already had 3 server technology transitions.  But because each one of those was simply a way to record a digital format, there was no loss (of the source format)  and the entire collections were able to be transferred.  The reason the entire collections were transferred is that it took one operation for the entire collection.  Basically, Copy *.*  and go to bed.....  Digital offers a great advantage over analog in that entire digital formats can be preserved easily, as to whether they will be understandable in 100 years, that's another story.  Photography is perhaps a better way to tell that story however.

What is my history with photography like?  Strangley enough, it's very similar to audio.  Analog formats ruled the day for about a 100 years and copy and preserving was an act of reproduction using the newer cameras and films.  Then came digital.  My digital collection is now close to 13 years old.  The only universal storage format is another compromise with data integrity, the jpeg format (very similar to the MP3 format in result).  The lossless formats exist, but each camera manufacturer uses their own.  Since we have owned five different brands, I now have five different lossless formats; CR2 (Canon), NEF (Nikon), ORF (Olympus), ARW (Sony) and RW2(Panasonic).  Each one of them has made the server transition intact. The question is whether knowledge of those formats will last for the next 100 years?

Yes, there is a universal, lossless digital image format from Adobe called DNG.  But I don't use that yet and along with the software ability to understand digital formats, that's another blog entry,.

A suprise Netflix find

L

Cthulhu 

Watched the "Call of Cthulhu", a sort (46 minutes) movie on Netflix.  I sort of expected a low budget horror film similar to all the other Lovecraft inspired movies I have seen, but this was quite different.  

http://www.cthulhulives.org/cocmovie/

II 

As you can see this was produced by the HPLovecraft Historical Society.  I highly recommend following the link to get to their site.  This will tell you all about the movie itself.  This blog is about my experience of the movie, it is not a review.

If you are a Lovercraft fan or got to this site by searching for Cthulhu you might be disappointed because you already know about this movie, the story it's based on and the HPLHS.  I can not add anything more to that mileau. 

Yes, this is a 46 minute black and white silent movie made in the 21st Century.  From the opening titles you know that the film makers are 'riffing' the past.  Within the first 10 minutes there are 4 jumps in time and place, consistently moving backwards from the thirties ( I guess) to 1870.  Maybe some historical silent movie has done this but I immediately thought we were in Inception territory.  Surrealistic sets at place in 20's German film are here but without the budget of Murnau let alone Lang.  Miniatures and movie monster technology out of 1925's "The Lost World" contribute to the period  re-creation style.

Being a sort of movie buff I found all this enjoyable.  After the film ended I researched at the web site and found a list of old movies which influenced this one. I had watched nearly all of the movies in the list.  I am most curious to know the reaction of someone with no experience of these silent masterpieces to a modern day version.

Not all is rosy (or sepia), the fight scenes would have embarrassed Fairbanks.  Faces are not as expression rich as in the best silents and sometimes the pervasive moodiness needs relief that does not come.  But this is all in character with how I remember Lovecraft stories and this is intended to be a faithful translation to the silent screen of the short story with the same title.

Still, as an inveterate lover of 50's 'B' movie Sci-fi and Horror one comes to appreciate low budget creativity and this movie punches well beyond its budget.

Well, after nearly a year of off time, Google reminded me that I better migrate to the new platform or loose access.  So that's what I did.  In the coming months I will be more active, I promise :)

bottom up community change

I read that a small village in Wales,  Llangattock, had won a  prize from British Gas to improve their energy efficiency.   The whole story is a little more complex, it involved seed funding, community organization and  a contest.   It is a good example of how small community led, but large organization incented,  change can create positive change.  Here is a report about that uses Llangattock as an example of community led change in the wake of what these authors believe -  "it is clear that both centralised state activity and unfettered markets are flawed when it come to achieving deeply embedded social change".  Full report here:

http://tinyurl.com/3exoqvf

The key phrase here is "flawed", i.e.  central activity and markets are not totally dismissed, but they are not sufficient nor perhaps even structured appropriately.

The story of the Green Valleys project in Llangattock demonstrates that a diversity of efforts, ranging from high tech - solar PV and micro-hydro to low tech revivals such as traditional woodlands managment can achieve remarkable results.  Here is their web site for you to explore:

http://www.llangattockgreenvalleys.org/

I was also intrigued by mention of a 'small machinery ring' as a future activity.  I did a google search and found this description of the Lothian Machinery ring.  Their purpose is to share expensive farm machinery since such machinery forms a large and fixed capital cost in farming.

What I really wanted to talk about today

So, I do get it, that there are arguments that can be made about how climate change is either not happening or it's just part of the passing parade of weather, don't worry about it.

Let me give an example,  the conditions or processes that can change climate are called 'forcings' and forcing can have feedback which either amplifies or attenuates the impact of the forcing.  Turns out there are a lot of forcings and a lot of feedbacks..  In the recent report from the Artic council about the state of the Artic cryosphere, we find this - "Of those feedbacks expected to have strong effects, eight lead to further and/or accelerated warming, and just one leads to cooling."

Now, I could go to two different web sites and find one which used as it's 'evidence' that  world is in imminent danger one or more of the positive feedbacks and another one which used the negative feedbacks as evidence that everything will be ok.

It's nice having these plausible physical explanations which can be incorporated into predictive models, but more important to me are these two facts, also embedded within the same report:

1) The intensity of feedbacks between
the cryosphere and climate are not yet
well quantified, either within the Arctic
or globally. This lends considerable
uncertainty to predictions of how much
and how fast the cryosphere and the
Arctic environment will change.

2) Model projections reported by
the Intergovernmental Panel
on Climate Change (IPCC) in
2007 underestimated the rates of
change now observed in sea ice.

Ok, so there is uncertainty and the models are wrong, but what's the trend?  Things are happening faster and we don't know how fast they will go or if they will slow down.

Seems like that is a pretty big question and we better find out soon. Prudent risk management behavior would be taking mitigation steps for what we do know is happening with a plan to either ramp mitigation up or close it down depending on the outcome of more research and observations.

Follow up on the IPCC blogo-mania

I re-read my post from yesterday (that means at least one person has read  my entries :) and realize that it could give the wrong impression.  What I meant to say is that the content of the blogosphere reactions is political not scientific, although it claims to be about science...

The level of cross-linking and commentary triggered by Mark Lynas's single post creates a hyper-storm of content.  This massive increase in information contains absolutely no increase in knowledge.  From this kind of behavior, it's easy to see that after enough of these hyper-storms, finding knowledge in the sea of information get's harder.  Already google searches return so much information that one has to be selective in what one chooses to follow-up on, and if you are trying to discern what is actually happening, then good luck.

Let me give a good example.  Yesterday I found a youtube video of what looks to be steam release from fukushima reactor 4 that took place on June 14th.  Some people called it an explosion, but I can't tell from that video what is happening. Searching the web for an explanation is nearly hopeless, the majority of links are  dire warnings for which this  'explosion' is yet more proof for the point of view of the author with a link back to the youtube video.

It's easy to say that no one bothers to check facts anymore, but when it's this difficult, it's hard to see any other outcome but increased paranoia among the people who routinely plug into this stream of self-referential links.  It's also easy to see how one can get the impression that the evidence for whatever you are talking about, in this case the 'explosion',  is overwhelming, why I found 25 links talking about it!