Too little time is the reality

Or not enough time....So how do things end up taking so much time.......

First, understand that what I am talking about is a complex system with 100's of components and 1000's of technical details, i.e. a medium size IT operation.....

The system is complex because there are dependencies among the components. The Network infrastructure is really a lot appliance computers dedicated to networking, with often remote services from outsource providers. Storage is often handled centrally by a SAN or a NAS and there are usually multiples of them at various stages of life cycle. Then there are other shared resources, authentication servers, time servers, database servers......some can be outsourced as well.

Each of these groups have multiple parts or future possibilities: speed, size, expandability, power consumption, space consumption, robustness, life cycle (depreciation) and technical compatibility with other components.

Here is an example of what I mean.

We want to expand one type of storage. This storage needs to be in a specific location because of networking constraints. That location has space but no power available. There is a set of servers in the space that are being retired and eventually replaced by servers in another location. That's where the power will come from, retiring these servers. To do that we need to schedule downtime with the users. Their next window is three weeks out. In the mean time, we realize that this server we are moving can't move because it's dependency on a local SAN storage is not moving with it. That's not moving because another server is using it that can't move in the same time frame. We now have to schedule a replace storage expansion at another location. That's going to put us beyond the first three week window, and the next window is a month away. So can we get this storage expansion (not the same storage as we started with) done before the next downtime window? And guess what? We are back to the beginning of this process, but with a whole different objective. And on it goes.....

Catch-up

Well, it was bound to happen. I have really slacked off in my postings. Am I just lazy, not inclined to on-line rambling, or something else?

Reason #1 is in the eye of the beholder.
Reason #2 is clearly not true, once I get started I perhaps ramble on too much.
Reason #3 has some legs in it.

I never really made my mind up what I was going to be when I grew up. Here my progression:

Starting off in college I was going to be an academic in abstract math and logic. During the idealistic late 60's I changed that to being an academic in real world changing (ecology and resource planning). My research agenda started with the quite modest goal of understanding how humanity came to be where I perceived us to be, on the precipice of self-destruction. To understand humanity I thought I needed to start with Australopithecus and work forward. Needles to say, I didn't get very far before being embroiled in the hot debates of the day:

Were Proanthropus and Australopithecus different species or just different ends of genetic expression?

Nature versus Nurture

Did language cause evolution or is it a consequence of it?

Gee what fun, now I had to understand just what a species is. Start with Darwin and LaMarck, switch over to paleo-botany and try to understand the Cretaceous angiosperm explosion.

Try to understand humanity from the inwards direction, study religion and psychology.

Go to talks by Noam Chomsky to find out about language but instead find out about another view of the politics of the day.

Okay, Okay, enough already......but you can see the pattern there. Lot's of research, forks in all different directions and pretty soon you run out of patience and time. In today's academic world, you can't be an expert in the big picture, you can spend an entire career on very narrow fork in the road.

So this continues to this day. I have a lot of things I would like to blog about, but I just have to do a little bit more research down this avenue, what I find there just might change everything I want to say.......but the avenues never end and side roads are abundant and the obscure lanes and alleyway's look enticing.

Stay tuned, I think I see some light up ahead.

The Obsolete Blues

After struggling for over a year to get a VMWare solution up and running robustly and then priced out for a recharge service, I find that our central IT group has done the job already and for not much more than we can do it for. Thus, there is really no reason to be in this particular business.

I did see this coming, it is pretty clear that cloud computing is the real next wave. VMWare is what established IT shops do, and will do, to make those 'private' clouds.

The real cloud computing outside of the private world, will be based on higher level abstractions than the Operating System. Since I have been an advocate of abolishing any user interface into an operating system (after all, are we not really trying to perform application logic?) it should come as no surprise that I would also advocate for cloud computing using interfaces or API's well above the operating system.

What people are going to want at the most primitive level is going to be Ruby on Rails servers, PHP servers, Java servers or some other programming abstraction (Hadoop?). Then they are going to want data management, but not file managment, I mean contextually relevant data management. That could be via database systems or it could be via something else. This is the developer level access to clouds.

But what even more people (non-developers) are going to want are applications that just work and do useful things. Contact managment systems, billing systems, mail systems, customer relationship systems, social networking, media sharing and purchasing, etc.....

So, my new career, should I choose to accept it, will most likely be trying to show end users in academic research, how to get what they want out of clouds. I would like to think that I will be part of building a private/public cloud to facilitate the transition, but I don't think that a group of my size can ever effectively be in the infrastructure business. I don't know what I was ever really thinking in trying to do that anyway....vanity maybe.

Everything you know is wrong (again)

More people are finding me on facebook now than twitter. I still don't really know how to effectively use facebook or twitter, so I continue to write this blog since writing lots of words, whether I succeed at communicating or not, is what I seem to be good at.

Yesterday, I signed the petition to encourage my federal representatives to support the use of VISTA as the core of a proposed new bill "Health Information Technology Public Utility Act of 2009"

Many, in the technical communities I travel in, find VISTA's core use of MUMPS as reason enough to ignore it. This is because MUMPS is an 'old' technology and everyone knows that old technology can't be as good as 'new' technology. Or at least we have built a market based on that, with computers 'lasting' just 3 to 5 years before they need to be replaced. That sort of technological imperative thinking is just too simplistic for me anymore.

(Bet you were wondering if I would get back to the title of this post :)

So, what else do we know that's wrong? One thing that really strikes me is the nearly universal notion that 'we have to get this economy back on it's feet'. I take that to mean, at it's simplest, that we need to get back to the way things were! You can see this everywhere: Banks are now making money so those high paid executives who created that innovative engine of growth, financial derivatives (say sub-prime mortgage's), need to be rewarded again. The automotive market simply has to re-structure itself for lower operating costs, as if alternative living, working and transportation arrangements that are demonstrably better along many important dimensions (health, energy consumption) no longer need to be encouraged. Finally, we need to spend a lot more 'stimulus' money to get all those retarded health care practitioners to adopt the latest technology.

Don't know where to start

Major re-thinks going on regarding the following:

Storage Services
Clouds for the academic research community
Operational costs of IT (OpEx)
Is green the new red?

Some random thoughts:

How come I just found out about the 'Prisoner' remake, it's in post production right now?

No one really understands service levels.

No one ever thinks about risk in a way they can communicate about.

'IT' as a concept space has grown too large to manage. We already knew that as a terminology generator it exceeded the DOD some time ago. But now I find that I hear terms, read articles, read analysts reports and come up with high level internal architectural scaffolds with which I understand what I am talking about and asking for. It's just not likely that the people I am talking to or requesting product use the same scaffold and model. The net result is that I am constantly either disappointed or feeling abused and/or ripped-off.

The little things strike again.

First, on the "my horse for a drink of water" front, our installation of close to a half million in hardware has been held up for lack of a 1 meter long fiber cable. Now have plenty of fiber cables, LC to LC and SC to SC, but what we don't have is SC to LC and that is what we need. The vendor was supposed to ship us one, but they don't have them either!

If you have not guessed, LC and SC are connectors! SC are bigger than LC and seem to be prevalent in switching gear, at least what we get from Cisco and Procurve. LC are prevalent at the server side of things, such as 10GbE cards or Fiber Channel cards. Yes, you guessed it, the 'enterprise' switch vendors are in an enterprise that consists of switching gear and no servers. How long have servers been needing switches to connect them to the network? As another example of this, just recently ProCurve announced a new switch line designed for the machine room, which mainly means it has front to back cooling, with the front defined as where all the connectors are. Up until now, both CISCO and Procurve switches had either side to back or back to front cooling while all servers have front to back cooling. Try mixing that up in your racks for some turbulent air flow. Of course, this is all a matter of perspective since while the 'front' of switches is the front of the rack for most installations, the 'front' of the server is bereft of any connectors aside from the occasional USB port. Servers put their connectors in the 'back'
So, as a guy who installs servers, I really don't care which perspective is right or 'better', just that everyone agree's!

As a footnote, not all server vendors put their connectors on the back, a rather small vendor (Capricorn) which supplies the Internet Archive with it's servers, put's all the connectors on the front. Seems like they must have visited a machine room......

Now, for something completely different......

The second thing is, you may remember our NFS problems? Look at past postings for more information. You may also remember our 10GbE issues (hardware, drivers) as well. We have now demonstrated on the 1/2 of 10 identical servers which can't transfer much NFS data, that they can't transfer much of any kind of data, FTP, SFTP, etc.....They can, however transfer data just fine over Infiniband and link aggregated 1 GbE links!

So we are certain that this problem is in the 10GbE path. Whether it's software, firmware, or chipset issues, we don't know yet.

Hey, didn't you say these servers were identical? Yes, they all came in on the same shipment, their BIOS and Firmware all seems to be the same and it doesn't matter which OS we load, official Solaris, OpenSolaris, NexentaStor the one's that have a problem have it with all three OS versions.

But, there must be some difference somewhere, more sleuthing is required. By the way, when these kinds of things happen, don't believe the vendors that they are ready to support you, you are on your own.

I know the enterprise switch vendors are trying to tell us that 10GbE is the server room fabric of the future, but personally, Infiniband is far more robust and even has better performance to boot! It's also cheaper! But eventually we need to get to Ethernet to get on the Internet, thanks CISCO and Procurve for ignoring Infiniband and leaving us small fry with endless grief.

LOST - in data storage confusion

Previously on LOST - The data center edition, we were having problems with our particular enterprise implementation of Active Directory and SUN's CIFS server. We are switching vendors in an attempt to keep this project on the rails as the derailment looming ahead would be the "BIG ONE" for us.

To accomplish that we switched our storage node inter-connect protocol from iSCSI to NFS(v3). We, being completely paranoid by now, decided to re-run our stress tests on the X4500 storage nodes using NFS this time. Well, things have gone from bad to worse. So far, out of 8 X4500's, we have had only 2 pass the stress test. The one's that fail, do so anywhere from 200GB to 385GB into the test. The two that work can transfer over a TeraByte with no errors. What we see are client side timeout errors which eventually lock the client side machine, forcing a reboot of it.

(Fast cut to another scene, in another part of the data center)

The Oracle systems folks are starting to scream, their systems are locking up, both client side and server side and nothing, not even a trip to the remote managment ILOM console or a mad dash to the data center to attach a serial cable, will bring these servers back to life. They must be power cycled. What's common here is that the Oracle servers are NFS mounting X4500's.

(Cut to the last scene, a despondent group of IT folks in a darkened conference room, staring at log data.)

The leader say's to everyone - does anyone know the difference between the servers that fail and those few that don't? The answer is not yet. What are our plans? Well, we need to exercise the vendor support path (we have two vendors we can try here and three options). We also decide to take a stab at running the venerable and slow moving official Solaris 10 release on one of our failing X4500's and re-run the tests. We also decide to try and swap in a SUN 10GbE card but we need to buy one and get fiber transceivers and packs for our switches.

The scene fades out with no joy in IT land, things are collapsing faster than we can send in the operatives to repair them.....deadlines are looming, the clock is ticking, we can see the broken tracks in our headlights, .... to be continued.