When All Else Fails

As you will probably be able to guess from this commentary, one of my other hobbies is amateur radio and emergency communications.  The following paragraph is a summary of a May 25, 2011 news report, FEMA Administrator Calls Amateur Radio “The Last Line of Defense”,  that appeared on www.arrl.org web site.

At a May 3, 2011 Federal Communications Commission forum on earthquake communications preparedness, FEMA administrator Craig Fugate described Amateur Radio operators as “the ultimate backup, the originators of what we call social media.”  He went on to say “I think that there is a tendency because we have done so much to build infrastructure and resiliency in all our other systems, we have tended to dismiss that role ‘When Everything Else Fails.’ Amateur Radio oftentimes is our last line of defense.” and   “we get so sophisticated and we have gotten so used to the reliability and resilience in our wireless and wired and our broadcast industry and all of our public safety communications, that we can never fathom that they’ll fail. They do. They have. They will. I think a strong Amateur Radio community [needs to be] plugged into these plans. Yes, most of the time they’re going be bored, because a lot of the time, there’s not a lot they’re going to be doing that other people aren’t doing with Twitter and Facebook and everything else. But when you need Amateur Radio, you really need them.”


The same can be said about the power industry.  We have done so much to build its' extensive infrastructure, reliability and resiliency that we cannot believe that it will even fail.  However sophisticated our systems get, they have failed in the past and will fail in the future.  The probability of failure is 1.00, the only question is when the failure will occur.  Tokyo Electric Power has had a hard lesson in the true probability of failure.  As we make systems more robust and reliable, nothing bad happen for long periods and we get bored, forget the reasons things have been done a certain way, slack off on maintenance on equipment that is never needed, and stop paying attention to the planning and training for the "could never happen" incidents.

The recent news reports on the Air France 447 flight data recorders seem to show the potential problems with suddenly dumping control of the aircraft on the pilot when the advanced flight control systems that actually do the flying of today's aircraft get confused due to "could never happen" failures.  We tend to place the operator in the same position by expecting him to correctly handle the situations that exceed the bounds of the automatic controls provided.

Are you prepared to respond "when all else fails"?

Where Does The Time Go

When I wrote the first entry in this blog, I did not expect it to be dormant for so long.  Certain constraints from my retirement were partially to blame but even without that, time was very scarce and the blog probably would not have gotten updated.  I wrote two short articles for the ISA Power Industry Division Spring 2011 Newsletter, one with the same title as this blog.  Never being one to waste words, here is that article.

It does not seem possible that I have been retired for a little over a year now. As I had been warned, I have been busier than while I was working. Does not seem possible. Wish some of the work would have generated a positive cash flow.

Since the first of the year, I have been reading a number of reports written by government commissions and boards on a number of incidents that have occurred. While only one directly involved a power plant, they all offered lessons learned and warnings for the power industry.

First up was the January report from the National Commission on the BP Deepwater Horizon Oil Spill and Offshore Drilling; Deep Water: The Gulf Oil Disaster and the Future of Offshore Drilling which can be downloaded at http://www.oilspillcommission.gov/final-report. I must admit that I skimmed the sections of the 398 page report dealing with the cleanup and environmental impact. However, the sections on the time-line, process equipment and decisions were very interesting.

To me, the most interesting discussion in the report is “Complex Systems Almost Always Fail in Complex Ways”. The tendency after a incident is to try to identify the 'single' point of failure, decision, or activity that caused the incident. Believing that there is a magic silver bullet that will prevent a repeat of the incident just re-enforces the complacency that caused the incident. The report references “Complex Systems Almost Always Fail in Complex Ways” to the Shuttle Columbia Accident Investigation Board Report. The six volume report is available at http://caib.nasa.gov/news/report/default.html. The referenced discussion appears in Volume I.

Next came the Chemical Safety Board's report on the Bayer CropScience Pesticide Waste Tank Explosion (http://www.csb.gov/newsroom/detail.aspx?nid=361) in 2008. Factors cited in the investigation reports included attempting the process start-up with a new DCS system that was still being checked out, instrumentation not calibrated, totally different look to HMI displays due to change of DCS vendor, etc. Other factors identified are cognitive tunneling and not seeing the big picture, task-shedding due to the workload experienced during the re-start, interlocks that did not work on the old system being retained in the new system where they were also bypassed, missing valves and instruments, and operating procedures that were not updated to the new equipment that had been installed.

The final item reviewed is the Chemical Safety Board's Safety Video; “Deadly Practices” (http://www.csb.gov/newsroom/detail.aspx?nid=363) that focuses on the ConAgra and Kleen Energy explosions that resulted from the hazards of fuel gas releases. The video focuses on the dangers of what appears to be a cavalier attitude about the explosive potential from fuel gas releases when charging lines or doing gas blows to clean line. The CSB has recommended changes in codes, standards, recommended procedures and laws to prevent additional incidents. Some of the recommendations have been implemented in the year since the Kleen Energy explosion. From the number of house explosions and fires that have been making the local news this winter, I have concerns that everyone has become too complacent in handling natural gas and propane.

Even though only one of these incidents involved a power plant, I think everyone will recognize many of the factors cited from their personal experience. These reports are recommended reading and worth consideration for everyone, regardless of their experience.

Hopefully, the next blog entry will not take as long.  I have been busy trying to grow the ISA Power Industry Division group on Linked In which was the subject of the other newsletter blog.  I also have a technical paper being presented in the division's 54th POWID Symposium in Charlotte next month and it's time to submit another article for the Summer Newsletter.  You would think that I like to write or was good at it.

One More Day

This coming Monday will be my last day with the only employer I have had since 1964. Only time will tell if the decision to participate in the VRIF was a good decision. I never had realized how much work it would be to organize my paper and electronic files for handover and clean out my office. Hope one day will be enough to finish. With normally being off most of December due to the holiday shutdown and burning off unused vacation, it probably won't really hit me until I don't go back to the office after January 1st.

Starting in December, I will have to figure out what to do; write a book, become a consultant, spend more time on the air as a ham operator, or finish up a few hundred of the many projects I never got to. Since I have not had time to golf since 1966, I doubt I will take it back up.

Maybe I'll use this blog to get up on a soapbox and comment on the automation & control business. Possible topics could be:

• Why are the DCS suppliers impeding the I&C business
• Alarm mismanagement by design
• Should operators be trained on situational awareness
• Why is anyone still just using Basic Process Control
• How to blow up your plant with a Safety Instrumented System
• Why are the technical societies unable to actract younger members
• Will ISA and the DCS vendors ever make up

In case any one reads this and wonders where I am coming from, I have spent most of my carer on I&C for burning "dirty" coal to make steam & electricity. Along the way I have also been involved with nuclear and solar boilers, carbon capture and automation of industrial ice cube makers.

Oh well, right now I have to get my office cleaned out by day's end on Monday so I can pack to accompany my wife on my "retirement cruise". Stay tuned to see what if any of my plans develop or if I apply to be a greeter at a big box store. The latter seems to be every retiree's goal any more.