Fail of the Week: A Candle Caused Browns Ferry Nuclear Incident

Hackaday
Fail of the Week: A Candle Caused Browns Ferry Nuclear Incident
Fail of the Week: A Candle Caused Browns Ferry Nuclear Incident

A colleague of mine used to say he juggled a lot of balls; steel balls, plastic balls, glass balls, and paper balls. The trick was not to drop the glass balls. How do you know which is which? For example, suppose you were tasked with making sure a nuclear power plant was safe. What would be important? A fail-safe way to drop the control rods into the pile, maybe? A thick containment wall? Two loops of cooling so that only the inner loop gets radioactive? I’m not a nuclear engineer, so I don’t know, but ensuring electricians at a nuclear plant aren’t using open flames wouldn’t be high on my list of concerns. You might think that’s really obvious, but it turns out if you look at history that was a glass ball that got dropped.

In the 1960s and 70s, there was a lot of optimism in the United States about nuclear power. Browns Ferry — a Tennessee Valley Authority (TVA) nuclear plant — broke ground in 1966 on two plants. Unit 1 began operations in 1974, and Unit 2 the following year. By 1975, the two units were producing about 2,200 megawatts of electricity.

That same year, an electrical inspector and an electrician were checking for air leaks in the spreading room — a space where control cables split to go to the two different units from a single control room.  To find the air drafts they used a lit candle and would observe the flame as it was sucked in with the draft. In the process, they accidentally started a fire that nearly led to a massive nuclear disaster.

Working with Inflammable Materials

You can build walls 30 inches thick, but you still need to get utilities in and out of the area. This was the case in the spreading room — the area where cables from all over the plant converged on the common control room.

The workers found a 2×4 inch opening near a cable tray. They stuffed the hole with foam and checked it again. There was still a draft and the flame was sucked into the hole, lighting the foam on fire. The inspector tried to knock out the fire, first with a flashlight and then with rags. By this time, the wall was on fire and several fire extinguishers were used to attack the problem but without success. The fire burned on. In fact, the fire extinguishers may have blown burning material out of the hole, making it even worse.

The Failure of the Fire Plan

Because of the efforts to put it out, the fire wasn’t officially reported for 15 minutes. There was also confusion about what phone number to use to report the fire. Perhaps most surprising is that for whatever reason, the operators elected to continue running the reactors despite the fire. According to the official report they then noticed that pumps in the emergency core cooling system were running:

Control board indicating lights were randomly glowing brightly, dimming, and going out; numerous alarms occurring; and smoke coming from beneath panel 9-3, which is the control panel for the emergency core cooling system (ECCS). The operator shut down equipment that he determined was not needed, only to have them restart again.

I wouldn’t operate my car like that, much less a nuclear reactor. After a few restarts, they started talking about shutting things down. Just then, the power output of unit 1 dropped for no apparent reason. They reduced the flow on the operating pumps which then promptly failed. Finally, the operators dropped the control rods to shut down the nuclear reaction.

Doing Everything to Cool the Cores

As you might expect, shutting down a reactor isn’t quick and easy. Electrical supply was lost to several systems in unit 1 including several key instrument and cooling systems. In unit 2, the panels were going crazy and there were many alarms. Then about 10 minutes after the unit 1 reactor started dropping its output, unit 2 followed suit.

Unfortunately, the equipment failed there too and they lost emergency cooling and control of some relief valves. Unit 1 was struggling with very little instrumentation and a reduced number of relief valves. The fear was that if the core did not remain submerged in water, it would melt down.

To keep the core underwater, they used the relief valves to drop the internal pressure from 1020 PSI to under 350 PSI so that a low-pressure pump could force water into the chamber. This decision was met with yet another problem; the low-pressure pumps were not working either so they had to rig up a workaround using a different pump.

In unit 1, the water level was normally about 200 inches above the top of the core, but it fell to about 48 inches. Unit 2 had more pump capacity, but it still wasn’t enough. They rigged up the same makeshift pump arrangement.

Domino Effect of Power and Control Failures

Before this all began, unit 2’s computer already happened to be down, and the unit 1 computer soon failed. With nearly all the instrumentation having failed, and the diesel generators down, they had very little on-site power. The phone system failed, preventing the control room from making outbound calls which were being used to send instructions to people operating valves and other key equipment manually.

Unit 1 under construction

Meanwhile, the fire was still burning. There was a built-in extinguisher that could be manually activated with a crank. But during construction, those activation cranks all had metal plates placed under them to prevent accidental activation of the extinguisher system. Almost none of the plates had been removed when construction was complete. By the time they were finally able to operate the system, it didn’t stop the fire completely and had the effect of driving thick smoke into the control room.

Two workers were tapped to investigate. They put on breathing gear and went into the spreader room to find that the neoprene covers on the cable were burning and emitting a thick black smoke. The quarters were cramped and one man described having to take the air cylinders off his back and push them along with the fire extinguisher in front of him to get under the trays about 30 feet to reach the flames.

The extinguisher system wasn’t the only safety equipment that was ill-prepared for an emergency. Many of the breathing masks at the plant were not working. Some had improperly filled tanks and others were missing parts. The main tank on site was apparently low on pressure and unable to completely fill the working tanks which resulted in about 18 minutes of air per fill for those trying to fight the fire.

The Red Truck Brigade

The local fire department arrived on the scene but they were not allowed to run the effort — presumably because you want people with specific training to fight a fire in a reactor. However, the fire chief did repeatedly suggest that water was the right way to put out the fire, as it wasn’t actually electrical in nature. However, plant management didn’t agree.

After the fire burned for over six hours, the plant personnel decided to try water. Unfortunately, the fire hose didn’t deploy fully so they were getting low pressure. In the heat of the moment, the workers erroneously decided the nozzle was defective and borrowed one from the fire department, but it had incompatible threads and would not stay on the hose. Even with these problems, water had the fire out in 20 minutes.

On the Verge of a Meltdown

You might think the fire being out is the end of the story, but no. The damage had been done — control of the two reactors was greatly inhibited and keeping the cores cool remained an emergency situation. The relief valves on unit 1 finally quit and pressure went up beyond the ability of the makeshift pump system to operate.

There was an ancillary pump operating, but it couldn’t keep up and a meltdown seemed likely. In retrospect, there was a way to use some of unit 2’s equipment but no one figured that out at the time.

Instead, it was luck that they were able to make repairs before time ran out. Workers fought to get the pressure valves back online and succeeded. This allowed the pressure to drop enough for the pump to continue providing fresh water.

Timeline of a Near Disaster

The candle flame started fire about 12:20 PM. Unit one reached full shutdown at 4 AM and that was the end of it. As much as it sounds like everything went wrong, it was even worse. There were a host of problems with equipment ranging from lights to tape recorders.

Speaking of tape recorders, there was one really interesting phone conversation between J. R. Calhoun, the chef of TVA’s Nuclear Generation Branch at the time and Frank Long of the NRC (and reported by a Canadian website):

Calhoun: Yah, you know everything for those two units comes through that one room. It’s common to both units, just like the control room is common to both units.

Long: That sorta shoots your redundancy.

Thanks to creative problem solving, it appears the incident didn’t pose a public risk — although many people have critiqued how the public was kept informed (or not). There was never any radioactive leakage from the plant reported.

Analysis

So many questions. Why were they using candles when other methods were available? Why were they using flammable material as insulation? The investigation turned up that flawed tests indicated the polyurethane used in the foam was resistant to fire… in solid form. However, the foam was highly flammable and many people knew this. Many people didn’t know that candles were used for leak detection.

Perhaps the worst bit of news is that two days earlier a similar fire had started but was put out quickly. The shift engineers had a meeting and had already decided to recommend a different way to test for leaks that didn’t require candles. But nothing had been done.

Needless to say, the Nuclear Regulatory Committee made many changes to their fire protection standards and mandated silicone foam for firestops. It even influenced practices in other industries, too. If you want to read the DVD the NRC released about the incident in 2009, you can.

The fire caused about $10 million of direct loss and as much as  $500 million in indirect costs. (That’s about $44 million and $2.2 billion dollars if you update to 1976 figures for 2018 value.) It took about 1000 man-years of effort during the 18-month recovery process.

I know the debate over if we should have nuclear power or not is polarizing and I won’t tackle that here. But it is amazing that a high tech piece of equipment — no matter what it does — could be taken down by a candle and some bad procedures. You know there were all sorts of safety devices and procedures and that everyone there must have known the possible consequence of something going wrong. Yet you had a known fire problem ignored, bad air and fire equipment, and a host of other problems.

So think about not only what balls you have in the air with each project, but ask yourself which of those are glass balls. Don’t forget to focus on the small seemingly inconsequential things. There’s also a danger in assuming that you don’t need at least some understanding of all the balls in the system. After all, if someone high up had realized foam caught fire and workers were using candles around it, this might have been a different story.

Обсудить
Leave a Reply

;-) :| :x :twisted: :smile: :shock: :sad: :roll: :razz: :oops: :o :mrgreen: :lol: :idea: :grin: :evil: :cry: :cool: :arrow: :???: :?: :!:

Fail of the Week: A Candle Caused Browns Ferry Nuclear Incident