1. Test safety system components even if they are new.
2. Do not rely on listening for valve movement as confirmation that a valve is closing; measure downstream pressure to determine if the valve is really sealing properly.
3. Lock the valves when working in gas manifolds.
4. Maintain a clear path to the exit in case of an emergency.

A hydrogen release of this type is a significant event. The event highlighted a number of procedural contributing factors that will influence the manner in which these fuel cell systems will be serviced in the future. A complicating factor in this event was that multiple companies were involved, and communications among them were inadequate. It is likely that the condition existed from the original manufacture of the fuel cell systems, and may even have been understood by the Company A fuel cell team, but the history is not fully known since that team no longer exists. Company B’s investigation also discovered that a similar leak had been experienced at the same facility and a similar replacement had been required, but there was no corporate memory of the repair or the underlying failure mode.

If a situation arises as a result of consolidation or equipment transfer wherein another entity takes ownership or service and support responsibility for fuel cell systems, the full design history and operating records of the systems must be fully documented and accessible. This will allow for proper knowledge transfer of underlying design considerations or problematic reliability or safety-related issues, and potentially prevent this type of avoidable incident from occurring again.

Another lesson relates to how high-pressure components within the hydrogen fuel storage system are qualified following a repair. It is envisioned that in the near future, there will likely be regional service centers equipped with re-manufacturing capabilities to support commercial fuel cell deployments. These repair shops would be equipped with the infrastructure to properly purge and pressurize equipment with small-molecule gas to test for leaks.

Procedures for safe handling of compressed gas cylinders, marking design of gas cylinders and connecting lines, and arrangement of cylinders were reviewed and modified as necessary. The spectrometer was returned to the manufacturer for a careful examination to assess the full extent of the damage. The affected laboratory area was taken out of service. Additional conspicuous markings were added to flammable gas cylinders and connecting lines. Specific training on safe handling of compressed gases was provided for all compressed gas users. The FTIR spectrometer was physically moved to a different laboratory where hydrogen cylinders were not used. All hydrogen lines and valve connections were color-coded red.

In addition to the probable causes listed above, the lack of a standard operating procedure for hydrogen leak detection was one of the probable causes of this incident. Additional contributing factors included the following:
- Severe pipe corrosion due to the presence of hot water in the pipe trench
- Hydrogen piping located in a concealed space
- Limitations of the flash-fire-resistant garments worn by plant employees.

Key findings noted in the CSB report included:

1. Significant accumulations of combustible iron powder fueled fatal flash fires when lofted near an ignition source.
2. Facility management were aware of the combustibility hazard two years earlier but did not mitigate the hazard with engineering controls or housekeeping.
3. The plant did not institute combustible gas monitoring or employee training to help avoid flammable gas fires and explosions.
4. OSHA did not include iron and steel mills in its Combustible Dust National Emphasis Program.
5. The 2006 International Fire Code (IFC) does not require enforcement of the more comprehensive and rigorous NFPA standards for the prevention of dust fires and explosions.
6. The state and city did not enforce recommended IFC practices.
7. The local fire department inspected the facility just months before the fatal hydrogen explosion and dust flash fire but did not cite or address the combustible dust hazards that were present.
8. The flame-resistant clothing provided to the plant employees did not adequately protect them from the hydrogen explosion and dust flash fires.
9. There was no corporate oversight regarding the management of combustible dusts even though there had been a succession of serious accidents at the facility in the past.

The CSB made recommendations regarding combustible dust hazards to OSHA, the International Code Council, the state, the company, the Metal Powder Producers Association, the city, and the local fire department. Recommendations to the company covered both combustible dust and flammable gases as shown below.

1. Conduct periodic inspection audits of the facility for compliance with the relevant NFPA standards (484, 499, 497, 2, and 2113), using knowledgeable experts, and implement all recommended corrective actions.
2. Develop a training program for combustible dust hazards for all employees and contractors.
3. Implement a preventive maintenance program and leak detection/mitigation procedures for all flammable gas piping and gas processing equipment.
4. Implement a near-miss reporting and investigation policy.

The company investigation revealed that the incident arose because insufficient water was added to the batch. This resulted in a rapid increase in temperature and evolution of hydrogen gas following the addition of aluminum powder in the last seconds of the mix. Despite the presence of a functioning level-control valve on the mixer, the hydrogen gas was ignited when the operator opened the hatch. The most likely source of ignition was the faulty lamp. The operator was acting in accordance with his training and following the company's written safety procedures.

The company took a number of measures to prevent a reoccurrence of this incident, including:

•  provision of intrinsically safe lamps
•  introduction of daily checks of the vent valve
•  minor modification to LEV and increased venting throughout the mixing process
•  lab testing by the aluminum supplier to evaluate system safety with regard to hydrogen generation for all reaction conditions and quantities of aluminum added
•  reprogramming/development of the software to improve both the safety of the operation and operator understanding of warning alarms.

The incident was the result of a combination of factors leading to exceptional temperature conditions that were not taken into account in the mechanical design of the reactor. Corrective actions that were implemented by the plant management included:

• redefinition of the appropriate tightening torque on flanges
• improved design of the leak collector on the flange (which failed during the accident)
• creation of a nitrogen injection system in the leak collector
• installation of a steam-injection system to protect the bottom part of the synthesis reactor.

The following actions were proposed as a result of this incident:

•  The company in charge of valve calibration and maintenance will be subject to approval of the plant service inspection team.
•  Plant operating procedures will be improved.
•  The specifications concerning the mounting and revision of valves will be strengthened.
•  An additional pressure sensor will be installed.

The project team concluded that the jar contained a sufficient vapor pressure of isopropanol to ignite when it came into contact with the decomposing hydride. The lesson learned was that hydrides react rapidly in air and can lead to combustion of any organic vapor that might be present nearby. Thus, the project team adopted a procedure that all hydrides must be submerged in mineral oil before they are removed from the glove box to prevent exposure to air before isopropanol treatment. Since this procedure was adopted for pacification/disposal of hydrides, there have been no more incidents or near-misses of this type in the laboratory.

1. The presence of other flammable impurities (e.g., oil carryover from compressors, hydrocarbon contamination of the gas) is an additional hazard and should be eliminated before cryogenic purification.
2. Potential sources of oxygen in the system include:
   1. Oxygen or air impurity entrainment in the main gas to be purified in conjunction with failure of the upstream processes to remove these impurities.
   2. Regeneration of previously desorbed oxygen in the activated carbon bed.
   3. Accidental ingress of oxygen or air in the upstream processes (e.g., during vacuum or low-pressure processes).
3. Risk mitigation methods for processes involving cryogenic activated carbon adsorbers include:
   1. Identify and eliminate by design the following:
      1. Potential formation of liquid oxygen and its contact with the activated carbon bed. Sources of liquid oxygen could be air ingress or oxygen impurity being entrained in the main gas to be purified/liquefied.
      2. Potential ignition mechanisms/sources (e.g., electric heaters used for regeneration).
      3. Use of alternative/substitute adsorbent that is noncombustible with liquid oxygen (e.g., silica gel, molecular sieve).
      4. Add an oxygen trap (e.g., silica gel, liquid nitrogen trap/knockout pot system) upstream of the activated carbon adsorbent bed.
      5. Avoid having sudden flow or pressure changes through the activated carbon adsorber, since these would increase the likelihood of ignition.
   2. Use sensors to identify the presence of oxygen or air ingress upstream and actuate alarms or system emergency stop when concentrations exceed predetermined set points.
   3. Purge the adsorber bed with an inert gas to reduce the risk of ignition if high oxygen concentrations are present.
   4. Alternatively, pull a vacuum on the adsorber to remove any adsorbed oxygen before the adsorber is put back online.
   5. Look for alternatives to or add-on features for pressure-reducing devices.
4. Years of satisfactory service without incident should not be taken as proof of safe operation of cryogenic activated carbon adsorbers.

Three root-causes were noted during the investigation: (1) the use of incompatible materials in the manufacturing of the PRD valve, (2) improper assembly resulting in over-torquing of the inner assembly, and (3) over-hardening of the inner assembly materials by the valve manufacturer. These problems could have been avoided by adequate quality assurance/quality control procedures during the design and safety reviews.

The canopy was added to the station as an afterthought, sometime following the HazOps review. The prestart-up safety review by all parties and the local authority having jurisdiction did not recognize the setback distance of the canopy. Had an engineering management of change, follow-up HazOp or other form of risk assessment been conducted, it is likely that the vent stacks adjacent to the canopy would have been raised in order to avoid any damage in the event of a fire.

Prior to reopening the station, physical changes were made using the correct PRD valves and higher vent stacks, and new and modified procedures were instituted to improve the timely communication of station status during emergency events. Additional training of personnel focused on improving the response time and effective communication between employees, first responders, and the hydrogen equipment supplier.

1. The hydrogen supplier installed a fire-resistant material board adjacent to the high-pressure hydrogen storage banks to prevent any potential jet flames from affecting adjacent high-pressure cylinders for several minutes. The 0.25 mm sandwich board of fiberglass-reinforced, lightweight concrete is easy to maintain and does not rot under outside conditions. This safety measure was implemented just three days after the incident occurred, although it had been planned for a long time.
2. The hydrogen supplier installed a semi-automated sprinkler system to cool the high-pressure hydrogen storage banks to prevent any potential escaping hydrogen gas that might ignite in jet flames from affecting other hydrogen cylinders. In addition, the dry piping system above the high-pressure hydrogen storage banks can be flooded with water by the fire department in case of fire or leakages in the high-pressure banks.
3. The alarm system was refined to send automated messages to relevant personnel informing them of gas/fire alarms.
4. The remote control room where service personnel are monitoring the fueling station is now equipped with an additional audio system to draw faster attention to alarms.
5. All plans and emergency procedures have been reviewed, adjusted and edited to document changes and fully capture the lessons learned.
6. Other learnings: Training for worst-case scenarios is recommended in order to be prepared for those siutations.

Personnel were focused on the AGES system test and results, not the compatibility of the test equipment. The manual valve was needed to successfully test the system, however the fact that this particular valve could not accommodate the full cylinder pressure was overlooked.

The following corrective actions will be implemented:

1. Evaluate the interfaces between engineering and operations systems and procedures to manage temporary modification work.
2. Develop and implement a procedure for engineering design of instrument systems.
3. Evaluate alarms/emergency response procedures for a relevant set of facility systems and revise, if necessary.
4. Evaluate the relevant facility building access training for appropriateness of the alarm response section and revise, if necessary.

This safety event suggests that temporary modifications, particularly those required for system testing, should be given the same level of attention and review as permanent modifications.

An important aspect of the reliability of a valve is the condition of the stem seal which tends to deteriorate with time and wear. Valves used in hydrogen service should be packed with the correct valve packing material and periodically checked for leaks as part of a regular maintenance program.

1. Carbon steel Nelson curve methodology cannot be depended on to prevent HTHA equipment failures and cannot be reliably used to predict the occurrence of HTHA equipment damage. Revisions to recommended practices should be considered regarding the use of carbon steel in HTHA-susceptible service and the verification of actual operating conditions.
2. Given the difficulty of inspecting for HTHA because the damage might not be detected, inherently safer design is a better approach to prevent HTHA.
3. Process hazards analysis (PHA) and damage mechanism hazard reviews (DMHRs) need to carefully consider all assumptions, periodically if necessary, to ensure that hazard identification, safeguards and control of hazards to prevent equipment failure are effective.
4. Effective programs need to be in place to manage and provide oversight for hazardous nonroutine work.

An investigative communication notes that "mechanical integrity programs at refineries repeatedly emphasize inspection strategies rather then the use of inherently safer design to control the damage mechanisms that ultimately cause major process safety incidents." Regarding the similarity of this accident to others, it is also noted that "while sulfidation is a well-known damage mechanism at refineries that requires regular inspection and monitoring, the segment that failed has no record of ever being inspected."

NOTE: This record is based upon an investigative report and related communications and will be updated, as appropriate, when additional investigative reports are completed, released and reviewed. Additional details regarding mechanical integrity programs and procedures in place at the time of the accident are expected.

The researcher's failure to pull the fire alarm was an oversight of required facility practice. The alarm should have been triggered in consideration of the potential for greater harm to personnel and facilities.

Hot, reacting ammonia borane produces hydrogen as well as other pyrophoric impurities. Reactions should be carried out in inert atmospheres or purged with inert gases. Furthermore, efforts should be made to prevent oxygen from coming in contact with the material while hot.

As a consequence of this incident, an updated procedure was put in place to check critical fittings before each subsequent test and to purge the apparatus with argon in the area surrounding the reactor/fittings.

The procedure for disposal of spent or partially spent AB has been modified so that it does not include the use of water. Instead, the AB is removed from containers and transferred for disposal by rinsing with mineral oil, silicone oil or other similar inert materials. It is then disposed of as a slurry.

Maintain an internal process for verifying component wetted material compatibility for intended use as part of the procurement process for hydrogen system equipment. It is critical that component parts be appropriately rated for the materials, pressures, temperatures, and other conditions experienced during operation of the system in which they are a part. Don’t rely solely on a manufacturer to provide appropriately rated materials and components. Verify components and their specifications as early in the design or procurement process as possible. Manufacturer-provided literature (brochures, instruction manuals, bills of materials, etc.) may not always identify the specific materials for each component, so verification may require Internet research or contacting the manufacturer to obtain the necessary information or certification. (Reference https://h2tools.org/tech-ref/technical-reference-for-hydrogen-compatibi…)

Implement rigorous assembly, verification, and documentation procedures for equipment.

Increase automated leak detection frequency.

Do not interconnect hydrogen drain trap lines with other drain lines

Thoroughly inspect equipment installation to manufacturer’s installation instructions

Warn against unauthorized field modifications

Consult the manufacturer and listing agency before any field modifications are undertaken