[Internet2] Major electrical outage at HPCf.

[Humberto Ortiz Zuazaga]

As most of you are aware by now, the HPCf suffered a major electrical
outage last week. I would like to describe some of the events that
happened. I am trying very hard not to assign blame. If a portion of
this message appears to assign blame, ignore it, or better, send me a
note and I'll apologize.

The gist of the matter is that a transformer in the substation that
supplies the building housing the HPCf and the building housing the
UPR-RRP Press (Imprenta) failed Monday, August 16, at approximately 7:30
AM.

The HPCf has a large UPS which kept the GigaPOP and the supercomputers
running untill around 8:30, when staff of the HPCf came in and shut down
the supercomputers. The HPCf has no electrical generator (yet, it's in
bidding), and the air conditioning system cannot be run on our UPS.

The AEE came Monday morning, and determined that although the
transformer in question was inside the substation, it was in fact,
University property. Servicios generales was notified, and called an
outside contractor to repair the substation.

The UPS drained the banks of batteries Monday afternoon. At that point,
all the Internet2 lines became inoperable.

The UPR campuses (campii?) had T1 lines before the Internet2 project,
and by Tuesday morning UPR-RRP and UPR-RCM had activated these
lines. UPR-RUM had some technical issues re-activating the T1 lines.

The HPCf purchased a small (5.5 KVA) power plant, and was able to
activate the core router Wednesday. Internet2 service was
reestablished. Because of the nature of the generator, power was
interrupted several times.

UPR-RRP leased a large generator, and power to the buildings was
restablished, however, this power was intermittent because of technical
problems with the generator.

Repairs continued in the substation for several days. Two transformers
and the feeder lines from the grid to the substation and the substation
to the building were replaced.

Friday afternoon, repairs to the substation and feeder lines were
completed. Air conditioning was reestablished Monday (August 23)
morning, although not at 100%.

Open bids for both a generator and new A/C equipment for the machine
room are currently underway at the HPCf, paid for by funds from the
National Institutes of Health BRIN program. Open bids for remodeling the
interior electrical infrastructure at the HPCf have been awarded, work 
begins in the next few weeks.

A planned expansion of the Internet2 network including redundant
Internet2 and commodity Internet connections to each campus is under
development, costs will be shared by UPR and the BRIN project.

-- 
Humberto Ortiz Zuazaga
Programmer-Archaeologist
High Performance Computing facility
University of Puerto Rico
http://www.hpcf.upr.edu/~humberto/


-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 241 bytes
Desc: not available
URL: <http://lists.hpcf.upr.edu/pipermail/internet2/attachments/20040825/71b30a9d/attachment.sig>