[Internet2] Re: [sysadmin] Major electrical outage at HPCf.

["José G. Conde"]

Thanks for the information, Humberto, and specifically for keeping the 
community informed.

José

Humberto Ortiz Zuazaga wrote:

>As most of you are aware by now, the HPCf suffered a major electrical
>outage last week. I would like to describe some of the events that
>happened. I am trying very hard not to assign blame. If a portion of
>this message appears to assign blame, ignore it, or better, send me a
>note and I'll apologize.
>
>The gist of the matter is that a transformer in the substation that
>supplies the building housing the HPCf and the building housing the
>UPR-RRP Press (Imprenta) failed Monday, August 16, at approximately 7:30
>AM.
>
>The HPCf has a large UPS which kept the GigaPOP and the supercomputers
>running untill around 8:30, when staff of the HPCf came in and shut down
>the supercomputers. The HPCf has no electrical generator (yet, it's in
>bidding), and the air conditioning system cannot be run on our UPS.
>
>The AEE came Monday morning, and determined that although the
>transformer in question was inside the substation, it was in fact,
>University property. Servicios generales was notified, and called an
>outside contractor to repair the substation.
>
>The UPS drained the banks of batteries Monday afternoon. At that point,
>all the Internet2 lines became inoperable.
>
>The UPR campuses (campii?) had T1 lines before the Internet2 project,
>and by Tuesday morning UPR-RRP and UPR-RCM had activated these
>lines. UPR-RUM had some technical issues re-activating the T1 lines.
>
>The HPCf purchased a small (5.5 KVA) power plant, and was able to
>activate the core router Wednesday. Internet2 service was
>reestablished. Because of the nature of the generator, power was
>interrupted several times.
>
>UPR-RRP leased a large generator, and power to the buildings was
>restablished, however, this power was intermittent because of technical
>problems with the generator.
>
>Repairs continued in the substation for several days. Two transformers
>and the feeder lines from the grid to the substation and the substation
>to the building were replaced.
>
>Friday afternoon, repairs to the substation and feeder lines were
>completed. Air conditioning was reestablished Monday (August 23)
>morning, although not at 100%.
>
>Open bids for both a generator and new A/C equipment for the machine
>room are currently underway at the HPCf, paid for by funds from the
>National Institutes of Health BRIN program. Open bids for remodeling the
>interior electrical infrastructure at the HPCf have been awarded, work 
>begins in the next few weeks.
>
>A planned expansion of the Internet2 network including redundant
>Internet2 and commodity Internet connections to each campus is under
>development, costs will be shared by UPR and the BRIN project.
>
>  
>
>------------------------------------------------------------------------
>
>_______________________________________________
>sysadmin mailing list
>sysadmin at lists.hpcf.upr.edu
>http://lists.hpcf.upr.edu/mailman/listinfo/sysadmin
>  
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.hpcf.upr.edu/pipermail/internet2/attachments/20040825/bff27265/attachment.html>