From humberto at hpcf.upr.edu Wed Aug 25 13:36:29 2004 From: humberto at hpcf.upr.edu (Humberto Ortiz Zuazaga) Date: Wed, 25 Aug 2004 13:36:29 -0400 Subject: [Internet2] Major electrical outage at HPCf. Message-ID: <200408251736.i7PHaT6P018960@daleth.hpcf.upr.edu> As most of you are aware by now, the HPCf suffered a major electrical outage last week. I would like to describe some of the events that happened. I am trying very hard not to assign blame. If a portion of this message appears to assign blame, ignore it, or better, send me a note and I'll apologize. The gist of the matter is that a transformer in the substation that supplies the building housing the HPCf and the building housing the UPR-RRP Press (Imprenta) failed Monday, August 16, at approximately 7:30 AM. The HPCf has a large UPS which kept the GigaPOP and the supercomputers running untill around 8:30, when staff of the HPCf came in and shut down the supercomputers. The HPCf has no electrical generator (yet, it's in bidding), and the air conditioning system cannot be run on our UPS. The AEE came Monday morning, and determined that although the transformer in question was inside the substation, it was in fact, University property. Servicios generales was notified, and called an outside contractor to repair the substation. The UPS drained the banks of batteries Monday afternoon. At that point, all the Internet2 lines became inoperable. The UPR campuses (campii?) had T1 lines before the Internet2 project, and by Tuesday morning UPR-RRP and UPR-RCM had activated these lines. UPR-RUM had some technical issues re-activating the T1 lines. The HPCf purchased a small (5.5 KVA) power plant, and was able to activate the core router Wednesday. Internet2 service was reestablished. Because of the nature of the generator, power was interrupted several times. UPR-RRP leased a large generator, and power to the buildings was restablished, however, this power was intermittent because of technical problems with the generator. Repairs continued in the substation for several days. Two transformers and the feeder lines from the grid to the substation and the substation to the building were replaced. Friday afternoon, repairs to the substation and feeder lines were completed. Air conditioning was reestablished Monday (August 23) morning, although not at 100%. Open bids for both a generator and new A/C equipment for the machine room are currently underway at the HPCf, paid for by funds from the National Institutes of Health BRIN program. Open bids for remodeling the interior electrical infrastructure at the HPCf have been awarded, work begins in the next few weeks. A planned expansion of the Internet2 network including redundant Internet2 and commodity Internet connections to each campus is under development, costs will be shared by UPR and the BRIN project. -- Humberto Ortiz Zuazaga Programmer-Archaeologist High Performance Computing facility University of Puerto Rico http://www.hpcf.upr.edu/~humberto/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 241 bytes Desc: not available URL: From jconde at rcm.upr.edu Wed Aug 25 13:53:08 2004 From: jconde at rcm.upr.edu (=?ISO-8859-1?Q?=22Jos=E9_G=2E_Conde=22?=) Date: Wed, 25 Aug 2004 13:53:08 -0400 Subject: [Internet2] Re: [sysadmin] Major electrical outage at HPCf. In-Reply-To: <200408251736.i7PHaT6P018960@daleth.hpcf.upr.edu> References: <200408251736.i7PHaT6P018960@daleth.hpcf.upr.edu> Message-ID: <412CD204.1030303@rcm.upr.edu> Thanks for the information, Humberto, and specifically for keeping the community informed. Jos? Humberto Ortiz Zuazaga wrote: >As most of you are aware by now, the HPCf suffered a major electrical >outage last week. I would like to describe some of the events that >happened. I am trying very hard not to assign blame. If a portion of >this message appears to assign blame, ignore it, or better, send me a >note and I'll apologize. > >The gist of the matter is that a transformer in the substation that >supplies the building housing the HPCf and the building housing the >UPR-RRP Press (Imprenta) failed Monday, August 16, at approximately 7:30 >AM. > >The HPCf has a large UPS which kept the GigaPOP and the supercomputers >running untill around 8:30, when staff of the HPCf came in and shut down >the supercomputers. The HPCf has no electrical generator (yet, it's in >bidding), and the air conditioning system cannot be run on our UPS. > >The AEE came Monday morning, and determined that although the >transformer in question was inside the substation, it was in fact, >University property. Servicios generales was notified, and called an >outside contractor to repair the substation. > >The UPS drained the banks of batteries Monday afternoon. At that point, >all the Internet2 lines became inoperable. > >The UPR campuses (campii?) had T1 lines before the Internet2 project, >and by Tuesday morning UPR-RRP and UPR-RCM had activated these >lines. UPR-RUM had some technical issues re-activating the T1 lines. > >The HPCf purchased a small (5.5 KVA) power plant, and was able to >activate the core router Wednesday. Internet2 service was >reestablished. Because of the nature of the generator, power was >interrupted several times. > >UPR-RRP leased a large generator, and power to the buildings was >restablished, however, this power was intermittent because of technical >problems with the generator. > >Repairs continued in the substation for several days. Two transformers >and the feeder lines from the grid to the substation and the substation >to the building were replaced. > >Friday afternoon, repairs to the substation and feeder lines were >completed. Air conditioning was reestablished Monday (August 23) >morning, although not at 100%. > >Open bids for both a generator and new A/C equipment for the machine >room are currently underway at the HPCf, paid for by funds from the >National Institutes of Health BRIN program. Open bids for remodeling the >interior electrical infrastructure at the HPCf have been awarded, work >begins in the next few weeks. > >A planned expansion of the Internet2 network including redundant >Internet2 and commodity Internet connections to each campus is under >development, costs will be shared by UPR and the BRIN project. > > > >------------------------------------------------------------------------ > >_______________________________________________ >sysadmin mailing list >sysadmin at lists.hpcf.upr.edu >http://lists.hpcf.upr.edu/mailman/listinfo/sysadmin > > -------------- next part -------------- An HTML attachment was scrubbed... URL: