VPP300 Downtime Information and Post-mortem

Remember that this Friday is the last working day of service on the VPP. Please take a copy of all the files you are likely to need and remove them from the vpp and covpp.

VPP downtime 5/06/2001 8am-2pm

The VPP was down for relocation in the machine room. The space will be used for the new Compaq GS server; another component of the APAC National Facility.

The VPP is back up and ready for work.

The home disk is being recovered from backup. There is a lot of data and it is a slow process, so the VPP will not be available till tomorrow morning.

The problem was that a fan in a disk enclosure failed, frying a home disk. The disk contents are being recoverd from backup.

The VPP is being shutdown to fix a problem with the home disks. This problem has been the cause of the disruption to service since the last reboot.

The I/O system of the vpp hung this afternoon. The symptoms were similar to yesterdays problems except using some equipment similar to that replaced yesterday. The equipment has now been replaced and we expect to be up shortly.

The Leonard Huxley Machine Room was powered down on Wednesday for work on the power grid. The downtime was extended until 3pm as some of the power cables were difficult to move. As a result of the downtime the vpp has been completely moved to the new power boards and so will not require a second downtime.

(3:15pm) The power has been returned to the vpp, however there is now a hardware fault in the I/O system. Replacement parts have been installed.

15/3The vpp was returned to users at 6pm.

The VPP suffered a hardware memory fault which crashed the system.

VPP300 suffered a CPU failure at 1:10pm, requiring an immediate system downtime. The system was returned to service at 10:15pm.

All ANUSF machinery was powered off in compliance with ITS Y2K policy.

The VPP300 will be down for system maintenance.

The VPP crashed at 10:30am. The main PE CPU stopped after an IO job tickled a bug in the kernel. The machine was rebooted and returned to service at 11:30am.

The VPP is down for a software upgrade. This will include a new revision of the NQS queueing system which will require all pending jobs to be cleared. The OS will be upgraded, and various Y2K fixes applied to the compilers. Any code with references to the 'date' system call should be recompiled.

Update The VPP had a hardware failure on Friday evening, and so was not fixed until noon, Monday. Some additional work was required to restore the system and so the downtime has been extended. We will be mailing out when the system is ready for use.

All ANUSF machines will be unavailable for a test of the fire systems in the Leonard Huxley Bldg.

During this downtime some Y2K patches were applied. A new 18GB disk for /short0 was installed (although won't be in production use until a later downtime). Preventative hardware maintenance was also performed including firmware updates to disk arrays and the PEs. The ATM switch had a memory upgrade during this downtime as well.

The VPP had a problem which shutdown the queues. The machine was returned to service at 5:30pm on Saturday. During the downtime we took the opportunity to replace the remaining older disk in /vflvol0.

The VPP will be unavailable on Sunday (13/9) afternoon for disk reconfiguration, and on Monday morning (14/9) from 8-11am for system maintenance and network reconfiguration.

Details:
On the Sunday downtime we will be doubling the size of /vflvol0 to 36GB, also using faster replacement drives. We will also be installing UXP/V V10L20 X98061, F90/VP V10L10 L98061, and ANALYZER V12L10 L98031.

During the Monday downtime the ATM card will be moved to PE12. This will enable two PE's to connect directly to the network and allow faster file transfers to Mass Data Store. This will also require covpp to be reconfigured and rebooted as well.

In addition there will be a new queue for large failsafe transfers to the Mass Data Store accessed using a new set commands called "netmv" and "netcp". Further details of this command will be posted soon, but if you are interest in this command please contact anusf@anu.edu.au

The VPP will be down for System maintenance on Monday 13/7 from 7:30am-11:30am. During this downtime PE10 will be fixed and brought back into service. In addition two VPP Fortran compiler patches will be installed.

The ANU VPP300 was down for system maintenance. In addition to the installation of a new version of F90/VPP, some disks were re-organised.

The ANU VPP300 was upgraded to Fujitsu's X98031 standard package. Details of this can be found in VPP News. Due to one of the requirements of the upgrade, all queues were cleared after the reboot. All mounts from store were restored after automounting was removed.