ICS Maintenance June 18, 2016

In order to accomodate a 12 hour facilities shutdown on June 19, 2006, ICS Computing Support will begin shutting down non-critical ICS infrastructure in the ICS Datacenter at 5pm on June 17, 2016 and performing general maintenance throughout the day on June 18th, 2016.

Facilities's Message

Facilities High Voltage Team has been performing preventative maintenance on many of the campus building transformers, switchgear, and breakers. While performing maintenance we have been able to identify and repair many equipment deficiencies and reduce the possibility of electrical failures and outages.

Facilities High Voltage Team has scheduled preventative maintenance on Computer Science T-18 & T-19 transformers and switchgear for Sunday, June 19th, 2016, from 5:00am-5:00pm

In order to perform this maintenance it will require a 12-hour electrical shutdown to the entire building. The only power will be emergency power.

Effects on ICS

ICS Co-Maintenance

ICS will be taking this opportunity to perform maintenance on multiple systems that would individually cause significant disruption for users.

Critical Infrastructure

Effects on Critical infrastructure will be not be affected bavailable during this period. That infrastructure includes:

  • ICS Home Page: http://www.ics.uci.edu
  • ICS Email Systems
  • ICS Home Directory (qualified with exceptions)
  • ICS Samba Service (IGB Samba will be unavailable)
  • ICS Samba Service
  • Many ICS hosted websites (contact helpdesk to see if yours will be available).

Plan on brief outages on several ICS systems during the day. A more specific schedule will be provided later. These items are required to implement network redundancy inside the data center and perform maintenance on systems:

Planned Maintenance

Summary

  • ICS NetApp Storage Outaages: Several service interuptions are expected to setup network switch redundancy.
  • ICS shared space: Move /net/tron/support0_tainer_sata/shared/ to net/cybertron/vol2/ics_shared
  • SGE: Central Spool data relocation
  • Subversion data relocation
  • Gitlab data relocation
  • Complete Data Center Reboot
  • Move MySQL
  • Move ugrad accounts
  • CentOS and Ubuntu OS Patches and Updates
  • New UPS installation
  • Bunker CentOS7 Upgrade

Gitlab Data Relocation

Copy file Update /etc/auto.gitlab

OS Patches and Updates

Begging on Monday morning, June 10th, we will be running updates on all CentOS and Ubnunt Linux clients. Beginning at 9pm, systems that have been updated will be shutdown.

Storage Shutdown

The following NFS servers will be unavailable beginning at 9pm on Friday the 17th.

  • bunker.ics.uci.edu
  • coachz.ics.uci.edu
  • gaia.ics.uci.edu
  • gigantor.ics.uci.edu
  • kraken.ics.uci.edu
  • nerve.ics.uci.edu

Scheduled Preparation Week of 6/13

Date Time Description
6/13 All Day Begin performing yum update on all non-infrastructure CentOS5 hosts
6/14 All Day Begin performing yum update on all non-infrastructure CentOS6 hosts
6/15 All Day Begin performing yum update on all non-infrastructure CentOS7 hosts
6/16 All Day Begin performing apt-get update on all Ubuntu 14.04 hosts
6/18 Midnight Undergraduate Home Directory Move
6/18 8am Disable all Nagios Monitoring Alerts
6/18 8am ICS 10Gb/s Network Upgrade
6/18 8am Baldig Group ome and Extra Directory Moves (/extra/baldig1, /extra/dock1, /extra/dock2, cerk)
6/18 9am SGESon of Grid Engine Shutdown and SGE Spool migration to Cybertron
6/18 9:15am Gitlab shutdown and directory migration
6/17 9:30am Move ICS Shared to Cybertron
6/18 10am (36 hours) Begin Rolling restart of critical infrastructure hosts
6/18 10am ICS MySQL Server Maintenance
6/18 10:30am Update Cybertron LDAP Configuration
6/18 11am Upgrade bunker to CentOS7 operating system with ZFS on Linux
6/18 10pm Shutdown non-critical hosts and infrastructure in the ICS DC
6/18 2pm IGB 10Gb/s Uplink Activation
6/19 5am We anticipate loss of commercial power and cooling to the data center
6/19 5pm We anticipate that power will be restored to the ICS DC. Some hosts will power on
6/19 10pm Power on hosts in the datacenter
6/19 10pm #59377: Fwd: Power on hosts for BIC
6/20 8am Restore Nagios notifications. Check Observium and Ganglia Reports
6/20 9am Rebuild SGE Queues
projects/maint-6-18-2016.txt · Last modified: 2016/06/29 11:58 by Hans
CC Attribution-Noncommercial-Share Alike 4.0 International
Driven by DokuWiki Recent changes RSS feed Valid CSS Valid XHTML 1.0