Developing the SAP Data Center : Data Center Physical Requirements

11/21/2011 9:22:45 AM
Physical requirements mean different things to different people. In the context of building your data center, I could go into minute details regarding actual construction of the facility, for example. In such case, physical requirements would exist with regard to the following:
  • Physical construction materials and related factors (fire codes, weight-bearing members for roof-mounted AC/Environmental Units, load-rating factors for the raised floor construction, cable risers or trays, and so on).

  • Location (avoid the first floor or top floor of buildings, and avoid flood zones, areas where access may be limited in terms of single stairs, elevators, roads, and so on).

  • Environmental systems (cooling, heating, humidity levels), including access to published thermal specifications for each component to be housed in the data center.

  • Accessibility (loading dock/freight elevator access, as well as double-protected public access points).

  • Physical security/access, including monitoring systems (card or other systems for doors and window access, and attention to vertical security above the dropped ceiling and below the raised floors).

  • Controlling systems (temperature/fire suppression, smoke, water sensors, and so on).

  • Lighting and plumbing, as required.

  • A central operations/monitoring station.

  • Access to high-bandwidth multi-path data communications circuits or network/Internet connections.

  • General dual power infrastructure, in terms of 208 volts AC versus other options, including the availability of generators, battery backup, and so on. This can also include access to two discrete city or state power grids, should high-availability or disaster recovery requirements dictate such a robust power infrastructure.

These details are best left up to the experts who design and build data center facilities. Ensure that at a minimum the preceding points are addressed, however.

Power Requirements

Power problems can plague an otherwise bullet-proof solution architecture and therefore power needs to be planned for the long term, not merely for the demands of Go-Live—your SAP environment will grow, grow, grow. When addressing the power needs of the SAP data center, it is helpful to first analyze each specific server, disk subsystem, network, or other hardware component requiring power, and then work “back” to the ultimate power source. For maximum availability, ensure the following:

  • Where the highest levels of availability are necessary, each hardware component must support redundant power supplies (otherwise, the remainder of this list is not of any use). Preferably, these power supplies should also be “hot swappable” or “hot pluggable.” In this way, in the event of a power failure, not only would the server remain available and powered up on its second power supply, but the failed power supply could be pulled out and replaced without incurring downtime.

  • Each power supply alone should be capable of keeping the hardware component up and running. That is, if the average load being pulled from one of the power supplies in a dual-power supply configuration exceeds 50% of its rated capacity, you actually don’t have protection from failure of the other power supply! The alternatives are clear—lower the capacity requirements by reducing the number of disk drives, CPUs, and so on, or increase the number of power supplies to three or more, or in rare cases simply replace the current power supplies with higher-rated alternatives.

  • Each power supply must have its own unique power cable. This is a very common oversight with some of the second-tier server and disk subsystem vendors, where highly available systems might be touted, but reality differs. These high-availability wannabes often provide only a single power cable receptacle even in their “redundant” power supply configurations. A single anything represents a single point of failure, and should be therefore avoided. Besides, I have actually seen a couple of power cables fail in the real world. As silly or unlikely as that sounds, it happens. And besides, even more likely is the potential to pull out a single power cable, effectively bringing down the most available server or disk subsystem.

  • As I indicated previously, color-coding or otherwise differentiating power cables makes it very clear to everyone when things are cabled correctly. The most common implementation of this involves a black cable cabled to the primary power supply, and a gray or white cable cabled to the redundant power supply.

  • Each power cable must be routed to dedicated separate power distribution units (PDUs, or power strips, and so on)—whatever is used by the company to centralize many power feeds into fewer larger-capacity connections. Each PDU needs to be analyzed to ensure that the load placed on this single component, should the other PDU fail, can still be addressed by the remaining PDU. And as I said earlier, the most common implementation of this is black cables to one PDU, and gray or white cables to the redundant PDU.

  • Each PDU must in turn be cabled to redundant uninterruptible power supplies, or UPSes. Like the PDUs, these need to be regularly tested and analyzed to ensure that they are indeed “redundant.” Note that a UPS tends to only be equipped to handle short-term power losses, thus necessitating our next power component.

  • Primary power for each redundant power supply should culminate in redundancy at the breaker boxes as well. That is, each power supply should ultimately receive its power from a dedicated breaker panel, like that illustrated in Figure 1.

    Figure 1. Here you see a completely redundant power infrastructure, from the servers/hardware components all the way back to the breaker panels.

  • The “back-up generator” is a necessity for mission-critical SAP shops. Whereas the UPS provides short-term relief from blackouts and brownouts, a generator can conceivably provide power for days, as long as fuel is available. Select a generator that runs on whatever is most easily accessible or readily available, including diesel fuel, propane, or natural gas.

It is of utmost importance that the generator and the UPS be properly sized to handle the loads placed upon them. Generators must leverage an Automatic Transfer Switch (ATS) to allow them to tie into both the power company (the power utility, or “utility power”) and the SAP data center, as you see in Figure 2. Critical systems need to be identified and earmarked for generator backup. These systems typically include emergency lighting, emergency environmental and safety controls, the critical SAP data center gear, and in some cases the contents of the entire data center.

Figure 2. Without an Automatic Transfer Switch, the actual usefulness of a backup generator is questionable, thereby impacting high availability.

Best practices also dictate that critical computing systems be isolated from main facility/operational power, but that each source of power “back up” the other. In this way, operational failures do not impact the enterprise system, and vice versa, and both are protected from failure by two power sources.

Using KVA for Accurate UPS Sizing

UPSes are rated by KVA or Kilo Volt-Amps. The formula to calculate KVA is amps × volts / 1000 = KVA. So if your rack is capable of pulling 69.5 amps × 240 volts = 16680 / 1000 = 16.6 KVA. Never allow your UPS to run above 80% capacity. 16.8 KVA is 80% of 21 KVA. So at a minimum, the rack should have 21 KVA worth of UPS.

Power Oversights in the Real World

One of my favorite enterprise SAP servers, the HP ProLiant, serves as my next example. A wealth of information is available on the ProLiant, in the form of something HP calls “quickspecs.” These technical specifications have been published and updated for years, and describe in great detail much of the minutia of little interest to anyone but hardcore techies once the SAP system landscape is in place. Prior to that time, though, these quickspecs fulfill a number of critical roles. First, quickly perusing this document reveals that the ProLiant DL760 8-CPU server will draw a maximum of 10 amps on a 208- to 240-volt line, while producing a moderate 5309 BTUs every hour. These little tidbits of information will help ensure that the data center facilities folks understand how many and what type of power circuits to run. The BTUs, on the other hand, should be fed into a simple model that will determine the minimum rating of the air handlers (cooling/heating system).

Another bit of information is provided in the quickspecs as well, the power plug connector. As with most servers, the power connector from the server to the PDU or UPS is ordered as an option with the PDU or UPS. The real challenge then becomes matching the PDU’s power plug with the appropriate power receptacle. Typically, either an L6-20P or L6-30P is called for—in the case of certain PDUs deployed at one of my particular customer’s new SAP data center sites, the L6-30P (a 30-amp circuit) was specified by the quickspecs.

However, the customer got ahead of themselves and in the interest of meeting deadlines for power, laid enough L6-30P power cables for the new Storage Area Network, too, which was to arrive shortly after the servers. When the SAN cabinets showed up, though, they couldn’t plug them in. The connectors were different—they required the L6-20P receptacles. To this day my colleagues and I are still not quite sure how our motivated customer managed this, but they actually forced their million-dollar SAN power cables into the wrong receptacles, and ran the system this way for perhaps a week before someone noticed that “something just didn’t look quite right” under the subfloor. Understand that these two connectors are quite different, and this little engineering feat will begin to sink in—they somehow managed to squeeze and twist the male plugs most of the way into the female receptacles. Not only did this pose a potential safety hazard, but they also risked blowing up their 20-amp SAN gear with 30 amps of juice.

They got lucky. The SAN had very few drives actually installed and spinning (and therefore drawing power) that first week. However, this simple oversight caused a one-week delay in the project plan, while the system was effectively shut down awaiting the proper wiring. In the end, the lost time was made up in the operating system and SAP basis installations, and neither gear nor people were any the worse. But the moral of my little story should be clear. Not only should the technical specifications for each piece of equipment be checked and verified, but we should never solve our power problems by brute force. Besides, because all of this information is just a click or quick-spec away, there’s really little excuse for misconfiguring or underallocating power and cooling resources.

Another common mistake illustrates how the redundancy of power-related components can be rendered useless through lack of attention to cabling and overall power architecture. My customer in this instance wanted to factor in redundancy at the physical layer of their SAP deployment. Their high-end servers, disk subsystem, and network equipment all supported redundant power supplies, so they took advantage of them. Each power supply on the back left side of each server and disk subsystem drive shelf was the recipient of a black power supply cable. This in turn was carefully routed along the left side of the rack enclosures to a power distribution unit dedicated for this purpose. Similarly, each power supply located on the back right side was fitted with a gray power cable, and these gray cables were also carefully routed to their own PDU. So far, so good—no single points of failure existed, in that half of the power supplies, cables, and PDUs could fail, and the system could still remain up and powered.

However, all of the careful preparation and planning that went into this phase of the project was tossed out the window after another few minutes of work. My customer plugged both PDUs into the same UPS, which then was cabled to two large redundant data center UPSes. Like the 1975 Chevrolet Corvette’s exhaust system, a dual-power approach to high availability is just “smoke and mirrors” after everything merges into a common pipe. The Corvette never realized its peak power potential that year, and my customer lost any hopes of achieving 100% reliability, even though the solution “looked good” from many angles. By plugging the PDUs into the same UPS, they defeated the purpose of redundant power—if their single-point-of-failure UPS were to fail, the entire system would grind to a halt.

Like power, the next layer in the solution stack also represents a basic necessity for supporting your SAP enterprise—cooling.

Cooling and Other Environmental Controls

One of the biggest causes of hardware component failure is heat. Luckily, planning for cooling requirements has become a lot easier with the popularity of the World Wide Web. That is, nearly every hardware vendor out there today publishes BTU/thermal specifications. Your job is then to simply pull down and “add up” these technical specifications on every piece of equipment you plan to deploy. Don’t forget to allow for future growth, either—with server and disk form-factors shrinking every year, the heat generated per cubic foot of data center space just continues to grow and grow. To conservatively address the next three years in your data center planning efforts, determine the average BTU output per cubic foot, and then double that number and apply it to any remaining floor space that could conceivably house incremental SAP gear. Don’t forget to factor in the fact that air might not move uniformly through your data center and any other cooling dynamics inherent to your facility. In doing so, you will be eminently ready for the day the VP of Operations says, “Hey! We’re gonna go ahead with that SAP PLM project, so make room for 20 new servers and a couple of SAN cabinets in the next few weeks.”

As with cooling, air handlers exist that allow for controlling and exhausting heat, monitoring and fine-tuning humidity, and so on. Ensure that new hardware additions to the SAP data center are plugged into the BTU model as soon as possible, so that any new requirements for cooling will be given an opportunity to be addressed. Air handlers and other large environmental gear like this require significant lead times when new components or upgrades/replacements loom in the future.

Don’t forget to load the proper OS-drivers or applets that may be required by your hardware system to shut itself down in the event of overheating or loss of cooling. In the case of the ProLiant, the Compaq System Shutdown Service shuts down the server when the heat exceeds a predefined threshold, acting in response to commands from the integrated management features inherent to the ProLiant server platform.

And consider some of the newer trends in air handling and monitoring. For example, HP recently developed a robot that literally rolls around your data center floor looking for hot spots. Upon finding a hot spot, the robot analyzes the conditions and may, for example, signal your cooling system to increase airflow to the area. Or it may instead communicate with your hardware systems to relocate workloads from one system to another. Utilizing a combination of these approaches, HP believes that it can reduce cooling costs for its customers in the neighborhood of 25 percent.

  •  Developing the SAP Data Center : Introducing the SAP Data Center
  •  Inventory of Broadband Phone Services
  •  Parallel Programming : Task Relationships (part 2) - Parent and Child Tasks
  •  Parallel Programming : Task Relationships (part 1) - Continuation Tasks
  •  BizTalk 2006 : Handling Ordered Delivery
  •  BizTalk 2006 : Implementing Dynamic Parallel Orchestrations
  •  Windows System Programming : The Registry
  •  Windows System Programming : File Locking
  •  SharePoint 2010 : Security - Secure Store Service & Using SSS with BCS
  •  SharePoint 2010 : Security - Claims Based Authentication
  •  SharePoint 2010 : PerformancePoint Services (part 2) - Using PerformancePoint
  •  SharePoint 2010 : PerformancePoint Services (part 1) - PerformancePoint Central Administration Settings
  •  Windows System Programming : Example: Listing File Attributes & Setting File Times
  •  Windows System Programming : File Attributes and Directory Processing
  •  Windows System Programming : File Pointers & Getting the File Size
  •  SharePoint 2010 : Business Intelligence - Excel Services (part 2) - Accessing Excel Services Over SOAP
  •  SharePoint 2010 : Business Intelligence - Excel Services (part 1) - Accessing Excel Services Over REST
  •  SharePoint 2010 : Business Intelligence - Visio Services
  •  Exchange Server 2010 : Perform Essential Database Management (part 3) - Manage Database Settings
  •  Exchange Server 2010 : Perform Essential Database Management (part 2) - Manage the Transaction Log Files
    Top 10
    Nikon 1 J2 With Stylish Design And Dependable Image And Video Quality
    Canon Powershot D20 - Super-Durable Waterproof Camera
    Fujifilm Finepix F800EXR – Another Excellent EXR
    Sony NEX-6 – The Best Compact Camera
    Teufel Cubycon 2 – An Excellent All-In-One For Films
    Dell S2740L - A Beautifully Crafted 27-inch IPS Monitor
    Philips 55PFL6007T With Fantastic Picture Quality
    Philips Gioco 278G4 – An Excellent 27-inch Screen
    Sony VPL-HW50ES – Sony’s Best Home Cinema Projector
    Windows Vista : Installing and Running Applications - Launching Applications
    Most View
    Bamboo Splash - Powerful Specs And Friendly Interface
    Powered By Windows (Part 2) - Toshiba Satellite U840 Series, Philips E248C3 MODA Lightframe Monitor & HP Envy Spectre 14
    MSI X79A-GD65 8D - Power without the Cost
    Canon EOS M With Wonderful Touchscreen Interface (Part 1)
    Windows Server 2003 : Building an Active Directory Structure (part 1) - The First Domain
    Personalize Your iPhone Case
    Speed ​​up browsing with a faster DNS
    Using and Configuring Public Folder Sharing
    Extending the Real-Time Communications Functionality of Exchange Server 2007 : Installing OCS 2007 (part 1)
    Google, privacy & you (Part 1)
    iPhone Application Development : Making Multivalue Choices with Pickers - Understanding Pickers
    Microsoft Surface With Windows RT - Truly A Unique Tablet
    Network Configuration & Troubleshooting (Part 1)
    Panasonic Lumix GH3 – The Fastest Touchscreen-Camera (Part 2)
    Programming Microsoft SQL Server 2005 : FOR XML Commands (part 3) - OPENXML Enhancements in SQL Server 2005
    Exchange Server 2010 : Track Exchange Performance (part 2) - Test the Performance Limitations in a Lab
    Extra Network Hardware Round-Up (Part 2) - NAS Drives, Media Center Extenders & Games Consoles
    Windows Server 2003 : Planning a Host Name Resolution Strategy - Understanding Name Resolution Requirements
    Google’s Data Liberation Front (Part 2)
    Datacolor SpyderLensCal (Part 1)