Categories
Uncategorized

Virtualization and power use and spreadsheet games (updated)

Here’s a thought experiment. Suppose each server uses X watts idle and X+Y watts busy. If you have N programs that
currently run on N servers that are Z% idle then in the best case you really only need D= (100-Z)/100*N servers. So in a perfect world you have D machines busy all the time for D*(X+Y) power use instead of N*(Z/100)*X + N*(100-Z)/100 *(X+Y).


Something like this

N servers= 200
Z idle rate= 70
D need= 60
X idle watts= 150
X+Y busy watts= 800
Power use= 69000
Consolidated Power use= 48000 Savings= 21000

Ok that rocks, 21KW/hour. But can you really get that savings?  If program 1 and program 2 need to run at the same times: you can’t save anything on them[1] and then the virtualization itself adds cycles for purely overhead and increases memory which burns more power. Suppose we now add K% as the overhead measure – real need R= D+D*K/100. If  K=20, we cut savings in half. But 20 seems wildly overoptimistic to me, so put it at 50.  We’re still only needing 72 servers instead of 200, but they are running flat out so we actually increase power use.

K overhead= 50
R need with overhead= 90
Consolidated Power use with overhead= 72000 Savings= -3000

That does not rock.   But here’s the kicker – if we reduce idle power to 25Watts, maybe by turning off idle machines, a 50% overhead rate for virtualization means virtualization increases power use by 20KW. If we can reduce idle power to 1 watt, even at 10% overhead, virtualization increases power use.  So what are the real numbers? I have not seen any published studies (of actual data centers) – maybe there are some.

If you want a copy of the xls, send me an email. Maybe I set it up wrong.

http://www.hpl.hp.com/personal/Lucy_Cherkasova/papers/final-perf-study-usenix.pdf

This claims lower levels of overhead

http://www.engineyard.com/blog/2009/10-years-of-virtual-machine-performance-semi-demystified/

but I’m unconvinced. In particular, I’m curious about memory usage.

This is funny.

The benchmark itself reported its elapsed time by calling a function to find the system time at both the beginning and end of the benchmark.  The elapsed time reported by the benchmark was less than the wall-clock elapsed time.  What we hypothesize is that, due to the unrelenting CPU consumption by the benchmarks, the virtualization layer was unable to update its clock with the virtual CPU clock ticks.  This phenomenon is mentioned in [10] and [11] but we feel that this type of CPU workload severely exaggerates the situation.

http://www.cmg.org/measureit/issues/mit39/m_39_1.html

See the last letter in this

http://serverfault.com/questions/135431/is-virtual-machine-slower-than-the-underlying-physical-machine

While it is obvious that load is critical in any analysis, it may not be obvious how for example memory usage can depend on scheduling. If VM1 and VM2 run serially, memory usage is the max of the two, if they overlap it is the sum – unless it’s ok to slow them both down with more VM operations – which will, of course, increase the time to complete which uses capacity since that time is now not available for a third VM etc.

It’s also important to understand whether overhead is per VM. Suppose that we have 2% overhead per VM, all roughly the same size and 10 VMs. Is this overhead additive? Clearly cpu time is additive and so is I/O time, memory pressure is fuzzier and depends on how many VMs we run at any one time.

Notes

[1] If you have multiple cores, which you do, then if there are enough cores to run VM1 and VM2 in parallel, no problem. And this brings up the question of the relative power use, say of 2 dual core machines versus one 4 core machine or other multiples. Note that it’s hard to get power savings by turning off 4 of the 8 cores of a 8 core machine, but possibly easy to power down the one 4 core machine.

Categories
Uncategorized

Power savings via software

This press release is particularly fluffy, but whatever the reality of this very vaguely defined algorithmic development the basic message is correct

to validate nine terabytes of data (nine million million or a number with 12 zeros) in less than 20 minutes, without compromising accuracy.  Ordinarily, using the same system, this would take more than a day. Additionally, the process used just one percent of the energy that would typically be required.

The way to reduce power use in data centers is to improve the ratio of  (useful clock cycles)/(overhead + idle clock cycles) and to optimize I/O.

But this requires a 100% change in the way that software is designed from the current method.

Categories
Uncategorized

The Amory Lovins bottleneck

Lovins observes that power inputs in many industrial processes go into a bottleneck that makes power conservation hard if you start at the wrong end.  The power goes into a long pipeline of process that emerges on the other end with some useful (in theory) work. If you start on the power input end, then reducing power x% requires percolating incremental improvements down the chain of linked machinery with each step reducing work at the step further down the pipeline. But if you start on the other end, changes automatically flow upward. The same, obviously, holds true for data centers. If you start by improving power efficiency of air-conditioning – a good thing in itself – you cannot obtain the scale improvements that can be gained on the other end of the pipeline by reducing the activities that use power and generate heat. That is, if you can increase work-done/computational-steps you drive savings up the pipeline. And the kind of large scale savings Lovins achieves in other industrial processes seem plausible: if you reduce power demand at the work end enough to reduce the inputs of cooling needed so that a smaller air conditioning unit can be used, you have a potentially greater savings than by improving the efficiency of the air conditioning unit.

Categories
Uncategorized

Instructions per joule – a good start

… multi-GHz superscalar quad-core processors can execute approximately 100 million instructions per Joule, assuming all cores are active and avoid stalls or mispredictions. Lower-frequency in-order CPUs, in contrast, can provide over 1 billion instructions per Joule—an order of magnitude more efficient while still running at 1/3rd the frequency. Worse yet, running fast processors below their full capacity draws a disproportionate amount of power:

Dynamic power scaling on traditional systems is surprisingly inefficient. A primary energy-saving benefit of dynamic voltage and frequency scaling (DVFS) was its ability to reduce voltage as it reduced frequency [56], but modern CPUs already operate near minimum voltage at the highest frequencies. Even if processor energy was completely proportional to load, non-CPU components such as memory, motherboards, and power supplies have begun to dominate energy consumption [3], requiring that all components be scaled back with demand. As a result, running a modern, DVFS-enabled system at 20% of its capacity may still consume over 50% of its peak power [52]. Despite improved power scaling technology, systems remain most energy-efficient when operating at peak utilization


Andersen, D. G., Franklin, J., Kaminsky, M., Phanishayee, A., Tan, L., and Vasudevan, V. 2009. FAWN: a fast array of wimpy nodes. In Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles (Big Sky, Montana, USA, October 11 – 14, 2009). SOSP ’09. ACM, New York, NY, 1-14. DOI= http://doi.acm.org/10.1145/1629575.1629577

Once you start thinking about instructions/joule, you need to also think about work/instruction. This number is going to make current system look a lot worse than even the dismal summary above because the immense amount of overhead is so overwhelming. It’s not even just OS and Virtual machine overhead, as horrible as that it. Consider a Java program. Anyone who has ever looked at production Java code can testify that the number of layers of “frameworks” and “libraries” and so on between a program and actual computation is not small. So consider a program that reads data from the network, processes, stores in a DB, and sends responses. This is not an unusual program model. Ok: we begin with an interrupt that must navigate the virtual machine layer to some OS that most likely contains code that replicates most of the functionality of the VM layer, then  layers of protocol fiddling, the sad fumbling over locks in the a “fine grained locking” OS, the clumsy process model and atrocious scheduling left over from the days in which CPUs were scarce resources, the IO layer (DEAR GOD!!), the pointless copying of every bit of data over and over again, and then the agonizingly bumbling progress of the intent of the programmer dripping down through geological layers of Java to a byte-code interpreter – and that’s just the start of this slow motion avalanche: electric power is being converted into waste heat with appalling recklessness.

I’m not sold on the “Fawn” model, but the problem analysis is right on the button.

Categories
software business

Amory Lovins on smart engineering

Data centers show up around minute 24.

[youtube=http://www.youtube.com/watch?v=28RmpedUYLk&w=425&h=344]

Categories
software security

Security for dummies – a lesson for smart grid

WASHINGTON — Militants in Iraq have used $26 off-the-shelf software to intercept live video feeds from U.S. Predator drones, potentially providing them with information they need to evade or monitor U.S. military operations.

Senior defense and intelligence officials said Iranian-backed insurgents intercepted the video feeds by taking advantage of an unprotected communications link in some of the remotely flown planes’ systems. Shiite fighters in Iraq used software programs such as SkyGrabber — available for as little as $25.95 on the Internet — to regularly capture drone video feeds, according to a person familiar with reports on the matter.

So what are the lessons. First, “security by obscurity” is equivalent to “hope is a plan”

The potential drone vulnerability lies in an unencrypted downlink between the unmanned craft and ground control. The U.S. government has known about the flaw since the U.S. campaign in Bosnia in the 1990s, current and former officials said. But the Pentagon assumed local adversaries wouldn’t know how to exploit it, the officials said.

Whoops!

And second : use standard protocols

Predator drones are built by General Atomics Aeronautical Systems Inc. of San Diego. Some of its communications technology is proprietary, so widely used encryption systems aren’t readily compatible, said people familiar with the matter.

Categories
software business

Green energy and smart devices

windpower5We’re starting to see a confluence between IT and energy that will change both industries. A windmill power data center is an interesting data point. At some time, we’re going to want to control the energy generation from the data center – for example, to run big batch jobs when the wind is blowing or to generate more power during peak billing periods or to shut down unnecessary heat producing computations during low energy periods. As smarter technologies become available for generating power from waste heat, and as carbon generation costs become integrated into prices for purchased power, the whole economics of running data centers will change and the data center will have to act like an intelligent factory – producing compute time against costs of heat production and power consumption. As we get there, we have to understand that one of the most important properties of the Internet comes from its “end-to-end” design.  Earlier networks suffered from the problem of being designed as layers, but the internet protocols and hardware were designed to solve the problem of moving streams and packets around networks of machines – considering the problem in totality, not as a set of layered components.  Modularity is not incompatible with end-to-end, but end-to-end requires an understanding of the applications and is incompatible with the component supplier view that dominates modern computer systems development.

Categories
software business

Cleantech and history of software

I’m on my way to Linz, Austria to participate in a panel at  an event called Cleantech Venture Forum. We’ve spent a lot of the last two years looking at ways to save power in data centers and mini-data centers (which may be a term nobody else uses).  If we look at the history of software, the key thing to optimize changes over time:

  1. Alan Turing Era – minimize processor time and memory use (machines are few and very expensive)
  2. Gene Amdahl/Seymour Cray Era – maximize throughput (big batch jobs in expensive machines)
  3. Gordon Bell Era – maximize responsiveness (suddenly we have terminals and impatient people screaming at them.)
  4. Bill Gates Era (client server era) – minimize programmer time and maximize functionality.  This is the era marked by three inspiring principles:

    1. Can’t we get these 4000 poorly educated, poorly treated code  monkeys to produce something valuable?
    2. No matter how crappy this is, it will work better after more memory and more processors.
    3. Get it out the door and let the customers debug it (or “Ready, Fire, Aim” as a former boss used to put it before our company went bankrupt!).
  5. The Al Gore Era -  power use/functionality becomes important.

Something like that.