"the purpose of all that shuffling is really only to get the deck clear so we can concentrate properly on the hard bits without getting distracted by crud".
I think this is missing the point. Refactoring give us tools for rigorously transforming code. There are several reasons why this might be useful. One reason is that the programmer before us was an incompetent buffoon Another reason is that we work in and with teams, and collaborative work tends to get messy with time. We need to perform housecleaning so the technical debt doesn't cause us to go bankrupt.
These reasons are largely in line with what Mike is suggesting, I think. However, there's another reason to perform refactorings on a regular basis, and it is crucial to how we work as programmers.
The code you wrote six months ago (or six weeks ago) solved one particular problem pretty well. This is probably not the exactly problem you're trying to solve anymore. The requirements may have changed (accounting suddenly really _does_ want to run reports against itemized receipts), or maybe you just saw a better way to code the whole thing. Now you've got to pull all the affected abstractions up to the new world order. Classes start to fit together differently. Design patterns get replaced by different patterns, or disappear altogether. Engines suddenly find themselves servicing wholly new systems.
The way forward is with the refactoring tools. Instead of rewriting the system to conform to our new way of thinking, we can transform it. We can mold it so it continues to fit our needs. And we prefer this to a rewrite because it is less costly, we can deliver with only half of the transformation done (usually) without unduly impacting features, and the new designs practically write themselves.
I think "the hard bits" Mike references are exactly where refactoring shines. Making sure the performance profile is acceptable is a refactoring task. Ensuring the application is extensible through plugins is an exercise in refactoring. Extracting out the timer behaviors from the license module to be reusable is refactoring; so is reconciling the 3 different timers you just found in the system, because they were disguised as something else. Refactoring touches nearly everything we do, so we really need to make sure we do it well.