We've been using puppet for a year or so now, and have had great success. The positives of puppet and other configuration management solutions is fairly well documented by now, but I'd like to talk briefly about some of the more troubling aspects of the solution which I've not seen documented elsewhere up till now.
One of the key issues I'm currently battling with puppet (which should likely apply to the entire realm of dynamic configuration management) is the issue of sand-boxing the configuration for testing purposes. As an example, firewalls. When the time came to puppetise our firewall configurations (as in Linux iptables) we had the option to use the puppetlabs firewall puppet module, or to simply push the /etc/sysconfig/iptables save file out to disk. We thought about the options, and here's what we found.
For the dynamic solution, using code: * You gain the ability to look-up data dynamically * You can therefore sandbox the configuration on a testing machine, altering the IP addresses and such to match the test environment.
On the other hand: * You make a lot of extra work if you're machines aren't similar in configuration (ours aren't). * You lose the ability to simply read the diffs the files from what's on disk to what's being pushed out, which makes keeping track of changes more difficult using puppet's reporting features.
The issue is really one caused by being half way down the road to configuration management, but as the amount of dynamic configuration increases, so too does the amount of work needed to maintain it. The net result is that we (and i suspect many others) take a more static approach to configuration management in the early stages.
What I'm slowly realizing is that configuration management is viral. If you make one part of your configuration dynamic, you most likely will need to make more of it dynamic. At some point the effort needed to maintain a codified ability to stay flexible doing so overtakes the ability of a sole admin or small team to update the state of the code.
The net result of which is that configuration management should be thought through thoroughly before being embarked upon, and this needs to take into account your wishes to create testing environments and such from the same puppet code base. You need to think through where you can cut corners and stay static, simply pushing out files, and where you need the dynamic ability to change things on the fly.
This may just be teething pains... We've got most of the basic system configuration automated now, and we're just heading up the layers towards the more customized and one-off software. At the same time the advantages of repeatability inherent in configuration management is being slowly nullified, and so we've reached an impasse.
How much is enough, and what price are you willing to pay for it ?