"Men, it has been well said, think in herds; it will be seen that they go mad in herds, while they only recover their senses slowly, and one by one." (Charles Mackay)
The majority in government, in regulation, and in the financial services industry were supported in their rosy view of the 1990s financial world by the fact that all 'the great and the good' shared their optimistic analysis of corporate behaviour and of the virtues of 'light touch regulation'. This was despite the fact that there were nevertheless a good number of perceptive commentators who did their best to warn of the forthcoming catastrophe. This is a good example of two very common psychological behaviours, often called 'herd behaviour' or 'group-think'.
Herd behaviour is not necessarily irrational. Herds usually run together for a good reason, and there is nothing irrational about humans seeking to watch and learn from what others are doing. Rational investors in an efficient market will therefore produce frenzies and crashes from time to time - but policy advisers are surely responsible for ensuring that their organisation does not follow the herd over cliffs and take lots of innocent customers, taxpayers and/or staff with them.
Group-think is more pervasive but also highly undesirable, for it is essentially the unquestioning acceptance of obviously wrong answers simply because it is socially painful to disagree. Large organisations require a good deal of conformity, of course. You can't have every single person asking questions. Decisions have to be made and implemented. And if everyone you know, every newspaper you read, every person you once admired, are all saying the same thing, it takes a real effort of will, and real courage, to argue back. But someone has to do it - and yet it is uncomfortable to be a contrarian. Indeed, dissent can lead to the dissenter becoming the subject of personal and sometimes humiliating attack. Camilla Cavendish (who had run Prime Minister David Cameron's Policy Unit) told the Today Programme in December 2016 that some officials "became very very angry and took it viscerally personally" when it was suggested that there might have been a better way of negotiating with the rest of the EU in advance of the 2016 Brexit referendum. Questioning their strategy was perceived by them as questioning their commitment.
For further information, you might like to read Mannie Sher's paper on The Psychology of Regulation, the first part of which deals with this subject.
The British reluctance to offer overt challenge or criticism doesn't help. Industrialist Sir Denys Henderson was famed for his unforgiving tongue but appears to have had little impact as a non-exec of the Board of Barclays Bank. One scholarly director told him "You are essentially an oratio recta [direct talking] man but Barclays is essentially an oratio obliqua [indirect talking] company." No doubt this contributed to Barclays' subsequent troubles. (The Senior Civil Service, too, suffers from too much indirect talking.)
Margaret Heffernan offers an interesting - almost hilarious - analysis of group-think in her excellent book Wilful Blindness. "I've even heard boards discuss how, and why, they are invulnerable to groupthink, oblivious to the irony inherent in their confidence. ... Dennis Stevenson, then chairman of HBOS, eulogised the outstanding board he chaired [at a time when] everyone knew the bank teetered on the edge of collapse. ... [Lord Stevenson cited as evidence] the fact that, even in this crisis, 'we are as one'. He seemed oblivious to the notion that the unity of his board may have been a contributory factor to the bank's mess in the first place."
The same book introduced me to the related concept of cognitive dissonance. This phrase refers to the stress that we all encounter if we try seriously to consider two incompatible views at the same time. We all therefore fiercely hold onto our preconceived beliefs even in response to intense external pressure. One result is that - as we almost all regard ourselves as good people, and essentially honest - we find it very hard to admit to ourselves that we or our organisation is behaving badly. Senior executives typically therefore push back very hard against any regulator's or other suggestion that their actions maybe doing unnecessary harm, including risking their own business.
Cognitive dissonance almost certainly accounts for the failure of many, including Alan Greenspan - a deep believer in the power of the markets, and a fan of financial instruments such as derivatives - to react to all the warnings that something was going very badly wrong in the period before the 2008 financial crisis.