Democracy manifest: should senior Whitehall appointments be made on a political basis?

Boris Johnson wanted to make Lord Frost, a Conservative peer, his national security adviser but the civil service was horrified

It is a commonplace of current commentary on the Whitehall machine that there is a creeping politicisation of the civil service, and the assumption, or at least very strong implication, is that this is a bad thing. The most recent outbreak was in September, when Liz Truss (remember her?), within two days of assuming the premiership, dismissed the long-serving permanent secretary to the Treasury, Sir Tom Scholar. His had been a glittering career. Still only 53, he had served as private secretary to the chancellor, UK governor of the International Monetary Fund and the World Bank, Downing Street chief of staff and adviser on European affairs to the prime minister. But he had unexpectedly run out of road, suddenly the personification of the “Treasury orthodoxy” which the prime minister had spent the summer castigating.

The government did not hint that Scholar was incompetent, although Lord Agnew of Oulton, who had until recently been a junior Treasury minister, fulminated at the outgoing mandarin’s “foot-dragging and passive resistance” to creating an office in Darlington and his “botched arrangements” for bounceback loans during the pandemic. Scholar’s predecessor as Treasury permanent secretary, Lord Macpherson of Earl’s Court, was more in tune with majority opinion when he described him as “the best civil servant of his generation”, while the former cabinet secretary Lord O’Donnell observed that “sacking someone with no notice for no apparent reason—someone held in high regard by chancellors of all political parties—is no way to earn the respect of the Treasury and the civil service”.

The most serious charge, in some ways, came from the venerable Lord Butler of Brockwell, who had served as cabinet secretary and head of the Home Civil Service under Margaret Thatcher, John Major and Tony Blair and knows a bit about a robust attitude towards officials. “I think they are behaving improperly towards the civil service,” he told Radio 4’s The World This Weekend. “It’ll weaken them, but it’ll also corrupt our system, because one of those great advantages of having an independent, loyal civil service will be compromised.” There it was, the worst curse of the mandarinate: politicisation.

Scholar’s dismissal was only the latest in a series of supposedly political moves towards the civil service. A significant one had arisen in 2020, one which, as it turned out, occasioned so much concern and pushback from the civil service and its allies that the government had changed its mind and climbed down. But it is worth looking at in some detail, to see if the headline of politicisation is really as applicable and regrettable as it at first seems.

When Boris Johnson became prime minister in July 2019, he inherited, as any new premier does, a Downing Street team shaped substantially by his predecessor. There has always been a degree of leeway in this regard. While our civil service is an apolitical and neutral one, it is accepted that personal relationships matter too. A substantial step in the direction of politicisation had taken place in 1997, when Tony Blair had brought in his political adviser Jonathan Powell to be Downing Street chief of staff. This new post took over the running of the prime minister’s office from the (civil service) principal private secretary, the occupant of which at that point, Alex Allan, was placed below Powell in seniority. This was achieved thanks to an Order in Council which, exceptionally, allowed Powell, employed under contract as a special adviser, to give instructions to civil servants.

The incumbent cabinet secretary and national security adviser in 2019 was Sir Mark Sedwill, who had enjoyed a very close relationship with Johnson’s predecessor, Theresa May. Indeed, their partnership went back before Downing Street to 2013 when he had become permanent secretary at the Home Office when she was home secretary. He had been her “first and only choice” for the top civil service post, it was believed, but he failed to gel with the new prime minister, not helped by the publication of remarks by him months earlier warning that a “no-deal” Brexit would be a disaster for the UK. By June 2020, there was irresistible pressure on Sedwill’s position, and he submitted his resignation (though his letter to Johnson said very carefully that “we have agreed I will stand down”.

Johnson made the decision to separate the posts of cabinet secretary and national security adviser which May had combined to considerable criticism. An advertisement was issued for a new cabinet secretary and head of the Home Civil Service, open only to UK nationals who were or had been permanent secretaries; The Times ran an article profiling eight potential runners and riders, though it managed not to feature the eventual (admittedly surprising) chosen candidate. In August 2020, it was announced that the new cabinet secretary would be Dr Simon Case, recently appointed as Downing Street permanent secretary and before that private secretary to HRH the Duke of Cambridge.

Sedwill’s other role, however, was national security adviser, a post which only went back to the beginning of the coalition government in 2010 so had no great tradition about it and was still evolving. The prime minister announced in June that the post would be taken by David Frost, who would also be nominated for a life peerage. Frost was at that point serving as Johnson’s Europe adviser and head of Task Force Europe, a 48-strong team drawn from a number of departments and based in 10 Downing Street, reporting directly to the prime minister. The task force was created from the remains of the Department for Exiting the European Union, which was abolished when the UK formally withdraw from the EU, and Frost’s deputy was career diplomat Lindsay Croisdale-Appleby.

It is worth pausing to examine Frost’s background before judging his suitability to be national security adviser. Graduating with a first from St John’s College, Oxford, he had joined the Diplomatic Service in 1987 after two years with KPMG. In a 26-year career, he rose with moderate speed, achieving some important and influential positions: he was private secretary to the permanent secretary, the gravel-voiced and chain-smoking Sir John Kerr, “a glittering start” in one minister’s words; in between a series of EU-related jobs in the Foreign Office, he also served as economic counsellor at the British Embassy in Paris early in the millennium, a taste of a very prestigious mission. Somehow, though, his career lost momentum and it became obvious, to others and probably to him, that he would not rise to the very top of his service. In 2006, he was named ambassador to Denmark, his first (and only) job as head of mission, but, to quote former FCO minister Denis MacShane, “for an ambitious FCO official… a backwater’s backwater”.

After three years seconded to the Department for Business, Innovation and Skills as director for Europe, trade and international affairs, he left the Diplomatic Service and at the beginning of 2014 took over as chief executive of the Scotch Whisky Association, the trade body for whisky distillers. On taking over, he stressed his trade credentials, referring to “a wealth of experience of global commercial issues which will stand me in good stead in my new role”. His colleagues in the Diplomatic Service knew he was dissatisfied with his progression, and he will undoubtedly have earned more money in the private sector, but ambition obviously remained unfulfilled.

When Boris Johnson was, to some surprise and not a little consternation, appointed foreign secretary in 2016, he approached Frost, whom he had encountered in Brussels when he was a young journalist and Frost was a diplomat, to be a special adviser. As with his appointment to the SWA, it was Frost’s trade experience which recommended him; he had written an article in The Daily Telegraph shortly before being selected by Johnson in which he had dwelt on the complex nature and importance of a future trade relationship with the EU after Brexit.

He joined Liam Parker and Ben Gascoigne in Johnson’s political team, but was paid a hefty-for-SpAds £120,000, not very much less than the prime minister’s joint chiefs of staff, Nick Timothy and Fiona Hill, were earning. He was clearly not from the stereotypical special adviser mould, in his fifties and with a substantial career already on his CV, mostly as a civil servant in the department to which he was returning. But the idea of his returning to Whitehall had been discussed; he had described himself as “a fan” of Johnson’s, though had cautioned that “it’d have to be a pretty good offer to get me back into the task that we’ve just been describing [Brexit]”. After accepting the post, he had spoken to former Europe minister Denis MacShane, who had indicated surprise. Frost had responded, “Come on, Denis. Whatever you think of Boris, it is going to be an exciting time for British foreign policy making and it will be fun to be at the centre.”

It may be that Frost made a miscalculation. When Johnson arrived at the Foreign Office in July 2016, the department had just been stripped of many key responsibilities—including, critically, Brexit negotiations—to two new creations, the Department for Exiting the European Union and the Department for International Trade. While Johnson was famous or notorious, according to taste, for blunt, tactless and embarrassing remarks (though he was never the one who was embarrassed), and there were concerns within the Diplomatic Service about this tendency, conversely he had no strong convictions or considered positions on any major foreign policy issues which came across his desk. Having won a competition organised by The Spectator for an offensive poem about Turkish president Recep Tayyip Erdoğan in May that year, Johnson had no qualms about visiting Turkey in September, expressing support for the government and dismissing his verse as “trivia”. As ever there was something admirable about his sheer effrontery.

However impressive his trade and commercial experience, Frost must have found little into which he could sink his teeth as Johnson’s special adviser. Incautious remarks about Gibraltar and Northern Ireland after Brexit, a tour of the British Virgin Islands, being shushed by the British ambassador in Myanmar and a skirmish with Russia over the Salisbury poisonings were thin gruel, and it must have dismayed even a fan like Frost when Johnson responded to business concerns about Brexit in June 2018 with a simple “Fuck business”. The foreign secretary’s tenure ended as it was always destined to end when, after a cabinet away day at Chequers, he joined the Brexit secretary, David Davis, in resigning from the government.

It is the fate of special advisers that they live and die by the fates of their ministers. We should not feel too sorry for them: they know the terms and conditions when they take the job. To borrow a vaguely apposite phrase from ILP legend Jimmy Maxton, if you can’t ride two horses, you shouldn’t be in the bloody circus. But Johnson’s resignation meant the sudden end of Frost’s second act in the Foreign Office. He had, of course, advised Johnson to do it: after the Chequers summit, the foreign secretary had met his three special advisers, Frost, Parker and Gascoigne, at his official London residence, 1 Carlton Gardens, to assess how to react to the so-called Chequers plan, the white paper agreed by the cabinet on how to deal with the EU. All three told him that he had to resign. If he stayed in the cabinet, especially if David Davis walked, he would lose all the Brexit credibility he had built up since the referendum in 2016.

However, in one sense, the two years as special adviser had been transformative for Frost’s image and reputation. Oliver Lewis, who had been research director for the Vote Leave campaign and would later by Frost’s deputy in Downing Street, made some interesting observations in an article in 2021. He said that his view of Frost before 2016 was “something quite disparaging about the ‘Scotch Whisky guy’”. Frost had, after all, in his role at the SWA, spoken of the danger for the industry he represented of a hard Brexit. But, says Lewis, “it became clear very quickly that this was someone with serious intellectual heft—someone who had spent years thinking about, and advancing the Eurosceptic cause”. Indeed, when he was introduced to the ex-SpAd in 2018, Lewis was told he was “‘the secret weapon’, the person who would solve the Brexit crisis”. It is impossible that Frost would have enjoyed this status without those critical two years advising the foreign secretary. And it was not all bad news. According to the FCO annual report and accounts 2018-19, he received a severance payment of £30,000 when Johnson resigned.

Redemption, whether earned or not, came swiftly. At the beginning of 2019, Frost, without obvious employment, was appointed chief executive of the London Chamber of Commerce and Industry. The press coverage of the new role dwelt roughly evenly on his diplomatic and governmental experience on the one hand and his recent tenure at the Scotch Whisky Association on the other. But politics grows more volatile and unpredictable with every passing year. Johnson, having proved his Brexit faith in the flames of resignation, was now a very clear contender for the premiership if a vacancy occurred. That seemed increasingly likely, as an exhausted-looking Theresa May tried to hold together a fractious parliamentary party.

In March 2019, May offered herself as a sacrifice for her Brexit deal: she told the 1922 Committee of Conservative backbenchers that she would resign before the next stage of EU negotiations if her deal was accepted by the party. She wanted the deal approved by Parliament before the summer recess, so her departure would be in July. She then pledged to bring this forward to 30 June. But a regicidal party given a taste of blood will not be satisfied with delayed gratification. She had planned to publish the bill formalising her deal after the European elections on 23 May, but had to defer this date till June. It was too much. Andrea Leadsom, the leader of the House, resigned on 22 May. It belied Leadsom’s influence that she was the proverbial straw: May announced her resignation two days later, vacating the party leadership on 7 June. She would remain prime minister until a successor had been chosen.

The contest can be explored elsewhere. An extraordinary 10 candidates were nominated, three more indicated an intention to stand but withdrew before the first ballot, and another 18, from the vaultingly ambitious to the terminally unrealistic, expressed an interest in standing. (Ironically, in all the mayhem of the contest, I don’t remember ever reading the name of the young Richmond MP who was a bottle-washer at DCLG, Rishi Sunak.) The race boiled down to a ballot of members to choose between Johnson and his successor as foreign secretary, Jeremy Hunt. Johnson was revealed as the winner on 23 July and kissed hands as prime minister the following day.

As Johnson, so Frost: the former’s elevation brought the latter’s renaissance. On the day he entered Downing Street, Johnson reappointed Frost as a special adviser, this time in a much more elevated role. He was named Europe adviser to the prime minister and chief negotiator for exiting the European Union, effectively replacing that Brexiteer hate figure, Sir Olly Robbins. But Robbins was a civil servant, having started in HM Treasury before performing a number of national security roles in the Cabinet Office (including deputy national security adviser from 2010 to 2014). Placing his responsibilities in the hands of a special adviser was a move as extraordinary as had been Blair’s appointment of Jonathan Powell as chief of staff in 1997.

Martin Stanley, a former senior civil servant and wise observer of Whitehall anthropology, highlights the unusual nature of Frost’s appointment and how it just—in a characteristically Boris way—managed to skirt the outer limits of propriety. It “broke new ground”, says Stanley, but his “activities did not exactly break the Spad Code of Conduct”. (The use of the word “exactly” is illustrative.) He acknowledges that Frost “did not formally manage or instruct any civil servants.  But he was very powerful and had a very important job in which he appeared to make some policy decisions whilst remaining entirely unaccountable”.

This was the strange Whitehall no-man’s-land in which David Frost was operating when Boris Johnson nominated him as national security adviser in June 2020. The context is important: he was a man in whom Johnson had placed faith again and again, who knew the corridors of power well enough from his days as a regular to navigate them, but spoke with the conviction and bluntness of a political irregular. On Brexit, he had been a true believer. Johnson’s praised these qualities when announcing the nomination, describing him as “an experienced diplomat, policy thinker, and proven negotiator, with a strong belief in building Britain’s place in the world”.

Even more importantly, given the Dominic Cummings-inspired “hard rain” on civil servants, Frost had proven himself in that perennial political panacea, “delivery”. Johnson put great emphasis on this ability. The designated NSA “negotiated the deal that finally enabled us to leave the EU in January and in his new role I am confident he will make an equal difference to this country’s ability to project influence for the better”. He concluded:

I have asked David to help me deliver this Government’s vision for Britain’s place in the world and to support me in reinvigorating our national security architecture and ensuring that we deliver for the British people on the international stage.

The language was more emollient than Cummings would have employed, but the message carried the stamp of the prime minister’s adviser. “Britain’s place in the world” would inevitably be defined by Brexit, and “reinvigorating” was barely more than a code word for the hard rain. Change, reform, revolution: whatever word you chose, the old ways were unlikely to survive. It is unlikely that Johnson had even the most diffuse notion of what kind of changes might be made to the national security apparatus and the role of the NSA, but he was not there to provide the details. Frost’s nomination was about mood, atmosphere, vibes. To place a political appointee in such a senior and sensitive role, dealing with the state’s darkest secrets, was as provocative as could be.

Frost himself, in his official response to his nomination, echoed the disruptive hints of his boss.

My aim is to support the Prime Minister in setting a new strategic vision for Britain’s place in the world as an independent country after the end of the EU transition period, and in championing that vision as we strengthen our international relationships.

“New strategic vision”, “independent”, “vision”, “strengthen”. There is a danger in reading too closely between the lines, but the message is clear. Change. Frost continued:

To do this effectively we need to strengthen and refocus our international policy apparatus, to ensure that we keep pace with others in the world… Implementing the Integrated Review of our international capability, and making sure we use the National Security Council to drive its results, are also essential and I look forward to leading both.

One observation: Frost is staking a claim to “leading” the implementation of the Integrated Review, and he was looking well ahead. The review had not yet begun in June 2020, and the call for evidence was not issued until December. The project was formally announced in September, but it was a Ministry of Defence lead, and had only been unveiled as a concept in the Queen’s Speech opening the new Parliament in December 2019.

It was not surprising that the announcement of Frost’s nomination drew criticism from many corners. The former cabinet secretary Lord O’Donnell (who, as we have seen, would go on to condemn Liz Truss’s sacking of Tom Scholar) said, with mandarin restraint, that he was “worried” by the appointment. Frost lacked “deep subject knowledge”, O’Donnell concluded, and there was a danger that, as a political appointment who owed his elevation to the prime minister’s patronage, he might be a “yes-man” rather than an independent, impartial, authoritative adviser.

Other veterans made similar points. Lord Ricketts, who had been the first NSA, was anxious that Johnson was trying to place favourites in key roles and that Frost’s “main credential for the role is that he is a trusted political adviser to the prime minister”, while Lord West, security minister in the Labour government, admitted cheerily that he would no have put Frost high on a list of candidates. More dangerous, perhaps, were harsh words from Johnson’s own predecessor Theresa May, enjoying an Indian summer as a senior backbencher. The shadow home secretary, Nick Thomas-Symonds, asked an urgent question in the House on 30 June, leading with Frost’s appointment but taking in “other senior civil service positions”. Michael Gove, Cabinet Office minister, responded for the government. He began by lavishing praise on the prime minister’s nominee.

David Frost has served for decades in our diplomatic service. A former ambassador, he has also been director of the Foreign and Commonwealth Office’s policy planning staff and principal foreign policy adviser to the Foreign Secretary. He is now, of course, the UK’s negotiator, shaping our future relationship with the EU, covering issues from trade and tariffs to security and defence co-operation. As NSA, David Frost will help to deliver this Government’s vision for Britain’s place in the world, supporting the Prime Minister in reinvigorating our national security architecture and ensuring that we defend our interests and values across the globe.

None of this was flat-out untrue, though it sparkled with Goveian glitter. He went on to point out, fairly, that national security adviser was a relatively new post, carrying little baggage, and that the prime minister of the day was, up to a point, entitled to make whatever arrangements he or she wished. But he veered off into virgin territory when he tried to explain the status Frost would have as NSA, neither a civil servant nor a conventional special adviser.

David Frost’s status will be akin to that of a special envoy representing the UK abroad, speaking publicly and setting the agenda for policy making. He will not be a permanent secretary or a special adviser, and the civil service will support him in the same way as it supports any other political appointee: with objectivity, honesty, integrity and impartiality.

This smacked rather of post hoc rationalisation. It was fair to point out, as Gove did, that the first civil service commissioner, Ian Watmore, had agreed that the NSA role could be a political appointment. But May was not convinced, and, after paying tribute to the departing Mark Sedwill, she was blunt with Gove.

On Saturday, my right hon. Friend said: “We must be able to promote those with proven expertise”. Why, then, is the new National Security Adviser a political appointee, with no proven expertise in national security?

Gove, a stalwart at the despatch box who is adept at making the best of a bad lot, mounted a defence, but it was thin.

We have had previous National Security Advisers, all of them excellent, not all of whom were necessarily people who were steeped in the security world; some of them were distinguished diplomats in their own right. David Frost is a distinguished diplomat in his own right and it is entirely appropriate that the Prime Minister of the day should choose an adviser appropriate to the needs of the hour.

There was, in fact, a problem with May’s challenge, in that it conflated two different but connected which cannot usefully be answered together. One is general, the other specific. First, was David Frost a suitable candidate to be national security adviser? Secondly, should the post of national security adviser be given under any circumstances to a political appointee?

The first question is easier to dispose of. By the standards of the four men who had preceded him—Peter Ricketts, Kim Darroch, Mark Lyall Grant, Mark Sedwill—Frost lacked both general seniority in the civil service and specific expertise in national security issues. On the general point, Ricketts and Sedwill had both been permanent secretaries, while Darroch and Lyall Grant had held important ambassadorships. On the specific point, Ricketts had chaired the Joint Intelligence Committee and been coordinator of security and intelligence in the Cabinet Office in the early 2000s, while Sedwill had run the Home Office, with all its security and counter-terrorism functions, and been the senior NATO civilian representative in Afghanistan. Frost, by contrast, had inhabited the Copenhagen embassy, and worked mainly on trade.

Naturally, experience is not everything. Aptitude counts for much, and one can learn on the job. But there was little in Frost’s career to suggest that he was particularly adept at the sort of tasks which would fall to the national security adviser: threat assessment, management of competing priorities, balancing the needs of different agencies, understanding and predicting long-term trends in geopolitics. Overall, it is part of the NSA’s job to be a kind of referee for all the intelligence agencies and their advice, to be the prime minister’s gatekeeper for national security issues. It was simply not his skill set, or if it was there was no evidence of it. Quite the contrary, Frost had made his name for passionate devotion to the concept of Brexit, an uncompromising negotiating style and a generally brusque and no-frills demeanour.

If he had still been a civil servant, Frost would have raised eyebrows as a weak candidate for one of the most important official jobs in government. But then, if he had still been a civil servant, he would never have been anywhere near consideration for the post anyway. The truth is that judgements had been made about Frost by his senior managers in the early 2000s which had quietly but irrevocably determined that he would never reach the very highest levels of the Foreign Office. The civil service, like the armed forces, is adept at managing people upwards or out.

Let us look briefly at some comparisons. When Frost left the Diplomatic Service in 2013, he was 48. Not at the end of his career but with maybe a dozen years left. He was on secondment to the Department for Business, Innovation and Skills, as director for Europe, trade and international affairs, an SCS2 (Senior Civil Service) post. He had been at that grade at least since returning from Denmark in 2008. Where were the other people who served as national security adviser at that age? It is not a perfect comparison but it in indicative. At the age of 48, Peter Ricketts was chairman of the Joint Intelligence Committee; Kim Darroch was EU director at the Foreign Office; Mark Lyall Grant was high commissioner to Pakistan; and Mark Sedwill was FCO political director (actually a director-general level post or SCS3). Frost was, to be blunt, not in the same race.

If Frost was not per se an appropriate candidate to be national security adviser, that is a specific judgement on him. But it does not answer the wider question with which it was conflated: is it appropriate to have a political, non-civil service candidate appointed as national security adviser? This is a more difficult and multi-layered question.

In general, the UK political system maintains a clear and strict division between those roles carried out by politicians or their nominees, and those assigned to civil servants. The grey area tends to be small and consists of often short-lived advisory and consultancy roles. The most significant change in (relatively) recent history has been the creation of the Downing Street chief of staff role, which is generally now held by political appointees, though some have been more in the civil service tradition (like Dan Rosenfield, who held the office for a year between 2021 and 2022 and was, like Frost, ironically, a former civil servant who had left to work in the private sector.

There are some exceptions. We have a long-standing tradition of allowing political appointments to major embassies, though it is relatively rare. If we look at the early 20th century, for example, James Bryce had been a Liberal cabinet minister under Rosebery and Campbell-Bannerman, and then in February 1907 was appointed ambassador to Washington. He was a respected author and constitutional expert, having written The American Commonwealth in 1888, so he was well received by the US government. The Washington embassy has seen more “political” appointments than any other mission: Lord Reading, Lord Grey of Fallodon, Lord Halifax, David Ormsby-Gore, John Freeman, Peter Jay, all were drawn from outwith the ranks of the Diplomatic Service to fill one of the Foreign Office’s most senior positions.

There have been other “political” diplomats. Churchill sent Sir Stafford Cripps to Moscow in 1940, partly to get rid of him, but mainly because he hoped the Marxist-adjacent Cripps would find a rapport with the Soviet leadership. Duff Cooper, the staunch anti-Municheer, was ambassador to France from 1944 to 1948, a devoted Francophile who just managed to maintain a working relationship with General de Gaulle. He was followed by fellow Conservative Christopher Soames (1968-72) who had lost his seat in the Commons in 1966. Ivor Richard, a Labour minister who lost his seat in the first general election of 1974, was appointed permanent representative to the United Nations until 1979. Most recently, David Cameron’s Downing Street chief of staff, Ed Llewellyn, was given a peerage and the Paris embassy in 2016, before becoming ambassador to Italy earlier this year.

An even more unusual experiment was tried by the ever-innovative Harold Wilson in 1964. Hugh Foot (brother of Michael, John and Dingle), who had enjoyed a respectable career in the Colonial Service culminating in serving as the last governor of Cyprus (1957-60) was appointed to a unique dual role: Wilson made him permanent representative to the United Nations and a minister of state at the Foreign Office. He was also given a peerage, becoming Lord Caradon. He was an exceptionally able man, gifted at administration, a persuasive diplomat and an impressive speaker. His extensive experience in British colonies gave him not only profound empathy with populations longing for independence but also a feel for the delicate business of managing territories towards self-government safely and securely. (Macmillan mischievously described him as “a governor who ran out of colonies”.)

One of Caradon’s greatest achievements at the UN was the drafting and acceptance of Security Council Resolution 242 which condemned the acquisition of territory by conquest (it was drafted in the wake of Israel’s victory in 1967’s Six-Day War) and then set out five principles to work towards a peaceful solution to the conflict: withdrawal of Israeli forces, “peace within secure and recognized boundaries”, freedom of navigation, a just settlement of the refugee problem and security measures including demilitarized zones. It was unanimously approved by the Security Council on 22 November 1967.

What was the purpose of Caradon’s role? It seems in some ways to blur the line which we deem important between ministers and officials. But then, Wilson was rarely worried about administrative neatness, preferring experimentation and “creative tension” within his government. In fact, though, Caradon being both minister and permanent representative was very much in tune with the mood of the early 1960s. The UK government had embraced, or decided to make the best of, the process of decolonisation since Harold Macmillan’s “Winds of Change” speech in Cape Town in February 1960. By 1964, self-government had already arrived for a number of countries including Malawi, Kenya, Uganda, Jamaica, Tanganyika, Nigeria, Cyprus and Malaysia, and British administrators looked with superiority at France’s agonising and bloody process of withdrawal. The smooth process was a major progressive cause which was attractive to Wilson.

By giving Caradon both jobs, Wilson made him a kind of “minister for the United Nations”, and enabled him to be an ambassador from the General Assembly to the UK as well as the other way round. He was able to speak in the House of Lords when he visited London, and his ministerial rank gave him extra weight at the United Nations. Did the experiment work? Up to a point; Caradon was a successful and able ambassador who was widely respected in New York. But the achievements to which he could point were largely personal rather than institutional. He knew several of the newly independent countries, and easily established rapport with fellow diplomats. He also spoke Arabic.

Lord Hurd of Westwell, in his maiden speech in the House of Commons in 1974, cast doubt on the Caradon model. Hurd had served at the UK Permanent Representation under Caradon when he was a diplomat, and, while he plainly admired him, he thought the dual role had not always been helpful.

That is borne out by the experience of the noble Lord, Lord Caradon, who went there with a great reputation, which he still enjoys, great eloquence and great experience of the United Nations. Yet I wonder whether that experiment was a success. It seemed to me that Lord Caradon was constantly arousing, through no fault of his own, expectations which the Government at home were not always able to fulfil.

Although it was a Conservative-dominated government which introduced the position of national security adviser, having first developed the new architecture in 2006, there was a general feeling that something along these lines was inevitable and absolutely invaluable if government policy was to be coordinated effectively. In February 2010, the House of Commons home affairs committee published a report entitled The Home Office’s Response to Terrorist Attacks. The report recommended the creation of a National Security Council and the use of “prominent appointed National Security Advisers who could also be fully accountable to Parliament”. It is not entirely clear what the committee meant by “fully accountable”: officials are, of course, expected to appear in front of select committees (of both Houses) when appropriate , though under the so-called Osmotherly Rules, it is accepted that committees will not ask for specific, named individuals. But did the committee envisage something more than this?

Gordon Brown had also pledged shortly after taking over as prime minister that he would create a National Security Council. Yet he seemed curiously reluctant to appoint anything identifiable as a national security adviser. Indeed, when Sir Richard Mottram retired in November 2007 as permanent secretary, intelligence, security and resilience, in the Cabinet Office, he was not directly replaced and his responsibilities were allotted to two new posts, Alex Allan as head of intelligence assessment and chairman of the JIC, and Robert Hannigan as head of security, intelligence and resilience. This seemed centrifugal rather than coordinating. In addition, the media occasionally applied the description of “national security adviser” to the director-general of the Office for Security and Counter-Terrorism, a unit based in the Home Office. The person occupying that post was Charles Farr, who had spent his career in the Secret Intelligence Service (SIS) before taking over the OSCT in 2007. He was an extraordinarily secretive and hyper-cautious man, reluctant to appear in public and very reluctant to give public evidence to select committees. But as head of the OSCT his duties were primarily for domestic security and counter-terrorism, rather than a broader portfolio.

(I wonder if Farr, whom I met once, was ever considered for the post of NSA in 2010? Perhaps it would have been too public for his tastes, as a new government created new structures. He was extremely polite but one could sense the impatience with non-spook mortals being controlled by an effort of will, and I was not surprised to find out that he was “unclubbable”, the word which must have no direct equivalent in any other language, and that he’d had ambitions to be C, the head of SIS, as well as director of GCHQ and, somewhat surprisingly, permanent secretary at the Home Office, which I can’t imagine he’d have enjoyed. He might have been an excellent C, though he was appointed chairman of the JIC in 2015, at which I’m told he excelled.)

Are there any reasons why a prime minister couldn’t appoint a political choice as national security adviser? There would be some administrative issues: another order in Council would be needed to allow them to direct civil servants unless they were given formal civil service status, like outsiders who become permanent secretaries. A non-civil servant would also not be able, under current rules, to have oversight of the Single Intelligence Account, which is the budget Parliament grants collectively to MI5, SIS and GCHQ. These seem minor and essentially technical issues were are not insuperable.

However, as Dr Joe Devanny of King’s College London pointed out when Mark Sedwill was appointed NSA, it was open to David Cameron in 2010 to make his new adviser either a ministerial post, and therefore accountable directly to Parliament, or, as Frost would have occupied it, as a special adviser. Devanny observes that “This would have formalised just such a deep connection between the prime minister and the NSA, separating the post from the presumption of more formal civil service procedures for appointment”, though he goes on to conclude that Cameron’s choice was probably the right one.

I could spend a great deal of time examining other countries and their systems, to explain how their national security advisers are selected and how they operate. But that seems a task for another day and a more systematic approach, rather than cramming a few amusing anecdotes into the interstices of this essay. Suffice to say, comparisons can be informative but are limited in applicability if the country you are examining doesn’t use our kind of parliamentary system (essentially the best comparators will be Commonwealth countries). But you can draw more general principles from other countries’ arrangement, such as the role and responsibilities of a national security adviser and his or her relationship with other senior ministers or officials.

One of the most frequent arguments against a political appointment as NSA is that it would rob the post of its independence and its ability to give the prime minister unvarnished and sometimes unpalatable advice. These are certainly important aspects of the job, but I am unconvinced that only a member of the civil service has the odour of sanctity which allows this revered independence. The departures of Sedwill and Lovegrove (and possibly Lyall Grant) have already demonstrated that, in extremis, the national security adviser serves at the prime minister’s pleasure, so it would seem unlikely that an explicitly political appointment would be any more beholden for his or her tenure.

How far, after all, can the idea be taken before it tips the balance? Until now, all but one of the national security advisers (Stephen Lovegrove) have been mainly or entirely Foreign Office men (though Sedwill, inarguably the best qualified, had also spent most time away from the FCO, even running another department). Suppose the next NSA, after Barrow serves his time, were a former C: would that be acceptable? Or what about a former service chief, would that be OK? Or a former Home Office permanent secretary, or MoD chief, or senior police officer? Going further, what about a defence and security focused academic, someone from RUSI or Chatham House? Then the great step: what if there were an MP or peer who was extremely well qualified to advise the prime minister: Tom Tugendhat or Rory Stewart or, a generation ago, Paddy Ashdown? Would their status as parliamentarians rule them out, or somehow make them incapable of independence?

We seem to turn instead to a matter of character, not status. Is a candidate independent-minded, does he or she have integrity, honesty, duty? If the answer is yes, they might be suitable for NSA whether they are diplomats or MPs, but if the answer is no, it doesn’t matter if they come from the heart of the securocracy, they will not be effective in the role. When Frost’s name was announced, those critics who focused on his status as a special adviser were missing the point. He was unsuitable, and had his name withdrawn, because he was not temperamentally suited to the job of national security adviser and he had far too little direct experience of the issues. The format of his contract was a detail.

We have only had a national security adviser for 12 years. Sir Tim Barrow, if the new prime minister decides he has confidence in him, can reasonably expect to serve three or four years as NSA, and still have time for perhaps one last job, perhaps ambassador to Washington or even cabinet secretary. So the Whitehall machine may not have to consider candidates for national security adviser until 2025 or 2026, and it could be Rishi Sunak or Sir Keir Starmer or Penny Mordaunt or Angela Rayner making the choice. The precedent of the non-appointment of David Frost tells us, actually, very little, except that a shortlist should consist of men and women with substantial experience specifically in national security matters. But cast the net as widely as possible, find all the promising candidates that you can.

It would be a depressing irony if, at a time when the fashionable gossip is of opening up the civil service, engaging with the private sector and drawing on the opportunities presented by outside candidates, the selection for this one post, a very senior and absolutely critical one, were to move in the opposite direction and be reduced to a pool of a few middle-aged diplomats and maybe a senior Home Office civil servant. Let us be aware of what matters and what we can change or influence about public servants, and what is outside our control. Diversity is this too.