As a backroom boy, Neil Kinnock’s chief of staff or even an education minister, his image didn’t much matter — indeed it added to the impression of a fearsome troubleshooter. But as a member of the smartest cabinet since Chippendale, the chairman is not quite in the image of his maker.

And now it seems that the knives are out. Last week he had just returned from the TUC conference in Blackpool, a foray into enemy territory if we are to believe the gossip.

In theory, a man with his no-frills style ought to be a splendid negotiator with unreconstituted union lads nostalgic for ballots, beery intrigue and deals brokered on the back of fag packets. But he has never been a lefty, or a faux-prole; even as a Cambridge student and National Union of Students’ president he was a moderate Tribunite; he has always been associated with the modernising project, if not its sharp tailoring.

Now, with strikes threatened and funding scrutinised, the unions are his big problem. They feel ignored, I suggest, they stump up to get the party elected and get no favours. “They may feel ignored, but they are not justified in feeling it,” he replies. With cheeky aplomb he puts the unrest down to higher employment under Labour and a corresponding increase in union membership; and to Labour’s fearless reform of public services, which demands that people change and therefore annoys them.

But shouldn’t he be providing aspirin for the migraine Derek Simpson of the Amicus union has threatened to give the government, rather than scoring points? In the same job Tony Blair would be at least pretending to feel their pain.

As party chairman since the last election, Clarke is charged with a raft of responsibilities (funding reform, election spokesman, party revitalisation) that depend largely on good relations with sometimes alienated supporters. In essence it is a public relations portfolio, and one wonders if he is temperamentally suited to it.

“Keeping people as friends is not the important thing,” he says of his relationship with union leaders. So Labour and the unions no longer need to be friends? “You can have a working relationship without being friends.”

This attitude might explain the whispering campaign. For the past few weeks anti-Clarke stories have appeared, best of which is a claim that he barged into a private dinner at Pizza Express, denounced John Prescott and Gordon Brown to their parliamentary private secretaries, and scoffed two pizzas.

I expected him to shrug off this monstrous slur, but the element of truth means he can’t; besides it was not the accusation of disloyalty that needled him but the hint of poor table manners. “I can never recall ordering two pizzas. I might eat one, and then feel like eating another. But to order two would seem a basic discourtesy, like claiming one knew that one wouldn’t be enough.”

Did he say Prescott was useless? “I really can’t recall.” Is he of that opinion now? “Not really.” Not really? And that Gordon would never lead the party? “I don’t remember . . . All that was over two years ago so it’s extraordinary it has been recycled now.”

Doesn’t it mean someone is out to get him? “I don’t think it’s a Prescott-Brown plot, it’s not how they operate. There are people who are cross with things I’ve done or said but I don’t think the unions dislike me. Besides, a paranoid politician is one who can’t get things done, so I intend to remain confident and optimistic.”

He is bolstered by his wife Carol and sons aged 12 and 15 whom he tries never to mention in public. The son of Sir Richard “Otto” Clarke, a Whitehall permanent secretary, he attended the private Highgate school, London, and read mathematics and economics at Cambridge. Affluent, intellectual and well-connected, then. Yes, but upwardly mobile, he quickly adds: his mother was the first person to go to university “from her valley”.

How does he define his class? “All politicians are upper middle-class, well-paid people. John Prescott is. Didn’t he have a row with his father about it? We’re all middle class.”

He denies that a caucus of disaffected lefty MPs ever attempted to propose him as deputy prime minister or that he wants the top job. To illustrate his lack of ambition, he says that having tried for a seat in 1983 he ignored other chances because his work in Kinnock’s office was more important for the cause. Believe me, if he had a shot at the big time, he’d grab it.

We are talking in his party’s new Westminster headquarters where his office is on the top floor. The lift is out of action, but since he arrives without a wheeze he must be fitter than he looks. He is one of only two cabinet members to have dispensed with the ministerial car.

He has never been advised to shed stones for presentational enhancement but has often tried for the sake of his health, and is oddly sensitive about his weight, murmuring that I might be “polite enough not to mention it”. No chance: the unmistakable silhouette of the Westminster operator, rotund, powerful, with the sort of unruly ears for which schoolboys get tormented, is very much the point.

He denies that he is abrasive, rough-tongued or a big boozer — three units a day he promises, ho-hum. “If I am, it comes from impatience. I’m a politician, I live in a political world, I have to be concerned with what is politically achievable as opposed to what is desired. But we all have our weaknesses.” Even the boss, he seems to imply. “There is no perfect person to lead us, there are right people for different times, that’s all.”

One can’t help contrasting this with his roaring enthusiasm for the prime minister Kinnock would have made. “Absolutely outstanding. He had the passion, purpose, decisiveness and courage without which the party would have fallen apart after ’79 and ’83. But Tony speaks for middle England in the way a Celtic politician never can.”

Clarke has been about as abrasive as treacle sponge but he can infuriate without even knowing it. When he talks, the mouth moves at a sprint, producing an effortless flow of reasonable-sounding sentences that defy interruption, but you feel the brain might be engaged somewhere more interesting, that there is nothing you could ask or observe that he couldn’t deal with in his sleep.

At the end, he announces that this had been a “relatively routine” interview — clearly pleased I have been efficiently dispatched — for which I could have cheerfully choked him with a quattro stagione or two. If that is the party conciliator’s idea of a charming adieu, don’t hold your breath for happy families.