Many academics feel privileged to be working in a university. “Knowledge workers” like us, it is often said, are intrinsically motivated: driven by the internal rewards of their work. This contrasts with those who might be more motivated by extrinsic factors, such as pay or other awards.

But, as we all know, even the most attractive jobs in the world can be spoiled by unpleasant colleagues and, in particular, by a difficult boss. And this matters. How satisfied you are with your job does not just affect your own sanity: there is a growing body of research showing that employee job satisfaction matters both to an individual’s productivity and to an organisation’s performance.

The evidence from our previous work suggests we need bosses who are experts in the “core business” of the organisation. This means people who are knowledgeable about the sector because they have spent many years in it, and who excel at the underlying activity itself. For example, research universities that are led by distinguished scholars outperform those led by presidents who either came from non-academic backgrounds or who went into administrative roles early in their careers, giving up on their research.

In work done with economists Larry Singell at Indiana University and John McDowell at Arizona State University, we have also seen this pattern replicated inside universities at the department level. Economics departments led by heads whose own research was well cited went on to be the most successful over a 15-year period. These studies use longitudinal data and adjust for a number of influences on people’s working lives. They build on work by sociologists Frank Andrews of the University of Michigan and George Farris of the Massachusetts Institute of Technology, who found as early as 1967 that the best predictor of a researcher’s creative performance was the leader’s level of technical ability.

This “expert leader” effect has been found well beyond the ivory tower. In a 2008 study co-authored with Larry Kahn of Cornell University and Andrew Oswald of the University of Warwick, one of us showed that the success of a basketball team was strongly associated with having a coach who was himself an outstanding player. And work with Ganna Pogrebna of the University of Warwick in 2012 showed that Formula One teams led by people with long previous driving careers outperform the rest.

In terms of job satisfaction, the link between that and having an expert boss was found in a recent study one of us carried out with Oswald and Ben Artz of the University of Wisconsin. Based on a sample of 35,000 randomly selected employees, matched with their employers, we found that the characteristics of supervisors are the single strongest predictor of employee well-being, overriding all of the more conventional influences on people’s job satisfaction, including education, earnings, length of time in post and the type of work done; indeed, the effect of boss competence had twice as much of an effect on employee job satisfaction as pay did. The key issues are the supervisor’s level of technical competence, whether the supervisor could do the employee’s job, and whether the supervisor rose up through the company (or owned it).

Two questions motivated our latest, as yet unpublished, study of 700 academics and their department chairs in research universities, mostly UK Russell Group (80 per cent) and Swedish institutions. First, could we replicate the above finding on job satisfaction, showing that leaders make as much difference to academics as they do to random employees? And, second, could we also start to explain how they have this effect?

The results are conclusive. Controlling for age, gender, tenure, discipline, position and even overall life satisfaction, we have found that academics report considerably higher levels of job satisfaction and morale when their head of department is either a “distinguished researcher” or a “highly distinguished researcher”. We have also found that having a distinguished head of department makes the academics less likely to move on.

The reason is that heads of departments who are themselves active researchers are more likely to create the right work environment for their faculty colleagues, protecting them against an encroaching managerial culture. Our survey respondents said that those heads they judged to be distinguished or highly distinguished researchers gave better feedback, consulted more effectively and, in general, were more understanding of the nature of academics’ work.

So we need good scholars to take on leadership positions such as department chair, dean, provost or president. The problem is that, as is the case with all experts and professionals, going into management is infinitely less appealing to a successful academic than doing the day job.

So we ran a small pilot study that asked faculty in a single university what might incentivise them into taking a management role. Our sample of 100 responded from a pool of 420 academics submitted to the last UK research excellence framework. Among women, 88 per cent rated “a reduction in other responsibilities such as teaching and administration” as the most important factor. For 80 per cent of men, the key incentive was a pay rise. However, pay was also very important among women, with 85 per cent listing it as a factor.

The third most common stipulation, from both men and women, was for the leadership role to be considered positively towards their promotion to professor. Women also reported that they may be more interested in taking on a shared leadership position. This seems to be an important consideration for increasing female participation in leadership. Taking on shared roles may seem less burdensome and help build confidence; there is quite a lot of evidence suggesting that women are less happy than men to compete for these kinds of roles, and may be less confident even when they are more skilled than men.

When asked about the disincentives to taking on a leadership position, 87 per cent of women and 72 per cent of men cited the resulting reduction of their research time. The second most common factor for both genders was the administrative burden it would create. But both genders overwhelmingly said they would be more willing to consider a position if they were asked by a senior colleague to apply for it.

The need for prospective leaders to undergo leadership training only makes the prospect even less attractive to the average academic. We have probably all received emails detailing the various courses available. The UK’s Aurora programme, offered by the Leadership Foundation for Higher Education, is one that targets women (who, of course, are underrepresented in senior roles). But its unattractiveness to academics is underlined by the fact that 80 per cent of last year’s intake at City, University of London, for instance, were drawn from professional roles. This year’s selection process required asking a manager and work colleagues to fill out a 360-degree assessment consisting of 131 extraordinarily detailed questions about the applicant: the idea that academics would know information sufficient to answer even five of these questions about a colleague brings a wry smile to the face. We barely know this stuff about our partners, husbands and wives.

When we asked what a “good” leadership programme would look like, the top priority of an overwhelming majority of women was one that excluded professional staff. This should not be viewed negatively; academics and professional staff have very different career trajectories and so need to be managed differently. Academics may feel more relaxed about discussing certain issues with other academics, and professional staff will undoubtedly feel the same. Both men and women said they would also prefer the training to be delivered in an external location – away, perhaps, from prying ears. And both men and women said they would be more likely to sign up for a training programme if other responsibilities were reduced and, once again, if it were viewed positively towards their promotion. Women were particularly keen to learn about “how to have a difficult conversation”, while men’s priority was to learn “how to manage people”. Both genders were keen to have a coach or mentor. But, again, women were particularly concerned about losing research time. This sense that they have much less spare time than men may partly account for their scarcity at the top of universities. But as research productivity is key to academic promotion, it is important that female faculty do maintain their research productivity. If they don’t, not only will we have few female leaders, we will also continue to have a shortage of female professors.

One of the things that worries academics contemplating going into management is the popular comparison of doing so with succumbing, Darth Vader-style, to the dark side of the Force. This identity crisis is also well documented among other professionals, such as doctors.

At the same time, heads of department in particular have lost power over the years as those above them in the hierarchy have gained it. Much of the power acquisition by senior management over the past 20 years has been necessary, but a common view abounds that the pendulum has swung too far. For instance, in a 2016 paper published in the journal Prometheus, Ben Martin, professor of science and technology policy studies at the University of Sussex, sets out the view that universities have become overly hierarchical, with centralised, top-down management structures, increasingly bureaucratic procedures, teaching to a prescribed formula, and research driven by assessment and performance targets. His paper, “What’s happening to our universities?”, also says that departmental autonomy has been greatly weakened; this may partially explain some of the recent faculty unrest at University College London about restrictions on expenses. It may also explain academics’ reluctance to take on a head of department role that has only diminished in influence as it has grown in bureaucratic burden.

Martin points to the irony of the fact that as universities have striven to become better managed, they have actually been moving in the opposite direction to that which their own business schools’ research suggests they should take. Current thinking in management studies is that success comes from a decentralised model, allowing for local autonomy and decision-making that is both more appropriate and, if done right, more efficient than diktats from the centre. For example, a head of department who fears the imminent departure of a star faculty member needs to act fast. If she or he has to go through two months of bureaucratic process to offer that person a higher salary or improved conditions then they might as well wave goodbye to them. Such situations are made all the more infuriating by the fact that hiring great faculty requires many hours of investment by department chairs.

In this environment, the university leader has a direct influence on the head of department’s job satisfaction – and their willingness to take up and continue in the position. That influence will also partly be exercised through their role in appointing all the pro vice-chancellors and deans that stand between them and the head of department. It is important that these individuals, too, be good researchers – but only presidents who are themselves good researchers are likely to recognise the importance of that, and to act accordingly.

Yet even when presidents have been good researchers, it is easy for them to forget what it feels like to be part of the rank and file. Hence, when they push more demands (often from the government) down through the system, they cannot easily appreciate what it is like to be the head of department on the receiving end of them. Perhaps the time has come to offer university presidents “reverse mentors”: heads of department who they can shadow and consult to remind themselves what being at the administrative coalface is really like.

Arguably, university leaders have extremely challenging jobs, which are unappealing to many. One of those jobs is to try to motivate academics to complete annual surveys assessing job satisfaction. But by taking a few conceptual steps back into the chair of department’s position, they might just find that those satisfaction rates rise far higher than any whizzo new initiative from the centre could hope to aim for.

Author Bios: Agnes Bäker is assistant professor in the department of business administration at the University of Zurich. Amanda Goodall is senior lecturer at Cass Business School, City, University of London. The authors are grateful to the Leverhulme Trust for funding this research.