Why do we still have digital skills gap" when Harold Wilson identified both problem and solution?

1.    Until earlier this year we were trapped in Groundhog Day

There have been regular enquiries into shortages of what we now call “digital skills” for 50 years.  A cyclical pattern was identified in the 1980s. Each recession after that of 1979 – 83 accelerated the decline in demand for old “digital skills” skills and delayed investment in training for the new generation of “digital” skills that as taking their place. Each recovery (from 1982 onwards) saw a “crisis” and another attempt “to predict the unpredictable” in order to better target public funded education and training. 

Training levies and grants were scrapped in the 1980s as a bureaucratic waste of money which did not lead to more training but proposals to reward employers for recruiting trainees and retraining existing staff were regularly turned down by those whose status depended on dispensing public funds via hierarchies of committees. [see the blog continuation, section 5 onwards, below for a potted history, with links].

The biggest obstacle to employer organized training programmes is still the cost, particularly staff time, to organise supervised work experience before trainees and/or apprentices can be trusted to work on their own. Hence the need for generous tax incentives (e.g. exempting apprentices from National Insurance and Income Tax), on top of good quality mentoring support via local training providers, to help cover that cost. The current limited exemptions for low paid level 2 apprentices are a good start but the approach needs to be extended to those on higher level contractual training programmes (e.g. those in line with the main test case Strathclyde Regional Council v. Neal  )   

2. Why do employers  import contractors instead of hiring/training Britons?
Tax, Tax and Tax.    

Until about a year ago it was more economic for most employers to compete for skilled staff or import from overseas, rather than train their own. Now it can be cheaper to train than to recruit – but only if you can find those able to help you deliver relevant and well-targeted blended learning modules when and where you need them.  No attempt has yet, however, been made to level the playing field between those who recruit trainees and retrain existing staff and those who import supposedly skilled “contractors”.  We still appear to have a situation where some of the latter can be paid tax free allowances for travel and accommodation and exempted from national insurance up to year . This enables their employers to save 50% (sometimes even more) compared to UK staff or contractors with equivalent take-home (after tax and expenses) earnings.

In looking at big business bleats about attempts to control the Tier Two visas used to import IT skills in supposedly short supply, it is as well to note that over 36,000 of the 60,000 in the UK at this time last year were from India and nearly 7,000 were from one company alone 
apparently outnumbering their indigenous workforce by about 2:1.        

Last year, in my evidence to the House of Lords Select committee, I said that we needed to copy our overseas competitors in exempting employees following professionally and technically accredited training programmes from income and payroll (c.f. National Insurance) taxes and allowing individuals acquiring new skills, not essential to their current jobs, to offset the cost against current and future earnings. We also needed the same tax and expenses regime for imported staff and contractors as for their UK counterparts. The changes needed also include addressing how IR35 penalises those contractors seeking to keep abreast of changing demand for skills.

Since then we have seen welcome tax breaks for apprentices but not yet for the growing number of subcontractors, including those working direct for the public sector (at a fraction of the cost of those employed via waterfall framework contracts), who need to keep their skills up to date. [see section 7 onwards for what else has changed, or not, since the last century!] 

3.    Why our failure to distinguish between digital education (disciplines) and digital training (specific skills) cause such problems.

One of the reasons so much public spend has been wasted has been the common failure to distinguish between core disciplines (which change slowly, if at all, over time) and technology, product and service related skills where demand can change before the curriculum headings, let alone content, is agreed.

The core disciplines for the successful  use of complex technologies have changed little, it at all, since structured programming and programme/project  management were added to the grounding in organisation and methods, operational research and systems thinking methodologies recommended by those who organized the systems analysis and programming courses of the 1960s. Those who helped me concoct the original Micros in Schools Programme (Cashing in on the Chips, 1979) saw programming as the new Latin or Algebra, much as those running Code Clubs today – then we had the obsession with computer literacy – alias how to use the office software products of the day – which turned off a generation.
The subsequent neglect of such basic principles as the separation of code and data and of “public” and “private” processes,  in favour of  transient techniques and technologies, as “professional” and “hobby” computing “merged” in the 1980s, lies behind many, perhaps most, of our current reliability, resilience and security problems.

4.    We need to find better ways of assessing current and future demand

We also need to find better ways of relating publicly funded and accredited qualifications and courses to current and emerging skills needs and employers’ recruitment and training plans, without overloading those who do seek to plan ahead with “consultations” asking questions they cannot answer. The solution entails pooling budgets for demand assessment and forecasting via, for example, consortia of Sector Skills Councils and LEPs, to enable the use of industry strength market research

Few employers can forecast their needs more than a year ahead in the detail needed to plan conventional courses and qualifications. Those able to do so commonly wish to mix and match modules for just-in-time delivery (to meet immediate skills needs) with those for longer term career development across academic and professional disciplines. This presents challenges to colleges, universities and funding agencies. Those willing and able to respond can derive significant earnings from the delivery of short course modules (both residential and on-line) within the global apprenticeship and continuous professional development programmes of major engineering and financial services employers.

They are however FORBIDDEN from doing so. The English Trailblazer apprentice programme requires adherence to the Skills Funding Agency Qualifications Guide This bars (Para 19) using the vendor or commercial (including those organized by professional bodies or trade associations) qualifications and associated materials  that are more commonly required by digital employers than those funded by the agency.

The other fundamental problem is that the definitions used in almost all public sector accreditation, let alone funding, planning and forecasting, processes are too broad for employers to understand the competence they can expect. More-over the delivery programmes are segmented into technical and professional career development “drainpipes” that do not fit a world in which employees and students need to acquire patchworks of skills to meet changing needs. The need is to mix and match just-in-time delivery modules from a variety of sources in ways that depend more on the ability to respond rapidly to changes in demand than to predict those changes. I have blogged on this in the context of the cyber-security skills needed by financial services employers a couple of times.

Those seeking to prepare current and potential customers to use the product and service innovations they are planning should be taking responsibility for ensuring the availability of the necessary modules. But they are constrained by demands to fit UK into accreditation and funding frameworks which cannot respond in time to meet evolving needs. In consequence there is a basic split between those who are seeking to meet global skills needs (whether the UK delivery of courses and materials developed elsewhere or the UK development of courses and materials designed for overseas delivery) and those responding to the demands of UK-centric funding agencies to produce courses and qualifications that will fit government targets, including for accreditation revenue streams and royalties.

5.    The need to break out from Ground Hog Day

There have been enquiries into problems with regard to shortages of “Digital Skills” (variously defined), every few years since 1965, when an interdepartmental working group of the Department of Education and the Science and the Ministry of Technology was established. Their 1967 report into an expected shortage of Systems Analysts by 1970 led to the establishment of the National Computing Centre, which later ran studies into the similar (but not identical) skills shortages of the early 1980s.

The underlying pattern of cyclical skills shortages was identified in the 1980’s and was summarised, with supporting statistical examples, in the 1996 IT Skills Trend Report  (pages 13 – 18) as follows:

“Except for the decimalisation boom and bust, UK IT recruitment effort over the past thirty years has followed the trade cycle with the magnitude of each boom and bust growing more extreme as each wave of technology change hits the industry … The first sign of economic downturn, before it shows in Government statistics, is the scaling down or cancellation of graduate recruitment programmes among those employers who plan ahead … a well established pattern of “boom and bust” with the trauma of transition, as each wave of changing technology and associated skills hits the industry, made more painful by the economic cycle.

[The graph below is based on analyses of recruitment advertising in Computer Weekly from 1975 – 1991 and across the Trade and National Press from 1991 – 1995. The extrapolation from 1995 was based on discussion with recruitment consultancies and was later confirmed accurate save that the crash with the dotcom bust was sharper and the recovery slower. The subsequent switch to on-line recruitment complicates attempts to look at recent trends]

Recession accelerates the decline in demand for old skills and delays investment in training for the new skills that are taking their place. The effect of the trade cycle is to magnify the problem and complicates any solution.

The basic cause of the periodic crises is that not enough users take on trainees so that even in recession we can have shortages of skilled and experienced staff in the areas of future growth. The problem is compounded by the low level of refresher and update training given to most of those already in the workforce. Employers who do not train their own staff bid up the price for skilled staff in order to poach from those who do train, whether as permanent staff or, more usually, as short term contractors.


Meanwhile those individuals who have not been able to update their skills find it difficult to retrain and harder still to obtain the necessary practical experience to re-enter the workforce, if they have been made redundant.

6.    How the debate on “solutions” has evolved (or not) over the past 40 Years

I first participated in an exercise on what are now called “Digital skills” when ICL sponsored me on the Masters Programme at the London Business School in 1971, during the digital slump that followed the decimalisation skills “crisis”. I used live data from a previous employer (STC Microwave and Line Division) for a Manpower Planning project to illustrate why the best practice of the day would not have helped during the run up to that “crisis”.

Shortly afterwards I was able to collate the views from around 50 organisations (albeit all in a single industry sector) while running the MinTech study to produce “A Computer Development Plan for the Regional Water Authorities” (published May 1974 by ICL and DTI). This was the first and (and may have been the last) study to factor the problems of recruiting staff with the necessary skills and experience into a “Digital” strategy.

It recommended a policy of retraining user staff rather than seeking to compete in the open market: “Since the majority of such staff [analysts and programmers] are not nearly as mobile as people think, this is probably sufficient to induce a sharp rise in salaries in a confined market … The speed of progress indicated … will not be practical [unless the new Regional Water Authorities pooled resources in collaborative partnerships] …. because the development staff are not available and cannot be trained in time”

In 1979, (when my day job for the Welcome Foundation included looking at “Medical Informatics” including opportunities from using technology to help look after an ageing population), I was able to get a recommendation that: “extra funds [be] earmarked at school level to give education in living with and using the new technology by providing at least one micro computer with appropriate teaching material to support staff in every secondary school in Britain by 1982” embedded in both Conservative and Labour policy papers and the DTI briefs for who-ever won the election. Part of the reasoning was summarised in the Conservative post-election discussion paper “Cashing in on the Chips

In 1981 Donald Michie challenged me to take at the impact of technology change on the education system and its values. The result was “Learning for Change”  and “No End of Jobs“, Learning for Change sold  out after a vitriolic attack in the Time Higher Education Supplement In speaking of the tension between developing individual creativity and developing social skills and the new skills required. I had said: “The second new skill is problem structuring and solving, and in particular group problem solving of the kind used by the class “cheat”, who knows which classmate’s homework to copy in which subject. By definition, this skill is selected against in our educational system and thus its most skilled practitioners frequently end up working against society as rebels, criminals or parasites rather than in the key management posts which they should occupy”.

The reviewer had not quoted the first new skill “the concept of simulation … leading to the understanding of how computer models can be run backwards, from the desired ends to identify and test the logic, assumptions and premises which lead to that end.”

He had also ignored  “Thorough and imaginative approaches to group problem identification, structuring and solving are going to be essential in the factory of the future where quality control is going to be one of the main occupations. Ensuring that complex computer controlled products are functioning correctly, and that the specification of the control program is adequate under all circumstances and not dangerously inadequate under even the most unlikely circumstances, may well become the most labour intensive part of the production process.”

I suspect, however, that what had really angered him was the comment that in a world of multi-career lives, teaching should be thought of as a second or third career, not a first career. He confused my meaning with with second or third class. What I meant was that those with practical experience (and pensions which mean they were not scared of being fired) were often (but not always!) better teachers than young graduates.

In 1987 an NCC team summarised employer’s views on what needed to happen in “The IT Skills Crisis – a prescription for action” . The headline conclusion was: “both IT users and suppliers believe that priority for Government action should be given to tax incentives and improved information services to encourage training initiatives and expenditure by industry rather than additional direct public spending or intervention. The main exceptions are to do with local initiatives in regions of high unemployment, including inner London, training for school leavers and Government’s own training and staff development responsibilities as the largest direct and indirect employer of IT applications staff.”

Part of the objective of that survey was to gauge the interest of the NCC’s then 1,800 user and 200 supplier members in action on skills. It was agreed that a 2% response would indicate apathy and anything over 5% would indicate genuine interest. Most skills surveys, then and now, are based on response rates of around 2% (i.e. they are not statistically significant and indicate either a faulty sample or a lack of interest on the part of the target audience). That survey had a response rate of 10%. I suspect that a similar survey study today would attract a similar response.

From 1988 – 92 the first Women into IT Campaign helped raise the proportion of girls taking IT related degree courses from barely 10% to nearly 30% and female employees (including returners) to over 25%, before the Department of Education and Skills decided not to build on the DTI pump-priming and the Women into IT Foundation had to be wound up before the Trustees became personally liable. That success demonstrated that action to improve careers advice and guidance, led by employers seeking to portray themselves as employers of choice, can change perceptions in schools and universities. It also showed that evidence of clear employer support is not sufficient to overcome departmental opposition. At that time the opposition was to “affirmative action” and performance measures (placement into work) which threatened established skills programmes.   

At about the same time (1992) the West London Training and Enterprise Council obtained funding to use computer assisted telephone interviewing for its Labour Market survey and to attach sections on computer use, training needs and how they were being meet. The result was the only statistically reliable snapshot of private sector employer views: a response rate of over 50% from a structured sample of 1,500 local firms. It showed a far greater use of technology, including among small companies, than expected but also an almost total lack of either training or professional support. The headline of the published management summary  was “The users have taken over the system“.

The main report (not published) demonstrated that the IT skills programmes of the day were not seen as relevant by employers. What they most wanted was help in identifying well educated (literacy and numeracy) and motivated (including time-keeping and social behaviour) recruits. If the “TEC” wished to use computer literacy as a euphemism for basic skills, that was up to it. Most of the small minority of employers willing to work with the TEC on planning publicly funded programmes wished to attend student selection meetings and sit in on classes to spot those who they might wish to recruit. They were then happy to pay for computer courses to meet their own needs, as part of work experience placements, for those they wanted to recruit.

From 1991 until 1997 the Women in IT Foundation and (later) IMIS (The Institute for the Management of Information Systems, later taken over by the British Computer Society) produced annual IT skills surveys (based on analyses of recruitment advertising and surveys of salaries and staff turnover). After a presentation of the 1995 report to the then Shadow Chancellor, the 1996 report was upgraded to collate thirty years of material on trends with the technology forecasts being used by corporate planners in major suppliers and users, to cover the action needed to address the expected skills “crisis” during the run up to “Y2K”.

Work on checking which systems had not already been modified to handle Y2K, (the year 2000 had a different leap year routine and software with two digit date routines might also need re-writing), was expected to coincide with that on EMU (the amendments needed whether or not the UK joined the European Monetary Union). At the same time the industry was expected to face one of its cyclical crises, exacerbated by competition for the skills to enable businesses to begin the transition to Internet-based systems for on-line transactions.  

That report was one of the few on IT skills to sell more than a handful of copies. The report and the events and publicity organised around its launch, helped bring forward by several months the work (across the EU as well as in the UK) which enabled serious problems to be avoided. Among the most effective programmes was that to train 40,000 Millennium Bugbusters.

This transformed the UK supply of micro-computer support technicians, using courses contracted only to suppliers who could demonstrate (to a panel of assessors with experience of running both IT operations and IT training programmes) their ability to deliver short, intensive, hands-on training, delivered by named staff with relevant experience. The nature of the quality control was controversial because few of the organisations then running mainstream Government-funded skills programmes met the criteria required and the Treasury, who had provided ring-fenced funding, refused to allow any relaxation.

Section 8 of the 1996 Skills Trends report contained 40 pages of action plans for:
•    employers – including integrating skills acquisition, development and retention
•    suppliers – including the supply of skills to handle current and future products lines
•    individuals – including taking charge of your own career path
•    training providers – including “clearing the certification jungle”
•    government – including to: 
   i.    cut the pre-tax cost of training by rationalising funding around programmes using inter-operability standards for modular delivery and
   ii.    cut the post-tax cost of training e.g. “that those following professionally recognised training and/or update courses …. be exempted from national insurance and/or income tax” and
   iii.    support activities to improve careers advice.

About the same time EURIM (now the Digital Policy Alliance) produced a short paper, with help from the British Computer Society, IMIS, IEE (now IET) and others, on “Re-skilling Europe for the Information Society” study. This called for action to use the challenges of Y2K and EMU to help bring UK and EU education and training into the 21st century, to cope with a world in which demand for skills can be expected to change before curricula are agreed, let alone the courses delivered:

The first three recommendations were:
•    Education and training policy must better distinguish between core educational and academic skills (which rarely change over time) and vocational and technology skills (where traditional methods can no longer cope with the pace of change).
•    Vocational curricula and course content and delivery must be better related to current and emerging skills needs andemployers’ recruitment and training plans.
•    Commercial market research and quality assurance techniques should be used to assess skills demand and the relevance and effectiveness of training delivery.

The initial funding to enable the embryonic ITNTO (IT National Training Organisation) evolve into one of the most successful of the Sector Skills Councils was announced by John Healey, Minister for Adult Skills at an IT Skills Summit organised by IMIS and Comptia in 2002. Ministers have since made many speeches similar to his opening comments on the need for both mass market programmes to prevent social exclusion and focussed programmes for technical and professional skills but few have been so succinct and clear.

Unfortunately he was reshuffled two hours later. By the time the funding emerged (over 18 months later) momentum had been lost and the initial support group of 20 Human Resources Directors from major ICT employers had lost patience. It took nearly a decade for momentum it to be recovered. EURIM (now DPA) was among many industry-supported groups which suspended attempts to influence Government skills programmes other than in support of activities planned by e-Skills.

That momentum has now been recovered and many of the Sector Skills Partnerships (including the Tech Partnership for digital skills) have strong support from employers. But they are once again threatened by a fragmentation of Government funding for initiatives and consultations which fail to recognise the cost and difficulty of obtaining representative inputs from employers and students on their needs and expectations.

A key finding from the summary report of the 2001 Skills Summit, used in the submission to get John Healey to make his announcements to the second summit in 2002, was:  

•    There are too many surveys, initiatives, programmes and organisations at every level but particularly with regard to the identification of employer needs. We are suffering from study and initiative overload. We need to do less but do it better. We also need to achieve economies of scale without compromising quality or losing local enthusiasm and personal tutorial support.  

•    The result is a mix of ignorance and paralysis. Even organisations with large industry education teams (e.g. Microsoft with a team of 12) cannot respond to more than a fraction of the invitations and/or surveys they receive from government departments, universities, colleges, schools and industry bodies. Those who try to do more have no time to do their day jobs.

•    There are far too many funding agencies and programmes with complex rules, expensive bidding processes, slow decision processes. Too much is capital spend when the need is usually for a flexible mix of capital and revenue spend. This leads to waste of public funds in parallel with mounting frustration at lack of funds to exploit success.”

The then Head of the Microsoft team told the 2001 Skills Summit that they had logged over 3,000 requests in the previous year for inputs to those planning courses and qualifications. Hence the very strong support from all the users and suppliers present for a wholesale rationalisation of consultations and surveys and the suggestion that, instead of demanding duplicated evidence of employer support, those planning bids for public funding should quote evidence of local demand using material collected and collated via the relevant sector skills councils based on industry strength market research.

Good market research is, however, more expensive than “consultations” and semi-academic desk studies based on collating the results of surveys with response rates too small for statistical significance. Such studies can be (and often are) systemically misleading.          

7.    What has changed since the last century and what has not?

It is now much easier and cheaper to organise affordable on-line, multi-media material for  delivery when, where and how needed by employers and students. But the first three recommendations of the 1997 Eurim briefing “Reskilling Europe for the Information Society” remain valid:

•    Education and training policy must better distinguish between core educational and academic skills (which rarely change over time) and vocational and technology skills (where traditional methods can no longer cope with the pace of change).

•    Vocational curricula and course content and delivery must be better related to current and emerging skills needs and employers’ recruitment and training plans.

•    Commercial market research and quality assurance techniques should be used to assess skills demand and the relevance and effectiveness of training delivery.

Today we face the very real risk that predictions of need based on current Internet Business Models (novel in the last century, now dominant and supported by growing armies of lawyers, regulators and compliance officers) will not survive much beyond 2020 before they crumble. 

If 14 years was an appropriate period of copyright and patent protection for the fast moving 18th Century, why should longer be needed today – except when safety testing (as with pharmaceuticals and other complex products) may take a decade or more. In the latter case, surely the protection should go to those who do to the testing and bring the product to market, not to those who patented a compound or technology but made no attempt to bring it to market. In the 18th Century failure to bring a patent to market or keep a book in print was grounds for loss of protection. 

It is impractical to criminalise whole generations of teenagers. Meanwhile the growing use of ad blockers and anti-spyware combined with concerns over privacy and security may be about to destroy the “big data” industry. 

I spent five years in the late 1970s and early 1980s as a corporate planner. I enjoyed participating in ten and twenty year forecasting exercises. I even wrote one for 50 years: as an entry for an essay competition on the 50th Anniversary of Leo (the first commercial computer). My Director, however, thought that the pace, nature and direction of short term fluctuations was such that my predictions should be used for strategic positioning and risk avoidance unless and until I could identify the evidence that would identify whether my scenarios were about to happen for real. I like to think that I became rather good at that.       

Today, business models that assume global players running centrally controlled networks and big data clouds are being “complemented” by models tht assume a future based more on locally interconnected, resilient  meshes, which can not only offer more security (including from remote surveillance or attack) but are also capable of supporting smart homes, cities and those physically dependent on tele-care and welfare – regardless of whether national and international networks are still running. Hence my views on broadband investment.

Those who hedge their bets may be more likely to survive but those providing solutions which will work both ways are more likely to prosper.  

8.    The “real” questions for politicians looking at skills policies include:  

•    How do we make it more attractive for employers to retrain existing staff and recruit trainees than to recruit from their competitors or customers or toimport skills or export jobs?

•    How do we make it easier for individuals to identify and demonstrate that they have acquired the skills in current and prospective demand?

•    How do we make it easier for those at all stages and levels of the UK education and training supply chains to deliver what is in current demand, when, where and how it is needed?

•    How do we ease the pain of moving from timetabled courses to lifelong learning networks that enable students (of all ages) to acquire the skills in current demand, when and where needed?

•    How do we ensure socially inclusive, local access to world class learning and training: from school, through apprenticeship, college, university to continuous professional development?

•    How do we provide accurate, attractive and realistic careers advice when pre-planned technical and professional career progressions are becoming the exception rather than the norm?

•    How do we help those employers who do plan ahead and can offer career progressions to better identify, motivate and exploit talent that is currently going to waste?

I believe that the answer, in almost every case, is to mix national programmes, led by national employer driven sector skills councils in flexible partnerships cross-cutting local skills partnerships – with a focus on helping employers to organise the training they need for their employees, as opposed to that which meets funding agency targets.

We are beginning to see the rhetoric and the tax regime is beginning to be more favourable but the small print of the Skills Funding Agency “guidance”, particularly its prohibitions appears to be straight from the 1970s. I also tried to get my head round the 79 pages of “guidance” on the new Tralblazer apprenticeships , particularly why the rules are so much less flexible for English employers than they are for those in Scotland, Wales or Northern Ireland. I got as far as understanding that employer specific apprenticeships are not permitted, each has to involve at least ten participants, including two SMEs.and thsoe with apprentices in both England and Wales have to have separate schemes …

At that point I gave up.   

Join the conversation


Send me notifications when other members comment.

Please create a username to comment.


I enjoyed your history of the failure to address the "digital skills" shortage, and agree that it will never be solved using today's Groundhog Day approaches.

Fortunately there is a better way to address the shortage that builds on structured programming.

The work of Michael Jackson, JSP/JSD, can be mapped on to Communicating Sequential Processes (CSP) by Tony Hoare, i.e abstract graphical models mapped on to mathematical proof.

This is how engineering works for the complex artefacts of the Industrial Age, e.g. Boeing knew the Dreamliner would fly long before they started to build the first instance, with the designs being stored using CAE tools.

An engineering approach needs to be applied for the Digital Age and would raise the skill levels. CAM tools would also eliminate manual coding, reducing the number of people required.

What has also changed is that the computing platform was initially a local computer, but is now the global computing network.

In public services, this would enable the myriad legacy applications operating in hospitals, GP surgeries and pharmacies to be replaced by a single healthcare application. Again, dramatically reducing the number of people required.

So, in summary there are only two problems with today's "Digital" agenda.

1. The existing software was not designed for today's hardware platform.

2. The software is not engineered.

These are the two problems that need to be addressed.

See my blog entry for more detail.

As ever,


Thank you for this. I completely agree.

In my own submission to the Select Committee I drew attention to the need to distinguish between educating in basic disciplines (including structured programming) which change slowly if at all, and training in transient skills (to handle the techniques and products of the day) which can change more quickly than publicly funded courses can be agreed.