The Paradoxical Work Force

The Paradoxical Work Force

Kurt Cagle 14/02/2018 10

It's a recession when your neighbor loses his job; it's a depression when you lose yours, Harry S. Trumen. After several months where work seemed stalled on several fronts, things finally started to become busier again recently, enough so that I've had only minimal opportunities to update my LinkedIn posts. However, I've been thinking hard about the nature of work for a while now, and have come to some disturbing conclusions about what I think is happening structurally worldwide.


One of the most disturbing charts I've seen recently shows that while employment has risen anemically in the last six to seven years, profits from manufacturing have climbed dramatically. Why? During the Great Recession of 2008-2012, companies were laying off people heavily. As the economy began to improve, however, those same companies, operating on lean time assumptions, invested heavily into automation. They discovered that the automation had improved in that period to a point where they did not in fact need to hire more workers.

The same thing happened in high tech fields, though with significant differences. The in-demand professions in the software field were not, in general, programmers. They were for subject matter experts with deep analytic skills who could perform meaningful analysis upon the sea of data that companies were now awash in, and there are signs that even here (and given that such expertise is largely chimerical in nature) the scale is tipping towards the emergence of analytic toolsets that largely automate even the expertise to know what kind of statistical analyses need to be done.

Machine learning, deep learning, computational linguistics ... this has been a great time if you happen to be graduating from university right now, because what has largely been theoretical and academic now has immediate application. The late Marvin Minsky, one of the Fathers of AI, would be positively giddy. This is not to say that this technology is mature (it's not, by a long stretch) but certainly what has been under the rubric of Artificial Intelligence for a long time is now finally shifting over to just Computer Science.

However, this also heightens the point. The point that most people miss about data scienceis that it is not a programming profession. It is a profession where a subject matter expert gains the software skills and tools necessary to perform analytics within their domain. A business analyst, a population biologist, a political analyst, a military analyst, a city planner and so forth may all use similar sets of tools, because the kind of things that need to be analysed are usually stochastic (probabilistic) in nature, but it is their expertise in business, biology, politics, military actions, city planning and so forth that ultimately is their strength, and these skills often take years to learn because they are experiential in nature.

This is increasingly true in most professions. A special effects artist is a film animator that has become proficient in modeling, shading and animation tools. Ditto game designers, automotive designers, shipping analysts and so forth. In general, what is happening today is that for any given domain, you're seeing a stratification of concerns along the lines of designers, analysts and decision makers. The designers envision, the analysts recommend, the decision makers execute. Most of the actual production is becoming automated.

Dilbert, by Scott Adams

Sales is becoming automated. Brokers, those who mediate deals, are disappearing as the Internet becomes the broker of choice, as are middle managers. You can tell this is happening in great part because the actual wages of new people entering into these fields are considerably less than they would have been even a generation ago, if there is even work, especially when accounting for inflation.

The reason that doctor salaries have remained relatively high (and they are also dropping) is not because doctors have any more expertise than nurses, they are just held to a higher standard of liability, and as demand for health care goes up, paradoxically this is likely to result in nurses being given many of the responsibilities that doctors have now ... along with the resultant liabilities. Why? Because nurses are cheaper, and hospitals and clinics are increasingly under pressure to make a profit for their stockholders. This in turn means that the medical profession is similarly fracturing into the research designer | analyst | executive trichotomy. ICD-10 code analysts are one of the hottest jobs in medicine right now, because they require both an extensive domain knowledge and the ability to optimize using computer systems.

One of the great shibboleths of technology boosters has been that, while yes, technology destroys pre-automation jobs, it also opens up post-automation jobs. The problem with this is that ultimately there are far fewer of these jobs (about 1/5, by most economist's calculations) that are created than are destroyed. In a pre-automation environment, sales people could become very wealthy because sales is a very time-intensive business, and you often needed a number of sales people working a population to score those big wins.

In a post-automation environment, the salesperson has been replaced with the SEO analyst, who's forte is understanding the rules of various social media and then optimizing for that. Cold calling has become the province of sweatshops, because while the likelihood of a sale is low, the costs are also low. Even here, AI chat bots are going to encroach on this domain (they are already taking a huge chunk out of recruiting). Indeed, the biggest challenge for India, Inc. is that its business model of providing low cost technical labor is now colliding with AI systems that can automate a huge amount of the pre-qualification process. A high recurring cost (wages) gets replaced by an initially higher but one time cost (AI systems), which can then be amortized to return a return on investment quickly.

Will companies lose potential sales on AI systems? Some, yes, but remember that these are already considered marginal revenue streams to begin with, the equivalent of strip mining knowing that most of what will be mined will be dross, but there will be enough retrieved in terms of metals and similar resources to make it worthwhile long term.

So this is a factor creating the paradoxes in the workforce today. We have a workforce where companies are screaming for high end analysts because they are seen as an indispensable part of the modern data-driven corporation but are passing over programmers who were hot only a few year before because they no longer really have the urgent needs to automate basic tasks that they did earlier. A workforce where 60 hour weeks are not uncommon among those that do work, because that workforce is also young, trying to prove themselves, and only marginally aware that their actually wages are only 2/3 of their stated hourly rates. A workforce where a high school degree is no longer enough for most jobs, because automation is rapidly overtaking those things the average 18 year old can do, resulting in an increasingly barren wasteland at the forty+ year old level who tend to make the best analysts, having gained the experience in the field to see the patterns.

I've been expecting that the economy will start to falter starting this quarter, though initially felt that as recessions went it might end up being fairly mild. I think I was a little too early (by perhaps a couple of quarters) though there are already signs that the economy isfaltering. However, I also now think that the recession to come will actually prove quite severe.

On the retail side, companies are still opening new stores, but about three times more stores are being closed as no longer profitable than are being opened. A huge amount of retail corporate debt is coming due in the next two years (in a pattern very similar to the amount of mortgage debt that came due in 2008), and retail companies (now owned largely by holding companies) have fallen way behind in paying that off. Couple that with the Amazon effect and the dis-intermediation of human brokers, and that very large portion of the economy is already distressed; it will get much worse.


The economy has benefited from a global oil production glut, but that has not come without cost. Relative to its low, gas prices in the US have increased about fifty percent in the last year, and most gasoline in the US is now above $3.20 a gallon. There's a fairly strong correlation that shows that oil prices (and hence gas prices) spike between eighteen months and two years after entering ten year local minima, which raises the possibility of high gas prices (in excess of $4.50 a gallon) by the 2nd or 3rd quarter of 2018.

This may seem like good news for oil workers, except that automation has had its way here as well. More and more deep water rigs are autonomous - rather than requiring 40-50 man crews per well, they need 4-5, and that primarily to perform occasional maintenance.

This then characterizes another paradox of the economy. Most seasoned programmers are well aware of the fact that work is governed by impulses. Software development involves the creation of a product intended to reduce the complexity of a process. Once the software is written, the need for a development team no longer exists. If they are lucky, there may be other work, but if a good developer does their job right, they will put themselves out of work on the task at hand. Many jobs are like that - building a house, creating a movie or video game, writing a novel. The reality is that most jobs are episodic.

The genius of modern capitalism is that it managed, for a while, to give the illusion that work could be stable. The forty hour work week was the equivalent of the Social Contract - people might work beyond that allotted period for a while to pay for those times when work was slack. It began to break down for those positions that provided lesser value (or where labor pressure was weak), leading to the distinction between full time work and part time work (which is simply another way of saying contingent labor). Benefits - health care, pensions, equity plans and so forth - became only available to those people who represented a significant investment and challenge to fill, in exchange for demanding more hours from that labor - in effect, subsidizing the higher wages.

However, even this was an illusion, as layoffs became a standard way of trimming labor excess during slow periods. This created a second tier of contingent labor, with a period of 18-24 months, which affected the processing and eventually technical professions especially hard. Not surprisingly, the bulk of consultants now operating independently (or as part of a consultancy) comes overwhelmingly from this second tier.

Outsourcing represents a third layer of contingency worker - inexpensive, with no real ties to the company other than a salary, that can be fired readily with little consequence. A final layer of contingency workers that have become especially popular of late have been internships, where in essence young workers take a position with companies in exchange for experience and the possibility of being hired even in a part-time capacity.

It's easy to point the finger at management (and shareholders), and there is quite a lot of justification to do just that. When an executive's compensation package is the equivalent of a hundred times the compensation for an average worker (especially when that executive is NOT the founder) then there are clearly problems. No executive, by him or herself, makes that large a contribution to the company, and that imbalance is clearly exacerbating an already dire situation.

However, the reality for most companies is that, outside of perhaps a hundred giant corporations, margins are thin, sales are down, and companies in general are surviving primarily because of accounting gimmicks that put off the final reckoning for another year, but that don't in fact do anything to staunch extreme debt situations.

The paradox that the economy is thriving comes about due to the use of such accounting practices. A recession does not occur out of the blue. It happens when an impulse hits the market that shakes the faith by people that the status quo is stable. Ironically, it usually happens when the economy seems to be at its most stable and complacent. When you have paradoxes such as that we seem to have full employment, the stock market is pushing record highs, people in high profile positions are complaining that they are working 70 hour weeks, and new billionaires seem to be minted daily.

The reality is darker - that permanent work (lasting longer than a decade) has disappeared for all but perhaps 5% of the population, that, accounting for inflation, real wages have fallen for nearly fifty years (corresponding roughly to the creation of the microchip), that most companies are only barely solvent, and that automation is eliminating about 10% of net jobs per decade.

I'm personally not an alarmist with respect to automation. What I think is important to understand, however, is that automation is transforming our society far faster than our social or economic systems can cope with. As a society, we need to be asking ourselves whether providing the needs for an intelligent, healthy, well-informed society can be achieved given the current system where we are compensated primarily on the basis of making money for other people, where virtue is assigned on the basis of net worth and where the investor has prior claim over all others for compensation for work that is done.

For a while, the answer was yes, because it corresponded to the notion that has been at the heart of the American Dream - that hard work, loyalty, careful investment, and a willingness to delay gratification could, over the course of decades, make that investment pay off. Increasingly, however, the answer seems to be shifting into the no column; that putting in 60 hours a week is a fool's errand because you won't see the fruits of your labor, that if your job function can be replaced less expensively with automation that it will be, that loyalty will not be rewarded and in fact may even be seen as a fault, that careful investment has the potential to run afoul of unregulated con men, hidden fees, squandered taxes and hidden biases, and use it or lose it becomes the default operating paradigm.

The thing about paradoxes is that they seldom are actually paradoxical. Instead they represent a system in transience, where information about one part of a system has not yet been caught up by the rest of the system. This is where we are right now. Human beings are notoriously better at picking up the pieces after things fall apart than they are in preparing for when they do. Even so, it's probably a good time to be preparing.

On that cheery note ... until next time, enjoy!

Share this article

Leave your comments

Post comment as a guest

terms and condition.
  • Thomas Verrati

    We humans may not be perfect but we are the product of millions of years of evolution that furnished us with valuable instincts, such as empathy and common sense – traits that are impossible to code in a machine, at least in the foreseeable future

  • Anthony Dupuy

    Important decisions that affect people should always be filtered by human leaders before being finalised and executed.

  • Fred Pusil

    Priority has to be placed on the creation of open data platforms in organisations, coupled with HR leaders managing the necessary cultural shift that will lead to data-driven processes.

  • Jack Baudry

    As innovation-driven disruptions become widespread, it is essential to ensure that we are ready to engage with a future that is full of promises and uncertainties

  • Wesley Wood

    It is true that every new innovation and technology disrupts, impacting jobs and employment, but the innovations and technologies behind those ideas, in fact, create more work and multiple ways of engagement that cannot be imagined while thinking in a linear fashion and working within conventional silos.

  • Pierre Moulin

    Every new shift forces us out of our comfort zones to know more, learn more and adapt more.

  • Cody Gallows

    History is witness to innumerable examples of similar phenomenon where shift in technology and rethink about its application has opened up unimaginable possibilities. So guys do not panic ;)

  • Ronald S

    Digitisation is getting even smarter, democratising the process of content creation and the destruction of more jobs

  • Bradley Sparks

    Up-skilling and re-skilling are indispensable to stay relevant in the future job market.

  • Sam Hunter

    Automation, artificial intelligence, machine learning, robotics, 3-D printing, Nano Technology, Internet of Things and the widespread use of data and disruptive innovations are redefining the way citizens will learn, earn and live.

Share this article

Kurt Cagle

Tech Expert

Kurt is the founder and CEO of Semantical, LLC, a consulting company focusing on enterprise data hubs, metadata management, semantics, and NoSQL systems. He has developed large scale information and data governance strategies for Fortune 500 companies in the health care/insurance sector, media and entertainment, publishing, financial services and logistics arenas, as well as for government agencies in the defense and insurance sector (including the Affordable Care Act). Kurt holds a Bachelor of Science in Physics from the University of Illinois at Urbana–Champaign. 

Cookies user prefences
We use cookies to ensure you to get the best experience on our website. If you decline the use of cookies, this website may not function as expected.
Accept all
Decline all
Read more
Tools used to analyze the data to measure the effectiveness of a website and to understand how it works.
Google Analytics