Machine Learning’s Complexity Problem

I’ve been experimenting with machine learning lately.   For someone who started writing code in the early 90’s and witnessed firsthand the explosion of the web and all the software engineering practices that evolved from it, I find amazing how machine learning  flips traditional software engineering on its head.

Traditional software engineering taught us to divide and conquer, minimize coupling, maximize cohesion while artfully abstracting concepts in the problem domain to produce functional and maintainable code in the solution domain.  Our favorite static code analysis tools helped keep our code (and its complexity) in check.

Similarly, traditional software architecture taught us to worry less about code complexity and more about architectural complexity for it had farther reaching consequences. Architectural complexity had the potential to negatively impact teams, businesses and customers alike, not to mention all phases of the software development lifecycle.

Yes, this was the good ol’ world of traditional software engineering.

And machine learning flips this world on its head.  Instead of writing code,  the engineering team collects tons of input and output data that characterize the problem at hand.  Instead of carving component boundaries on concepts artfully abstracted from the problem domain,  engineers experiment with mathematics to unearth boundaries from the data directly.

And this is where machine learning’s complexity problem begins.  Training data sets rarely derive from a single cohesive set.  They instead depend on a number of  other data sets and algorithms.     Although the final training data set may be neatly organized as a large table of features and targets, the number of underlying data dependencies required to support this can be quite dramatic.

Traditional software engineering became really good at refactoring away dependencies in static code and system architectures in order to tame the complexity beast, the challenge now is to do the same for data dependencies in machine learning systems.

In conclusion, the paper “Machine Learning: The High Interest Credit Card of Technical Debt” summarized this and a number of other ML complexity challenges nicely:

No inputs are ever really independent. We refer to this here as the CACE principle: Changing Anything Changes Everything.”

AI 3.0

AI, Machine Learning (ML) and Deep Learning (DL) are all the hype these days, and for good reason. By now we know progress in AI accelerated over the past decade thanks to a convergence of factors including Big Data and compute power. And results are impressive as a recent Economist article highlights:

In February 2015 DeepMind published a paper in Nature describing a reinforcement-learning system capable of learning to play 49 classic Atari video games, using just the on-screen pixels and the game score as inputs, with its output connected to a virtual controller. The system learned to play them all from scratch and achieved human-level performance or better in 29 of them.

Over the next two years, many businesses will continue ramping up their ML/DL initiatives with the hope of improving every aspect of their business performance. These companies will follow a path similar to Major League Baseball’s pursuit of sabermetrics, or Wall Street’s appetite for algorithmic trading.

I think at some point in 2018, the latest wave of AI hype will peak and begin receding thereafter. Ongoing issues with model accuracy, as well as high costs required to operate less-than-stellar model performance will be two of the primary reasons behind this. I also believe decision-makers will feel increasingly vulnerable as AI effectively detaches them from understanding and refining the theories underlying their business performance.

This will usher in a new period of enlightenment  where companies adjust their be-all-end-all expectations of AI in favor of empowering their people to effectively coexist with AI.  This will be good news for workers too as Tyler Cowen suggests in Average is Over:

As intelligent-analysis machines become more powerful and more commonplace, the most obvious and direct beneficiaries will be the humans who are adept at working with computers and with related devices for communications and information processing. If a laborer can augment the value of a major tech improvement by even a small bit, she will likely earn well. This imbalance in technological growth will have some surprising implications. The key questions will be: Are you good at working with intelligent machines or not? If the answer is yes, then your wage and labor market prospects are likely to be cheery. If the answer is no, but you have an unusual ability to spot, recruit, and direct those who work well with computers, then the contemporary world will make you rich.