How pattern detection and pattern exploitation might elevate each other to a new level
Although most of us don’t see it, optimization algorithms (OAs) are at work everywhere. They plan shelve stocking for our grocery stores, create airport schedules, and give us the shortest route to our holiday destination. Exact algorithms in particular do very well at exploiting known structures — e.g., convex structures — finding solutions even in massive decision spaces with many constraints. Over the past decades, the combination of hardware- and algorithmic improvements yielded massive speed-ups in the order of millions. A planning task that might have taken a computer months to complete in the 90’s could just take a second today.
Similarly, machine learning (ML) has taken an incredible flight in the last decade or so. MuZero showed the capability to learn superhuman game-playing policies without knowing the games’ rules, Graph Neural Networks learn complex relations unperceivable to the human eye, and Transformers gave rise to ChatGPT and its competitors. The commonality is that these algorithms are all able to detect patterns from their environment, be it text databases or video games. Novel and highly complicated architectures are introduced on a regular basis, often solving new problems and offering unparalleled performance. Despite all successes and breakthroughs, for many real-world problems, end-to-end ML struggles to achieve competitive results. Tailored OAs often still beat ML, but may require substantial computational time.
There is no need for the two approaches to compete though. Interestingly, optimization algorithms excel at exploiting patterns, whereas machine learning shines at detecting patterns. Instead of pitting them against each other as benchmarks and see which one outperforms the other, wouldn’t it make sense to marry the two complementary halves instead?
When merging optimization and machine learning, it often boils down to statistical learning being used to improve optimization routines in one form or another. This way, we can speed up the search by exploiting patterns that we learned. The development of such integrated solutions has become an emerging research field in recent…
