LinkedIn's AI Breakthrough: Why Small Models & "Multi-Teacher Distillation" Trump Prompting

In the ever-evolving landscape of Artificial Intelligence, the quest for more efficient, accurate, and personalized solutions is relentless. LinkedIn, a titan in professional networking and a long-time pioneer in AI recommender systems, recently shared insights into a significant breakthrough that redefined their approach to building next-generation AI products. Forget the hype around simple prompting; LinkedIn's success lies in a sophisticated technique they call multi-teacher distillation, a method that has unlocked unprecedented levels of quality and efficiency.

The Prompting Problem: A Non-Starter for Sophistication

While large language models (LLMs) have captured the public imagination with their conversational abilities, LinkedIn found that relying solely on prompting for complex recommender systems was a dead end. Erran Berger, VP of Product Engineering at LinkedIn, revealed in the "Beyond the Pilot" podcast that the company didn't even attempt to use prompting for their next-generation recommender systems. The reason? It simply wasn't capable of delivering the necessary accuracy, low latency, and efficiency required for a platform handling millions of job searches and candidate profiles daily. Prompting, while useful for many applications, lacked the granular control and deep customization needed to align with LinkedIn's intricate product policies and user needs.

The Breakthrough: Multi-Teacher Distillation Explained

Instead of brute-forcing with prompts, LinkedIn embarked on a more methodical journey. The core of their innovation lies in a process that starts with a comprehensive product policy document. This document, meticulously crafted by product managers and engineers, serves as a blueprint, detailing how job descriptions and candidate profiles should be scored across various dimensions. This wasn't a quick process; it involved numerous iterations to capture the nuances of LinkedIn's recommendations.

This detailed policy was then used to fine-tune an initial, large 7-billion-parameter model. However, the innovation didn't stop there. This initial teacher model was then used to train a second teacher model focused on click prediction and personalization – crucial elements for any recommender system. The real magic happened when these two distinct teacher models were used to distill a much smaller, yet highly optimized, 1.7-billion-parameter student model. This process, known as multi-teacher distillation, allowed them to:

• Achieve High Affinity: The student model retained the crucial aspects of the original product policy.
• Master Click Prediction: It effectively learned the nuances of user behavior and personalization.
• Modularize Training: The process became repeatable and componentized, allowing for easier iteration and development.

Berger likens this to training a chat agent with two teachers: one focusing on factual accuracy and the other on communication style. By blending their expertise, the resulting agent is far more capable and can be improved independently in each area. This methodology has proven so effective that it's now being adopted across various AI products at LinkedIn, creating a standardized "cookbook" for AI development.

A New Era of Collaboration: Product Managers and ML Engineers Unite

This technical breakthrough has also spurred a significant shift in how teams collaborate at LinkedIn. Historically, product managers focused on strategy and user experience, leaving the intricate details of model development to machine learning engineers. However, the success of multi-teacher distillation hinges on the deep integration of product domain expertise and ML engineering. Product managers and ML engineers now work hand-in-hand to define and refine the product policy, which directly influences the teacher models. This unified approach ensures that the AI not only performs technically but also aligns perfectly with user needs and business objectives. Berger emphasizes that this collaborative model is a fundamental shift, serving as a blueprint for all future AI initiatives at LinkedIn.

Key Takeaways from LinkedIn's AI Strategy:

• Prompting Limitations: For complex, real-time systems like recommender engines, prompting alone is insufficient.
• Product Policy is Paramount: A detailed, iterative product policy document is crucial for guiding AI development.
• Multi-Teacher Distillation: This technique allows for the creation of highly accurate and efficient smaller models by learning from multiple specialized teacher models.
• Collaboration is Key: Close partnership between product management and ML engineering is vital for aligning AI with business goals.
• Optimized R&D: LinkedIn has focused on streamlining its R&D process to achieve results in hours or days, not weeks.
• Flexibility in Models: Developing pipelines that support plugability and experimentation with different models is essential for agility.

LinkedIn's journey highlights a critical evolution in AI development. It's a testament to the power of innovative techniques like multi-teacher distillation and the indispensable role of cross-functional collaboration in building AI that truly understands and serves user needs.

For more in-depth insights, you can watch the full podcast or listen to "Beyond the Pilot" on your preferred podcast platform.