Headlines: OpenAI founder dropouts, Compute mania in Generative AI, Google competitor Perplexity scaling, How big tech is circumventing regulation in M&A, New architecture in Neural networks, Long read - Generative AI industry dynamic, Deep Learning Foundations book, Explainer - Attention is all you need

AI in Businesses

• Is this just a coincidence or something sinister, we won't know yet. OpenAI co-founder Schulman leaves and Brockman is on extended leave. the leadership at the company is through troubled times forever (Kyle Wiggers, Techcrunch)

• Training Llama 4 is expected to need 10X of that of Llama 3, potentially necessitating around 160,000 GPUs based on estimates from Llama 3's requirements. To support this expansion, Meta is significantly boosting its capital expenditures, with Q2 2024 seeing a 33% year-over-year increase to $8.5 billion (Simon Sharwood, The register)

• Elon Musk's vision for the future of AI extends beyond the current technological landscape, as his AI initiative, xAI aims to surpass human cognitive abilities with AI by the end of 2025 (and be at par with OpenAI and Deepmind by the end of 20240. xAI is building a 100,000 GPU cluster called "Gigafactory for compute" after his "Gigafactories" for EVs (Tristan Greene, CoinTelegraph)

• Perplexity AI has revised its revenue expectations for the year to $35M from $5M in 2023, also changing to an advertisement revenue model from a subscription-based model. Perplexity is positioned as the google competitor to disrupt its ad-based search business using AI tools (Financial Times)

• How big tech is circumventing antitrust regulators by clever 'acquihire' strategy? there are 3 start-ups where big tech (Microsoft - Inflection, Amazon - Adept, Google - Character.ai) brought most of the rights to the start-up's product and also became the initial investors. Once the start-up got to its scale they hired the key talent into the company and left the skeleton of the original company still in place without giving any room for litigation (Tanay Jaipuria, Substack)

Technology updates from AI

• A new architecture in deep learning neural networks. The new networks are called Kolmogorov-Arnold Networks (KANs), after two mathematicians who studied how functions could be combined.

   

  "In the new architecture, the synapses play a more complex role. Instead of simply learning how strong the connection between two neurons is, they learn the full nature of that connection—the function that maps input to output. Unlike the activation function used by neurons in the traditional architecture, this function could be more complex—, a “spline” or combination of several functions—and is different in each instance."

   

•  Why would we care - the resulting network is several orders of magnitude smaller compared to similarly performing MLP models (depending on the task performed - mathematical problems, image recognition). In one case it was as much as 100 times.

• Not just that, the resulting networks are a lot more explainable as now there is a way to inspect the functions rather than absolute 'weights' itself.

• However, The architecture won't go mainstream yet given that it is not inherently parallelisable as the MLP networks and takes lot longer to train. A disadvantage while training LLMs. 

   

   (Matthew Hutson, IEEE Spectrum)

Long read - Generative AI Industry Dynamics

• It's just not clear if there will be a long-term, winner-take-all dynamic in generative AI. Who owns the generative AI platform is an insightful investigation into competitive advantage and long-term revenue stability at various layers of the generative AI business. The below paragraph summarises it well.

• "There don’t appear to be any systemic moats in generative AI today. As a first-order approximation, applications lack strong product differentiation because they use similar models; models face unclear long-term differentiation because they are trained on similar datasets with similar architectures; cloud providers lack deep technical differentiation because they run the same GPUs; and even the hardware companies manufacture their chips at the same fabs."

• "There are, of course, the standard moats: scale moats (“I have or can raise more money than you!”), supply-chain moats (“I have the GPUs, you don’t!”), ecosystem moats (“Everyone uses my software already!”), algorithmic moats (“We’re more clever than you!”), distribution moats (“I already have a sales team and more customers than you!”) and data pipeline moats (“I’ve crawled more of the internet than you!”). But none of these moats tend to be durable over the long term. And it’s too early to tell if strong, direct network effects are taking hold in any layer of the stack."

   

  (Matt Bornstein, Guido Appenzeller, Martin Casado - Andreessen Horowitz)

Resources

• Deep Learning Foundations and concepts book written by Chris Bishop, Technical Fellow at Microsoft & Director of AI4Science at Microsoft. Foreword by Geoffrey Hinton, Yann LeCun, Yoshua Bengio (Chris Bishop & Hugh Bishop)

• A pictorial tutorial for the "Attention is all you need" paper where sensational "transformer architecture" in deep learning is first described (Jay Alammar)