What is an AI accelerator and why does it matter?

An AI accelerator is specialized hardware — such as a neural processing unit (NPU) or dedicated AI chip — designed to run machine learning workloads efficiently. As these chips become more powerful and energy-efficient, they enable AI tasks to run locally on devices like smartphones, laptops, and industrial equipment without sending data to the cloud.

Will on-device AI replace cloud AI entirely?

It is unlikely that on-device AI will fully replace cloud AI in the near term. More resource-intensive tasks such as large model training and complex inference will continue to rely on cloud infrastructure. However, edge AI will handle an increasing share of real-time, privacy-sensitive, and latency-critical workloads.

What are the business implications of the shift to on-device AI?

Businesses will need to reconsider their AI architecture strategies, weighing the cost savings and privacy benefits of edge processing against the flexibility and scale of cloud solutions. This shift also affects vendor relationships, data governance policies, and how organizations manage and update AI models deployed across distributed devices.

AI Accelerators Are Just Getting Started Wi...

AI Accelerators Are Just Getting Started With Powering On-Device AI. Will They Replace the Cloud?

Hardware advances are forcing a fundamental reckoning over where artificial intelligence processing happens and who controls it

This story was produced through MarketScale. See how Engineering & Construction teams put it to work with Partner & Channel Enablement.

By Software And Technology · May 16, 2024, 3:06 PM UTCAi AcceleratorsCuriosExperts TalkIntel

Key takeaways

AI accelerators are enabling powerful on-device inference, reducing dependence on cloud processing.

The shift to edge AI has significant implications for data privacy, latency, and operational costs.

Cloud and on-device AI are likely to coexist rather than one fully replacing the other.

In a rapidly evolving technological landscape, the discussion around AI accelerators is heating up. This timely analysis comes as industry leaders and researchers explore the incredible potential of on-device AI, which promises to transform computing by reducing reliance on cloud infrastructure. As the debate continues, key experts weigh in on the future of AI accelerators and their implications for businesses and consumers.

How soon will edge computing chips be powerful enough to eliminate the need for cloud-based AI computing, and what does this mean for the future of technology?

In this clip from a full episode of MarketScale's Experts Talk, we hear from:

Grant Powell, Founder of Curios
Mark Beccue, Top AI Market Research Analyst
Joel Polanco, Segment Manager at Intel Corporation

Key Takeaways

On-Device AI is Just Getting Started: The development of on-device AI is still in its infancy, but significant progress is being made. This shift marks the beginning of a move away from centralized cloud computing towards more capable edge devices.
Market Leaders' Strategic Moves: Companies like Apple have been proactive in preparing for this shift. For instance, Apple has included neural processing units (NPUs) in iPhones for years, even though they remain underutilized. This foresight demonstrates a commitment to future-proofing their devices.
Broader Integration Across Devices: The integration of AI accelerators is not limited to smartphones. There is a growing trend of embedding these technologies in PCs, laptops, and desktops, indicating a widespread adoption of on-device AI across various platforms.
Emerging Applications and Benefits: As edge computing capabilities improve, we can expect a surge in applications that benefit from local AI processing. This will lead to reduced latency, enhanced privacy, and a decreased need for constant internet connectivity.
Strategic Business Considerations: Businesses must consider the balance between cloud and edge computing when investing in AI technology. The potential for reduced cloud costs and increased operational efficiencies through on-device AI should be factored into their strategic planning.

Video TranscriptExpand ↓

At what point, you know, how far out in the future could the edge computing chips have enough processing capacity to not need AI cloud computing? And is that something that's taken into consideration, when educating people about either investing in AI accelerators or making a decision for their company and how they're gonna structure their their models and their requests and their hardware. For clarity, for me, I call that on device AI. So if you you're talking about more of a, personal chips and personal computers and, you know, mobile phones and edge devices. And to me, those are two different things, and I I'm not sure I would call those accelerators. I don't know what Joel and David think. Yeah. I I would, comment, like, you know, one one little known fact is, actually, Apple has had a NPU, a neural processing unit sitting in the iPhone unused, for several years now. And so it shows you the build ahead, the thought that's going into the future here with some of these companies. And, you know, now you see within the PC market large launches, for neural processing that is gonna go into our laptops and our desktops. And so, you know, I think with regards to edge computing or on device, it's really just getting started. And and it's always been something that's present where you have, central computing in in the cloud versus your edge computing, and it's a constant push and pull. Right? And, on the edge side right now with AI, we're just like, it's really just getting started. And so you're gonna see more and more applications that land on these devices and start to pull some of that compute away from the the cloud.

About the author

Software And Technology

AI Accelerators Are Just Getting Started With Powering On-Device AI. Will They Replace the Cloud?

Key Takeaways

You just read one expert. Imagine publishing your whole team.

Explore Channels

More from Engineering & Construction