Benchmark results report that Microsoft’s “Copilot + PC” does not have as high AI processing performance as claimed – GIGAZINE

Microsoft is developing PCs suitable for performing AI tasks, including its AI assistant Copilot.Copilot+ PC” and is strengthening its presence in the AI-oriented PC market. However, it is actually sold as Copilot+ PC.Microsoft Surface Pro (11th generation)An AI hardware development company said that when it ran a benchmark, it found that the execution performance of the AI ​​was poor.Useful Sensorsis reporting.

GitHub – usefulsensors/qc_npu_benchmark: Code sample showing how to run and benchmark models on Qualcomm’s Window PCs
https://github.com/usefulsensors/qc_npu_benchmark

Microsoft says about Copilot+ PC, “40TOPSmore thanNPUWindows 11 device withis defining. In other words, even if a PC is equipped with a GPU with high AI processing performance, a PC without an NPU will not be recognized as a Copilot+ PC.

“NPU” is a processor designed to speed up AI inference processing. GPUs can also execute high-speed AI tasks, but because they have built-in units that handle not only AI processing but also various calculations, power consumption increases. On the other hand, NPUs specialized for AI processing have the minimum number of calculation units required, so even when performing the same AI processing,Consumes much less powerThat’s what he said.

Additionally, “TOPS” is a unit used as an index to express the processing speed of an AI chip. The computing power of traditional computers isfloating point arithmetic” indicates how many times a second can be performedFLOPS” was expressed. On the other hand, since AI tasks often perform integer operations rather than floating point operations, “TOPS”, which indicates the number of integer operations per second, is used to indicate the processing speed of AI tasks.

The “40TOPS or more NPU” listed as a requirement for Copilot+ PC means “an NPU that can perform more than 40 trillion integer operations per second.” Therefore, Useful Sensors actually conducted benchmark measurements of Microsoft Surface Pro (11th generation) using a test script.

Useful Sensors uses Python to run test scripts, but Python, available in the Microsoft Store as of October 2, 2024, does not support Arm architecture processors. Therefore, it is not suitable for running packages that access Qualcomm’s NPU installed in existing Copilot+ PCs. Useful Sensors instead uses the officialPython.org installerIt is explained that it was used.

Benchmarks are designed to resemble real-world AI models and are based on OpenAI’sWhisper6 giant layers similar to the most time-consuming layers in Transformers models, includingmatrix multiplicationIt is said that it will be executed. It works by measuring the actual time it takes to run a model from start to finish as latency, and then calculating the number of operations per second from that latency.

As a result of the test, it was found that Qualcomm’s Snapdragon X Elite (12 cores: clock frequency 3.40 GHz) running on Microsoft Surface Pro (11th generation) does not have as high AI processing performance as claimed. Benchmark results showed that the number of operations per second was 573 billion (0.573TOPS), which was far short of the 45 trillion operations per second (45TOPS) promoted in marketing materials.

On the other hand, when the same benchmark test was run on an Nvidia Geforce RTX 4080 installed on a notebook PC, it was reported that the number of operations per second was 2.16 trillion (2.1TOPS).

Useful Sensors says, “We’ve seen how the underlying hardware works so effectively on other platforms, such as Android, that in the future we’re seeing how these things work at either the application, framework, or driver level. We hope that software changes will be made that will improve results.”