人工智能与神经网络BenchMark--MLPerf

What is MLPerf

MLPerf is a benchmark suite that is used to evaluate training and inference performance of on-premises and cloud platforms. MLPerf is intended as an independent, objective performance yardstick for software frameworks, hardware platforms, and cloud platforms for machine learning.

The goal of MLPerf is to give developers a way to evaluate hardware architectures and the wide range of advancing machine learning frameworks.

MLPerf Training benchmarks

The suite measures the time it takes to train machine learning models to a target level of accuracy.

MLPerf Inference benchmarks

The suite measure how quickly a trained neural network can perform inference tasks on new data.

The MLPerf inference benchmark is intended as an objective way to measure inference performance in both the data center and the edge. Each benchmark has four measurement scenarios: server, offline, single-stream, and multi-stream. The following figure depicts the basic structure of the MLPerf Inference benchmark and the order that data flows through the system:

 Server and offline scenarios are most relevant for data center use cases.

while single-stream and multi-stream scenarios evaluate the workloads of edge devices. 

MLPerf Results Show Advances in Machine Learning Inference Performance and Efficiency | MLCommons

Today, MLCommons®, an open engineering consortium, released new results for three MLPerf™ benchmark suites - Inference v2.0, Mobile v2.0, and Tiny v0.7. These three benchmark suites measure the performance of inference - applying a trained machine learning model to new data. Inference enables adding intelligence to a wide range of applications and systems. Collectively, these benchmark suites scale from ultra-low power devices that draw just a few microwatts all the way up to the most powerful datacenter computing platforms. The latest MLPerf results demonstrate wide industry participation, an emphasis on energy efficiency, and up to 3.3X greater performance ultimately paving the way for more capable intelligent systems to benefit society at large.

The MLPerf Mobile benchmark suite targets smartphones, tablets, notebooks, and other client systems with the latest submissions highlighting an average 2X performance gain over the previous round.

MLPerf Mobile v2.0 includes a new image segmentation model, MOSAIC, that was developed by Google Research with feedback from MLCommons. The MLPerf Mobile application and the corresponding source code, which incorporates the latest updates and submitting vendors’ backends, are expected to be available in the second quarter of 2022.

MLPerf Suite Categories

MLPerf Training v2.0 is the sixth instantiation for training and consists of eight different workloads covering a broad diversity of use cases, including vision, language, recommenders, and reinforcement learning.

MLPerf Inference v2.0 tested seven different use cases across seven different kinds of neural networks. Three of these use cases were for computer vision, one for recommender systems, two for language processing, and one for medical imaging.

Image Classificataion:

MLPerf Traininguse ResNet v1.5 with the ImageNet dataset.

MLPerf Inference use ResNet v1.5 with the ImageNet dataset.

Object Detection (Lightweight)

MLPerf Training uses Single-Shot Detector (SSD) on the COCO 2017 dataset.
MLPerf Inference uses SSD-MobileNet-v1 (low-res, 0.09 MP), with the COCO 2017 dataset.

Object Detection (Heavyweight)

MLPerf Training uses Mask R-CNN on the COCO 2014 dataset.
MLPerf Inference uses SSD-ResNet-34 (high-res, 1.44 MP) with the COCO 2017 dataset.

Biomedical Image Segmentation

MLPerf Training use 3D U-Net with the KiTS19 dataset.

MLPerf Inference use 3D U-Net with the KiTS19 dataset.

Automatic Speech Recognition (ASR)

MLPerf Training use RNN-T on the Librispeech dataset.

MLPerf Inference use RNN-T on the Librispeech dataset.

Natural Language Processing (NLP)

MLPerf Training uses Bidirectional Encoder Representations from Transformers (BERT) on the Wikipedia 2020/01/01 dataset.
MLPerf Inference used BERT with the SQuAD v.1.1 dataset.

Recommendation

MLPerf Training use the Deep Learning Recommendation Model (DLRM) on Criteo 1TB dataset.

MLPerf Inference use the Deep Learning Recommendation Model (DLRM) on Criteo 1TB dataset.

Reinforcement Learning

MLPerf Training uses Mini Go benchmark based gameplay.

MLPerf Inference suite

The latest branch for MLPerf Inference is MLPerf Inference v2.0 (submission 02/25/2022)

GitHub - mlcommons/inference: Reference implementations of MLPerf™ inference benchmarks

MLPerf Mobile Inference suite

This benchmark suite measures how fast systems can process inputs and produce results using a trained model. 

GitHub - mlcommons/mobile_app_open: Mobile App Open

GitHub - mlcommons/mobile_models: MLPerf™ Mobile models

GitHub - mlcommons/mobile_datasets: Scripts to create performance-only test sets for MLPerf™ Mobile

GitHub - mlcommons/mobile_results_v2.0

Scenarios and Metrics

In order to enable representative testing of a wide variety of inference platforms and use cases, MLPerf has defined four different scenarios as described below. A given scenario is evaluated by a standard load generator generating inference requests in a particular pattern and measuring a specific metric.

Benchmarks

Each benchmark is defined by a Dataset and Quality Target. The following table summarizes the benchmarks in this version of the suite (the rules remain the official source of truth):

Each Mobile benchmark requires the single stream scenario. The Image classification benchmark permits an optional Offline scenario.

Divisions

MLPerf aims to encourage innovation in software as well as hardware by allowing submitters to reimplement the reference implementations. MLPerf has two Divisions that allow different levels of flexibility during reimplementation.

The Closed division is intended to compare hardware platforms or software frameworks “apples-to-apples” and requires using the same model as the reference implementation.

The Open division is intended to foster innovation and allows using a different model or retraining.

inference-mobile-20 results

https://mlcommons.org/en/inference-mobile-20/