Tiiny Pocket AI Model Benchmarks

1. Model Overview

2. Test Environment

3. Speed Benchmark

Measured on real code and general_qa samples at 4K, 64K and 128K context lengths, with concurrency 1, prompt cache disabled and thinking enabled. Each row reports the average of 10 samples. The complete benchmark dataset is publicly available at Hugging Face.

4. Accuracy Benchmark Comparison

Both builds were evaluated with the same prompts and chat template. The reference was run on an A100 in BF16. All accuracy benchmarks were run with the EvalScope evaluation framework.

1. IFEval (Prompt Strict): Measures a model's ability to strictly comply with verifiable formatting, structural, and language constraints in prompts (all rules must be met).
2. BFCL v4 (Multi-turn Base): Evaluates function-calling (tool use) capabilities across multi-turn conversational interactions, testing tool selection, parameter accuracy, and context maintenance.
3. MMLU-Pro: Benchmarks advanced multi-step reasoning and domain expertise across 14 academic/professional subjects using rigorous 10-choice questions.
4. Needle in a Haystack: Evaluates long-context information retrieval by testing if a model can find a specific target fact hidden inside a massive block of irrelevant context.
5. HumanEval+: Measures Python code synthesis correctness using an expanded, heavily augmented test suite to prevent false positives.

1. Model Overview

2. Test Environment

3. Speed Benchmark

Measured on real code and general_qa samples at 4K, 64K and 128K context lengths, with concurrency 1, prompt cache disabled and thinking enabled. Each row reports the average of 10 samples. The complete benchmark dataset is publicly available at Hugging Face.

4. Accuracy Benchmark Comparison

Both builds were evaluated with the same prompts and chat template. The reference was run on an A100 in BF16. All accuracy benchmarks were run with the EvalScope evaluation framework.

1. IFEval (Prompt Strict): Measures a model's ability to strictly comply with verifiable formatting, structural, and language constraints in prompts (all rules must be met).
2. BFCL v4 (Multi-turn Base): Evaluates function-calling (tool use) capabilities across multi-turn conversational interactions, testing tool selection, parameter accuracy, and context maintenance.
3. MMLU-Pro: Benchmarks advanced multi-step reasoning and domain expertise across 14 academic/professional subjects using rigorous 10-choice questions.
4. Needle in a Haystack: Evaluates long-context information retrieval by testing if a model can find a specific target fact hidden inside a massive block of irrelevant context.
5. HumanEval+: Measures Python code synthesis correctness using an expanded, heavily augmented test suite to prevent false positives.

1. Model Overview

2. Test Environment

3. Speed Benchmark

Measured on real code and general_qa samples at 1K, 4K, 64K and 128K context lengths, with concurrency 1, prompt cache disabled and thinking enabled. Each row reports the average of 10 samples. The complete benchmark dataset is publicly available at Hugging Face.

4. Accuracy Benchmark Comparison

Both builds were evaluated with the same prompts and chat template. The reference was run on an A800 in BF16. All accuracy benchmarks were run with the EvalScope evaluation framework.

1. IFEval (Prompt Strict): Measures a model's ability to strictly comply with verifiable formatting, structural, and language constraints in prompts (all rules must be met).
2. BFCL v4 (Multi-turn Base): Evaluates function-calling (tool use) capabilities across multi-turn conversational interactions, testing tool selection, parameter accuracy, and context maintenance.
3. MMLU-Pro: Benchmarks advanced multi-step reasoning and domain expertise across 14 academic/professional subjects using rigorous 10-choice questions.
4. Needle in a Haystack: Evaluates long-context information retrieval by testing if a model can find a specific target fact hidden inside a massive block of irrelevant context.
5. HumanEval+: Measures Python code synthesis correctness using an expanded, heavily augmented test suite to prevent false positives.

1. Model Overview

2. Test Environment

3. Speed Benchmark

Measured on real code and general_qa samples at 4K, 64K and 128K context lengths, with concurrency 1, prompt cache disabled and thinking enabled. Each row reports the average of 10 samples. The complete benchmark dataset is publicly available at Hugging Face.

4. Accuracy Benchmark Comparison

Both builds were evaluated with the same prompts and chat template. The reference was run on an A100 in BF16. All accuracy benchmarks were run with the EvalScope evaluation framework.

1. IFEval (Prompt Strict): Measures a model's ability to strictly comply with verifiable formatting, structural, and language constraints in prompts (all rules must be met).
2. BFCL v4 (Multi-turn Base): Evaluates function-calling (tool use) capabilities across multi-turn conversational interactions, testing tool selection, parameter accuracy, and context maintenance.
3. MMLU-Pro: Benchmarks advanced multi-step reasoning and domain expertise across 14 academic/professional subjects using rigorous 10-choice questions.
4. Needle in a Haystack: Evaluates long-context information retrieval by testing if a model can find a specific target fact hidden inside a massive block of irrelevant context.
5. HumanEval+: Measures Python code synthesis correctness using an expanded, heavily augmented test suite to prevent false positives.

1. Model Overview

2. Test Environment

3. Speed Benchmark

The benchmark was run as a service evaluation over five audio datasets. A100 (GPU) is the reference side kept for comparison. Both sides scored the same fixed clips, and throughput and latency should be read together with the evaluation set and run setting in section 2. Accuracy is reported separately in section 4.

Dataset Details
1. AISHELL-1: Mandarin speech recorded in quiet indoor conditions; listed as aishell1 in this report.
2. AISHELL-4: Multi-speaker Mandarin meeting speech recorded with a microphone array; test set listed as aishell4_test.
3. AliMeeting: Mandarin meeting speech; this benchmark uses the far-field evaluation split, listed as alimeeting_eval_far.
4. AMI IHM: English meeting speech, individual headset microphone (IHM) recordings; test set listed as ami_ihm_test.
5. WenetSpeech: Multi-domain Mandarin speech corpus; this benchmark uses the meeting test set, listed as wenet_meeting_test.

4. Accuracy Benchmark Comparison

Both builds were evaluated with the same prompts and chat template. The reference was run on an A100 in BF16. All accuracy benchmarks were run with the EvalScope evaluation framework.

1. Model Overview

2. Test Environment

3. Speed Benchmark

The benchmark evaluated image generation performance across three test suites, totaling 1,040 images. The A100 GPU serves as the reference baseline. Both platforms were evaluated using the same fixed prompts. Throughput and latency should be interpreted alongside the test environment and run settings described in Section 2. Accuracy results are reported separately in Section 4.

Dataset Details
1. GenEval2: Tests text-to-image alignment for objects, attributes, relations, and counting; this report uses 300 prompts scored with Soft-TIFA.
2. T2I-CompBench++: Tests compositional image generation, including attribute binding, spatial relationships, and numeracy; this report uses 640 cases across eight categories.
3. Internal test suite (based on Qwen-Image-Bench): Used for the internal MUSIQ quality, semantic accuracy, and rendered-text exactness evaluations in this report.

4. Accuracy Benchmark Comparison