Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

llama : remove beam search #7736

Merged
merged 1 commit into from
Jun 4, 2024
Merged

llama : remove beam search #7736

merged 1 commit into from
Jun 4, 2024

Conversation

ggerganov
Copy link
Owner

The beam-search functionality should be implemented in user-code, utilizing the llama_kv_cache_ API. The existing implementation is likely not used at all, so I think it is ok to skip the deprecation phase

martindevans added a commit to martindevans/LLamaSharp that referenced this pull request Jun 4, 2024
@martindevans
Removed beam search. Upstream is removing it: ggerganov/llama.cpp#7736
283358c
@martindevans martindevans mentioned this pull request Jun 4, 2024
Merged
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jun 4, 2024

📈 llama.cpp server for bench-server-baseline on Standard_NC4as_T4_v3 for phi-2-q4_0: 550 iterations 🚀

Expand details for performance related PR only
  • Concurrent users: 8, duration: 10m
  • HTTP request : avg=8474.72ms p(95)=20396.97ms fails=, finish reason: stop=498 truncated=52
  • Prompt processing (pp): avg=93.34tk/s p(95)=398.57tk/s
  • Token generation (tg): avg=45.29tk/s p(95)=46.06tk/s
  • ggml-org/models/phi-2/ggml-model-q4_0.gguf parallel=8 ctx-size=16384 ngl=33 batch-size=2048 ubatch-size=256 pp=1024 pp+tg=2048 branch=gg/remove-beam-search commit=fb1e7b08e8cfe84ecf5b9a3225b8dc84ab436fd2

prompt_tokens_seconds

More 
---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 550 iterations"
    y-axis "llamacpp:prompt_tokens_seconds"
    x-axis "llamacpp:prompt_tokens_seconds" 1717506705 --> 1717507327
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 457.05, 457.05, 457.05, 457.05, 457.05, 692.93, 692.93, 692.93, 692.93, 692.93, 686.31, 686.31, 686.31, 686.31, 686.31, 712.45, 712.45, 712.45, 712.45, 712.45, 775.99, 775.99, 775.99, 775.99, 775.99, 772.18, 772.18, 772.18, 772.18, 772.18, 792.06, 792.06, 792.06, 792.06, 792.06, 806.49, 806.49, 806.49, 806.49, 806.49, 822.94, 822.94, 822.94, 822.94, 822.94, 823.46, 823.46, 823.46, 823.46, 823.46, 827.84, 827.84, 827.84, 827.84, 827.84, 813.09, 813.09, 813.09, 813.09, 813.09, 824.2, 824.2, 824.2, 824.2, 824.2, 833.16, 833.16, 833.16, 833.16, 833.16, 845.88, 845.88, 845.88, 845.88, 845.88, 847.93, 847.93, 847.93, 847.93, 847.93, 843.64, 843.64, 843.64, 843.64, 843.64, 840.61, 840.61, 840.61, 840.61, 840.61, 819.69, 819.69, 819.69, 819.69, 819.69, 825.31, 825.31, 825.31, 825.31, 825.31, 831.55, 831.55, 831.55, 831.55, 831.55, 831.88, 831.88, 831.88, 831.88, 831.88, 833.12, 833.12, 833.12, 833.12, 833.12, 829.18, 829.18, 829.18, 829.18, 829.18, 831.63, 831.63, 831.63, 831.63, 831.63, 833.67, 833.67, 833.67, 833.67, 833.67, 845.88, 845.88, 845.88, 845.88, 845.88, 846.57, 846.57, 846.57, 846.57, 846.57, 846.77, 846.77, 846.77, 846.77, 846.77, 850.25, 850.25, 850.25, 850.25, 850.25, 851.82, 851.82, 851.82, 851.82, 851.82, 848.37, 848.37, 848.37, 848.37, 848.37, 853.36, 853.36, 853.36, 853.36, 853.36, 860.22, 860.22, 860.22, 860.22, 860.22, 865.08, 865.08, 865.08, 865.08, 865.08, 870.75, 870.75, 870.75, 870.75, 870.75, 865.62, 865.62, 865.62, 865.62, 865.62, 863.91, 863.91, 863.91, 863.91, 863.91, 865.04, 865.04, 865.04, 865.04, 865.04, 868.05, 868.05, 868.05, 868.05, 868.05, 872.74, 872.74, 872.74, 872.74, 872.74, 873.23, 873.23, 873.23, 873.23, 873.23, 872.42, 872.42, 872.42, 872.42, 872.42, 870.4, 870.4, 870.4, 870.4, 870.4, 869.8, 869.8, 869.8, 869.8, 869.8, 872.22, 872.22, 872.22, 872.22, 872.22, 873.13, 873.13, 873.13, 873.13, 873.13, 871.5, 871.5, 871.5, 871.5, 871.5, 871.36, 871.36, 871.36, 871.36, 871.36, 870.03, 870.03, 870.03, 870.03, 870.03, 871.77, 871.77, 871.77, 871.77, 871.77, 869.45, 869.45, 869.45, 869.45, 869.45, 867.63, 867.63, 867.63, 867.63, 867.63, 867.13, 867.13, 867.13, 867.13, 867.13, 868.02, 868.02, 868.02, 868.02, 868.02, 867.57, 867.57, 867.57, 867.57, 867.57, 868.71, 868.71, 868.71, 868.71, 868.71, 870.05, 870.05, 870.05, 870.05, 870.05, 869.33, 869.33, 869.33, 869.33, 869.33, 871.75, 871.75, 871.75, 871.75, 871.75, 872.08, 872.08]
Loading
predicted_tokens_seconds
More 
---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 550 iterations"
    y-axis "llamacpp:predicted_tokens_seconds"
    x-axis "llamacpp:predicted_tokens_seconds" 1717506705 --> 1717507327
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 43.47, 43.47, 43.47, 43.47, 43.47, 43.47, 43.47, 43.47, 43.47, 43.47, 31.04, 31.04, 31.04, 31.04, 31.04, 30.28, 30.28, 30.28, 30.28, 30.28, 31.95, 31.95, 31.95, 31.95, 31.95, 33.14, 33.14, 33.14, 33.14, 33.14, 33.95, 33.95, 33.95, 33.95, 33.95, 34.86, 34.86, 34.86, 34.86, 34.86, 35.23, 35.23, 35.23, 35.23, 35.23, 35.22, 35.22, 35.22, 35.22, 35.22, 35.12, 35.12, 35.12, 35.12, 35.12, 35.06, 35.06, 35.06, 35.06, 35.06, 34.38, 34.38, 34.38, 34.38, 34.38, 34.26, 34.26, 34.26, 34.26, 34.26, 32.2, 32.2, 32.2, 32.2, 32.2, 30.14, 30.14, 30.14, 30.14, 30.14, 29.78, 29.78, 29.78, 29.78, 29.78, 29.94, 29.94, 29.94, 29.94, 29.94, 30.27, 30.27, 30.27, 30.27, 30.27, 30.35, 30.35, 30.35, 30.35, 30.35, 30.51, 30.51, 30.51, 30.51, 30.51, 30.46, 30.46, 30.46, 30.46, 30.46, 30.69, 30.69, 30.69, 30.69, 30.69, 30.64, 30.64, 30.64, 30.64, 30.64, 30.93, 30.93, 30.93, 30.93, 30.93, 31.13, 31.13, 31.13, 31.13, 31.13, 30.96, 30.96, 30.96, 30.96, 30.96, 31.13, 31.13, 31.13, 31.13, 31.13, 31.34, 31.34, 31.34, 31.34, 31.34, 31.52, 31.52, 31.52, 31.52, 31.52, 31.51, 31.51, 31.51, 31.51, 31.51, 31.61, 31.61, 31.61, 31.61, 31.61, 31.72, 31.72, 31.72, 31.72, 31.72, 31.77, 31.77, 31.77, 31.77, 31.77, 31.64, 31.64, 31.64, 31.64, 31.64, 31.16, 31.16, 31.16, 31.16, 31.16, 30.89, 30.89, 30.89, 30.89, 30.89, 30.83, 30.83, 30.83, 30.83, 30.83, 30.92, 30.92, 30.92, 30.92, 30.92, 31.06, 31.06, 31.06, 31.06, 31.06, 31.24, 31.24, 31.24, 31.24, 31.24, 31.24, 31.24, 31.24, 31.24, 31.24, 30.92, 30.92, 30.92, 30.92, 30.92, 30.34, 30.34, 30.34, 30.34, 30.34, 29.97, 29.97, 29.97, 29.97, 29.97, 29.44, 29.44, 29.44, 29.44, 29.44, 29.43, 29.43, 29.43, 29.43, 29.43, 29.37, 29.37, 29.37, 29.37, 29.37, 29.29, 29.29, 29.29, 29.29, 29.29, 29.3, 29.3, 29.3, 29.3, 29.3, 29.37, 29.37, 29.37, 29.37, 29.37, 29.35, 29.35, 29.35, 29.35, 29.35, 29.22, 29.22, 29.22, 29.22, 29.22, 29.25, 29.25, 29.25, 29.25, 29.25, 29.27, 29.27, 29.27, 29.27, 29.27, 29.41, 29.41, 29.41, 29.41, 29.41, 29.52, 29.52, 29.52, 29.52, 29.52, 29.64, 29.64, 29.64, 29.64, 29.64, 29.73, 29.73, 29.73, 29.73, 29.73, 29.74, 29.74, 29.74, 29.74, 29.74, 29.75, 29.75]
Loading

Details

kv_cache_usage_ratio

More 
---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 550 iterations"
    y-axis "llamacpp:kv_cache_usage_ratio"
    x-axis "llamacpp:kv_cache_usage_ratio" 1717506705 --> 1717507327
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.12, 0.12, 0.12, 0.12, 0.12, 0.38, 0.38, 0.38, 0.38, 0.38, 0.15, 0.15, 0.15, 0.15, 0.15, 0.11, 0.11, 0.11, 0.11, 0.11, 0.16, 0.16, 0.16, 0.16, 0.16, 0.11, 0.11, 0.11, 0.11, 0.11, 0.15, 0.15, 0.15, 0.15, 0.15, 0.14, 0.14, 0.14, 0.14, 0.14, 0.15, 0.15, 0.15, 0.15, 0.15, 0.17, 0.17, 0.17, 0.17, 0.17, 0.15, 0.15, 0.15, 0.15, 0.15, 0.29, 0.29, 0.29, 0.29, 0.29, 0.34, 0.34, 0.34, 0.34, 0.34, 0.46, 0.46, 0.46, 0.46, 0.46, 0.43, 0.43, 0.43, 0.43, 0.43, 0.35, 0.35, 0.35, 0.35, 0.35, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.18, 0.18, 0.18, 0.18, 0.18, 0.22, 0.22, 0.22, 0.22, 0.22, 0.23, 0.23, 0.23, 0.23, 0.23, 0.14, 0.14, 0.14, 0.14, 0.14, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.14, 0.14, 0.14, 0.14, 0.14, 0.11, 0.11, 0.11, 0.11, 0.11, 0.13, 0.13, 0.13, 0.13, 0.13, 0.12, 0.12, 0.12, 0.12, 0.12, 0.16, 0.16, 0.16, 0.16, 0.16, 0.21, 0.21, 0.21, 0.21, 0.21, 0.1, 0.1, 0.1, 0.1, 0.1, 0.16, 0.16, 0.16, 0.16, 0.16, 0.23, 0.23, 0.23, 0.23, 0.23, 0.32, 0.32, 0.32, 0.32, 0.32, 0.39, 0.39, 0.39, 0.39, 0.39, 0.24, 0.24, 0.24, 0.24, 0.24, 0.16, 0.16, 0.16, 0.16, 0.16, 0.14, 0.14, 0.14, 0.14, 0.14, 0.1, 0.1, 0.1, 0.1, 0.1, 0.2, 0.2, 0.2, 0.2, 0.2, 0.51, 0.51, 0.51, 0.51, 0.51, 0.51, 0.51, 0.51, 0.51, 0.51, 0.44, 0.44, 0.44, 0.44, 0.44, 0.34, 0.34, 0.34, 0.34, 0.34, 0.21, 0.21, 0.21, 0.21, 0.21, 0.28, 0.28, 0.28, 0.28, 0.28, 0.25, 0.25, 0.25, 0.25, 0.25, 0.19, 0.19, 0.19, 0.19, 0.19, 0.18, 0.18, 0.18, 0.18, 0.18, 0.19, 0.19, 0.19, 0.19, 0.19, 0.32, 0.32, 0.32, 0.32, 0.32, 0.11, 0.11, 0.11, 0.11, 0.11, 0.25, 0.25, 0.25, 0.25, 0.25, 0.13, 0.13, 0.13, 0.13, 0.13, 0.19, 0.19, 0.19, 0.19, 0.19, 0.13, 0.13, 0.13, 0.13, 0.13, 0.16, 0.16, 0.16, 0.16, 0.16, 0.15, 0.15, 0.15, 0.15, 0.15, 0.22, 0.22, 0.22, 0.22, 0.22, 0.29, 0.29]
Loading
requests_processing
More 
---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 550 iterations"
    y-axis "llamacpp:requests_processing"
    x-axis "llamacpp:requests_processing" 1717506705 --> 1717507327
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 2.0, 2.0, 2.0, 2.0, 8.0, 8.0, 8.0, 8.0, 8.0, 2.0, 2.0, 2.0, 2.0, 2.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 6.0, 6.0, 6.0, 6.0, 6.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0, 5.0, 5.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 1.0, 1.0, 1.0, 1.0, 1.0, 2.0, 2.0, 2.0, 2.0, 2.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0, 5.0, 5.0, 2.0, 2.0, 2.0, 2.0, 2.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 8.0, 8.0, 8.0, 8.0, 8.0, 3.0, 3.0, 3.0, 3.0, 3.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 3.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 2.0, 2.0, 2.0, 2.0, 2.0, 6.0, 6.0, 6.0, 6.0, 6.0, 3.0, 3.0]
Loading

@ggerganov ggerganov merged commit 0cd6bd3 into master Jun 4, 2024
75 checks passed
@ggerganov ggerganov deleted the gg/remove-beam-search branch June 4, 2024 18:23
@AdolfVonKleist AdolfVonKleist mentioned this pull request Jun 7, 2024
Closed
@thxCode thxCode mentioned this pull request Jun 18, 2024
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
examples
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@ggerganov