shell script to collect Benchmarks

[logan-keede]

Which issue does this PR close?

• Part of Run DataFusion benchmarks regularly and track performance history over time #5504

Rationale for this change

Here is a suggestion on how to proceed with this project:

1. Create the converter from bench json --> line protocol (e.g. Export benchmark information as line protocol #6107)
2. Write a script that runs the bench.sh script to gather the clickbench performance numbers over the last 5 releases:

git checkout 40.0.0
./bench.sh run clickbench_partitioned
git checkout 41.0.0
./bench.sh run clickbench_partitioned
...
git checkout 45.0.0
./bench.sh run clickbench_partitioned

And then load/plot that data using Grafana.

Originally posted by @alamb in #5504

What changes are included in this PR?

Just a simple script to collect benchmarks for last 5 releases

Are these changes tested?

Yes, by running the script on my laptop.
using sh collect_bench.sh tpch.

Are there any user-facing changes?

Nope.

[logan-keede]

One problem that I sometimes encounter is that cargo decides to use arrow-arith v53.4.0 for particular releases which ends up giving compilation error.
I’m not sure why this happens. Sometimes, for the same script and same release, cargo decides to use some other version of arrow-arith and everything works fine.
If anyone knows a workaround, please let me know.

[alamb]

Sorry @logan-keede -- I am very excited to try this one out, but I ran otu of time today

[logan-keede]

Sorry @logan-keede -- I am very excited to try this one out, but I ran otu of time today

No problem at all! I look forward to your feedback whenever you have the time.

[comphead]

Thanks @logan-keede that is pretty neat, appreciate if you document the script and its usage in benchmarks/README.md

[logan-keede]

Just noticed that benchmarks/README.md is not included in prettier CI check, is that intended?