arrow-select: add support for optimized concatenation of struct arrays

[asubiotto]

Struct array concatenation previously fell back to concat_fallback, which uses MutableArrayData. This is an issue because any nested types within the struct will not benefit from optimized concatenation. Dictionaries within structs, for example, would not be merged and could result in excessive memory usage.

This commit adds no tests because there are already existing tests for struct concatenation.

Which issue does this PR close?

Closes #7516

What changes are included in this PR?

This PR adds concat_structs and uses it in the concat match statement.

Are there any user-facing changes?

No

[alamb]

Thanks @asubiotto -- I suspect the CI failure will be fixed by

• Fix Clippy in CI for Rust 1.87 release #7514

(not related to your PR)

[asubiotto]

Thanks @alamb, will rebase once that's in.

[alamb]

I wonder would it be possible to get some benchmarks (ideally added as another PR) for concatenating structs in

• https://github.com/apache/arrow-rs/blob/main/arrow/benches/concatenate_kernel.rs

Also, note to myself and other reviewers, I verified there are already functional tests for concatenating structs
https://github.com/apache/arrow-rs/blob/915e9928091c24a0116b135abb2596d6905a980d/arrow-select/src/concat.rs#L798-L797

[alamb]

Thanks @asubiotto -- I think the only thing that is needed is another test for concatenating arrays without nulls.

A benchmark would be really nice too

[asubiotto]

Thanks for the review @alamb. Added a test for struct concatenation with no nulls and will work on adding a benchmark in a separate PR.

[asubiotto]

Added the benchmark in: #7520.

Result of this PR:

concat struct with int32 and dicts
                        time:   [3.9948 µs 4.0098 µs 4.0257 µs]
                        change: [-13.181% -12.415% -11.227%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) high severe

This was around a 20% improvement without the primitive dictionary field, but I thought I might add that to the benchmark to see the improvements once #7519 is also in.

[alamb]

🤖 ./gh_compare_arrow.sh Benchmark Script Running
Linux aal-dev 6.11.0-1013-gcp #13~24.04.1-Ubuntu SMP Wed Apr 2 16:34:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing asubiotto-sconcat (1c779e8) to 1a5999a diff
BENCH_NAME=concatenate_kernel
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench concatenate_kernel
BENCH_FILTER=
BENCH_BRANCH_NAME=asubiotto-sconcat
Results will be posted here when complete

[alamb]

🤖: Benchmark completed

Details

group                                        asubiotto-sconcat                      main
-----                                        -----------------                      ----
concat 1024 arrays boolean 4                 1.03     28.7±0.03µs        ? ?/sec    1.00     28.0±0.05µs        ? ?/sec
concat 1024 arrays i32 4                     1.14     15.0±0.03µs        ? ?/sec    1.00     13.1±0.18µs        ? ?/sec
concat 1024 arrays str 4                     1.06     57.5±0.46µs        ? ?/sec    1.00     54.2±0.33µs        ? ?/sec
concat boolean 1024                          1.00    337.4±3.09ns        ? ?/sec    1.05    354.9±0.42ns        ? ?/sec
concat boolean 8192 over 100 arrays          1.00     44.2±0.06µs        ? ?/sec    1.15     50.9±0.06µs        ? ?/sec
concat boolean nulls 1024                    1.00    656.7±0.86ns        ? ?/sec    1.21    795.6±2.34ns        ? ?/sec
concat boolean nulls 8192 over 100 arrays    1.00     96.1±0.17µs        ? ?/sec    1.14    110.0±0.13µs        ? ?/sec
concat fixed size lists                      1.07   709.4±17.74µs        ? ?/sec    1.00   660.1±23.12µs        ? ?/sec
concat i32 1024                              1.00    441.7±1.03ns        ? ?/sec    1.03    454.1±1.08ns        ? ?/sec
concat i32 8192 over 100 arrays              1.00   215.5±14.35µs        ? ?/sec    1.01    218.0±4.25µs        ? ?/sec
concat i32 nulls 1024                        1.11    803.8±2.22ns        ? ?/sec    1.00   721.3±23.45ns        ? ?/sec
concat i32 nulls 8192 over 100 arrays        1.03   280.2±10.21µs        ? ?/sec    1.00    273.0±3.40µs        ? ?/sec
concat str 1024                              1.00     14.1±1.30µs        ? ?/sec    1.09     15.3±1.16µs        ? ?/sec
concat str 8192 over 100 arrays              1.00    105.2±0.73ms        ? ?/sec    1.01    106.1±1.13ms        ? ?/sec
concat str nulls 1024                        1.00      6.4±0.69µs        ? ?/sec    1.11      7.2±0.88µs        ? ?/sec
concat str nulls 8192 over 100 arrays        1.00     52.0±0.60ms        ? ?/sec    1.01     52.3±0.67ms        ? ?/sec
concat str_dict 1024                         1.02      2.7±0.01µs        ? ?/sec    1.00      2.7±0.02µs        ? ?/sec
concat str_dict_sparse 1024                  1.03      7.2±0.15µs        ? ?/sec    1.00      6.9±0.03µs        ? ?/sec

[Dandandan]

Collecting a.as_struct() here into vec is not strictly needed I think?
We can use arrays.iter() in the later loops?

[asubiotto]

We need to have these as StructArrays to call column(i) below so we might as well downcast here I think. Happy to change this if I'm missing something.

[alamb]

Can you also please merge/rebase this PR against main so it contains the latest benchmark?

THe run that just finished doesn't seem to include the new struct concat kernel

[alamb]

(I would be happy normally to do the merging / rebase but I don't have write access to the polarsignals repo)

[asubiotto]

Thanks for the additional review and apologies for the delay. I didn't have the bandwidth to work on this until now. Comments addressed + rebased.