[Format] Inline parent validity bitmap into child validity bitmap

[takaaki7]

Describe the enhancement requested

Currently, when struct itself is null, child field's validity is unknown. So client must compute AND of those bitmaps to know child's validity.

I think inlining parent validity into child validity is more effective for query engines.

Component(s)

Format

[takaaki7]

Accessing Arrow Struct children requires checking both parent and child validity (p[i] && c[i]), which can feel cumbersome. For example, Java's StructVector.getChild(), needing explicit parent.isNull(i) checks.
I think setting the child's validity bit to null at the format level when the parent is null would reduce burden and overhead for users.

ref:
https://arrow.apache.org/blog/2022/10/08/arrow-parquet-encoding-part-2/

[felipecrv]

Currently, when struct itself is null, child field's validity is unknown. So client must compute AND of those bitmaps to know child's validity.

That is a feature, not a bug. The validity bitmap (aka "nulls") is a mask on top of the children arrays. Setting a value to null only requires updating the mask, not recursively traversing the child arrays to set them to null as well.

A similar pattern exists in other nested arrays like List: offsets[i + 1] - offsets[i] is not necessarily zero when the i-th list is null.

Even though it's a bit cumbersome when reading (bitwise AND necessary), preserving this invariant you're proposing on every operation that builds new arrays from scratch or from other arrays would be almost impossible.

[emkornfield]

This might be better to discuss on dev@ mailing but but I'd also add that changing this invariant is non-trivial in terms of versioning. I'm not sure there would be much interest in doing so.