Add low-level create_dataframe_from_blocks helper function

[jorisvandenbossche]

See my explanation at #56815

• closes #xxxx (Replace xxxx with the GitHub issue number)
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

[jbrockmendel]

Can we add a docstring (module and/or function level) to the effect of "we discourage this for everyone except pyarrow. if you think you have a use case for this, let us know at [...]"

[jorisvandenbossche]

@phofl might also have a use case in dask (I don't know id you already have a better idea now if that would be the case?)

[phofl]

Yeah we are working on changing how we shuffle data were this would be helpful (we will get a huge number of small data frames, so overhead is painful), but I agree that we should strengthen this a little bit that makes it clear that end users shouldn't need this

[jorisvandenbossche]

I already added a more generic note "For almost all use cases, you should use the standard pd.DataFrame(..) constructor instead." without naming specific libraries that use this.

What would we gain with a "if you think you have a use case for this, let us know at"? Learning about use cases where people would use this is certainly valuable, but in the end it will be public developer API and so if we would in the future change or remove it, we need to go through normal deprecation processes anyway, I think.

[jbrockmendel]

Hopefully we'll never have to revisit this again. But if we do, there is evidence that discussions around a deprecation here would be more painful than elsewhere. It would be helpful to know ahead of such a discussion if anyone else was using it. Moreover, the "let us know" is a chance to try to talk anyone out of using this.

[jorisvandenbossche]

Added "If you are planning to use this function, let us know by opening an issue at https://github.com/pandas-dev/pandas/issues."

[jbrockmendel]

Not my favorite thing in the world, but better than the status quo so i'm on board.

[jorisvandenbossche]

I added some basic roundtrip tests and tests for the corner cases that I am aware of (the numpy arrays instead of EA for datetime/timedelta, and passing 1D EAs for cases that are stored 2D internally).
(and using this in pyarrow is also passing the test suite there)

[jorisvandenbossche]

@jbrockmendel @phofl could I get a more in-depth review (if there are remaining comments)? I would like to get the change to use this included in pyarrow 16, so we don't have to worry about deprecation warnings when merging #57754 for 3.0 (for the case of using the latest released version of both libraries), but then I need to get this in in the coming days.

[jorisvandenbossche]

Small reminder here

[phofl]

thx @jorisvandenbossche