BUG: preserve dtype for right/outer merge of datetime with different resolutions

[jorisvandenbossche]

Follow-up on #53213, fixing one more corner case for the right/outer joins making the key column object dtype.

Expanded the test added in that PR to test the different join types.
The whatsnew added in #53213 should also already cover this.

[jorisvandenbossche]

This is an ugly band-aid solution: the proper fix is to handle this in find_common_type. But I am not sure if changing find_common_type would be in scope for a bug fix release? (xref #46587 (comment))

cc @jbrockmendel

[jbrockmendel]

Changing find_common_type would affect concat, so im hesitant to call it a bugfix. But avoiding object may be nicer behavior, so id be open to it as an api change longer-term.

Do we know that both are tznaive at this point? If one is tzaware then object is correct (though i guess we should be able to determine that the merge will be empty?)

I've given a little bit of thought to something similar in the context of Index.get_indexer with mismatched resos. Instead of converting both to object, could find a fill_value not present in left and convert right to left.unit, filling non-convertible entries with that fill_value. The "finding a fill_value" part seems really hacky though. (update: when i wrote this paragraph I thought this chunk of code was in get_merge_keys)

[jorisvandenbossche]

Do we know that both are tznaive at this point? If one is tzaware then object is correct (though i guess we should be able to determine that the merge will be empty?)

I think so yes, normally we should raise an error earlier in the code if you try to merge on a tz-aware and tz-naive column:

In [1]: df1 = pd.DataFrame({"key": [pd.Timestamp("1970-01-01")], "a": [1]})

In [2]: df2 = pd.DataFrame({"key": [pd.Timestamp("1970-01-01", tz="Europe/Brussels")], "a": [1]})

In [3]: pd.merge(df1, df2)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [3], in <cell line: 1>()
----> 1 pd.merge(df1, df2)

File ~/scipy/pandas/pandas/core/reshape/merge.py:146, in merge(left, right, how, on, left_on, right_on, left_index, right_index, sort, suffixes, copy, indicator, validate)
    129 @Substitution("\nleft : DataFrame or named Series")
    130 @Appender(_merge_doc, indents=0)
    131 def merge(
   (...)
    144     validate: str | None = None,
    145 ) -> DataFrame:
--> 146     op = _MergeOperation(
    147         left,
    148         right,
    149         how=how,
    150         on=on,
    151         left_on=left_on,
    152         right_on=right_on,
    153         left_index=left_index,
    154         right_index=right_index,
    155         sort=sort,
    156         suffixes=suffixes,
    157         indicator=indicator,
    158         validate=validate,
    159     )
    160     return op.get_result(copy=copy)

File ~/scipy/pandas/pandas/core/reshape/merge.py:735, in _MergeOperation.__init__(self, left, right, how, on, left_on, right_on, axis, left_index, right_index, sort, suffixes, indicator, validate)
    727 (
    728     self.left_join_keys,
    729     self.right_join_keys,
    730     self.join_names,
    731 ) = self._get_merge_keys()
    733 # validate the merge keys dtypes. We may need to coerce
    734 # to avoid incompatible dtypes
--> 735 self._maybe_coerce_merge_keys()
    737 # If argument passed to validate,
    738 # check if columns specified as unique
    739 # are in fact unique.
    740 if validate is not None:

File ~/scipy/pandas/pandas/core/reshape/merge.py:1397, in _MergeOperation._maybe_coerce_merge_keys(self)
   1393     raise ValueError(msg)
   1394 elif not isinstance(lk.dtype, DatetimeTZDtype) and isinstance(
   1395     rk.dtype, DatetimeTZDtype
   1396 ):
-> 1397     raise ValueError(msg)
   1398 elif (
   1399     isinstance(lk.dtype, DatetimeTZDtype)
   1400     and isinstance(rk.dtype, DatetimeTZDtype)
   1401 ) or (lk.dtype.kind == "M" and rk.dtype.kind == "M"):
   1402     # allows datetime with different resolutions
   1403     continue

ValueError: You are trying to merge on datetime64[ns] and datetime64[ns, Europe/Brussels] columns for key 'key'. If you wish to proceed you should use pd.concat

[mroeschke]

Thanks @jorisvandenbossche