last day (23 days later) » 

15:53
0
A: How to merge two dataframes with overlapping data and special requirements - replacing intervals/rows using second dataframe?

jquriousIt looks like you want to create ranges from df2 and match those against df1? # Create a range for each From/To # There's probably a better way to do this ranges = df2.assign( Range=list(map(lambda row: list(range(row[0], row[1], 5)), df2[["From", "To"]].to_numpy())) ).expl...

This is exactly what I was trying to convey/ask! The rest, do I put if Change = True then Q_x = Q_y, otherwise use Q_x (same for all variables, of course in python language, not Excel)
I can edit the answer to add in the next step - I just left it like that to make it easy to tell if that was the output you were expecting.
ok I tried it and added a change column for each variable. It's almost all the way there. Now I only want to replace if I have the same length of replacement in each. For instance, if df1 (x) is missing in MRC-17 from 420-430, I see I have a 420-430 that matches in df2 (y) so I want to replace but in MRC-17 from 400-405 and 405-410 RM_x is missing, I don't have 400-410 interval in df2 (y), I have a 400-420 but I want it to equal the run of missing, if that makes sense? I don't want to replace 10 ft with 20 ft. Does that make sense?
I think I understand - it's possible for the smaller interval to match inside the larger interval - but it should not be considered a match.
Also, your expected output contains MRC-22 and MRC-23 whereas my example doesn't. Does this mean rows from df2 that are not in df1 need to be added?
Yes, the small interval in x I don't want to replace unless it has an equal interval in y. I did figure out the script to add the unmatched IDs back in but if you have a more efficient way to add it into this, I'm all ears. I think I pasted it at the end of my question with code. Thank you so much for all the help!
15:53
Perhaps you could add a data example for the smaller interval case, that would be helpful.
its already in there for smaller. For RM column you'll see MRC-17 at 400-410 and 415-420 it wants to replace but we can't replace since df1 (x) has a value in the middle at 410-415.
Hi @jqurious, it said to move this discussion to chat
Ah okay.
I tried what you had and expanded it to all 3 variables and it looks like it works on the first variable "Q" but not on RM, RQ. It's just filling those in and maybe I didn't expand right
I used chat for the first time earlier today, I didn't realize they had it on stackoverflow.
ranges = df2.assign(
   Range=list(map(lambda row:
         list(range(row[0], row[1], 5)),
         df2[["From", "To"]].to_numpy()))
).explode("Range")

matches = (
   df1.merge(
      ranges,
      left_on=["ID", "From"],
      right_on=["ID", "Range"],
      how="left")
)

# Keep only repeating -99 sequences
matches1 = matches.loc[
   (matches["Q_x"] == -99) &
   ((matches["Q_x"].shift(-1) == -99) | (matches["Q_x"].shift(+1) == -99)),
]
matches2 = matches.loc[
   (matches["RM_x"] == -99) &
   ((matches["RM_x"].shift(-1) == -99) | (matches["RM_x"].shift(+1) == -99)),
15:58
So, MRC-17 has an RM missing sequence of length 2. 400-410
And you need to check if 400-410 exists in df2
Does this mean in the current example None of the MRC-17 values should be updated?
16:46
let me look, I have a meeting. I'll be back later :) sorry and thank you for all the help! I work in the mining industry so only been using python about 10 years but normally whats needed is already built and this is not :(
17:06
No worries. It's a pretty interesting problem - the RM part is hard to visualize at the moment though, catch you later.
 
1 hour later…
18:10
# Group consecutive values together
RM = df1.assign(missing=(df1["RM"] != df1["RM"].shift()).cumsum())
# Only interesting in -99
RM = RM.loc[RM["RM"] == -99].groupby(["ID", "missing"])
# Get From/To for sequence
RM = pd.concat([RM["From"].first(), RM["To"].last()], axis=1)

"""
From To
ID missing
MRC-10 4 15 20
MRC-15 8 100 110
10 115 145
MRC-17 4 400 410
6 415 445
"""
This finds sequences of -99 in RM from df1 and turns it into the format of df2 - which may be part of solving the problem
Perhaps it can be done by converting df1's format into df2 - instead of expanding the ranges in df2 and turning it into df1's format. Or perhaps both approach need to be used.
I have to close this now, chat again later.
 
3 hours later…
21:36
Q = df1.assign(missing=(df1["Q"] != df1["Q"].shift()).cumsum())
Q = Q.loc[Q["Q"] == -99]

RM = df1.assign(missing=(df1["RM"] != df1["RM"].shift()).cumsum())
RM = RM.loc[RM["RM"] == -99]

columns = [
"ID", "From", "To_x", "Q_x", "RM_x", "RQ_x",
"From_y", "To_y", "Q_y", "RM_y", "RQ_y", "missing"
]

Q = pd.merge_asof(
df2.sort_values("From"),
Q.assign(From_y=Q["From"]).sort_values("From"),
by="ID",
on="From",
direction="nearest"
)[columns]

Q[["From_y", "To_y"]] = Q[["From_y", "To_y"]].astype("Int64")
I think going this way may be easier.
If you look at RM here - we can see From_y To_y is smaller than From To_x
 
1 hour later…
22:48
k I'm back, had a very long meeting ugh
k letme try this last piece
 
1 hour later…
23:55
but we would want to exclusively update. Just some background info since this is a chat and not public. the two datasets are analytical in a lab data.
We trust df1 over df2 because we trust the 1 lab over the 2 lab. But... if we don't have the data from lab 1 we want the data from lab 2 (only if the sample size was the same size). If I take a sample and take a big and small sample from it (lengths From To) I won't get the same answer analytically. So that's why I'm trying to fill 1 with "same size" from 2 variable by variable for each test. Just in case that helps

  last day (23 days later) »