last day (22 days later) » 

13:16
0
Q: Pyspark to map the exchange rate value in dataframe based on group of values

f.ivyI would like to convert values from a currency to a currency based on the Following logic: #df1# id from_curr to_curr Date item_number value_to_convert 1 AED EUR 2017-01-12 10 2000 1 AED EUR 2017-01-12 20 189 2 UAD EUR 2021-05-18 10 12.5 3 DZD EUR 2017-01-12 10 130 5 GBP EUR 2017-...

isn't this same as this question of yours?
Nope, actually here we may have two rates exchanges for the same currency, and an id have several values for the same currency.
Can you check the example of the output cited below?
okay, some tweaks needed but generally the idea remains the same
Yes, i used the approach cited here: stackoverflow.com/a/73082581/15076492
are you saying there are some currencies that have 2 dates but only 1 exchange rate, and that they get dropped due to the min criterion? i don't think that should happen.
if you feel the solution doesn't work, please share the use cases (data samples) where you see it breaking. based on the inputs provided, the solution matches the expected output
13:16
Yes, thanks it's turning on the dataset
There is still a problem, as you can see (in the modified version of my question), the same ID may have several items with different amounts for the same date, after applying the last approach i got the output in my annoncement.
please update df1 and df2 in the question with your use cases instead of screenshots
I have updated the dataframes, you can check it please.
Hello Samkart
how did you get the converted values, as in which rates were used?
suppose for these records
We take the value to convert we devide it by the exchange_rate
both of them have the same date, so they will be mapped to the same closest date from exchange df
so, both of them will have the same exchange rate, correct?
13:19
yes this is why the code is giving me the min value
yes
but as you see each value is stocked for a different item
390.39 for item 10 id 1
and 19.89 for item 20 id=1
so, -5.123 will be used for both, right?
yep
the code works fine
no it's giving me this
13:22
which code?
did you use
the one in the answer
look
table_calcul = (
purchase_order_item
.select(
F.col("last_modification_date"),
F.col("purchase_order_id"),
F.col("transaction_currency"),
F.col("group_currency"),
F.col('item_number'),
F.col("net_amount_in_transaction_currency"))
.join(

currency_rates_exchange,
["transaction_currency", 'group_currency'],
"left")
.withColumn(
'dt_diff',
F.abs(F.datediff('last_modification_date', 'currency_date')))
.withColumn(
'min_dt_diff',
F.min('dt_diff').over(wd.partitionBy(['transaction_currency', 'last_modification_date'])))
purchase_order_item = df1
currency_rates_exchange =df2
net_amount_in_transaction_currency = value_to_convert
the date in the partitionBy() should be from the currency (exchange rate) table
from_curr_start_dt?
yes, from_curr_start_dt is from the exchange_rate df
13:29
but in the 3rd line you using over(wd?partionnedBy('from_curr','dt')
and 'dt' is from df1
oh right, my bad
how can i send yo screen sot ?
shot
there's an upload button beside the message box
save the ss on your computer and upload it
I dont have it never mind
okay, confirm if this is correct
13:34
This is the output am looking for
but i'm not getting this
i don't have it hahaha
maybe because i don't have all privilèges
maybe you're not replicating the use cases here or not replicating the query correctly
did you check my last df1 and df2
yes, the result I pasted above is using that data
13:38
Look at my post again i added the screen shot there
i will deleted it after
this is what am getting
keep the exchange rate and value_to_convert in the result_df
for the EUR case
keep all columns in the screenshot
dates, rates, values
ok
U can check now
for id: 0013425466, converted values are duplicated and reversed for some
4 0013425466 EUR EUR 00030 2018-12-03 199.50 null null 199.500000000000
5 0013425466 EUR EUR 00030 2018-12-03 199.50 null null 22.500000000000
You there?
13:53
you needn't worry about that -- it isn't a code error, should be platform specific
notice that 2018-12-03 has 2 records for 22.5 and 2 for 199.5
wait i will check
yes it has two records, but as you see the converted value should remain the same
since it's from EUR to EUR
but it took another amount
if value to convert is 199.50 the converted value should remain 199.50
it's a platform specific issue, nothing to do with the code
see value for 49
2020-07-03
yes but notice that the purchase_id is different
why not take a distinct after your select?
0019374142 and 0019374530
13:58
```
purchase_order_item
.select(
F.col("last_modification_date"),
F.col("purchase_order_id"),
F.col("transaction_currency"),
F.col("group_currency"),
F.col('item_number'),
F.col("net_amount_in_transaction_currency")).distinct().join()
```
i will try it
i got the same problem
what if we change the coeslcse
with when. otherwise
@f.ivy this works the same way coalesce
why you not getting duplicates
like me
the df1 you posted does not have any
yes it doesnt
what does coalesce do?
14:06
replaces null with provided value -- link
am checking the data for 0013425466
There is no problem with the data for this id
why don't we use another way of doing this logique
something like : withcolumn('converted_value', func.when(curr_from == 'EUR', 'value_to_convert').otherwise(conversion formula)
 
2 hours later…
15:57
Your code was correct
i just got the error i made

  last day (22 days later) »