last day (45 days later) » 

10:17 AM
-1
A: Hive Query partioned and non partitioned column in where condition is not working

Kanagaraj dhanapalit might be external table issue. try: SELECT * FROM( SELECT * FROM mytab where dt_id = '20170101' and job_id = '100' )extble WHERE col1='XYZ';

 
Say what?! ....
 
The ultimate of aim the above query is we have to filter data based the three columns. so without knowing internal bug of hive we can achieve the results using subquery's
First it will filter the values based on partition columns and from the filtered data we can apply the next level of filter
 
And what makes you think that the optimizer does no unfold the sub-query, generating the exact same execution plan as the original query?
 
The execution plan is different here. Step 1: it will filter based on partition column, Step 2: it will filter the data from Step 1.
Stage 1: inner query and Stage 2: outer query
it will work parallel also
if you want i can remove my commands. i dont know why you downgrade my command. sorry
 
Have you actually checked the execution plan?
 
10:17 AM
yes i have lot query's like this
Could tell me what i have made the mistake here?
do you want me to exist this conversion or do you want me to delete the post
 
Run the following code, see that you are wrong and then delete the answer. create table t (i int) partition by (p int); explain select * from (select * from t where p=1) t where i=1;
 
i thought that it might me internal issue in hive beenline, so i have presented alternative way to do that. thats it
dear friend. it is different from what we are talking
Take 1 from 100 numbers
if you partitioned int(n/10)
there will be 10 partitions
if you want a number 50th partition i.e 55 to 59,
First we are filtering the 50th partition and then we are filtering the number as we want
is it make sense?
yes i am on the chat
 
Please go ahead and create any demo you want - and than check the execution plan. I would like to see a scenario where the data is being read from the relevant partitioned and then, on a different stage it is being filtered for the requested value.
 
are you there?
 
10:25 AM
could you tell me what will happen if everything on the single query
there will be only one stage right?
 
Sorry, I don't understand the request
Oh, O.k.
Yes, there will be one stage.
Do you currently have access to Hive?
 
yes i have
 
So please execute my demo
 
hive> explain select * from t where p=1 and i=1;
OK
STAGE DEPENDENCIES:
Stage-0 is a root stage

STAGE PLANS:
Stage: Stage-0
Fetch Operator
limit: -1
Processor Tree:
TableScan
alias: t
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Filter Operator
predicate: ((p = 1) and (i = 1)) (type: boolean)
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
expressions: 1 (type: int), 1 (type: int)
outputColumnNames: _col0, _col1
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
hive> explain select * from (select * from t where p=1) t where i=1;
OK
STAGE DEPENDENCIES:
Stage-0 is a root stage

STAGE PLANS:
Stage: Stage-0
Fetch Operator
limit: -1
Processor Tree:
TableScan
alias: t
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Filter Operator
predicate: ((p = 1) and (i = 1)) (type: boolean)
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
expressions: 1 (type: int), 1 (type: int)
outputColumnNames: _col0, _col1
 
10:28 AM
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 is a root stage

STAGE PLANS:
Stage: Stage-1
Map Reduce

Stage: Stage-0
Fetch Operator
limit: -1
Stage: Stage-0
Fetch Operator
limit: -1
 
As you can see, both execution plans are the same.
 
this will select the partition folder
STAGE PLANS:
Stage: Stage-1
Map Reduce
Map part will scan the table
Reduce part will be filter the third column values
is it make sense?
 
hive> explain select * from (select * from t where p=1) t where i=1;
OK
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 is a root stage

STAGE PLANS:
Stage: Stage-1
Map Reduce

Stage: Stage-0
Fetch Operator
limit: -1

Time taken: 1.062 seconds, Fetched: 12 row(s)
ok buddy
both are same only
i am telling that
there may be bug due to external table
so i am suggest to try my logic
thats it
 
Whatever the issue is, it is unlikely that it will be solved by a variation on the original query that produces the exact same execution plane.
 
10:38 AM
is it same my execution plan and yours?
 
Your execution plans seem to be cut
Can you add a snapshot like I did?
 
i am new to here
i dont know how add the image
could you help me ?
hive> explain select * from (select * from t where p=1) t where i=1;
OK
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 is a root stage

STAGE PLANS:
Stage: Stage-1
Map Reduce

Stage: Stage-0
Fetch Operator
limit: -1

Time taken: 0.084 seconds, Fetched: 12 row(s)
hive>
 
You have an upload button right to the test window
text
 
10:59 AM
No i dont have button like that
sorry
the above messages is execution plan from my hive shell
it is different from yours
so it will make the difference when you execute query
if you want take it
otherwise just skip it
thanks
bye bye
 
I cant see it like that
Can you just sent it by email?
dudu.markovitz@gmail.com
 
sent
dear friend , why did you downvote me?
 

  last day (45 days later) »