last day (15 days later) » 

9:00 PM
2
A: HDFS -extra data after last expected column

MikeIf you append the following to your external table definition, after the ENCODING clause, it should help to resolve the issue where a small number of rows fail due to this issue: LOG ERRORS INTO my_err_table SEGMENT REJECT LIMIT 1 PERCENT; Here is a reference on this syntax: http://gpdb.docs.p...

 
The reason is we do not want to discard bad rows . instead want to see in Bad rows also in target system.
@JonRoberts please can we talk in chat.
only concern is we have limitation to perform this in format clause only. Additionaly, wanted to know about can we pass group of delimiters in external table format clause for ex something like this : format(delimiter ';' quote '~!@#%^&*(')
@Jon Roberts Also I tried doing variation in format clause. but it does not solves the purpose. still complaining about extra data column error . I also observed this Corppoaccountnumber column =ROC NATION; NH. because of this additional semi colon is coming . but not sure how to remove this in external table
 
Ali
I had similar issues in the past, it was easier to clean the data inflight from SQL Server to HDFS, if you have the option then do so. Or else as Sung said - use Pig or Spark to count/clean it up. Just fyi, I used ',' as delimiter and had to remove \u0000 \r \n \f characters from the data. I also noticed you're using ';' as quote character, is it intentional? From the docs .. i could see QUOTE Specifies the quotation character for CSV mode. The default is double-quote (")
 
@Ali we are doing ETL for data movement (using talend tool). If I have to read external table as mentioned in question what will be the Format clause here. still trying to find out the solution.
what quote,delimiter should I use here ?
 
Ali
You cannot format this data if it genuinely has an extra semi colon and your data is not quoted. I do not see your bad lines being quoted. By quoted it means if you've got ';' character in the data then the column value must be quoted using the quote character - default is double-quote. So if you have data value with semicolon e.g. some;thing then your field value should be "some;thing" notice the semi-colon containing field is quoted using double quote. You mentioned quote as ';' which is not correct, remove it from the definition and use the character talend uses to quote the delimiter.
You said you want to see the bad value in the target system - i'm not sure that will be possible - if PXF encounters bad row it would put an entry in Error table and move on to the next row. It your job to go to the error table and do the necessary action to fix the bad rows.
Try the following format clause FORMAT 'csv' (delimiter ';' null '') Besides i looked up talend it does provide CSV Options where you can specify what kind of escape character and Text enclosure you want to use. Text enclosure should match Quote and Escape character must match Escape in FORMAT clause.
Here is the talend reference
 
@Ali Thanks to you . FORMAT 'csv' (delimiter ';' null '') - it works for me. also in talend Hadoop setting field separator as ;
Hi ali
 
Ali
9:09 PM
Hi
What component in talend u're using to export data, tell me about your pipeline
 
We are using tmssqlserver -->thdfsoutput -->tgreenplumrow -->tgreenplumcomit
Do we use Madlib in gpdb using talend for automating forecast just like r's arima model
 

last day (15 days later) »