last day (14 days later) » 

2:33 AM
0
Q: CSV file with different number of records each line (CSV Reader)

Urie P.I am using this library: CSV Reader but the problem is the deformity of the .csv file. Example: ,UDEQPT,,PROMIS,,,,,,,,,,,,,,,,,,,,,,,,,10:20:15,27-Dec-2015, ,UDEQPT,,DELAY,,,,,,,,,am24134_1_drift,am24134.1_drift,229,19,,,3176.00,164.78,,,,,,5, 1.00,1,06:16:16,15-Jun-2016,,,,,,, ,UDEQPT,,DELAY...

 
How to handle that should be a business logic, which means case by case. Some will ignore the whole line, some may just reject the whole file. Maybe you can tell us how you want to handle and let see what we can help
 
@Alex The current code throws exception because there is no index 31 on the first row. It iterates until 54 which is the column count.
@AllanS.Hansen I'm fairly new to parsing so how exactly should I propery customize it based on what's the content of csv file. Is there a way to do it using that library?
@grek40 Index error since the fixed column count is 54 but the first row is only until index 30 so it throws error.
@grek40 It doesn't.
 
May I know the exception and also which line throw this exception?
 
@Alex Error: An unhandled exception of type 'System.ArgumentOutOfRangeException' occurred in LumenWorks.Framework.IO.dll. Additional information: Field index must be included in [0, FieldCount[. Specified field index was : '31'. on this line: Console.Write(string.Format(i + ". {0} |", string.IsNullOrEmpty(csv[i]) ? "MISSING" : csv[i]));
@Alex i tried to try and catch it but it doesn't read the values in indexes 31 until 53. Any idea how to count the fields by row/record?
Note that I also used csv.FieldCount instead of 54 but it only reads until index 30.
 
Could you try to put a full record (54 fields) at the first line for testing?
 
2:33 AM
@Alex I did that and it doesn't throw any exception.
@Alex In cases like the one above, is it possible to alter the first row of a .csv file to add comma delimiters just to match the field count to column count?
 
Great, I think the problem is the logic in CsvReader, seems it have ignore the Columns when you using ReadNextRecord, and get the first line for the field count. So, will it be ok append a full record to file before you give it to CsvReader? And you may treat it as header.
 
@Alex Actually not only the first row, but there are other rows unequal to 54 columns. How do I append it?
 
You mean it still throw exception if second line unequal to 54 columns? From source code, seems it will get the field count based on first line...
 
@Alex For that particular for loop it doesn't but I'm also writing it to SQL server and it says Received an invalid column length from the bcp client for colid 13.
 
Have you check the SQL accept null? But it seems you are not just read it, and need to process it. Be aware that is very dangerous to use these kind of malformed CSV for processing, cuz you never know where the missing field located (2nd field? last field? or in the middle)
 
2:40 AM
Should it be more efficient if I edit the contents of .csv file or create a data table then write to server?
 
You may need to figure out how you should handle it first, get the business logic clear before you coding
For example, you need to process a csv with columns (id,date,amt,user), you have 2 lines:
1,0,Alex
0,Alex
 

last day (14 days later) »