last day (15 days later) » 

9:28 AM
0
A: Fastest way to process a file & DB insert - Java Multi-Threading

lexicoreOther answers have already pointed out that you should use batch insertions. I think for fastest imports you shouldn't actually go through Java at all. See Fast Database Import in the H2 documentation: To speed up large imports, consider using the following options temporarily: SET L...

 
have updated my insert script, using executeBatch, it's taking around 5 seconds for 250,000 records
cannot use CSVREAD as i have other data as well, which i need to ignore before inserting them to DB. so PreparedStatement is the ideal one for my requirement.
 
What do you mean by "other data"? If it's unnecessary columns, you can configur it in the query.
 
other data means other columns - which are not supposed to get into DB. BTW, i am getting TXT file as input, which contains 10 columns with space(multiple or single) difference, out of them only 4 columns i need to pick and insert into my DB. Have . updated my original post with the actual logic.
after using LOG=0;CACHE_SIZE=65536;LOCK_MODE=0;UNDO_LOG=0 - data is getting loaded with in 4 seconds (1sec saved) for 250,000 records
 
@Kumar You can insert a subset of collumns, it's not a problem. In CSVREAD you can configure delimiters so maybe you can read your TXT file as a CSV file. Maybe not.
@Kumar If your code is already working, maybe you could post it on Code Review and ask if further optimizations are possible.
 
Hi lexicore
@lexicore, that's the code i have which i posted in original thread, that needs to be optimized, it's taking about 4 seconds to load 250,000 records, can it be reduced further ? Please suggest.
 
9:32 AM
From my point of view it's quite fine, but maybe someone else would have an idea.
 
where can i post it for Code Review as you suggested ?
 
I would really suggest that you convert one of your files to CSV (I think it's quite easy to do manually, just search/replace). And then try CSVREAD. You'll see if it makes sense. If it does (like giving you your <2 seconds), consider parsing TXT and saving it as prepared CSV and the doing CSVREAD. Converting 30Mb TXT to CSV should be very fast.
See this for their on/off-topic policy:
https://codereview.meta.stackexchange.com/questions/5777/a-guide-to-code-review-for-stack-overflow-users
But I think that since you have a working code and a specific goal (optimization) and welcome any improvement, this should be on-topic.
Sorry, have to go now. Best wishes and good luck.
 
But how can i pick up perticular columns from CSV, i don't need entire 10 columns, i just need 4 columns
 
When you do SELECT FROM CSVREAD you can specify columns there, I think. Not sure exactly how, I'm not that familiar with H2, check their docs. Must be possible.
 
sure, thanks lexicore
nice chatting with you
 

last day (15 days later) »