« first day (1785 days earlier)      last day (3380 days later) » 

03:22
0
Q: Bulk insertion from tibco activespace to Cassandra fails after 3k-4k

RemboI am able to do insert operation of 50k records into tibco active spaces by using a thread pool of 25 threads where the pool is executed by ExecutorService. but, How can we perform bulk database insertions in Cassandra? I tried in following way: Created a subclass to tibco Persister, using it...

 
2 hours later…
05:41
@ug_, hi.
ug_
ug_
hey
So you have some ideas about my question, right?
ug_
ug_
Yea well, a few questions first. Im curious how your data is structured
Having it in one mega file is... kinda wild
That's true. I was thinking about some way of storing document as separate parts referencing one another.
But it should be defined by user in some sort of type definition language.
ug_
ug_
Have you looked at XML?
05:45
Yes, XML is next option when it becomes clear to me that jSON doesn't solve what I need.
But XML is not as simple as JSON though.
ug_
ug_
the problem I see with JSON is scale-ability. If your inserting that much data into it there really is no way to keep read/write times down.
I should say thats the problem with storing all your data in one file. Unless you have a specific way to jump to a position to find an entry
I had some experiments with extracting some sub-data parts from JSON according some JSON-path like rules.
For large documents like 1gig it takes about few seconds to get 1 meg sub-document.
ug_
ug_
might I ask what will be accessing the data and what kind of speeds it will require?
If you go through it as token sequence
ug_
ug_
also what and where the application will be distributed.
05:51
It's good question. I think ideally there should be some way in future to register user's defined service (in docker image) and user's defined UI widgets accessing the data. In loooooong term though.
It will be distributed on cluster in Berkeley (more precisely LBNL). And it will be (it is actually already) open source project called KBase (github.com/kbase).
ug_
ug_
hey nice LBNL, I have done some work with them ;)
So you see, there are some scientists trying to find good solution. :)
ug_
ug_
What is going into this JSON format? Is it files as base 64?
Biological info, it's typically human readable text data, but it would be nice to have it structured and typed.
I'm sure though that even if there are 1 gig dna strings in data there should be some middle layer caching and helping to get fragments of it by request. So it might be processed on server side during save operation.
ug_
ug_
yea
I would say look into trying to shove it into some sort of database. Having an application which already has data permissions, caching, multiple language interaction support and a team of people doing bug fixes on it is a huge +
There are XML database such as baseX. I havent used them but I have heard good things about them. I would definitly use XML over JSON as well, mostly because of the strongly enforced structure.
06:08
The idea is to save it into MongoDB gridfs (just as stream with mongo document wrapper having basic metadata).
But how about SAX parsers for XML. How do they work with very long strings?
ug_
ug_
poorly
haha
but you can use StAX parsers to solve most of those issues
I mean, I wouldnt count MongoDB out yet. If you can figure out how to break apart your data into smaller chunks it should work pretty well
It's possible to store really large text data into MongoDB GridFS.
regardless of whether it's JSON or XML
ug_
ug_
Yea of course you can store it. But if your talking 1gb of data thats going to get tossed through a parser every time you access any data from the application thats pretty crazy, I would be suprised if you can get decent speeds from that. Unless your data is formatted in such a way where you only access the first 10mb 90% of the time.
Or have lots of caching with a big heap space on your box. Its 1gb now but if you deploy the application and that grows to 5gb it might get big quick. Your going to needs some fast iops to pull that off
But what if I distribute all this in map-reduce like way? Each doc will be stored on some node and access operation will be performed on that node too.
And you're right it all looks crazy from set of requirements.
ug_
ug_
06:30
I dont know enough about MongoDB or your programs requirements to really provide a good answer to how you should or shouldnt store your data if you deiced to try it. I would say take a good look at how Mongo allocates documents regarding their size docs.mongodb.org/manual/core/storage/… and your program requirements and see how it might affect how you might design your database
Also, are your 1gb files a genome?
GTGCCTAGAGTTCCATT
It could be genome, yes. It could be many more types like Short Read Archive, Meta-genome, Proteome, Blast Comparison Results, Multiple Sequence Alignment, Phylogenetic Tree, Regulation Graph, Taxonomy Tree, ...
There is no limit to variety of biological types.
ug_
ug_
yea I can see why json would be a good format for your specs
Exactly!
ug_
ug_
Some quick last thoughts on it, 1. dont build something that can handle anything
And when you develop some algorithm later or UI widget, you don't need to load whole doc, you need some parts of it
ug_
ug_
06:39
its the dream but not very practical
RAX
RAX
hi I have some issue compiling Apache httpClient can I ask it here ?
ug_
ug_
Im thinking make your program have modules of some kind which rely on a database collection where they are stored. So you might have a class like

class GenomeDatatype extends Datatype

This datatype knows how to find and retrive its data. This will break your programs datatypes into simple classes which you can add with ease
@RAX sure
With any program you DONT want a god table, god class or anything class or datatype which claims to do everything. It ends up doing most things but doing them poorly. You will quickly run into issues expecially if you insert large ammounts of data into that thing. Think like 5 genomes and a bunch of meta data into one collection O_o. I can see it now, the 5gb pull from Mongo into Java followed by a heapspace error
@ug_ But there is a problem. On client side it's not just java. We should support at least javascript and python on top of java.
ug_
ug_
wait so are you only designing the database?
Nope
But I have some impact on decisions regarding it.
ug_
ug_
07:07
@rsutormin what do you mean python ontop of java?
or heck even javascript ontop of java.
oh, I meant python and javascript plus java.
Oh look, new faces :)
ug_
ug_
hery
hi
@ug_, thank you for your help!
ug_
ug_
07:24
yea no problem. Good luck with your project. Looking through Github it looks like you have some work ahead of you ;)
what repo are you working on for this issue?
but there is a dependency to github.com/kbase/java_common containing a lot done for json
ug_
ug_
cool, ill check it out. Im interested to see whats going on with all of this
it's somehow working now in production already (but not scalable yet): narrative.kbase.us
And there is no third party dev support at all (custom widgets and services)
And a lot of decisions were made before I joined the project 2 years ago
ug_
ug_
yea I saw you guys had several different servers for authentication!
I'm currently trying to find some ways how to optimize what's done and to switch to something else in some parts
ug_
ug_
07:30
like a 3 step process lmao
And to find out how to make all this scalable :)
Auth is cool but to be honest, it's open for outsiders for registering new user and start to work right away. So this way harm could be done instantly no problem. Why to bother...
Ok, I go to bed. See you later.
ug_
ug_
alright, checkin out the website right now tho, pretty cool.
good night
there are many people working on it (~10 full time or so)
08:26
Morning
Morning ^^
08:55
Morning!
Anything planned for the weekend?
09:31
hello
09:51
JSF frontend
ok
front end is client side javacript, back end is java, node is back end javascript, and JSF is front end java
and they say rocket science is confusing
Indeed
 
1 hour later…
10:54
Greetings...
And what's with you being here on a weekend?
I woke up, a tab was open to this page...
...Then again, every day is a weekend...
 
3 hours later…
13:38
Morning, Java!
/about
Dang it.
13:55
OakBot Online.
/roll 2d20
1, 17
Total = 18
hi.. @Michael how are you? after a long time i met you :)
Hi there, @Jude!
Algorithms module; we have to build an algorithm for given scenario. It's a program scheduler
Is it ok to use data structures which are already available than to use only plain arrays ?
What i have so far was only a code which gives the expected output. Not an algorithm.
:9
ᕙ(⇀‸↼‶)ᕗ
14:03
Well, can you do it using plain arrays?
if it ain't broke, don't fix it
@Uni! :D
hiya ^^
would be much pain
We have a saying in English.
14:04
anyone used ActiveMQ ?
"No pain, no gain."
screw your gain, spend your time somewhere better
@Unihedron you left the college?
life is too short to worry about what your teachers wants
@JudeNiroshan I aced it.
@Unihedron no surprise on it :D
14:07
Dang it @Uni, you didn't go to no college.
I'm still in high school.
But the thing is, the word college has a different meaning under our institution system here.
@Unihedron why the world would exists when things like this uttered by some people :P
Need to go guys! Have a good time. Bye Java!
14:10
Bye @Jude! See ya 'round!
15:00
Dead chat.
 
1 hour later…
16:06
Ah, that's better.
I fixed all of the ugly looking tables in ez-vcard's wiki: github.com/mangstadt/ez-vcard/wiki
Well done!
Anything planned for the weekends?
hi all
is it possible to send a file through an http request?
public void doPost(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {...}
can you show me some tutorials to do that?
Can I just give you working code?
16:11
@Unihedron I want to release a new stable version of ez-vcard. Haven't released one since last year. O_O
curl --upload-file uploadfile http://www.example.com/receive.cgi
well, i appreciate a lot, i you give me a snippet
Dang it @Uni, stop being cheeky!
@YokupokuMaioku You'll have to stream the file data using the response object.
@Michael can you show me a little example?
First, you need to say that kind of file it is, using the setContentType() method.
2 messages moved to Trash can
/javadoc ServletResponse#setContentType
16:17
@Michael void setContentType(String type): Sets the content type of the response being sent to the client, if the response has not been committed yet. The given content type may include a character encoding specification, for example, text/html;charset=UTF-8. The response's character encoding is only set from the given content type if this method is called ...
before getWriter is called. (1/3)
Then, use the getOutputStream() method to write the file data.
1 message moved to Trash can
Easy-peasy.
Right, @Uni?
16:44
Welp, off to lunch. Bye Java!
That's so rude.... @Michael never brings us anything after lunch...
He better bring himself back after lunch
17:00
...but I'm HUNGRY!
17:14
Then eat him out like he's groceries or something.
I heard that's what the kids do these days
17:44
Whatever you do, @Joe, please do NOT eat my booty like groceries.
I have this string > 3 12 2 51 2 35 2
I want to tokenize and get pairs like this 12 2 , 51 2 , 35 2
What would be the best way? is there any change to use regex ?
@Michael K
 
1 hour later…
19:00
@Michael I didn't say anything about a booty

« first day (1785 days earlier)      last day (3380 days later) »