last day (14 days later) » 

6:41 AM
0
Q: Visualizing Multiple Indexs on an ELK Stack (Ubuntu 14.04)

gabeHere's my situation. For years I have generated leads for a particular customer base and I've been struggling to properly visualize the data. My margins have been steadily undermined each year by increasing ad costs. I'd like to drop the losers and increase my spending on the winners. However, m...

 
Val
Maybe a dumb question but why not having a single index with adwords data, then enriched with visitor data, then further enriched with form and customer data (since they all share common unique ID)? I don't know the specifics of your case but that's probably how I would approach the thing, i.e. denormalize the data so that each of your documents are mostly self-contained with everything that pertains to them.
 
I doubt that it is a dumb question. I've only started looking into ELK in the last few days. I'm still trying to think of it differently than a DB where I would stored everything in its own DB and run queries across them using the unique id. Can you be more specific or provide a link to what you mean by 1 index of AdWords data enriched by the others? (The AdWords data lacks the unique ID so it may need it's own index.)
 
Val
Ok, so maybe two indices might do indeed. The base idea is to not think in terms of traditional DB development when you approach a problem with Elasticsearch in particular (and NoSQL in general). Since correlating data in different indices is not straightforward, the more you can keep your data correlated (i.e. collocated in a meaningful document) the better you'll be able to exploit them later on (via Kibana...). I don't have any specific example handy to share that might fit your case though, as this is still a pretty generic question that can be attacked from many different angles.
Maybe you can start at step 2, i.e. create the document based on the visitor data, and then you can enrich them with what adwords drove them there, then based on the unique ID you can add the form data and finally the customer data. Something like that, but again, it's just my own guesswork based on your question description.
 
I've edited my question with more detail. The searches I've done on enriching data so far all center on GeoData. The geo data is in a separate index it seems and then when a customer does something the relevant geo data is queried from that geo index. However, this isn't what I want since I don't think that would allow me to visualize it properly in Kibana. But maybe I have underestimated Kibana.
I would think for the log data use case in particular you would want to have an index for the logs and another for geo data linking IP addresses to locales. Then as you visualize the log data and ask questions like where did all of these port scanner hits come from Kibana would show you 1000 come from Russia, 800 from China, ... If you had to enrich each log entry in ElasticSearch you would storing a lot of repetitive data.
 
Val
Gabe, I'm moving this discussion over here if you don't mind. Since this is quite an open question that doesn't really fit into the SO model.
The idea of enriching data is not necessarily limited to geo data only.
The idea was just to start with what you know (visitor data) and slightly enrich it with more and more data (form, customer, adwords) etc, until you get the "golden record" you're after
So your last paragraph pretty much sums it up. You first store the data you're logically starting from, then you take the next set of data and enrich the existing data with, and so on
But again, at this level I can imagine it's all pretty abstract. but think about how that "golden record" would look like in your case. Once you figure it out, it should be straightforward for you to know how to build it. The traditional DB world always forced us to only INSERT data once it's complete and consistent. It's very different in the NoQSL world where you can have your data evolve as you see fit
 
6:51 AM
Thanks. The GeoData example was just an easy one to grasp. It was simply - 2 indices with 2 queries vs supplementing an existing index with data that might end up being redundant. From what I can tell, you are recommending that I not worry about the redundant data.
 
Val
redundant data is not an issue at all, hence why I'm calling for denormalizing your data, i.e. putting everything you need (collocating) into the same document, it's not a problem at all
I don't know how your data looks like, though, so it's not evident to talk "concretely"
 
Thanks. I need to sleep on it but I think the answer is actually a separate index for each data set so I can get data specific answers (mostly for testing purposes). Then another index with the golden record.
 
Val
You don't need to be afraid to store redundant and repetitive data in Elasticsearch, it's built for that
oh yes, definitely, don't be afraid of creating many "temporary" indices as well and one with your golden data. that's maybe the way to go
 
As far as concrete data goes, the AdWords data is aggregated at a daily level. Customer data has a row for each lead with unique ID, name, phone, zip, etc. The Visitor data has all the data from a couple of PHP globals - I think _SERVER and _REQUEST - plus the unique ID. The Form data has all of the form fields plus the unique ID. (The ID is an md5 hash.)
The Visitor and Form data is already JSON. The Customer and AdWords data is Excel/CSV and will likely be parsed using Python Pandas.
Anyways, thanks for the help. I'll get started on this tomorrow and just mess around for a while on each individual data set.
 
Val
No worries, feel free to chime in here if needed
 

last day (14 days later) »