« first day (1970 days earlier)      last day (688 days later) » 

2:06 AM
derp.
 
 
6 hours later…
8:20 AM
@TehShrike sir i am php developer and i want to improve my database skill, so what i do should i go for oracle certifiation or something else yo want to suggest.
 
 
6 hours later…
2:04 PM
@Nasir you use Oracle databases?
 
2:22 PM
hi guys, i didn't know where else to ask this. i have been desperately trying to find a clear answer to this. Anyway, i've seen around the internet and SO benchmark tests, where people create example databases and load them with (i'm assuming) random data, and then run queries while the database is being populated. here's an example of what I mean... http://stackoverflow.com/questions/4419499/mysql-and-nosql-help-me-to-choose-the-right-one/4421601#4421601

how do you do this? How did Jon Black load the tables with data? Was it just a loop? What tool was used?
 
2:41 PM
@MikelG It's a tough problem, because for good real-world benchmarks, random data isn't sufficient - you need to load in some data that is realistic to your software
So that the selectivity of the indexes and whatnot will all reflect what you'd see in the real world
 
i understand. my plan was to create a test database based on my real one
 
Yeah, that would be the most useful thing to benchmark against
 
that's easy, but i don't know how to simulate the DB being used...
 
I don't follow you there
What do you want to simulate?
Fire up a database on some box with similar specs to what you'll deploy to and load the data in
 
i want to test the performance of certain queries, on some particular tables that i know will be taxed heavily
 
2:44 PM
getting test data is usually the hard part
 
well, that's the issue... where do i get data to load in? i'm in development phase, so my DB is essentially empty
 
I thought you said you had real data you were planning on basing your test data on
@MikelG what did you mean by this?
So, loading in realistic data is the issue you're dealing with?
 
i'm not being clear here
please have a look at the link i posted
i want to do what jon black did in his answer
create a database, fill it with massive amounts of data, and run queries against it while rows are being inserted, to see the performance of the queries
 
2:59 PM
does that help clarify?
@TehShrike
 
@MikelG yeah
what is the difficulty in doing that?
 
i don't know how to
 
do you use MySQL?
 
yes
and i have MySQL workbench
 
The first step is to download MySQL and install it dev.mysql.com/downloads/mysql
 
3:07 PM
ok, got it
 
The second step is to set up a fresh copy of your schema
 
yup, got it
 
The third step is to populate it with hopefully-realistic data
 
and that's what i don't know how to do
do you create a loop? is there a built in function? etc.
 
ah, then you should have said "my difficulty is I don't know how to populate it with realistic-looking data" when I asked :-P
there is no built-in function
like I said above, filling it with realistic-looking data is difficult
for some general types of data, like names/addresses, you can find tools online that will generate you realistic-looking data
 
3:09 PM
my apologies... not the best at communicating this stuff yet
 
but you would still have to figure out how to get large quantities of those names/addresses into your specific schema
but yeah, you could start with making a script that would insert a thousand rows at a time into all of your tables
 
a script? what do you mean by that?
 
code
 
rather, let me clarify that lol
 
in whatever language is easiest for you
I can't really give much more specific help without knowing lots of specifics about the data you need to generate
 
3:11 PM
i did not think to do that... i was in the mindset of working purely in mysql
 
because otherwise, the code is just for (int i = 0; i < bigNumber; ++i) { insertIntoTable(generateNewRealisticRow()) }
 
well, that's easy enough to do
i guess i can do that via PHP and then use workbench to run the queries and performance bench marks
i didn't think that would be the "proper" way because of the potential overhead from the PHP side... hence trying to do it purely in MySQL
 
well, once you have it populated with data, there's still the question of how to best benchmark
if you want to benchmark queries under load, you have to find some kind of script to run lots of simultaneous queries to simulate the load you're expecting
 
well honestly, i'm most concerned with running the queries WHILE it's being populated
yes, there is mysqlslap... but that seems to work based on the premise of the data already being loaded previously
 
will that be realistic? Are you really going to be having rows constantly inserted?
 
3:15 PM
i'm looking to run queries while data is still being inserted... kind of a real-time scenario
 
if that's your goal, then you need to write a script that simulates the activity of your application when inserting rows
 
in one table, yes
 
as opposed to just writing a script to insert a bunch of data
so your script should be running the same insert queries that your app will be
if that's what you want to benchmark
 
right
well now that i'm thinking about using PHP to handle the inserts... this is almost a trivial matter
 
Sweet :-)
 
3:16 PM
i can do this without issue if i use PHP
and i THINK i can split the execution time... isn't there a way to have the database return a query execution time? that way i can separate the PHP execution time from the database execution
smh... this is what happens when you think "within" the box... you get stuck... with PHP, this won't be an issue at all, that never occurred to me... smdh
just looked at the documentation... even better. my framework logs DB query times. yeah, this will be easy now.
thank you @TehShrike
like seriously
 
You're welcome! :-)
 
while i have you, can i ask another question
 
Absolutely
 
alright, so i have a stats table
it logs how many times a post is clicked in a given data
so it looks like this
date, post_id, clicks
what is the best way to find out which post_id has the highest number of clicks over a given time period? (specifically 7 and 30 days)
i'm trying to build a "most popular this week, this month, and all time" page
i've been racking my brain for a clever way to do this that isn't too taxing
and i'm thinking the best way would be to run an event daily (cause i suspect this would lock the table)
but i don't even know what that "event" would look like
any ideas/suggestions?
@TehShrike
 
@MikelG do you have set start/end dates for the date ranges, or do you want to ask a question like "what is the 7 day period with the highest number of clicks"
@MikelG ah, you answered that here - if it's per-week or per-month then you can know the start/end ranges
 
3:28 PM
yes, definitely know the ranges
 
for how many date ranges do you want to query at once?
Like, do you want to get the top posts across the last 10 weeks?
Or just last week?
 
just last week, and then last 30 days
and all time
i imagine the queries would look the same, just the time period would differ
 
@MikelG in that case you can just put an index on your date column and query WHERE date >= ? AND date < ? or whatever
is that table unique on date, post_id?
 
yes
here's what i imagine... find all unique post_id's for a given date range, loop through each post_id and sum their clicks.

but i'm wondering if there is a better/clever way to do it? cause i imagine that to be quite a bit of work
 
3:47 PM
@teh
@TehShrike am i thinking the right way or is there better?
 
 
1 hour later…
4:48 PM
Sorry, got distracted by several conversations here
@MikelG yeah, you don't need to loop
you can SELECT MAX(clicks) FROM stats WHERE date >= ? AND date < ?
and then SELECT post_id FROM stats WHERE clicks = ?maxClicksFromLastQuery AND date >= ? AND date < ?
which should return 1 or more post ids
assuming you had any clicks on that day
 
5:39 PM
it's fine, i understand
hmm, i don't think that'll solve what i'm looking for
i'm not looking for the max within the time period on a single day, but rather the max over the entire period
if i understand you correctly, that will give me the most popular post on a given day between that time period
 
6:38 PM
@MikelG nope, that query is for an entire range
 
so it'll add up all the clicks for the entire range?
 
no - I thought you wanted the most popular post
 
yes, i do, but for the time period... not just the day
 
you just want the total clicks over the period?
 
YES
 
6:39 PM
oh I see
right, I needed SUM
 
right
that's why i was thinking the loop... cause you have to find the sum for each post_id within the range
 
total clicks over the period, per-post
 
bingo!
 
SELECT post_id, SUM(clicks) AS total_clicks
FROM stats
WHERE date >= ? AND date < ?
GROUP BY post_id
ORDER BY total_clicks DESC
LIMIT 1
2
 
oh wow
how beautifully simple
i was definitely over thinking that
thank you @TehShrike
 
6:45 PM
You're welcome :-)
 
although i'll test this for myself... just curious, do you think that query would take long for a large table? (asking about the "all time" scenario)
 
@MikelG it depends on how many posts you have clicks on per-day
is date a DATE or DATETIME?
 
right right
date
 
so that makes your indexes smaller/more efficient at least
 
alright
i'll test and find out
what's more, that query is only going to run once a day anyway
 
6:47 PM
you could test an index on date, post_id too
 
and i'll do it in the morning when the traffic is low
ok, will do
thank you @tehshrike i wish i could give you rep or something
 
no worries, I like to be useful :-)
 
indeed you are. take care. i'll look for you when i need help again
 

« first day (1970 days earlier)      last day (688 days later) »