« first day (2349 days earlier)      last day (1969 days later) » 
00:00 - 13:0013:00 - 18:00

13:00
by the way the official doc says: Never, never, NEVER use Python string concatenation (+) or string parameters interpolation (%) to pass variables to a SQL query string. Not even at gunpoint.
for psycopg2 but I was wondering is it ok to do string concatenation just for visual intendation if the query is long and fixed?
Like "Select * " + <this would be on new line> "FROM my_table"
or is this also dangerous?
no, you can concatenate your own strings, just never concatenate input from users into a string, use parameters
Nice thanks :)
Btw is it normal that 15Mil rows with 27columns take 70min to insert using bulk inserts with a page_size of 1000? This seems way to slow for me dunno
most columns are ints and bools, like 3 str
which database and what insertion mechanism
I should say there are 4 indices that need to be built too. 2 on one column and 2 on 2 columns
I'd expect to import 15m rows in a matter of minutes at most
13:06
postgresql and using psycopg2.extras.execute_values
so postgres, you are running insert statements per row?
now as i understand it execute_values groups them.
there is a param page_size which i set to 1000
but its still insert statements, and on a single connection, so its single threaded insertions in batches
Where is it getting the data from?
13:07
files, oh the files are on a shared drive
hmmm this could be the thing. but i feel like it is not it
e.g. do you have a measurement that already indicates that the speed issue is on the DB vs the process getting the data in the first place
because using sqlite3 also bulk inserts i got it in like 15 min or so
if i compare it to the time used with sqlite3 then yes i think it's the db that's slow, because the rest i didn't change
You need to get some measurements / instrumentation in place so you know how long each part of the process is taking, otherwise you are guessing where the bottleneck is.
but sadly i don't have pycharm professional, so i can't do easy profiling.
but true talking without numbers makes no sense, just wanted to get a feel for the times. but if you say it should take minutes i will for sure look into this at some point
It shouldn't take 70 minutes, it shouldn't even really take 15 minutes - you have at least 1 issue, probably more
13:10
^^
Of course, if you run it on very slow hardware / disk etc then it can be slow, I'm used to higher end spec kit
and we have no notion of how you have postgres set up, what disk array is backing it etc
it's a nuc with a i5 and i think an ssd
yeah ssd
16gb ram
so not high end but should be sufficient
@Andrew Do you have a mn?
13:35
possibly
13:55
Experince + intelligence = @Andrew
14:19
Good night everyone.. Have a nice sleep.. :D
u2 :)
14:40
Hello, I know this is a noSQL question, but there is no noSql chatroom, so this seemed most appropriate:

Using Cassandra, I am trying to aggregate rows which each contain a column of Set<BIGINT>. I would like the result to contain the sum of all sets, where null would be equivalent to the empty set. I expected sum(my_set_column) to do this, but the set type is not supported by this aggregate. Does anyone know of a way to aggregate this using existing cassandra built-ins? Thanks!
Do Views get deleted after a certain time? I can't find information about this online. I know that views are just an alias for a more comlex query. But does a view that is generated once stay there forever?
hey folks, anyone fitter in RECURSIVE CTE's than me and could spare a minute? :)
@Hakaishin because they stay
views will remain until they are dropped
15:05
@Hakaishin : A view should remain until dropped, but depending on the nature of how it has been coded, it could become rapidly deprecated / not fit for purpose
Oh ok thanks
16:05
Small rant, but I thought the whole idea of using a db and a query planer/optimizer is that I DONT have to think too much and fight too much with the details. But it looks like I still have to do everything...
Without proper indexes nothing works
@Hakaishin it should but maybe slower? :)
16:38
hey guys I have a question i am getting a syntax error could someone point out the problem? INSERT INTO contacts-submitted (name, phone, email, topic, message)
VALUES ('$name', '$phone', '$email', '$topic', '$message' )
17:09
nvm appreanty when writing sql you cant use - \
17:41
use "contacts-submitted" instead...
00:00 - 13:0013:00 - 18:00

« first day (2349 days earlier)      last day (1969 days later) »