last day (14 days later) » 

5:33 PM
0
Q: Formatting Mixed Type Python Array based on Type

eberkundI am trying to format a Python array as a string to be inserted into an SQL query. I want to be able to handle string and integer data but I cannot figure out how to to format the array. This is what I have right now columns = ['owner_id', 23, 'activity', 'rest'] val_str = ', '.join("%s" % (v) f...

 
There's no typeof in Python. You probably meant type, but that's still the wrong approach for this problem. Use the DBAPI to create parameterized SQL statements instead of string formatting. For example, cursor.execute('INSERT INTO activity_analyzed VALUES (%s)', columns)
 
Thanks, yep I meant type. That seems to be what I am after but when I use cursor.execute('INSERT INTO daily_activity VALUES (%s)', columns) I get "TypeError: not all arguments converted during string formatting".
 
Ah yes, that should have been (%s, %s, %s) - the number of placeholders needs to match the number of arguments. If you want a generic approach with a variable number of arguments, you should probably use named placeholders (see the available paramstyle options) and a dictionary instead of a list, and a an approach similar to this to build the column names and value placeholder strings.
Correction: (%s, %s, %s, %s) (4 placeholders). Apparently, I'm really bad at counting arguments.
 
Yeah unfortunately I need a generic method. I have already is actually a dictionary containing the values I need, so I set paramstyle doing db.paramstyle = "named" and then when I do cursor.execute(...) it will expect a dictionary?
 
No, you don't need to set paramstyle - that's just a documentation attribute to indicate what sort of parameter style your DBAPI conformant database driver supports. In order to use a particular parameter style you just have to use the corresponding syntax. So in order to use the named style, you'd use INSERT INTO activity_analyzed (owner_id_col, ...) VALUES (:OWNER_ID_KEY, ...), where owner_id_col is the actual name of the column in the database schema, and OWNER_ID_KEY the name of the key in your dictionary that should be used.
(I just used different spellings to make the distinction clear - they may very well be named exactly the same). And yes, if you use the named (or pyformat) style, you just pass a dictionary as the second argument to .execute(operation [, parameters ])).
 
5:33 PM
And passing a dictionary will let me automatically use those values to insert? I cannot specify the keys for each one because those will change. How would I do that? It only shows how to shows how to use a single parameter here: python.org/dev/peps/pep-0249/#paramstyle
 
hey
 
Hi, thanks for all the help so far
 
No problem. Yes, those examples in the paramstyle docs are for a WHERE clause, so there's only one parameter. But the principle is the same, you can just use multiple column names and value placeholders separated by comma.
If you use a dictionary, it won't automatically create a mapping from dictionary keys to DB column names for you however, you need to provide that yourself.
 
Would it be possible to do this without specifying the column names?
 
No - though that's not really a limitation in Python, but rather SQL
Unless you provide ALL the required values for a table row (in which case you can just provide the sequence of values in the same order as the columns), you'll need to provide it with some sort of information on how to map the values to columns
 
5:40 PM
for example I just want the final query to look something like "INSERT INTO daily_activity VALUES ('TEST', 1, 2, 3)" which I have tested to work
yeah exactly, I will specify all the columns in the row I am inserting
 
But basically this answer contains a function that solves this for the generic case: stackoverflow.com/a/32751768/1599111
ah, you'll always provide all the values?
The problem is, dictionaries aren't ordered. So if you want to just provide all the values, they need to be in the exact order of the columns in the DB.
The code in the linked answer is a bit hard to read - here's a cleaned up version:
    def build_insert_stmt(table, dct):
        """Build an insert statement using the `named` paramstyle, assuming the
        dictionary keys exactly match the names of the DB columns.
        """
        cols = ', '.join(dct.keys())
        vals = ':'+', :'.join(dct.keys())
        stmt = 'INSERT INTO %s (%s) VALUES (%s)' % (table, cols, vals)
        return stmt

    stmt = build_insert_stmt('daily_activity', dct)
    cur.execute(stmt, dct)
    db.commit()
 
Thanks, yeah I get what you are saying about the dictionary order
 
So if you had your values as tuples in the correct order, that would work too. But that's probably not trivial to get from the code you already have, so I'd still go with the dictionary route
 
I am using this
 
The above function should be generic enough, assuming your dict keys match the DB column names, you provide all required (non-NULL, non-default) values, and your DB driver supports the named style
 
5:49 PM
cols = ', '.join(params.keys())
vals = ':'+', :'.join(params.keys())
query = 'INSERT INTO daily_activity (%s) VALUES (%s)' % (cols, vals)
print query
cursor.execute(query, params)
ya so same thing basically
 
Which DB are you using BTW?
 
and getting this query
INSERT INTO daily_activity (time, activity, rest, owner_id) VALUES (:time, :activity, :rest, :owner_id)
I'm using MySQL
 
Ok. Yeah, Python-MySQL should support the named style as far as I'm aware
 
but I am still getting an error when I run it
_mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ':time, :activity, :rest, :owner_id)' at line 1")
Does that mean the DB is throwing an error? Is there a way to check the actually query that is being produced and make sure it is indeed inserting the right values for the named placeholders?
 
Yeah, that's the DB driver raising an exception based on an error from the backend
hmm. mysql-python.sourceforge.net/MySQLdb.html isn't very clear on paramstyle. But they only mention the pyformat style, not named
Adapted the function for the pyformat style. Can you try this?
def build_insert_stmt(table, dct):
    """Build an insert statement using the `pyformat` paramstyle, assuming the
    dictionary keys exactly match the names of the DB columns.
    """
    cols = ', '.join(dct.keys())
    vals = ', '.join('%%(%s)s' % key for key in dct.keys())
    stmt = 'INSERT INTO %s (%s) VALUES (%s)' % (table, cols, vals)
    return stmt

stmt = build_insert_stmt('daily_activity', dct)
cur.execute(stmt, dct)
db.commit()
(I'm assuming here that you're actually using mysql-python.sourceforge.net/MySQLdb.html - there's several different MySQL drivers for Python)
 
5:58 PM
Yeah that is the one I'm using
I will give that a try
Wow, thank you so much. That worked!! :D
 
There we go :-)
Yeah, creating parameterized SQL unfortunately isn't exactly trivial, but it's worth it
(blog.codinghorror.com/… is also a good read, by one of the founders of StackOverflow)
 
Yeah, well the end result code isn't too bad. No more than it would have been to iterate the list and make a string like I was trying before
 
Now that you got the "manual" way down of creating SQL by hand: You may want to have a look at SQLAlchemy: sqlalchemy.org
 
Thanks for the resources. This is my first time working with Python and I had to use it to work with some other code not written by me
Would you like to post that solution on my question?
 
It's mostly known for it's ORM functionality, but it also contains a core part that just deals with building SQL, talking to different DBs, etc. in a portable way, and almost entirely abstracts away the DB specific details for you
Well, it's a bit tricky to put into an answer that will be useful for anyone but you, that's why I chose the chat
I answered quite a few similar questions though in the past, and I noticed that this really isn't documented well, especially good examples are lacking. So I might try and write up a generic enough answer, but I probably won't get around to it this weekend ;-)
 
6:05 PM
Okay well thanks for all the help I really appreciate it. Take care!
 
You're welcome. You too, see you round!
 
Yeah I would suggest maybe just saying why parameterized is better and maybe explain the code bit if you decide to answer it otherwise I might put something there
 
I'll be happy to upvote it if you can summarize it in some way
BTW, here's how you'd build your INSERT statement using SQLAlchemy: daily_activity.insert().values(**dct) - docs.sqlalchemy.org/en/latest/core/…
 

last day (14 days later) »