« first day (3744 days earlier)      last day (1187 days later) » 

12:20 AM
For projects with multiple subpackages and a src directory, is it conventional to use imports of the form src.package.subpackage.module?
Or should the top-level directory of the project not be treated as a package/subpackage?
 
cbg. Anyone know why BeautifulSoup lxml.parser can't handle this XML? Is it choking on the double-hyphen XML comments '<!-- hearst/article/article_main.tpl -->'? I don't want to have to rely on 'html.parser':
from bs4 import BeautifulSoup as BS
from urllib.request import Request, urlopen

uri = 'https://www.sfgate.com/news/editorspicks/article/vaccination-plan-San-Francisco-Bay-Area-counties-15851854.php'
req = Request(uri, headers={'User-Agent': 'Mozilla/5.0'})
http = urlopen(req, timeout=20)

page = http.read() # NOTE: no .decode('utf-8')

#soup = BS(page, 'lxml-xml') # doesn't work on this XML, seems to choke on '<!-- hearst/article/article_main.tpl -->'
soup = BS(page, 'html.parser')

soup.findAll('h2')
 
12:51 AM
@NordineLotfi one of my coworkers just wrote a small TUI using picotui, didn't look too bad
 
1:08 AM
@alkasm I see, thanks for the suggestion! Didn't know that one, other one i thought was urwid because of better support for some things curses doesn't have by default afaik
 
np!
 
1:34 AM
@NordineLotfi not terribly helpful information, but my preferred debugger (pudb) is built with urwid
 
@AndrasDeak interesting! I kept trying to find different implementation of "visual debugger" so this one might comes in handy :D
 
It's like gdb --tui on steroids. I love it.
 
@AndrasDeak yeah, but there a lots of complicated stuff in gdb :/
what i wish i could have would be something closer to this but in python, instead of javascript (found fork of the original code for this on github but it was only in javascript afaik)
nvm, i meant this:https://cscircles.cemc.uwaterloo.ca/visualize#mode=display instead.

Still apply though since it's in javascript too :/
 
1:57 AM
@smci Partial self-answer: XML parsing requires bytes not str, so we have to do page = http.read().decode('utf-8') # convert bytes->str. That works with 'html.parser' but still breaks BS lxml-parser. Any advice guys?
 
@smci is HTML XML?
I don't know web stuff so I'd naively expect an html parser to be required for parsing html.
 
oh yeah, is it normal that sometimes, certain function that use () and return some values, then return "tuple" as type, when either i use it in a variable or use print(function) to show/output?
I know tuple use (), but in those context, i wasn't using actual tuple (afaik)
 
@NordineLotfi sorry, I have no idea what you're asking
Do you have a specific example?
 
ah, guess I'll find a working example hmm
 
@smci it seems that the comments are actually parsed just fine, which makes sense, because they seem to be valid XML comments w3.org/TR/REC-xml/#NT-Comment. It stops parsing when it reaches the first HTML tag(?). I think it's just that HTML is not XML.
 
2:13 AM
@AndrasDeak Doh thanks, you're right. Per BeautifulSoup: what's the difference between 'lxml' and “html.parser” and “html5lib”? and links to this BS reference overview. I was reusing some code I'd written that had run on XML. I have to robustify my client code to auto-detect the format, in general.
 
 
2 hours later…
4:05 AM
Cabage! I have a Google Colab log regarding the VM restarting. How do I interpret it, in order to understand the cause of the problem? Follow:
WARNING:root:kernel 0eeb769f-9eef-4840-8c5a-bea867fa4cc6 restarted
 
 
3 hours later…
6:36 AM
@user76284 No, absolutely not. You don't want src to be a package. That is a terrible name for a package. Imagine if every project was structured like that; they would all clash with each other because you'd have to import all of them as import src.something
I don't think it's ever a good idea to have multiple packages in the same project. Instead, I'd make one package with a bunch of subpackages
 
 
2 hours later…
8:46 AM
@user76284 Someone has helpfully created a bogus src package to prevent dependencies using that package name. That should tell you something.
 
Imagine if the stdlib was implemented that way
from src import abc
from src import datetime
from src import json
from src import time
from src.pathlib import Path
 
My recent review of current PEPs seems to have missed PEP 582 -- Python local packages directory. It's a draft of having lightweight venvs per-executable(s).
 
Not a bad idea, but not particularly important either IMO
Small personal victory: After wrestling with various design decisions and refactoring/rewriting large portions of my library for... how long has it been... a few years, I've finally arrived at a point where the code is no longer a tangled mess of spaghetti. I just fixed a bug in less than 10 minutes! A few refactors ago that probably would've taken me hours.
 
Nice, thats awesome
 
9:11 AM
Yup. It was a mess because I constantly discovered new things that couldn't be implemented with the current design, so I had to rethink the implementation from the ground up multiple times. Now the foundation is finally solid enough that I can "lock it in" and write tests for it
 
 
4 hours later…
1:06 PM
Hello, I am currently subscribed to Python Weekly (pythonweekly.com) and I learn a lot from it. What other lists are worth reading to be up to date in the new developments and packages?
 
 
1 hour later…
2:16 PM
mornign cabbages, folks
 
@Aran-Fey You mean you haven't been writing tests all along?!!! Shock, horror!
 
@holdenweb I am into a world where we don't write tests sometimes at all :(
I know it is horrible because the lack of tests makes us do somethings over and over again. They don't provide time for tests and nothing has tests here.
 
You don't need tests if you only write bug-free code.
 
spoken like a true mad genius!
 
2:32 PM
I have tools like 10 sometimes 20 years old. We are living under the ground, we swarm into bugs :)
There were some GOTO in the code before I fix them recently
 
how did you escape the velociraptors?
 
@inspectorG4dget I still can't, I am hiding under the shadows. I expect them not to find me. I hope.
@inspectorG4dget Lmao, good reference.
 
2:50 PM
@fedorqui'SOstopharming' I just lurk in this room, and scan the PEPs from time to time. and I really enjoyed that one pycon I was able to go to.
 
@Alper :O Would it be indelicate to ask how you know it works?
 
3:15 PM
@holdenweb No, it is not. Bdd. Manually testing the behaviors of the tool. It is more unpleasant than it sounds.
 
It already sounds unpleasant enough!
 
Just had to roll back halfway through a critical system reconfiguration because we do not test thoroughly enough. Test properly, folks!
 
3:43 PM
@Alper that's essentially what i do when tinkering with implementation or ideas (in python or other langs)
I only write test when I'm 100% sure there nothing else to add or remove
btw, i made a little sort toy implementation in Python, but, while i manage to make it work, i can't seem to rework the structure so it continue in a loop until the last character:
import sys

def move_char_by_increment(chars, char, increment):
    # Convert character sequence to list type.
    char_list = list(chars)
    # Get the current index of the target character.
    old_index = char_list.index(char)
    # Remove the target character from the character list.
    char = char_list.pop(old_index)
    # Insert target character at a new location.
    new_index = old_index + increment
    char_list.insert(new_index, char)
    # Convert character list back to str type and return.
this works but only for a single character, for the string "hello"
it print this:
hello
ehllo
elhlo
ellho
elloh
essentially, I can't seem to find/think of a way to make this work for every character in a loop, instead of just one (in this case, h)
 
can you clarify what you are trying to do?
move every character to the end of the string?
 
I'm moving each character, one by one until the last offset/place in the string
@MisterMiyagi yep
I managed to do it for just one, but I'm not sure how to make this into a loop so it do it for every character, like it did for h
 
@NordineLotfi so you want to end up with the original string when you are done?
 
@MisterMiyagi well, yeah, that would be the last result in my testing
 
@NordineLotfi Well, how would you do something else "for each character in this string"?
 
3:59 PM
@MisterMiyagi can you elaborate?
you mean how would i make the above work for every character or?
I guess, if the char variable was using actual number relative to the offset for each character (which it isn't in this case, as it use the actual character, in this example h) then i could just increment that in a loop and do it for everything like that?
or maybe i could do it the weird way, and make a separate list out of the original string "hello", and use a loop that use each as the variable "char", so it does it for each character?
I guess the later could work but it feel slower than just using the offset of the current list instead
 
@NordineLotfi I mean, take a step back. Don't think "how do I do this specific thing for each character". Just think "how would I do something for each character".
 
E.g., how would you code "for each character in this string: print the character"?
 
for i in range(0,total_char_input):
      do something
      total_char_input - 1
 
Ever seen something like this?
 
4:05 PM
How about
 
@MisterMiyagi something like this? though I'm unsure on where i should decrease the total_char_input variable
 
for character in "Hello World":
    print(character)
 
@MisterMiyagi oh yeah, but wouldn't that create another list instead of using the current one?
 
There's no list in my code block.
 
O
I see, so i guess i could do it like that
 
4:08 PM
Python loops are designed not to require indexing. It's quite usual to build one structure while processing another. When you've finished, re-assign the output to the source variable if required.
e.g.:
result = []
for c in char_input:
    result.append(do_something_with(c))
char_input = "".join(result)
assuming, of course, that do_something_with returns string values ...
Does this get you any nearer to answering your question?
 
@holdenweb yep, think i got it thanks to you two's suggestion :D
thanks!
oh, something i remembered i wanted to ask since a while now:
there this repo: https://github.com/amaljoyc/micro

and the file: https://github.com/amaljoyc/micro/blob/master/micro.py
has the "self" argument in each function...i can't wrap my head around because of the way it's used everywhere
any idea what it is for? beside being an argument?
I'm especially curious about what this "self" is getting replaced with, like:

If i was using "arg" and running "sys.argv[2]" in it's place, it would make more sense
import sys
import binascii
import re

def dec2bin(arg):
        print(bin(int(arg))[2:])

if sys.argv[1] == "-dec2bin":
        dec2bin(int(sys.argv[2]))
sys.exit()
something like this, using "arg" make more sense to me...don't know about "self" though
 
You will observe that the functions with self as a first argument are all defined inside a class definition. These so-called methods are defined on the class, so when they run they need to know which instance's data to operate on. This is achieved by the interpreter passing an automatic first argument which is the instance concerned.
 
@holdenweb so, does that mean self in the above repo's code is getting replaced with...the file used as argument when running it? or something else?
or do you mean it's getting replaced with any first argument?
I'm saying this because:
 def save(self,exit):
        y, x = self.window.getmaxyx()
        f = open(self.get_filename(), 'w')
        for y in range(y):
            f.write(self.window.instr(y, 0).rstrip())
            f.write('\n')
        f.close()
        if exit == 'die':
            sys.exit()
would make sense if this was the file here
 
4:23 PM
Suppose my_object is an instance of class MyClass. When you write something like my_object.method(a, b, c) this results in a call MyClass.method(my_object, a, b, c).
Does that help?
 
@holdenweb yeah, i think it does :O
 
Try writing a simple class yourself. You won't break anything even if the code fails.
The methods of MyClass are called with the relevant instance as the first argument, allowing methods to access the instance's attributes relative to self.
Play around a bit, have some fun!@
 
yeah, you're right!
looked at stackoverflow.com/questions/18867494/… think i get where i was confused now
 
There are worse places to start than docs.python.org/3/tutorial/classes.html
It needs some improvement, though, as there's far too much discussion before you get to see code examples.
I'd suggest you skip to the code, then backfill by reading the bits of the text you need to fill in the gaps. Someone else here might know of a better resource.
 
4:42 PM
@holdenweb yeah, essentially i would read/use the docs alone, but i noticed i learn things faster and better if i directly try implementing stuff, then solving problems along the way
 
Generally a good approach. The interactive interpreter can answer many questions!
 
yeah, i use ipython a lot more than i thought i would. "help" help a lot
I can't seem to be able to use jupyter though, there always either the kernel or something that break, and i can't always troubleshoot it because of lack of "reproducibility" or because of "time" constraint
 
5:05 PM
oh yeah, does anyone know any plugins or way to see the relative documentation for what i already typed, but when the "blinking" cursor is moved over what i already typed?
in say, vscode
Because i can only see the docs of what i type when i type it not after, unless i use the mouse
(but want to use the keyboard cursor instead, if that make sense)
at best, if there is an editor who has that feature/possibility for python programming, then I'd switch over right away!
 
@NordineLotfi Do you use virtual environments yet?
 
@holdenweb you mean venv/env?
I only use it for pip packages, never used it much beside that
 
That's the most common way of creating them, yes.
When you say you "use it for pip packages" what do you actually mean?
 
@holdenweb I mean, when i install a pip packages? (the official way i mean)
I guess it probably doesn't count as "using" then since it's normally automated
 
If you work in an activated venv then you're using a venv. Otherwise no.
 
5:09 PM
@AndrasDeak I see :O
 
But without actually creating a virtual environment pip installs everything in your standard Python interpreter, which may be why you experience difficulties getting Jupyter working. Are you a Unix user, or Windows?
 
@holdenweb Unix, Ubuntu
 
Try the following:
cd /tmp
python -m venv my_env
source my_env/bin/activate
pip install jupyter
jupyter notebook
The source command activates the virtual environment. You can deactivate it with the command deactivate, and you'll be back to your standard system Python environment (or you can simply terminate the shell). This does not affect other processes.
Nowadays jupyter installation is pretty reliable.
 
@holdenweb I see. Guess I'll try it and report back then :D
 
Please do! Virtualenvs can be created and deleted at will, so don't worry about it being in /tmp (though of course it doesn't have to be).
 
5:25 PM
Hi. Guys. I have a question of how could I make a better structure for my notepad python app. https://github.com/zaxoavoki/pypad
Is it overkill to divide the global menu into different parts (classes) as I did?
 
ah, found the answer for my last question here: stackoverflow.com/questions/37654532/…
 
AAB
Hi all,
I have a django app with 2 models, with one-to-many relationship
I have to populate a table with name and all numbers. Doing person_obj.mob_numbers.all()
This runs a select for each person, Is it better to use a an aggregate function in postgres?
like select name, jsonb_agg(jsonb_build_object('number', number) from t1 inner join t2 on t1.pk=t2.pk
Is it better to create a view in db for the above select query and a model for the same in django?
 
5:58 PM
Does anyone know the solution to these BeautifulSoup parser woes: seems you don't get treebuilder functionality invoked by default with any parser other than 'lxml'. Hence when you use 'html.parser', it doesn't read any tree structure, everything's a leaf node and its .children is empty. This is totally contrary to what the BS documentation promises :
... "Kinds of objects: BS transforms a complex HTML document into a complex tree of Python objects... BeautifulSoup.__init__() builder parameter: A TreeBuilder subclass to instantiate... instead of looking one up based on featuresparameter. You only need to use this (builder) if you've implemented a custom TreeBuilder."
typo/community-specific/misunderstanding stackoverflow.com/questions/58898597/…
 
Sam
Hey guys
I was wondering, can you supply a function a list of arguments to be chosen from? I dont know where I think I saw it, I believe it was JavaScript.
like def SAVE(text, type) where type can be log, error and other
 
6:18 PM
Cabage! I have a Google Colab log regarding the VM restarting. How do I interpret it, in order to understand the cause of the problem? Follow:
WARNING: root:kernel 0eeb769f-9eef-4840-8c5a-bea867fa4cc6 restarted
The moment the VM restarts, a message from Google Colab appears that an unknown error has occurred, and the only supplementary information regarding this is that information contained in the log. As there is no error message from Python itself, I can't debug the error.
 
@Marco Don't know, doesn't seem like a Python question, so offtopic for this room. Please look through github.com/googlecolab/colabtools/issues or suchlike.
 
@smci thanks. I thought of a way to deal with this situation, but I don't know the procedure. How do I debug Python code in order to execute the code line by line? Maybe that way I can check the error before the VM restarts.
 
6:35 PM
@Marco I don't know, I said please see the colab documentation, not ask about it here.
 
@smci I imagine that the line-by-line debug question is a Python issue, isn't it?
 
AAB
also have some trouble understanding django pagination
the last example here by the looks of it it seems like we are pulling everything from the db
In pagination dont we use offset and limit to move over the records?
 
6:50 PM
I suspect that Django is doing that automagically for you
Then again.... Django. It's been at least 3 years before I even tried to create an app with it, so I know close to nothing about it
 
@Marco No, it's a Google colab question. If you're talking about debugging the VM. (whereas if you're talking about debugging your Python source code, not under Colab or in a VM, that's different, use any decent debugger or IDE. Please see the tutorials.)
 
7:15 PM
how do i access the last line from print() output?
let's say from something like this:
for character in "Hello World":
    print(character)
I know i could use a list and query for the last offset/key in the list, but i prefer other method if possible
 
Why would you prefer another method?
 
@roganjosh because i don't want to change the type of the string "Hello World" to a list?
 
You don't have to
 
I mean i don't mind using lists, but was just curious if there was other methods
 
Strings are sequences that can be indexed just like lists
 
7:26 PM
so i guess i could use index() or try this?:
for character in "Hello World":
    print(character[-1])
though that doesn't work hmm
 
hint: if you're looking for the single final character of a string, don't write a loop...
 
Why would that work? You're giving a direct index, so why would you need the for loop?
 
@AndrasDeak it's actually just an example that use a single string...what i want to do is just to get the last line from a print() output :)
@roganjosh just was trying things out :D
 
@NordineLotfi I think I've told you before that print returns None. Somehow conflating print with accessing data passed to print is very confusing (confused)
 
again, i know how to do this using list, but I'm curious about doing this without it (if possible)
@AndrasDeak it's confusing but that's the kind of workflow i did for testing stuff in POSIX/shell scripting
 
7:31 PM
@NordineLotfi You're not listening to us
 
It's like "I photocopied my ID card. Using the paper copy how do I look at the other side of my ID card?". You don't. You take your ID card and flip it over.
 
Please take a step back and look at the output you got from this code. Can you explain to me how you think you're getting the output that you are?
 
import sys

def move_char_by_increment(string, char, increment):
    # Convert character sequence to list type.
    char_list = list(string)
    # Get the current index of the target character.
    old_index = char_list.index(char)
    # Remove the target character from the character list.
    char = char_list.pop(old_index)
    # Insert target character at a new location.
    new_index = old_index + increment
    char_list.insert(new_index, char)
    # Convert character list back to str type and return.
this is the code i made and posted a part of earlier here
essentially, i asked for printing the last line from print() output because i wanted to do it on this
@roganjosh sorry if it seemed that way, i just didn't give full detail because i wanted to fix it myself, which is why i used a "simpler" example
 
@NordineLotfi so you want to use python as bash. No problem. Write a python script that prints the last line of stdin. And then feed this python script the output of your first python script. Easy.
 
@NordineLotfi That's fine, but I am curious on you explaining to me why you think the code that you tried does what it does
In other words, what is this doing?:
for character in "Hello World":
    print(character[-1])
 
7:36 PM
@roganjosh you mean the one with the index number on print()? I wasn't sure it would work since i seen this work in certain instance (i managed to do it once using binascii for converting number) and i just thought i might do something here, related or not to what I'm trying to do...
I never said this work though :D
 
I never said it worked either
 
I mean, it doesn't give the desired output. But it does actually run. I'm asking you to tell me, in your words, what it actually is doing, even if it's not what you want it to do
 
@roganjosh I'm not actually sure in this specific example (by looking at the output, nothing change, unless i place two :: on both side of the number)
for example here:
def dec2bin(arg):
        print(bin(int(arg))[2:])
the 2: would remove the first two character in the output, afaik
 
So we have a phrase for this in the UK - "throwing spaghetti at the wall until something sticks". In other words, you're literally just guessing. You need to spend some time looking through tutorials
 
7:40 PM
@roganjosh I mean, yeah, but i usually learn more by my failure, unless what i want to do is specifically written in a existing tutorial
 
That's fine, I think all people learn best that way. But it's a private process until you get stuck
 
exactly
 
And I know for sure that there are resources out there that you could be reading that will solve this for you. It's not a problem for the Python chat room, so please go do your research
 
alright
though, i already know how to solve my problem (using list) but i was really just curious (aside from my failed attempt earlier) to know how to do it differently...
 
So you've said for the third time, at least as I've been in the conversation. I haven't once suggested using lists
 
7:44 PM
fair enough
 
8:11 PM
@holdenweb Well, on one hand, it would've been a waste of time, because the code kept changing so much. But on the other hand, trying to write tests for my mess might've helped me realize that my code wasn't only mess "under the hood", but my high-level API as well
In fact it took me quite a while to realize that I had a testable high-level API. It's basically a wrapper around GUI toolkits like tkinter and gtk, so there is a lot of stuff that automatic tests can't cover
 
AAB
@roganjosh thanks but in the django shell when I do modelobj.objects.all() all the records from db are present. Is this similar to pymongo cursors? like as you iterate records are brought from DB? In py mongo cursor object the same was mentioned clearly but for django I cant seem to find it in the docs:/
also regarding my other query on which is better use a view for table select aggregate query or use the django models with fk approach
I need to just print the data
Is it bad practice to use views that way? I am using postgres DB and using agg to create json object array
 
I agree that it's not clear in the docs and it seems to suggest that it does pull the data into memory and then lazily send it to the frontend (which is handy for avoiding things like FOUC) but don't take my word on any of this
 
AAB
FOUC?
 
Google? :P
 
AAB
TIL what is FOUC even though I have seen it happen so many times :P
 
8:24 PM
For things like dataTables, it will do pagination in JavaScript but it sometimes depends on how much data you're sending. If you send a lot, it'll be splattered across the screen while it sorts itself out
Aug 20 '20 at 15:41, by roganjosh
@holdenweb I've tried passing pagination off to dataTables on the front-end before. It's kinda fun; people talk of "FOUC" but if you pass 1000 rows of 60 columns, it's more like watching a zombie burst from a grave and slowly haul itself out. Great user experience :P
I suspect, though, that there are others better qualified to go through the django codebase and actually understand what the pagination object does. The docs aren't definitive (at least IMO)
 
AAB
Is there any thumb rule on how many rows we should pull from a database
Is something like 200 records fine for modern database?
 
It's not the database, it's the browser of the person viewing your site
200 results are nothing, provided the backend is set up correctly. Rendering it in a structured table is entirely different. I don't know how much that relies on the users' browser etc.
 
AAB
I see I was under the impression that frequently getting 200-300 records from DB puts a lot of load on it :P
 
Well it will, but so will executing the same query thousands of times and getting back only 10 results each time
Databases are pretty good at this kinda thing, though, so I suspect the restriction is more on UX and how quickly it can be loaded in the browser
 
AAB
cool thanks
 

« first day (3744 days earlier)      last day (1187 days later) »