« first day (4224 days earlier)      last day (947 days later) » 
01:00 - 21:0021:00 - 00:00

01:09
hey, how to delete the 3rd character from a string ?
s = "123456789"
x = s[0:3] + s[4:]
print(x)
is that is the right way to do it ? I feel like there is a better way
That seems reasonable, but make sure to DRY it up if you repeat the code, aka write it as a function.
not sure what do you mean by that?
I need to save sensitive int data temporarily and I thought if I could inject some digits in the middle whenever I need to store them and then remove it by the way above if I needed the real code, that would be a good way.

also if I need to search via this data, I would just add the "salt" that I have in mind and I know it's location, then search the salted code as it's how I stored it.

unless there is a better library for that?
I don't want to sound mean, but not knowing what the acronym DRY refers to hints that you probably shouldn't be writing security critical code :-/
sadly I'm not a cryptography expert
but always the the solution is to use some code somebody else wrote :-)
01:25
if your aim is to break my spirit you failed, lol
but thanks for the tip, I know I'm not the best on security either, but I just know enough
`I'm not the best on security either, but I just know enough`

Famous last words :-)
admittedly if you're a contractor, you can probably submit the code, flee and the only victims will be those who lost their passwords
@Mikhail it's not a password, it's a simple party member code, it's sensitive but not too sensitive.
@Mikhail lol
i was thinking instead of keeping the code on the database, I would inject few digits here and there just in case my security has been breached, the data wont make much sense to the hacker.
01:47
Hi, does the following make sense?
    def choose():
        try: pos = int(input('>>> pos: '))
        except ValueError: tryAgain()

    def tryAgain():
        print("\nWrong input. Try again!")
        choose()
For some reason, my text editor is pasting it with tryAgain as a subfunction under choose. I'm assuming this is not the case?
I believe it should be a function invocation of an undefined function, and not a subfunction.
choose() and tryAgain() are separate function definitions, yes that is correct.
Making it one recursive function fixes the issue:
def choose():
    try: pos = int(input('>>> pos: '))
    except ValueError:
        print("\nWrong input. Try again!")
        choose()
Yet, two would have been nicer because I could reuse the tryAgain() in other cases.
 
2 hours later…
user13428955
04:18
Hello Mentors.
Would you please have a look at my code, please?
https://stackoverflow.com/questions/72156287/sample-size-valueerror-a-must-be-greater-than-0-unless-no-samples-are-taken?noredirect=1#comment127491266_72156287
05:02
Hmm... while I got my importer working for pyparsing, it's failing from dill. It works perfectly if I rename the file to dillx and import dillx.
05:36
@MisterGeeky Stop using recursion. It makes your code harder to debug; just use a loop instead.
@LoopingDev and how do you plan to undo that security feature to get your data back to its original form? You would have to store the position and length of characters you inserted?
 
2 hours later…
07:38
I am trying to create a many to many relationship between two models. i am using the foreign key in my assosiation table. when i try to migrate i got this error sqlalchemy.exc.NoReferencedTableError: Foreign key associated with column 'user.nf_set_id' could not find table 'nf_sets' with which to generate a foreign key to target column 'id' ? i do have the table called nf_sets
@SumathiJ the model associated with nf_sets must be imported into the module where the relationship is being defined
okay
thanks
08:02
@Aran-Fey noted. Thank you plentifully and have a fair day.
 
2 hours later…
09:52
Hi guys. I'm trying to create an exe with pyinstaller and when I do pyinstaller --onefile my_file.py I get a module not found error for configargparse. Even though the import is not a second level import, like stackoverflow.com/questions/25733467/… would suggest. So why does pyinstaller not found configargparse?
I have no idea, but I've also recently tried to use PyInstaller and also had a bunch of weird problems with it
 
1 hour later…
11:14
Last answer here was the trick: stackoverflow.com/questions/50121330/… Pyinstaller has many weird failure modes
Last sorted by what? Score?
11:31
Hey hey hey, we don't do that here in the world of IT. We don't tell each other how our text is encoded or how our csv values are separated or how our answers are sorted. You just have to know, sheesh
12:12
haha "It's not you, it's us" has to be the softest title of a job rejection letter
Bonus points for being so upfront about it. I hate skimming through the email searching for the word "unfortunately", that's buried two paragraphs deep into the email
I have some database code which is obviously vulnerable to sql injection, assuming the bad guy can call search_widget_names with any string he wants. For example, he can use "%" to get the name of every product, or "' and price > 1;--" to discover bits of information about non-name columns in product_catalog.
What else can he do? Can he insert new values into a table? Can he drop tables? Can he get values out of the sensitive_user_data table? I tried a couple approaches, but anything along the lines of "Robert'; DROP TABLE Students; --" crashes with sqlite3.Warning: You can only execute one statement at a time.
12:29
You can execute multiple statements if you use executemany, for example
I'm focusing on just execute for now
Not sure how useful this is, but you can guess a value and then confirm if it exists with something like x%' or 'sensitive_user_data' in (select name from sqlite_schema);--
Since this produces output, the attacker now knows that a table named "sensitive_user_data" exists
print(db.search_widget_names("foobar' UNION ALL select name || ';' || password from sensitive_user_data --"))
#result: [('Kevin;Cabbage',)]
Uh oh
Yeah, that whitespace at s = is scary
12:39
Ok, so here's what I think happens: I import dill.py. Since exec can see sys.modules, it sees that dill is already imported and doesn't try to find it. I tried to fix this with importlib.invalidate_caches() in dill.py, but it didn't change anything. I then tried deleting sys.modules['dill'], but that just lead to recursion.
@AndrasDeak--СлаваУкраїні Oops, transcription error. That shouldn't be a keyword param to begin with.
Taking a page out of Aran's playbook, I can view all table names using "foobar' UNION ALL select name from sqlite_schema --"... I can basically execute any select statement I want as long as the result has one column containing strings
Hackers gonna be like "Only one column?! And it must be a string?! Nothing we can do boys, let's pack up and go home"
I'm curious what the attacker could do if he didn't know how many columns the select is returning, or what types. For all he knows, I'm selecting 17 different fields but only returning one from my method.
12:49
according to security through obscurity you are legally obliged to disclose your schema to any attacker who asks for it
Fair.
I think the types of the columns don't even matter, so all they have to figure out is the number of columns. And that's easily doable with a bit of trial and error
Ah, then the problem space is reduced from (num_types^maximum_sane_number_of_columns) possibilities to just (maximum_sane_number_of_columns). A big win for the bad guy.
Anybody: "Hey, can you pass me your salt and pepper?"
IT guys: *start sweating*
"I'll give you your own personal salt, but first you'll need to give me your email and select all images containing a bicycle"
The other guy: [nervous, but doesn't start sweating because he is a robot]"
huh
time for that long-awaited chat overhaul :P
Unfortunately, he probably won't be working on Chat. But at least his knowledge is returning to the system.
I'd settle for being able to upload .webp images
Actually that might be handled by the imgur api, so there's a real possibility that it's already been made possible since the last time I tried
Is there a way to force Python to not check if foo is in sys.modules and find and import the module? I tried importlib.invalidate_caches, but that didn't help.
I assume "deleting foo from sys.modules" is not an option
13:06
Yeah, I tried that, and it just leads to Recursion errors.
Well, then there's something wrong with your importer
@DrownedSuccess, I wonder if virtual environments would be useful. Not for the specific problem of manipulating sys.modules, but for the broad problem of ensuring modules can't interfere with one another.
@Aran-Fey perhaps deleting during the module's import
@Kevin I want to make it self-contained, so I don't know if that helps.
assuming something like that could lead to recursion errors...
13:07
I'm basically four rungs up on the XY Problem ladder, taking note of any interesting scenery
@Kevin not having access to a subset of the modules because they are not installed is a foolproof way of ensuring they don't interfere
@DrownedSuccess Virtual environments are great at making things self-contained :-)
@Kevin I think that applies to all of us, we've only been getting random scraps of information for a while now
yet I don't think that's what they are working against
Just another day in rooms/6
13:09
Basically, I have foo.py, which inside generates a module foo using the ZipImporter I talked about before. When I run import foo, I think Python is trying to import from the foo.py already there.
If I rename it to foox.py, then it works.
foo.py generates a module called foo?
To replace itself?
Hopefully, yes.
Have you explained yet why?
Hmmm... can't you just use a module-level __getattr__and offload every lookup to the zipimported module?
it would still be cursed, but it would probably work
Well, the problem lies in zipimporting the module in the first place, doesn't it?
no, I think the problem is replacing foo with foo.foo
13:13
Any problem description containing the words "generate a module" is at least two XY problems away from my current position
I don't think that's it, that's literally as simple as sys.modules['foo'] = the_zipimported_foo
and would that work while you're still executing foo?
Yeah. At the end of the import process, python looks up the module from sys.modules and returns that.
OK, then we're back to no MCVE.
(It's kind of a semi-official feature that exists for this exact purpose)
13:16
$ cat not_math.py
import math
import sys
sys.modules['not_math'] = math
$ python
>>> import not_math
>>> not_math.pi
3.141592653589793
can confirm
I figured something out --- I put my finder in the beginning of sys.meta_path
That certainly explains the recursion error
No I mean, I moved my finder to the beginning, instead of at the end, and it worked.
So I forced python to look there first
Proposal: name the generated module _foo.py. On the very last line of foo.py, add from _foo import *. Result: when anyone does import foo, they get all the members of the _foo module.
@Kevin and all the cruft also in foo
13:19
On 2nd thought, it might not explain the recursion error after all. I'll shut up and eat my alphabet soup that's missing the letters M, R and E
#foo.py
with open("_foo.py", "w") as file:
    file.write("def frobnicate(): return 23")
from _foo import *

#main.py
import foo
print(foo.frobnicate()) #result: 23
@AndrasDeak--СлаваУкраїні Yes. I suspect you can do something with __all__ to sweep the cruft under the rug. I don't know the specifics myself.
__all__ would only affect from _foo import * and would have to be defined in _foo. The cruft already there in foo will be there unless you delete it.
One way to avoid that is to, wait for it, replace foo with _foo entirely.
What if we put some caution tape around the cruft by putting underscores in front of the names? The user will still be able to access all of it, but at least they'll feel vaguely guilty about it
Maybe put it in a class so they have to unmangle the names, and feel double guilty
or put a glittery bow on the cruft instead and call it an art installation
Somebody get this man a grant for the arts
13:26
cbg
#foo.py
def initialize():
    #almost everything goes inside this function
    with open("_foo.py", "w") as file:
        file.write("def frobnicate(): return 23")
#these are the only three lines that may appear outside of initialize
initialize()
del initialize
from _foo import *

#main.py
import foo
print(foo.frobnicate()) #result: 23
print(foo.initialize) #AttributeError: module 'foo' has no attribute 'initialize'
v1.1, now with marginally less cruft
I also fixed my pyparsing issue --- previously, I put newlines between every line. Not doing that fixed my issue: don't know why.
I hope you don't mean newlines between every line of Python code
Yes I did (this was a code generator).
As long as it works now I guess
13:35
@Kevin I'd go for a import _foo; (lambda _foo: globals().clear() or globals().update(vars(_foo)))(_foo)
it has the fewest lines so it wins
Well, if you're dying to bind a bunch of variables in foo's scope and you don't feel like writing a ton of del statements...
module-level __getattr__ starting to seem like a sane alternative
Is that possible? I tried doing something like that for an unrelated abomination experiment the other day, but I couldn't figure it out.
13:39
👀
all = the = cruft = 'cabbage'
_foo = get_the_foo()

def __getattr__(name):
    return getattr(_foo, name)
untested though
won't be great for tab completion though, probably
Hmm! Not what I predicted:
#foo.py
import types
all = the = cruft = 'cabbage'
_foo = types.SimpleNamespace(frobnicate = lambda: 23)
def __getattr__(name): return getattr(_foo, name)

#main.py
import foo
print(foo.frobnicate()) #result: 23
print(foo.cruft) #result: cabbage
I guess there's a bit of magic where it searches foo's actual scope after our __getattr__("cruft") gives an AttributeError
Obviously we should instead raise an AttributeErrorAndWeReallyMeanIt
@Kevin hmm, same. So it doesn't work.
It's the other way round, it does that first. __getattr__ is the last resort
I must examine the scriptures of dunder for wisdom
13:48
It's consistent with the way __getattr__ works in classes
I'm back from the scriptures. I agree with you.
Also of note, docs.python.org/3/reference/… is one way to do it
... I think. The scriptorum only has candlelight, and their complimentary half-moon glasses don't come in my prescription.
I knew about __dir__ once...
I don't think I've played with __class__ before. I sense the opportunity for mischief.
class Dog:
    def __init__(self, name): self.name = name
    def bark(self): print(f"{self.name} says: woof")

class Tree:
    def __init__(self, name): self.name = name
    def bark(self): print(f"{self.name} photosynthesizes contentedly")

fido = Dog("Fido")
fido.__class__ = Tree
print(type(fido)) #<class '__main__.Tree'>
fido.bark() #"Fido photosynthesizes contentedly"
[Dynamic typing intensifies]
14:13
I thought you could only do that between subclasses or something
Well I can tell you that fido.__class__ = int doesn't work
TypeError: __class__ assignment only supported for mutable types or ModuleType subclasses implies the existence of both mutable and immutable types, which I didn't know was a thing
@roganjosh something like that
## salting
org = "123456789"
salted = org[0:2] + "5" + org[2:] + "5"
print(f"org:{org}  - salted : {salted}")
 >>>>> org:123456789  - salted : 12534567895
## reverse
reversed = salted[0:2]+salted[3:-1]
print(f"org:{org}  - reversed : {reversed}")
  >>>>>> org:123456789  - reversed : 123456789
@Kevin Interestingly, that still works if the classes have __slots__. But the slots must have the exact same names
class Foo:  __slots__ = ('foo',)
class Foo2: __slots__ = ('foo',)
class Bar:  __slots__ = ('bar',)

obj = Foo()
obj.__class__ = Foo2  # works
obj.__class__ = Bar   # throws TypeError
__slots__ confuses me. If I have to think about them for more than three seconds, that's a red flag
@LoopingDev For the record, that's not what salting is. That's just a very weak encryption
A salt is a bit of data that you hash along with your actual data. But you're not hashing anything
14:19
@Aran-Fey yes I know, but I'm not trying to salt or protect the password info, more like the userId, I need to search the user Id as part of the process, like if I wanted to look for user with userId 123 , instead for look for user 123 right away, I would add my salt "12535" then look for the database for the new salted value because that is how I stored it.
Why not use a proper encryption algorithm?
@Aran-Fey that is why I'm open for suggestions that would protect the userId without making it impossible to search like passlib, I need it to be searchable
if there is a library that would do that, I'm open for it.
I mean, it's fundamentally the same thing as your idea. If you want to search for "123", you first encrypt it (which returns "12535" or whatever) and then search for that
@Aran-Fey cool, then I'm on the right track, it'd be better if I do it myself at least I would know how it works.
It's fairly easy to search a list of encrypted values to find a perfect match for some input. But it would be pretty hard to find a partial match. e.g. if the cleartext of your input is "foo" and you want to find a value whose cleartext is "foobar"
14:32
that is why I'm saying before I look for "foo" in the database I would add the "salt" which in this case "bar" then search for a username "foobar" instead of just "foo"

:54535381 correct me if I'm wrong
userId : "123"
encrypted userId(123): "ifnnowfinweorifnwerf093wJFYTGVef3HFG4f93"
encrypted userId(123): "msf089uj45t0i4jg0iwj23078hr204ij2-3h230r8"

the same user everytime with a different hash, If I turn the userId/username into a hash, I don't think I would be able to look for it in the database.
Re: "it'd be better if I do it myself". I am normally a staunch advocate of reinventing the wheel for educational purposes. But for security in general and crypto in particular, I would not rest easy unless I was using a reputable and well-tested third party library.
See security.stackexchange.com/questions/18197/… for much discussion on the topic.
@LoopingDev Which hash did you use for this?
@LoopingDev You keep talking about salting and hashing, and I don't understand why. Let's clear this up: Do you want to hash the userIds or encrypt them?
Perhaps you can solve your specific dilemma without cryptography. AFAIK, user ids are typically not hashed or salted or encrypted etc. You can see the user id of every person in this room just by hovering over their avatar.
It sounds like your user id is somewhat more sensitive than this. Perhaps knowing that Kevin's userId is 12345 tells the bad guy that he lives in district #12 of the third oldest city of the 45th oldest country.
In that case, you shouldn't use it as the primary key for your table. Use a regular blind autoincrementing key, whose value confers no data other than the row's position in the table. put the sensitive id value in a different column. Now the bad guy can't tell I live in district 12 just by hovering over my name, but you can still write queries that search for users in district 12.
14:42
cbg
Kevin definitely raises a good point. What is sensitive about your usernames? Is it because you're doing something like tying real people back to user names?
(and their real name is sensitive, PII style information)
It may be useful to consider precisely what kind of attacks you want to defend against. For example, using a blind key can thwart eavesdroppers that are scanning network traffic for sensitive data. But it won't do anything against a bad guy that breaks into your office and steals the server rack.
Salting* your data with decoy characters will also not thwart the server rack thief. He's a tricky lad. I suggest a really good door lock and maybe a few attack dogs.
That's two factor authentication right there, the burglar needs both a lockpick and a dog biscuit
You mean attack pythons, surely?
Dogs patrol the exterior, pythons roam the interior halls. Their ability to open doors makes them naturally suited for indoor surveillance.
They're also harder to bribe. Dog biscuits are easier to sneak in than, say, a rat. Or whatever it is that snakes like to eat
14:52
@LoopingDev Can we have some context, please? Is this software a program that will be used by real people? Or is this just a toy program you're writing as a learning exercise, and no real people could potentially be harmed if you implement bad security?
Whatever it is, it's probably bigger than a breadbox. Not an easy parcel for an aspiring thief.
A little girl goes into a pet shop and asks for a wabbit. The shop keeper looks down at her, smiles and says:

"Would you like a lovely fluffy little white rabbit, or a cutesy wootesly little brown rabbit?"

"Actually", says the little girl, "I don't think my python would notice."

—Nick Leaton, Wed, 04 Dec 1996
Hmm, rabbits. I guess you could probably sneak one of those in without too much trouble around easter time
@LoopingDev If you're just playing around & want to explore crypto stuff, you might enjoy CryptoPals. cryptopals.com
A few years ago, several of the room regulars had fun doing the CryptoPals challenges... for a while. Eventually, we gave up, at various stages, mostly because the challenges were getting too ambiguous, and didn't provide enough info. But the first 10 or 20 or so weren't too bad, and most of us ended up learning some crypto stuff we didn't know before.
@Arne thx, bruh
15:16
I'm seriously considering putting a bounty on this question: Will a watermelon decompose on the Moon?. The Moon gets pretty hot during the day, so I think it'd probably explode. But maybe a watermelon is tough enough to contain the pressure.
stackoverflow.com/questions/72188813/… Please tell me there's a canonical for this. Everything I can find is a near-miss
Probably something akin to "missing 1 positional parameter" could be a dupe.
1k rep user, 9 minute old question with 7 answers
beautiful. I thinking of at least three "subtle put-downs"
A question that nobody should ever have if they've looked at a python OOP tutorial
OP is a "lead software developer", he's clearly too busy for any of those tutorials
15:20
Guys I've been trying to debug this for hours, with no success: I'm doing a scipy.differential evolution optimization, and there's 2 NonLinearConstraints being considered, but one of them functions is never being called when DE runs, even though I'm 100% sure it's being added to constraint list, and both of them are not identical. Any ideas?
The mcve I've built is working correctly, so I suppose it's something with the structure of the code (classes, etc).
@KarlKnechtel Hammer it with what seems like a very good match.
Kinda depressing though...
reading it again, I'm not actually clear on whether OP actually instantiated the class.
or understands the purpose of methods (and classes/objects) in the first place (it seems common for people to treat class as just a namespace)
@vaultah Maybe "lead developer" means "the only developer in the whole company"? Would explain a lot...
@PM2Ring just don't tell poke that... he likes his bunnies :p
@KarlKnechtel I think they did. myclass would be the instance to MyClass. Not that the answers cared...
15:23
> The problem with this sort of thing is that the ideal question needed to introduce the topic properly is not one that people would naturally ask, because it requires a particular insight - the kind that typically comes from already knowing the material.
@MisterMiyagi I think so too, but someone who was summarizing code could have just made a capitalization error/typo
@PM2Ring indeed :) now you see my near-daily frustration with the site
OP seems to have a "blog" on github.io that looks to be basically a scratchpad for taking random notes on whatever he's studying, except public for some reason
just outlines of what look like study notes and code snippets, no attempt to actually write explanatory paragraphs about anything.
Very true, but that's a frequent problem, and a flaw in the Stack Exchange model. And trying to find appropriate dupes can be really hard. You want an old good question that's similar enough to the new question, with answers that actually explain the problem, not just cryptic code solutions. It doesn't help that Google is heavily biased to search for questions, not answers.
@MisterMiyagi I'm with you on that 100%, if you know a library that would help with that, I'd highly appreciate that.
@MisterMiyagi passlib does something like that, but in this particular example, I just wrote it myself.
@LoopingDev Take a look at docs.python.org/3/library/hmac.html You can read more about HMAC on Wikipedia. Then learn about key derivation docs.python.org/3/library/hashlib.html#key-derivation
I just wanna securely store the userId, so far salting was a good start. but if there is a better option I would go for it.
the important key here is to store it in the right way so I can look userId later using flask-SqlAlchemy without running in any issues.
15:32
have we seen poke recently btw?
@PM2Ring it is to be used by real ppl.
@JonClements Not for ages. A few months, at least.
looks like he's been on the site but just not chat
@PM2Ring I'd love to get a tip or two, thank you for sharing!, but I'm kinda on a clock here, so I'd grab what I can for now and upgrade later.
@LoopingDev In that case, please be careful. Even if you use crypto libraries written by experts, you need to understand the concepts & tools to use them properly.
15:37
yes!, i think even the simple function that I wrote, I could also twist few digits there or reverse part of the org string which would make it much harder to steal

but if there is any tool that has been proven by expert I'd rather use that, and make sure that I take my time in learning how it works.
Can any of you guys spot a bug in this code example of the structure of a Multi-objective Differential Evolution optimization with dynamic constraints? pastebin.com/DPavgeFa
well, what happens when you run the code, and how is that different from what is supposed to happen?
I have no idea of the purpose of your program. But if it stores actual passwords in any way, you must do it properly.
@LoopingDev I'd definitely go with PM2's advice here... cryptography/encryption are a science on their own
you can't just knock something up in a few minutes that's even going to be worthy of any formal scrutiny
@KarlKnechtel are you asking me?
15:47
Most likely. I too have that exact question
it's not a hard read either...
some of it is a bit put your thinking cap on kind of thing but most of it is explained nicely
@LoopingDev Even if you use a good strong encryption like AES, it may not be secure if you don't use it properly. The simplest way to use a block cipher like AES is called ECB, Electronic Code Book. It is not secure! There's a great demo of its vulnerability here: en.wikipedia.org/wiki/… Of course, decent crypto software will use AES properly, by default. But if you don't understand what you're doing it's easy to screw up.
Even if your site is low risk, it can still lead to problems for your users, if your site is compromised. Partly, that's because users aren't perfect, and may re-use sensitive passwords on multiple sites. That's not your fault, but you still need to make your site as strong as you can, to protect your users from their own blunders.
Sadly, the last couple of panels of this comic are no longer as funny as they were originally.
16:20
A couple months ago I wanted to do some cryptography in a project of mine. I was 100% committed to doing it the right way, following every best practice, etc etc. Except... I couldn't tell what the right way was. I had hashlib and cryptography ready to go, and those are strong building blocks to be sure... But a house made of strong blocks can still fall apart if you don't know how to use them.
hey guys i've got a problem of scraping a site containing React.js , while also , there is pagination which is not the usual page/2 , rather just a a tag, when clicked it loads the element into it.
i see selenium can do this, but its heavy for such small task.
anybody have idea on this type of scraping ?
You'll find plenty of articles online describing how a system got breached despite using the strongest and best-tested encryption on their passwords, because they forgot to salt before encrypting. I could just imagine being in that position. "Kevin you fool, you salted and peppered the passwords, but you forgot to garnish them! Everybody knows that encryption is worthless without garnish"
@Becauseihatemyself you could use something like Selenium for that.
I would personally use Selenium. If you don't like that it opens up a browser window, it's possible to run it in headless mode.
oh ok thank you, so i've only selenium choice and no other.
scrapy ?
16:28
You could also reverse engineer the protocol that the page uses to talk to the server asynchronously, but that too is heavy for such a small task.
oh , tell me more about that kevin
I'd definitely start with that though. Open up the network inspector, look at the request and the response, then decide if you want to reverse engineer it or use selenium
Aran-Fey how reverse engineer
Oh, conveniently, the scrapy documentation has some tips for reverse engineering. docs.scrapy.org/en/latest/topics/dynamic-content.html
Do the request in your browser, then look at the URL and the headers and everything else you need to emulate it
If all of this is spanish to you, use selenium
Thank you guys
Aran-Fey gives a good summary of the usual approach
ok i'll use seleniu if not :)
I'd personally avoid selenium if possible but...
yeah... I'd personally go for looking at the traffic and emulating that
16:32
I'll revise my preferences. First I would look at the page's source and network history etc to see if it looks easy to reverse engineer. If not, then I'd use Selenium.
Kevin is reverse engineering means Selecting dynamically-loaded content or ..
in this context
Yeah
don't think it's been updated in a while but requests_html is also kind of nice
curiously it says "Only Python 3.6 is supported."... fairly darn sure I've had it running for a particular job just fine on 3.9
Recently I was trying to scrape a webpage (for non-evil purposes) and I worried it would be very difficult because it had infinite scrolling. But in a stroke of luck, I discovered the page also had an atom feed. It had all the data I needed, in an xml-based format that was only moderately irritating to navigate. No fiddly scraping / parsing / regexing required.
16:53
guys amazing, i found that website which was using react, with different a loading contents, have response, when i look into response , there was their backend site like bk.something.com/?category=""&&page=1... , and it is literally easy
is that developer not knowing to leave it alive , i mean backend api can be accessed into browser on other peoples .
The website wouldn't work without that
i mean why i can access their backend json api, isnt it inaccessable other than their client or domain name and cors.
The website has javascript code that uses that API
correct, but api isnt protected , even i can access it , isnt it? or i'm wrong
Of course you can access it. Javascript runs in your browser. If you can't access it then there's no point
16:59
ohhhh
Aran-Fey, if the data is in json and i could use just their api , to get all GET request data, then why use selenium ?
Because using selenium is easier
And sometimes more resistant to changes. The website isn't as likely to change as the internal API is
if i made like django rest framework backend api, for login logout i use frontend, then like if somebody uses my that login api, to send data, anybody can do it right , with my backend
Sure. Anything you can do with a browser, you can also do without a browser. Just not as easily maybe.
whoa i didnt know that , ifeelstudpi
i thought like if i make a domain, like hat.com , and if my backend api gives data only and only to domain that is hat.com and nothing else
is it possible if? :/
like if i made login api, that is only accesible to hat.com , and if other want to access that api just block them.
Then why even make an API in the first place?
If you're the only one who's allowed to use it, just don't put it on the internet
17:11
🫣
like those twitter gives api only if we signed it , otherwise they block to give their api how do they do it then?
I don't know how twitter does it, but generally speaking, you ask for some kind of authentication. A password, or a session token, or an access token, something like that
oh ok , i hope i'll understand it soon.
when i scrape the site in different langauge i get this alien words why is that \u0906\u092f\u094b\u0917\u0932\u0947 \u092d\u0928\u094d\u092f\u094b : \u092e\u094c\u0928 \u0905\u0935\u092
It is in nepali language. But api dont giving this, what is the idea here?
It should look right if you print it
>>> print('\u0906\u092f\u094b\u0917\u0932\u0947 \u092d\u0928\u094d\u092f\u094b : \u092e\u094c\u0928 \u0905\u0935')
आयोगले भन्यो : मौन अव
whoa
cant we save it as nepali langauge rather than jibbrish?
in json
I think you can pass ensure_ascii=False
17:23
ok thank you. :)
If there is given authentication based or something api, then i need to ask the owner to use the api, otherwise i get public in sites that doenst need session token or anything to let api consume.
:)
I'm in a similar position with a site I'm interested in scraping (for non-evil purposes). 90% of its content is available to anyone, and you need an account for the remaining 10%. I have an account, so in principle I could access 100% of the content programatically.
But of course nothing is documented and half the parameters are one letter long and it does not spark joy in me
17:42
@Kevin is this just one of your quests into obscure issues or is that your actual code and you're not going to/unaware of how to fix it?
On a technicality, sqlite3 does not support multiple statements in execute() so you would be ok on that side
It's a simplification of actual code that I don't own. I notified the maintainer and they said "nice catch, we'll fix it eventually. Not too high priority, since the table being selected from has no sensitive information. "
Now that I know that you can also access data in other tables, I may be able to get them to bump it up the importance list.
Ah, that makes more sense :)
I'm pretty sure they know how to parameterize queries properly, so it should be fine once they direct their attention to it
Probably done in anger with the % messing with wildcards if you have an f-string anywhere in the query (even for legitimate purposes for the things you can't parameterize)
Some of the comments are in all caps, so anger may indeed be involved
18:29
How unmotivated do you have to be to hear about SQL injection and put it off until later? Sheeeeeeesh
Arguing about it probably takes more effort than fixing it
Fixing bugs in your code in response to bug reports is a sign of weakness
Considering the declining insect population, it's probably safe to assume that only around 20% of bug reports are real bugs anymore at this point, right?
19:11
@Aran-Fey Only the weak fix their bugs instead of arguing about them
 
1 hour later…
20:14
I have two numpy arrays of type uint8, and I want to find the absolute difference between the two. Is there a better way to do it than abs(arr1.astype(np.int16) - arr2)? That creates 3 copies of my 2500x3500 image...
20:24
Hmm, probably not... If you're worried about copies you could probably use numexpr. Not sure if that supports type conversions.
perhaps you could use ufuncs with an out parameter
let's see
As in, create a int16 copy and then continuously modify it in-place?
That seems to work, great!
I think I'll have to avoid numexpr because it requires a C compiler. No way I'm setting that up on 20 PCs at work
Hello fellas! I've been working on this bug all day but ain't able to solve it. I have just created a runnable mcve that reproduces it. Would any of you be kind enough to try and help? pastebin.com/EwzthzMj
20:31
You can indeed use -= without explicitly calling a ufunc, but for the abs you probably need np.absolute(res, out=res). If you do res[...] = abs(res) you're still creating a temporary.
@lupus that's a CVE at best
I would be able to further simplify it, but because I don't know where the bug is I have used the exact same structure I use in real code :/ Maybe you can skip the intro.
I thought the intro contains your actual question
The question is on the "Problem" and "Expected behavior" bit
@Aran-Fey maybe you can do it using SymPy, see this: docs.sympy.org/latest/modules/numeric-computation.html
20:35
The intro being more of a context. But perhaps disposable. Is it recommended for a MCVE to dispose of all abstractions? I could perfectly use variable names like x, etc, but I initially thought it would hurt comprehension ease.
@AndrasDeak--СлаваУкраїні oh, sorry, but why?
@lupus It's not a problem if your variable names are meaningful. What might be a problem if your MCVE has 30 lines defining constants. I opened that, saw 173 lines in a pastebin, and closed it, because I don't have time for that much code right now. Seems like your problem is "I think some of my constraints in differential evolution aren't being called" which can probably be reproduced in less code. I'd probably try to whittle it down while I wait for replies.
@Marco because sympy is for symbolic calculations. Doing math like "a + x == b". That page explains how you can turn symbolic expressions into actual numbers by substituting things. If your problem is already numerical (because you have a numpy array of numbers), there's no way adding sympy to the problem would help. Symbolic math is expensive. It's slow. It's only to be used when you have a symbolic math problem.
Or when you have reason to believe that e.g. integrating a formula symbolically and then evaluating that is superior than a brute-force numerical integration.
@AndrasDeak--СлаваУкраїні got it, thanks, I just made a bet, I thought that SymPy would solve the problem of copies of arrays.
that only the result would be evaluated.
@AndrasDeak--СлаваУкраїні What would be a simpler way to include the constants in the MCVE though? In this case the values are all dummy numbers and don't matter at all, but if I do not include them the code won't work at all. Should I have a single line with all of them, and a comment telling readers to ignore because it's dummy constants, perhaps?
I'm sorry anyway. I'll do some changes to it and eliminate the whole context and abstraction and I think people will still be able to help, with much less code indeed. Brb
@lupus it's not really about the constants per se. I just noted what struck me as "too much for me now" when I looked at your dpaste. It's perfectly plausible that others here will see it and will be able to help anyway.
@lupus if you have time I'd do that, because the more focused your example is, the more people can try to help you
if you strip it down and the problem goes away you can 1. still stick to the original example, and 2. start suspecting that what you removed was responsible for the problem
20:58
s = b"\x01\x23\x45\x67" the amount of \x makes it hard to read. Is there an alternative way to define it?
Is bytes([1, 35, 69, 103]) more to your taste?
01:00 - 21:0021:00 - 00:00

« first day (4224 days earlier)      last day (947 days later) »