« first day (2776 days earlier)      last day (2397 days later) » 
00:00 - 19:0019:00 - 00:00

19:00
it's all about json nowadays
that is my only complaint
im on board with deep mlp on the cloud
Someone needs to write a JSON equivalent to XSLT and XSS.
Maybe get your ML in the cloud on designing and building that JSLT for us.
idk, protobufs sounds like an interesting alternative
Protobufs are more of a replacement for schemas than processors. And the “generate tons of unreadable code that does the parsing” thing feels like a throwback to ASN.1 or Corba.
Not so bad in Go, where anything takes a ton of unreadable boilerplate, so having a program generate that boilerplate is a win. But in Python, the choice is between auto-generated unreadable boilerplate and no boilerplate, so… a lot less win.
wim
wim
somehow I survived this long without knowing that stdlib html.parser is a bunch of regexes
(regices?)
19:07
Regexen
I think you’re supposed to use ironic Old English with stuff from the Unix world, ironic Greco-Latin with stuff from the Lisp world.
Reading other languages' docs always makes me appreciate the quality of the python docs.
fun fromLiteral(literal: String): Regex

Returns a literal regex for the specified literal string.
Gee, thanks
wim
wim
feck it, I'm just gonna use bs4 and lxml , looks like they have sorted out their cp36 bdist_wheel anyway
Tcl usually has good docs. Smalltalk used to. Can’t think of anything else; the next best after Python that people actually use is probably Go and Switt, and they’re both always lagging behind the actual library.
wim
wim
I hate people who comment their code like this
# call foo with argument bar
foo(bar)
and their commit messages are like this:
hate the code, not the coder. Love, peace, etc etc
wim
wim
19:13
modified the_file.py
I imagine they are the same ones who write docs like that ^
I love comments like that. A while ago there was a question with a piece of code from a newbie that was commented like that, and it had me laughing my socks off :D
From what I understand, the Swift team uses a tool that scans new comments and docs, and anything that (heuristically?) looks like it describes nothing more than the types goes into a “garbage list” to get “garbage collected” in the next sprint.
I suppose that’s hating the code, not the coder? But no reason you can’t also garbage collect the engineers who trigger the tool too often…
wim
wim
the tool could get their email address from the commit and sign them up for a few spammy mailing lists
open('the_file.txt') # open the file so that we can read the inputs which we will use later kinda comments are perhaps humorous on the surface but when you try to actually understand the question and fix their issue, it's a major distraction
@wim Really, it's not as bad as it first sounds, for the reason that OneRaynyDay & abarnert gave. Using regex on isolated fragments of known HTML / XML is safe. Trying to parse whole arbitrary HTML / XML documents is where the madness lies.
19:19
perhaps Zalgo can explain how the world has been turning lately
wim
wim
ehh....one import re and the whole module is tainted
Can I interest you in import regex? :P
@Aran-Fey To be fair, comments on newbie code that explain what they expect each piece of the code to do can be useful when they actually misunderstand what the piece of code is doing.
That's true I guess
wim
wim
@Aran-Fey I used that once, thought I was really clever, and then a 500k rep user rightly pointed out a pathological counter-example that broke my regex.
still smarts. scared to try it again.
19:26
Where's the pathological example from the 500k rep user?
something something two problems
I've seen people use regex for the \k escape
which is IIRC "drop everything you've matched so far"
wim
wim
@Aran-Fey it's under the comments of their answer
>>> pat = r'\(.*?\)(*SKIP)(*FAIL)|/'
>>> re.split(pat, 'A/((a/c)D/E)')
['A', '((a/c)D', 'E)']
@wim Impressive.
wim
wim
^ wrong result.
in the end I gave up trying to do it with regex and wrote a stateful parser. which collected a grand total of +0/-0 votes.
19:34
Oh yeah, it's well-known that regex can't handle nested structures. I didn't even count that as a bug because it should've been obvious :P
wim
wim
it was not well-known/obvious to me :)
I'm not from CS background, learned programming at the coal face so to speak
They don't teach you the cool stuff in school anyway, so you didn't miss much
wim
wim
what do they teach?
sockets, oddly enough. For some reason there's always an assignment to make a socket server + client.
@coldspeed Hello Sherlock Bones :p
19:43
let's see, lex & yacc, sockets, cgi, and multiprocessing
I didn't learn any useful language features, but I learned how to write a socket connection in java, C, and C#. Priorities ¯\_(ツ)_/¯
yeah and some crappy OOP related assignments featuring adventure games/puzzles/inheritance
I don't know how it is in schools here, but that's how it was done in India. Maybe that's why there are so many programmers who know "how to do something" more than know "how something works"
and there's a big difference between the two
wim
wim
wow
I thought it was gonna be stuff like red black trees , time and space complexity, ..
you know like google interview question stuff
I'm also not from a CS background but there must be some value in structured learning rather than reactionary learning, even if it doesn't cover everything? My knowledge is constantly blown wide open because deadlines mean I generally only learn something in depth when it becomes an issue in what I'm trying to build.
In theory, structured learning is a great idea. However, I'm usually disappointed in someone else's interpretation of what that structure should look like.
19:49
@wim Yeah, they teach those as well. But they only teach you that once, and not once per programming language, which is why sockets were the first thing that came to mind
It's not that I choose it to be that way but there's plenty of things I wouldn't even be aware existed until I find myself in a problem
@roganjosh I'm very much the same. I'm a depth first learner
@wim you don't get that (in most universities) until you take a master's level course in algorithms
if you want to prep for Google interviews, you'll have to do it yourself, school will not help
wim
wim
you're intern at google? what kind of questions did you get
@piRSquared and re: the issue with structured learning, it does tend to be lazy. 4 years of Chemical Engineering here, I think my lecture attendance was < 20%. If you're considering it, just buy Coulson and Richardson books and you can self-teach :) I guess books are a good alternative to reactionary learning
19:55
it's slightly easier for interns. They generally give you broader problems that require you to come up with something on the spot and code it in a google doc. The solution may or may not require multiple algorithms/lower level algo concepts
you get to choose the language you want to interview in...
cbg.
Golf clap
wim
wim
20:05
20:17
@wim When i went to school in the stone ages, it was that, plus denotational and operational semantics and even further into theory. Most classes in Scheme or Common Lisp, a bit of standard ML, C, and MIPS and 68k assembly. I remember trying to show my instructors what you could do with Tcl and C++, and later Python, and they didn’t get why you’d do that.
Oh, and we only learned about sockets in operating systems class, when we had to add the equivalent of sockets to our filesystem.
20:29
@PM2Ring I cba discussing it in that comment chain because it's getting too long already, but I really don't see the need for summ = Complex(). It's an instance to be thrown away
Ah, good old 68k assembler. What fun I had making my Amiga crash with invalid / unaligned memory accesses. :)
@roganjosh He should be returning those instances though, like his dif method does, not doing that silly string conversion.
We had a microprocessor lab class where we built a simple computer around a 68k. I managed to set the board on fire.
Wow. Even better than "printer on fire".
@PM2Ring I think it's the string conversion that's confusing me
@roganjosh It's a mystery. The 3rd comment asks about it, but the OP hasn't responded:
Why do most of your functions return strings? — molbdnilo 22 mins ago
20:34
Yeah, I saw that, but I didn't realise it was spilling over into my own misunderstanding when I read it :P
rbrb
Whenever I see someone write functions that return strings just to parse them, or ask why dct[1+”][2”] doesn’t work for their 2D list, I think maybe Tcl was right after all.
wim
wim
^ dupe for the billionth time, and somehow got to +9/-0
someone do the needful .. ?
squashed it
wim
wim
thx
20:42
Uh, that's different
>>> [] + 'a'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: can only concatenate list (not "str") to list
Wow.
wim
wim
> list.__add__ is implemented that way to maintain the convention that + operations are commutative: a + b == b + a
^ wat
@coldspeed I augmented the target list. ;) I was a bit slow because I thought it was in our canonical collection, but it isn't. I guess it should be added, but I'm too tired to do it now.
@wim Well, it's actually the first time I see that question asked on SO
Not to say it was not asked a lot, I see the duplicates, but I guess some of us have not encountered it and find it interesting
thanks, pm. I was in a bit of a hurry to close it before it got any more attention than it should've
wim
wim
20:51
@OlivierMelançon surprising
I've been here for 2 years, but more active for only 6 months though
wim
wim
anyone can understand what the heck user wants? stackoverflow.com/q/50496639/674039
I think OP only wants the html of the page and not anything that the JS / styles which might be impossible depending on how the page is set up
Nope. We should try asking those two mystery users who upvoted that question
for example if the page only contains a simple "hello world" in the <body> and the rest is derived from JS and styled I think op would only want "hello world"
20:55
another deadbeat dupe. Anyone with dupe suggestions?
wim
wim
want to give them benefit of the doubt, but it's sounding more and more like the guy from php_ceo
@wim I read it and assumed they needed to poll some website to give a response that was part of their own SLA/target. The response takes too long so they think that they can just pull a subset of the data and it will speed up, but I really doubt that it's the amount of data they're reading
I think they should scrap the first part and only write "Is there any way to load the body part only for a url?"
It seems pretty obvious: he wants magic. If the web were designed like IMAP plus mime/multipart, or like MPEG4 containers, that magic would be possible, but of course it isn’t.
wim
wim
perhaps he wants "not http"
20:57
If anything, it will surely be more limited in the response time of the website in general rather than whether it returns <head> or not
Maybe tell him that he can’t control the server, so why not just only read the bytes he wants from the router? When he asks how he knows which bytes he wants… then he’s got the problem.
In fact, I don't think it could possibly be quicker?
Eh, another black hole in my knowledge. I use Flask to render templates using {% include %}. If I send a GET request to that (I can't test now), I wouldn't get a response until the entire template rendering is complete?
wim
wim
unless jinja is doing something insanely clever, yes
since they must send you the status code 200 OK or not, you have to know if the template rendered successfully right?
So my thinking is correct, there couldn't be a noticeable speed-up by disregarding the header because your GET request can't return a response until the whole template is created
wim
wim
you can't disregard the header it's part of the protocol
21:15
I assume they don’t know much about what servers do. There’s no reason servers couldn’t be built around slapping together independent pieces that could be generated by separate templates and stitched together. But in practice, the way to do that is to create an empty skeleton and just AJAX the pieces.
21:27
@wim it could use HTTP trailers
21:38
When building a package that is also executable (e.g. python -m my_package do_your_thing), should the logging config be part of the code?
It might be trivial, but I have a really hard time finding good examples of how to best do that. Most don't include things like how to resolve paths, and how should it behave in prod as opposed to dev.
Yeah I started to think about writing a scathing answer about how obvious a question that was, and ended up waiting, hoping to read an interesting answer by someone smarter than me.
I mean, on some level you can just grab a logger and log to loglevels, and let an environment variable tell you which to default to (directly, or indirectly by which environment you're on)
But, yeah. I don't know much more than that.
Good to know it might not be trivial, I guess I'll write it as a proper question on SO.
@RobertGrant My follow up questions would be 1) should the environment variable be passed through env.args? 1.1) If it is, how do I log messages during e.g. tests, where I can't control that (python standard logging defaults to the NullLogger, and Pytest grabs from stdout) 2) to which location should log files be written?
Yeah, log files are complicated, and that's why I like writing to stderr, stdout and letting some clever devops person figure that out :)
Probably a good call
But as soon as I involve them they ask me to write my log messages in json format. It's bad options all around =(
Heh yeah we're about to encounter that
Have the log format passed in as an env var, then you can override it locally :)
21:54
env as in os.environ.get?
Just noticed that up there I meant sys.argv, not env.args. what a mixup
wim
wim
22:17
Put the logging config call as the first thing in your entry point (usually main())
Do NOT configure logging at import time
@wim opinion on putting logger @ __init__.py for module-level granularity?
wim should start Strong Opinion as a Service
wim
wim
yes, logger should be created as module level global
but logging configure should not be called at module level
makes sense?
@wim elaborate on that point?
wim
wim
I don't know how to make it any clearer
importing your module should not trigger a logging configuration
22:20
Why not?
hey, at least that part I did correctly
wim
wim
that's the beauty of the logging design. you don't care where the logs go to, if anywhere.
if I may add a naive question: where does it go then?
Ah. Makes sense.
wim
wim
it's up to the user who imports your code to decide how to configure loggers.
22:21
ah, nowhere
wim
wim
@AndrasDeak not necessarily
depends on the framework
or in the __name__ == "__main__" of whatever your user is running?
wim
wim
for example stdlib will spam warning and above to stderr console
I mean to be called from there...or something
wim
wim
because everything will trickle up to the root logger, and it will automatically configure a sys.stderr handler at warning level if there are no handlers
but, don't use stdlib logging. please.
it's slow and bad and there are a tonne of better options avail.
22:23
logging is stdlib, logger is what you prefer instead?
not paying attention to logging chat is taking its price
@AndrasDeak hi pot i'm kettle
recbg
we can start our startup together
wim
wim
I work at a HFT logging is extremely important here
@wim So I shouldn't start every file with log = getLogger(__name__) ?
I hate this question as the canonical dupe for "Why does list.append return None?". Are there any alternatives?
22:26
what is HFT? Google says High Frequency Trading
I think it's that
oh nice
scripts doing stock market or something, I imagine
that would sound like something where logging is important
unless it's trading with commodities, in which case logging is important to obtain lumber
wim
wim
@coldspeed yes
@Arne that's ok
@AndrasDeak oh, that's probably why 80% of all loggers have a beaver as a mascot
22:28
LOL
Ahah
wim
wim
@Arne that is creating a logger. not configuring a logger.
creating a logger at the top of every file is fine (some would even say, best practice)
success, my disgusting accepted answer with globals and exec has been mod nuked
hmm, what's a non-stdlib logger?
google is failing me
@wim I still don't know how to configure logger for tests though
wim
wim
a logger that is not an instance of logging.Logger
@Arne you don't
22:31
in other words, I'd rather not depend on main() be called before any line of my code may be run
But are you talking about different modules, or just a non-default logger instance?
wim
wim
well, you can, but you don't need to.
capturing log events doesn't require logging to be configured
context: I have no idea about logging
lmao good times
Yes, it was either that or a downvote. Roll-back if you prefer the downvote. — wim Jul 6 '17 at 17:52
@wim does pytest? i thought it just grabs stdout when it fails
wim
wim
22:33
@Arne nope
so, the way it works is, at test setup the test runner will inject a log handler
this is caplog fixture's __enter__
built into pytest nowadays
caplog intercepts log records and makes them available as the attributes .records and .record_tuples , which you can make assertions against
you don't need to configure logging at all, because you don't really want logs to go to files nor io streams during test execution.
Hi! Does anyone have experience with pyspark? I have this code: https://pastebin.com/29fkGL5w and I am trying to count how many articles in my RDD contain the name of the language. (For the sake of simplicity we check that at least one word (delimited by spaces) of the article text is equal to the given language.)

So far I have the code in the link, but when I run it, it simply returns a 0 for each item. When it should be returning much higher numbers. I feel like my occurancesOfLang method is probably doing something wrong, but I'm not sure what.
wim
wim
(and this is one of the several reasons why you don't want logging configuration to get triggered at import time - you'll have to "undo" that configuration in the test runner, and it can be tricky)
@AndrasDeak different modules. structlog is my current favourite.
Thanks. I find it weird that I can't google any alternatives
wim
wim
@AndrasDeak the only line under if __name__ == "__main__": should be main().
22:38
@wim thanks for all the info, it helps me a lot
wim
wim
no probs
to give you an idea of the scale we log at, I just searched the last 15 mins events at WimCorp and we have 47,360,979 log records firmwide. and it's not even in NYSE trading hours right now, so it's not even close to the full firehose.
structlog looks fun
wim
wim
thats about 50k logging events per second
That sounds pretty epic actually.
I just imagine text files filling and filling and filling.
22:41
one last question, does it make sense to have different configs for dev/prod/.. available and pass dev/prod/.. as an argument to the application, or should there ever only be is it good enough to only have one config?
not bad I suppose
wim
wim
I wouldn't call it dev/prod (because you want to be able to tweak log levels for various handlers within an environment too)
@wim elaborate as in what's the rationale; got your rationale, makes sense
@wim Is args = parse_args(); main(**args) still acceptable?
wim
wim
but yes, you should be able to configure logging levels from outside the app (I usually do this with a collections.ChainMap of cmd line arguments, environment variables, config files, defaults)
22:43
and I never understood why there is so many logs in this company, but now I guess I get it
@wim that sounds like I'd just pass the full config for each application launch
wim
wim
@Aran-Fey no. argument parsing should go inside main.
@wim python -m my_app arg1 arg2 path_to_logger_config.json?
@Annabelle I guess nobody here ATM knows pyspark. All I can suggest is to replace that lambda x: x[0] == lang with a proper def function that can log or print stuff, so that you can check that lang and x contain what you expect them to.
Hmm okay. I'll try that. I have a week to mess around and figure it out.
wim
wim
22:45
@Aran-Fey otherwise you need an exec or a subprocess to test your argument parsing.
Huh? I don't follow. Why would splitting a method into two methods force me to use exec?
I think he means you can't test what's inside your __name__ == "__main__"
wim
wim
^
you won't get coverage of the parse_args() line
not without ugly hacks anyway
... that seems pretty inconsequential, honestly
wim
wim
if you don't strive for 100% test coverage, sure.
for my proj if the test coverage drops below 100% that's a CI failure.
22:50
=0
that's a nice way to delegate bugs into tests ;)
I prefer to separate argument parsing from the main method because it allows people to import my script and call the main function like a normal function, without having to encode the arguments into sys.argv
then you should have a main that does the parsing and call another almost_main() that does the rest?
just following what I imagine is wim's point
How do you get 100% line coverage without exec or a subprocess anyway? You have to test the if __name__ == '__main__': block somehow. Why does it matter if there's one or two lines of code in there?
wim
wim
@AndrasDeak yep
@Aran-Fey if __name__ == '__main__': is ignored by coverage.
so it doesn't dip you below 100% ;)
22:54
solution: cram everything into if __name__ == "__main__":, coverage problem solved
I see. So wrapping those two lines in a function would make it easier to test. I think I'll take my chances and not wrap them in a 2-line function.
wim
wim
not so much about ease of testing as about making parity between "your code that runs in prod" and "your code that runs in test"
I have a colleague who wrote tests for his tests
it's really wild, and quite fascinating
that's just extra
wim
wim
I had a colleague that does that too. This, I don't really agree with.
23:02
pytest provides some features for it, so it can't be too unusual
wim
wim
It suggests there's too much logic in your test code. Tests should be very dumb, preferably no flow control at all.
for loops and if statements in tests = code smell
i would have liked to argue that point as well, but i couldn't understand the meta test code enough
it felt like philosophy courses in uni all over again
you need to test the test to make sure a freak cosmic ray hasn't replaced the test code with a print that pretends to be a passing test
that's why we run tests twice, and only pass if both path
(we don't actually. but i might bring it up for fun)
are you writing in lisp?
awwww and you had to edit :D
<3
23:06
too good to not keep =D
Python actually has a built-in statement to make a passing test case: def test_something(): pass :D
5
the actual reason I didn't argue the meta testing was probably because he was so enthusiastic when he explained it. and maybe also because he's my senior.
nooooo when I make PJs everyone ignores them
anyways, rbrb. I learned a lot again =)
00:00 - 19:0019:00 - 00:00

« first day (2776 days earlier)      last day (2397 days later) »