« first day (3311 days earlier)      last day (1644 days later) » 

2:04 AM
cbg patch
 
 
3 hours later…
5:17 AM
hey guys, how should i go about taking data formatted like this and making a distance matrix out of it
	0	1	2
0	bmw	toyota	1.287049
1	bmw	pontiac	1.442010
2	bmw	sedan	1.402632
3	bmw	honda	1.304037
4	bmw	acura	1.489019
5	bmw	audi	1.704268
6	bmw	infiniti	2.009573
2 is our distances
so its like this in the end
             bmw     toyota      pontiac ...
bmw           0        1.28          1.44
toyota     1.28            0
pontiac    1.44
and so on
 
For this simple problem, can I have a better algo?
I have 2 arrays, and two sets of upper and lower indices will be provided. We need to form two subarrays from those indices. Not I need to collect all elements in both arrays which are not in either of the subarrays.
I have written a simple code here: pastebin.com/bwWLgU9D
Do anyone have any better idea?
 
 
3 hours later…
8:34 AM
@Skyler where are the distances from? If they're in the right order you could use docs.scipy.org/doc/scipy/reference/generated/… perhaps
@Quark side note: do you really have arrays, or do you have lists?
 
9:05 AM
(5 more minutes for the latter but there's nobody here anyway :P)
 
 
2 hours later…
11:15 AM
@AndrasDeak closed
Leaving the above one as it is for now since there's been an edit with code, not sure if it's sufficient, but seems okayish at least
If we're still having the meeting on 15th, I think with <7 days to go, the starred message should get re-pinned / highlighted to the top
 
11:42 AM
Reminder: we're having a room meeting on November 15th. Agenda items can be suggested here
 
12:00 PM
It'd be cool to try to pull more people in, there are a alot of dead hours here
btw, i took care of the distance thingy, and then I had to do it a second time later on and realized i was being a total idiot and there was an even easier way of doing it. It's not exactly a true distance, it's a dissimilarity distance based on lifts, and I did an MDS on it after to constrain the output to a 2d plot
 
12:12 PM
hi, why can not i run this code from idle:

import subprocess
subprocess.call([
r "C:\.......\notepad++.exe",
r "D:\.......\filename.py"
])
 
12:32 PM
I solved it! syntax error :D
 
@EnthusiasticEngineer for future reference "can't run"/"doesn't work" is never enough information
 
@AndrasDeak what more should I add except mwe?
 
Whether it raises an error or gives unexpected results. In the former case the error with traceback, in the latter case the actual output vs expected
 
@AndrasDeak ok. sorry, I will post more complete questions in future. :)
 
Thanks
 
1:01 PM
^ closed. ty
 
Hmm. Why would ping wikipedia.com work for me, but not nslookup wikipedia.com? Don't they both use the same underlying mechanism?
 
1:50 PM
I wonder how hard it would be to implement a very small HTTP server on top of the socket module. Using the example code in the docs, I can listen for connections, and entering http://127.0.0.1:50007/ into my web browser successfully sends data to the server. But I'm not sure what I need to send back to make the browser display anything other than "problem loading page". b"HTTP/1.1 200 OK" by itself isn't enough.
Presumably I need, like, response headers and content and stuff. developer.mozilla.org/en-US/docs/Web/HTTP/Messages is evasive about which ones in particular I ought to use. I guess it varies.
Obviously I could do all of this in ten seconds with Flask, but where's the fun in that
 
I copy-pasted the response headers from example.com and now I can serve a simple Hello World page. Trying to figure out which of these headers are actually necessary
I removed everything but content-length and the response body and the page still rendered, but then I noticed that the browser was ignoring the body and using its cache instead. So maybe the various cache headers were important
I'm getting a little ahead of myself because all of this only works if I try to access 127.0.01, rather than my ostensible local network IP, 10.0.0.38. If the page isn't accessible from other computers on my LAN, then it's not very useful.
Even a simple ping 10.0.0.38 gives me four General failure. messages, so I'm skeptical that it's really my address
 
3:17 PM
@Kevin I'll go on a limp and say cache headers aren't important, content-type might be quite imp.
Of course there's a SO question on this with a validating answer :) stackoverflow.com/a/25586633
 
Good to know what headers are mandated by the RFCs. I suspect that each browser has its own ideas about what's required, though
 
3:35 PM
You have to create a function that takes a positive integer number and returns the next bigger number formed by the same digits:

12 ==> 21
513 ==> 531
2017 ==> 2071

If no bigger number can be composed using those digits, return -1:

9 ==> -1
111 ==> -1
531 ==> -1
I have written a function to do so, and that is working. But it is running out of the time.
Here is the code.
Where it is taking maximum time? How can I improve this?
 
Your solution is basically: make every number possible, then get it's index in a sorted list, then take the next. It works, but usually problems like these expect you to think about some properties of the numbers themselves, rather than just muscle your way through it.
what happens if there's 20 digits? or even 10? That's why it's slow.
 
itertools.permutations is O(N!) time so it's very slow for inputs with more than like six digits
 
So, change your line of thinking. Can you try to think more along the lines of the numbers themselves, and more specifically, what decides whether a number is bigger or smaller, given two numbers made up of the same digits? Like, why is 321 bigger than 123? Just think about it, no need to answer. Can you utilize that information somehow?
 
Hmm, I am trying to think about it, thanks for pointing my mistake.
 
I wouldn't even call it a mistake. Your solution works, so you deserve kudos for that. It's just that after brute force, it's time to see if there's something better you can do.
 
3:45 PM
Miscellaneous observations that may be useful:
- A good permutation algorithm outputs results in increasing lexicographic order, if its input starts out sorted
- it's possible to find the Nth permutation of a sequence in O(1) time.
- if A is the Nth permutation of B, then it's possible to find N in O(1) time, given A and B.
Hmm on second thought those O(1)s might be O(log len(input)) or something in that neighborhood, but it's still way better than O(N!)
 
 
3 hours later…
6:26 PM
Quiet in here this evening
 
*cricket*
 
I've done my time on the main feed, I need a break. Can you do a merry jig or something for entertainment, Andras? :)
 
beyond my paygrade
 
I already watched dolphins chasing bubble rings around. Until last week, I didn't know this was a thing. Maybe I've been living under a rock, but toroidal bubbles are awesome
 
Seen the one with the jellyfish?
 
6:32 PM
That was what led me to finding out that dolphins can make them at will :)
 
yeah, and they're jerks
playful jerks, of course :P
they can also play basketball
 
Slightly more friendly than their bigger cousins :)
 
those are jerks too :P
 
 
1 hour later…
wim
8:19 PM
@Kevin Kevin's annoying interview question spotted in the wild stackoverflow.com/q/58783138/674039
 
I'd think the straightforward solution is to clean up the technical debt...
"no, we can't switch to pypy, I know it's 2030 and cpython is almost dead, but our hack from ten years ago relies on the C stack"
 
9:12 PM
if I have a structure like

/my_proj
/a/__init__.py
/a/my_mod.py
/bin/my_script.py

What's the right way to import my_mod.py from my_script.py if I run it like ./bin/my_script.py? Do I really have to muck with PYTHONPATH/sys.path ? I've read a half-dozen things on default python paths and the best practice still isn't clear to me
 
You can't do triple backticks to format in chat. See the formatting guide
 
import my_proj.a.my_mod
if it doesn't work, install your package (how depends on the build system you're using)
 
@Aran-Fey What do you mean install my package? Like turn it into a proper python package with setup.py and everything? I'm actively editing it, I shouldn't have to re-install the package every time I make a code change to my_mod right?
 
Yes, turn it into a proper python package. If you use a setup.py, you can install it in editable mode with pip install -e .
though I'd recommend checking out flit
 
Thanks, was not aware of -e flag or flit, I will check those out
 
> Really great find by Rui and Prof. Williams. When I wrote the scripts 6 years ago, the OS was able to handle the sorting.
what do I press to doubt?
 
"Incorrect" results were caused by the order of glob output?? And they "fixed" it by making the order consistent across operating systems?? Wut?
 
K for 100 doubts
 
Whatever faith I had in the correctness of this research is now gone
 
Hohoho, the magic conda develop $(pwd) takes care of all python path issues
 
9:36 PM
@Aran-Fey I read the story earlier. They looped over files in pairs, and assumed that the two kinds of files would come in the same order. Because of how parameters were included in the file name...
 
Hello!
 
at the end of the day it's just a bug, but "it used to work" is not really true
hello
 
I hope someone can help me
I am reading this article regarding dash(community.plot.ly/t/…) but I don't understand one thing in the script; if i import an excel file, can the function (some_function)iterate through all of the sheets?
 
@AndrasDeak Oof. Something tells me there are probably more bugs of a similar caliber in that code
 
well, this is a special kind of wrong assumption, doesn't mean that the sciency stuff is wrong
the paper revealing the problem is here pubs.acs.org/doi/10.1021/acs.orglett.9b03216
(it's paywalled but sci-hub.tw is available as ethical piracy)
 
9:43 PM
eh, I'm not interested enough to deal with a paywall. If they don't want me to read it then I won't
 
@Aran-Fey :(
 
scientiic journals don't want you to read it, hence sci-hub
 
You know it costs £1000s for researches to not have their research behind that paywall?
 
nope
 
Andras explained the publication process in here before but I can't think of how to look it up
Basically, I submitted my second paper for publication. We paid £2400 (IIRC) so that it's open-access
That's just so that others can read it without subscription. That's nothing to do with the time/effort/money in making a publication, it's literally just to remove the paywall once it's published
 
9:46 PM
Jun 10 at 19:32, by Andras Deak
If you see a paywalled academic paper try sci-hub (currently sci-hub.tw). It's the right kind of piracy.
 
I.... have to really meter my language when talking about the publication process. It's an exploitative mess.
 
Wow, that's a lot of money
 
@roganjosh my thoughts exactly
 
Do people who review papers (editors, peer-review folks, whatever) get paid?
 
Nope
 
9:48 PM
@Aran-Fey and this is almost always tax money payed by governments to advance science for all of humanity ¯\_(ツ)_/¯
@ThomasJohnson if you review a lot you maybe get $40 off for books bought from the same publisher
 
I guess I don't really understand the function of journals than. Why publish in a journal at all rather than just put your stuff on arxiv or whatever?
Or why hasn't this whole process moved to a non-profit model?
 
@ThomasJohnson peer review is important, but reviewers and researchers are both victims of the status quo. A posh journal ensures your research is high-quality...
 
Journals have an "impact factor" which is, roughly, the ratio of external references to their articles vs. internal references. That does a lot for people's career if they publish in high impact journals
As to how this corrupt system came into being, I'm less sure. There's prestige in also being an unpaid reviewer for a high impact journal
 
Nature has high impact because people know it has high impact so papers there get high impact
 
Meanwhile, the journal just skims off money. It's ludicrous.
 
9:52 PM
 
The one drawback I see about arxiv is lack of peer-review. You would think that if you could build some kind of anonymous PR process on top (like Open Review + Arxiv) you would be able to make a run at changing the system starting from the bottom/middle
 
Another aspect is that some agencies that employ researchers use metrics to determine the quality of pubs based on journal status, meaning that even if the researcher could not care less about a journal's name, the organization funding the work might
 
@ThomasJohnson yeah, but you can't just create a high-impact journal. The reason this current big dogs are big dogs is long-term history. It's a kind of mass delusion. Like money.
 
This is true for many federal research agencies in the US, which is terrible because the knowledge gains from taxpayer funded work are often disseminated in a way that taxpayers cannot access/afford
 
even if you start a free alternative, great research will not be posted there because people won't want their high-quality research to be wasted
 
9:56 PM
So really it's not just PR but also a filtering process. Like if Nature somehow got 100 amazing submissions all in the span of a month it's not like they're going to publish some kind of special double-issue to fit them all in. So it's more like "I got published in Nature, therefore I'm better than all my peers who tried to get published but couldn't"
 
I'm not sure whether there is a truly hard limit on what gets published
Likely, the flow of good/not-so-good articles coming in is balanced against the experience of reviewers looking at the papers and accepting/rejecting
 
Who actually pays the publication fees? The University as an organization, or does it come out of researcher-specific grants and such?
 
The university... via grants
For a small project, you might apply for, say, £15K and in that application, you'll budget £2.5K for publication specifically
 
So this is like the other side of tuition inflation
 
This will be different all over the world, I'm sure. This was in Engineering in the UK for me
How is this to do with tuition?
Tuition is a labour for universities. They are money-making institutions like any other. Once you get out of undergrad status, you might actually be useful in the workforce for what they actually want to do. Which is bring in grants and make money
 
10:04 PM
Tuition increases are largely driven by lower-cost borrowing. The cheaper cost of debt allows universities to charge more, while providing the same amount of value. I was guessing that a similar kind of thing existed for post-grads/researchers in terms of paying for publication
 
I didn't personally pay any of it. Again, this is the UK system that I have specific experience of. The research grant from EPSRC (in the UK) had a specific spend for publication
Which, ultimately, just comes from all the taxpayers in the country
(I did say the system was nuts) :P
 
Yeah I was kind of assuming that there was some (marginal) funding connection between the increased amount of money that universities make and the money used for paying pub fees, but maybe it's 100% taxpayer->grants->journals
 
It's not. If you get industrial backing and it's not some critical business secret for them, they may publish too. Even pharmaceutical companies will publish some of their findings, and you'd expect them to keep everything close to their chest
 
Yeah TBH I don't really understand why corporate labs (pharma, big tech, etc) publish either
 
Prestige
 
10:13 PM
Apple barely publishes, and they still have prestige
Actually I suppose it kind of makes sense if you haven't had really impactful results. Like Deepmind does all this RL research and publication, but it's biggest claim to fame in terms of actual product is like "we reduced google's datacenter cooling costs"
 
Or that they're pretty confident that before someone else can build Alpha Go, they'd already be 3 steps ahead
 
That's what I mean though - Alpha Go is interesting as a research project, but in terms of how valuable it is...seems pretty low. Like, almost zero.
 
What about self-driving cars that have to make a move?
 
Self-driving cars aren't built on RL though
 
RL == real life?
 
10:18 PM
Reinforcement Learning
 
err, then I don't follow your point
Didn't they have self-driving cars doing thousands of hours doing simulated drives just for that?
 
Self-driving cars use Deep Learning, but not much Reinforcement Learning. My coworker who used to be at Cruise said that it's all very much heuristics / control theory
medium.com/self-driving-cars/… for more details on RL
 
mm, I'm not so sure. Certainly I could be wrong, but I don't know enough of the fine detail to debate it with you. Self-driving cars was an off-the-cuff for why Alpha Go optimisation might have a real-world application
 
10:37 PM
If I were to purchase web hosting for a Flask project, what marketing buzzwords translate to "can execute Python scripts on the server"? I'm paranoid that the bronze-tier dollar-a-month plans will let me serve static .html files and nothing else
Do I need... The cloud?
 
Maybe. How's "python anywhere"?
(not a buzzword, but a service)
> Host, run, and code Python in the cloud!
consistency!
it might still have a free tier
 
I ended up having to buy the $20/month digitalocean server to start my server off because OSRM took 3GB of RAM to process the map data. But it starts at $5/month and I've not seen an issue with their service tbh
 
Ok, I'll look into PythonAnywhere. The free tier should at least prove useful during the prototyping stage. Thanks
Bah, someone took the username "Kevin" already
 
hey, long time no see :)
 
11:13 PM
@Kevin Why not spin up a server instance on Digital Ocean or something and use use gunicorn with nginx to serve the app. I set up a Django app with gunicorn and nginx and it was easy and cheap.
and fun :)
 
11:26 PM
I want to use the simplest thing that satisfies my requirements, which appear to be shaping up to:
- less than ten simultaneous users
- CRUD access to a database with under 50 MB of data
- semi-real-time inter-client communication, suitable for chat
I think PythonAnywhere can satisfy the first two. Unclear to me whether it will let me use Flask-SocketIO, which I understand is the conventional way to do bidirectional communication
A cursory googling indicates that P.A. would like to support SocketIO "eventually".
 
suspicious
 
> PythonAnywhere dev here -- you're right that we don't support websockets right now. Our stack is actually nginx and uWSGI, but as you describe in your last bullet point, websockets tie up worker processes/threads, so they wouldn't work with our seup.

We're planning support for async servers in the future, so at that point Flask-SocketIO (or Tornado or whatever) should work fine, but no timelines on that right now.
I'm a little surprised that they're seemingly not supported at all -- I figured you'd be able to hack it in if you had a subscription that permits always-on processes. (i.e. anything but the free tier)
 
@roganjosh the first scipy version is out which supports 3.8. So regarding this earlier question of yours, see for instance github.com/scipy/scipy/pull/10930 for a few issues that had to be fixed with 3.8
 
So serverless is the simplest option unless you want live chat and then maybe Django with Django channels or something similar on a self administered virtual server becomes the simplest option.
I'll make a mental not of this for all future endeavors
 
On another page, a dev suggests just making every client poll the server periodically. I guess I could do that, but it's yucky.
I wonder if ten polls per second, every second, for a month, would run afoul of my data limits.
 
11:42 PM
you could also have the serveless app and a separate slack channel dedicated to the app to handle everything with minimal effort
 
Yeah. "just use existing chat software like Slack" is definitely not off the table, and solves half my problems
 

« first day (3311 days earlier)      last day (1644 days later) »