« first day (3125 days earlier)      last day (1824 days later) » 

2:21 AM
Is there any simpler way to generate a for-loop range, excluding a set of discrete numbers, than for n in set(range(10)) - set([a,b]): ... which gives {0,1,3,4,5,6,7,9}. (Note a,b are variables, not fixed)
(Can't do set(range(10)).remove(a).remove(b) since those set methods don't return the set object.)
 
2:34 AM
@smci What's wrong with {*range(10)} - {a, b}?
it's a few characters shorter than what you have right now (I ask what's wrong because I assume you've thought of this already)
 
2:55 AM
@cs95 Brilliant, thanks very much! No I wouldn't have thought of that, I forget we can use {...} notation for sets, and I wouldn't think to use *range to unpack it.
 
anytime :)
 
As a followup, has anyone ever seen a need to define a range_less(start, stop, step, *vals_to_exclude) function to do all of this?
 
Haven't seen one, but if you wanted to, you could probably create a generator that iterates over range(start, stop, step) and yields whatever isn't in *vals_to_exclude.
 
@smci You can use filter to avoid creating a set of the whole range.
 
ah... yep, filter would be the tl;dr code equivalent of what I just suggested.
 
3:05 AM
@cs95 Not quite, because then we need to declare a clunky lambda to test if the number from first range is not in the second (exclude) set.
 
a statement of pain may a function gain
or something like that
 
It's the same difference, only incomprehensible to non-Python users: filter(lambda n: n not in [a, b], range(10)). Whereas a custom range_less/range_exclude says exactly what it does on the can :)
 
@smci create a PEP ;)
write an impl in C lol.
 
3:23 AM
@smci Not necessarily. You can pass filter a dunder method, but some people don't like that.
 
@Dair No I don't believe this is a common use-case for general users, I never said it was. But say for prime-sieving and congruences it is.
 
@smci oh I was only kidding about the PEP haha.
 
@PM2Ring I don't understand, how can you do filter(lambda n: n not in [a, b], range(10)) with a dunder? I'm talking about that specific example.
Hey guys also, how do I have a range-type iterator where step is itself an iterator? Say we want to generate all prime candidates in the range 10..29, we would want something like range(11, 30, [2,4,2,2]) or range(11, 30, iter([2,4,2,2])) except that isn't legal syntax. It could be done with a hardcoded generator, or maybe with coroutines (one coroutine to generate the step). What's 'best' (simplest syntax, least obfuscation, most intelligible to non-Python users, performance not imp.)?
 
@smci Yeah, for that you need filterfalse, or whatever it's called, from itertools. Sorry, I'm on my phone, in the sunshine, eating pizza, so it's hard to read my phone. :)
 
@PM2Ring Yeah I suppose. If you understand what I'm driving at, it's a set of intuitive-sounding language-agnostic primitive iterators for integer ranges, so we can do prime-sieving, factoring, congruences etc. So the absolute last thing a non-Python audience wants to see is quirky language hacks and bastardized functional programming :D
Has anyone ever tried to generalize range() beyond having three integer arguments? Like with step being an iterator? or else otherwise? (Yes I am searching SO as I write this, but not found anything useful yet)
@PM2Ring Oh also, when dealing with sets, it seems more intuitive at least mentally to deal with sets, than iterators/generators. It's also clearer on the page. But obviously less performant and scaleable for large or infinite ones. I suppose I just need to unlearn my assumptions.
 
3:40 AM
@smci Sounds like something fairly sophisticated for a typical user, I would expect there to be some functionality like this in numpy tho.
 
range(11, 30, itertools.cycle([2,2,4,2])) also fails with TypeError: 'itertools.cycle' object cannot be interpreted as an integer
@Dair Nope, I don't think so, I'm already using sympy.ntheory. If you clicked on the link I gave, you can see my sieving code on some prime puzzle from Puzzling.SE. It works, but it feels like the libraries are sorely missing some primitive iterators, before I start writing incomprehensible gloop in Python iterators which none of a general audience will understand.
@PM2Ring Ah ok, itertools.filterfalse([a,b].__contains__, range(10)). But man that is so ugly and unintelligible to non-Python users. And it puts the range second and the predicate first... and if the exclude-object is a set/list rather than a predicate function, it doesn't do the 'intuitive' thing and call __contains__ on it. And what if the if the exclude-object is an iterator...?
whereas smart 8-year-olds could intuit what range(start, stop, step, exclude) means. exclude could be an integer/set/list/iterator/predicate function/...
Anyone got any guidance?
Oh and btw since range() is always ordered (and almost always increasing order) (well when step is nonnegative), we could optimize .__contains__ using interval-search or some funky data-structure.
 
4:09 AM
@smci I definitely wouldn't do that as a one-liner, I'd write a proper generator.
@smci The set methods that accept a set as their arg are quite performant with any iterable, since they don't build a set from the arg, they just iterate over it directly, and most of the work is done at C speed.
@smci what if the if the exclude-object is an iterator...? Then you should build a set from it first.
 
@PM2Ring what if the if the exclude-object is an iterator...? No, you should just automatically call __contains__ on it, just like if it was a set. At least in my context.
 
FWIW, last year I wrote a recursive filter that generates primes. At each recursive call, the current filter chain gets passed in & a new filter chain is returned that sieves out multiples of the next prime. It works, but it's not easy to read, and it gets rather slow when there are more than 30 or so filters in the chain.
@smci How could that work? When you call __contains__ on an iterator, the iterator is consumed up to the point where the contained item is found. OTOH, you can do it on iterables designed to support that stuff, like range does.
 
@PM2Ring Thanks for the correction, I meant iterables. (and that works because they support reset, right?) Anyway I guess the point is that a range should be an iterable that behaves like an ordered-set, has O(logN) _contains_
 
@smci No, they just need to define a __contains__ that does the needful. Otherwise, 888888 in range(1000000) would be slow if it had to actually iterate over all the numbers up to 888888.
 
4:24 AM
@PM2Ring Ah, right. I'm trying to code up my suggested range2(start, stop, step_iter, exclude=None) as we speak...
... and yes it should implement _contains_
@PM2Ring Hey would you please post that on CodeReview.SE, so we can see its performance, intelligibility, code idiom etc. If each successive filter only tests for divisibility by prime p_n in a list of known-primes, isn't that just an underperforming way of testing for p in known_primes_so_far: if n % p == 0: # found a composite
 
Check out the infinite prime generators here: stackoverflow.com/questions/2211990/… They use dicts rather than filters, and are reasonably fast, considering. A proper sieve using a bytearray is at least twice as fast, but of course you have to specify an upper limit
@smci I won't post it on CodeReview because it's a stupid way of doing for p in known_primes_so_far: if n % p == 0: # found a composite. :) I only wrote it to see if I could make it work, and to get an idea of how slow it is. I never expected it to be useful, except as an exercise in filter abuse. ;)
If you want to see a fast prime sieve in plain Python, see chat.stackoverflow.com/transcript/6?m=43545047#43545047
 
4:42 AM
@PM2Ring Thanks for the awesome link. The code for erat2a isn't that somewhat similar to the ancient Python-2 dict recipe where D[q] stores the next-multiple of each prime? and hence has v nasty memory requirements.
 
And here's one that sieves over a range: chat.stackoverflow.com/transcript/6?m=43223618#43223618
@smci Yes. See the delayed sieve versions of Will Ness & Tim Peters versions, that reduce that overhead quite a bit.
 
@PM2Ring Ok but any sieve with O(NlogN), O(N^2) memory requirements or worse isn't scaleable
 
Those infinite prime generators do need to remember all the primes they generate, but the delayed sieves don't need to sieve on p until n > p^2
But really, they're just cute exercises. If you want speed, and a controlled RAM footprint, use a segmented sieve.
And if you really want speed, use C, or some other compiled language, not Python. :)
 
@PM2Ring Oh ok. Those algorithms could do with some comments, and explanations of what the big-O CPU and memory requirements are. I expected a (finite) sieve will store all primes known so far <= nmax. That's only O(logN) memory, which should be acceptable for finite ranges, well at least up to 32b.
@PM2Ring Ah great, thanks. Segmented sieve
 
5:00 AM
@smci It's easy to build a generator using my segmented range sieve that I linked before: chat.stackoverflow.com/transcript/6?m=43223618#43223618 I thought I posted the whole generator program here, but I didn't.
 
Related: what's a very memory-efficient way (also with fast lookup) to store a table of (say) all primes with <= 32 bits? Do we do something crude like a reduced bitmask of prime candidates for each residue modulo some primorial, e.g. p_7# = 2*3*5*7*11*13*17 = 510510?
 
A proper segmented sieve calls the range sieve in a loop.
@smci You can do that. But the memory savings diminish as you add more primes, and the encoding / decoding is quite fiddly. Using just 30 = 2*3*5 is adequate. Apart from the 1st block of 30, in each subsequent block there are only 8 residues mod 30 corresponding to primes: 1, 7, 11, 13, 17, 19, 23, 29. So each block of 30 corresponds to a byte.
I wrote C code to do that a few decades ago, originally on the Amiga, but it runs fine on Linux. I have all the primes upto 3 billion on my HD, in a set of files, each file contains 10 million bytes.
 
wim
@piRSquared the most polished one is probably this one
 
@wim Looks good. Unless you want to flatten a bytes string or bytearray into a sequence of individual bytes. ;)
OTOH, I can't think of a good reason to do that. :)
@smci The C code that accesses the primes files is pretty fast, but the bit twiddling does slow it down. For testing single numbers for primality, my Python Miller-Rabin code is faster, and covers a larger range. And to test a bunch of numbers in a narrowish range, my range sieve is almost as fast as the table lookup.
 
5:34 AM
cbg
 
@PM2Ring Obviously, but what if you did it hierarchically? Consider the first three primes 2,3,5. Level-two could tabulate all the possible residues of primes in the range (6,30), since p_3# = 2*3*5 = 30. There are only 7 hits on the range (6,30). Hence if level-three is going to cover the range 2*3*5+1, 2*3*5*7*11+1) where 2*3*5*7*11 = 2310, it only needs to tabulate possible residues which will not be covered by level-two, ...
... hence (7/30) * 2*3*5*(7*11 - 1) = 532 candidate residues for primes, of which only 40 are hits. And so on. A hierarchically-compressed table of residues, modulo some successive product of primes. I presume that's been implemented, is it competitive? (for efficiently storing a table of known primes, with fast lookup)
 
@smci Sounds messy, and I'm not sure I see how that'd work for an efficient table. But see en.wikipedia.org/wiki/Wheel_factorization
 
@PM2Ring Ooh thanks. You are a trove of knowledge. What's your background?
@PM2Ring (Yes, but the bit twiddling could be done on hardware like Xilinx FPGA or custom ASIC, at least for small ranges like 32b, 48b. I'm pretty sure that was well explored 20 years ago.)
 
FWIW, when doing prime factorization I often use a possible_primes generator to generate the trial factors. I normally just generate 2,3, and then numbers of the form 6n-1 and 6n+1. I've also tried doing it with 2,3,t, thence the residues mod 30, but the speed difference is marginal, and for factorizing lots of smallish numbers it actually works out slower than the mod 30 version.
@smci Oh, ok. If we're talking custom hardware, that's a different story.
@smci I've been an amateur number theorist since the 1970s, so I've accumulated a few bits of number trivia over the decades. :)
Gah. I've also tried doing it with 2,3,5, thence the residues mod 30
 
Funny, these days I answer questions on tags i don't really know, i don't really know regex, and yesterday i got a 5 up-votes and accepted answer on it, as well as beautifulsoup, i just answered one 2 or 3 hours ago that got accepted and up-votes super quickly.
 
5:48 AM
@PM2Ring the concept seems to be well-known: The Prime Spiral Sieve. That author (Gary Croft) tabulated modulo 90, period-24 digital root repetition cycles of the domain "allows us to sieve and sequence by digital root (dr)", as listed below (and documented by the author on the On-Line Encyclopedia of Integer Sequences in 2018).
 
@U9-Forward That can be a great way to learn stuff. There's a post on the starboard about that.
 
@PM2Ring I am happy with that as well, see stackoverflow.com/…, i got my most up-voted today, and with regex, stackoverflow.com/…, i got the best as well
2 days ago
 
@smci Sure, I've just never heard of using the wheel principle for building a table like you mentioned. OTOH, I'm not saying it can't be done, so please feel free to give it a try.
 
Damn, forgot the reply thingy.
 
Turns out he implemented it as pyprimes package back in 2012, for Python 2.5-3.2. Sad if it's become abandonware.
Wow it's fast. list( pyprimes.nprimes(10**5) ) is instant.
Hmm, it seems to slow down around 32b
 
6:01 AM
Nothing is perfect, lol :-)
 
pyprimes package is designed to speed test non-probabilistic prime number sieving algorithms (i.e., deterministic sieves that precisely account for all prime numbers within a given range). These include the Sieve of Eratosthenes, Wheel Factorization and the (Gary Croft's) Prime Spiral Sieve. So, good for toy recreational use...
 
Morning cbg.
 
which extension do you suggest for VS code? The default one is not coloring well
 
6:16 AM
@smci Big integer arithmetic slows things down.
 
6:33 AM
@PM2Ring I know that, I'm trying to say it already starts slowing down around 24b, which is well below 32b. But it's lightning fast below 16b. So, kind of a recreational toy for non-probabilistic primes. Sigh. But it proves that progressive wheel factorization is viable as an efficient exhaustive sieve for general non-probabilistic primes.You know, Ulam spirals for nerdy bathroom designs, and suchlike...
 
@Kevin No I still have a reference to it...
 
Meanwhile, taking blunt instruments to ambiguous tags:
1
Q: What on earth is [step] tag, it's not synonymous with [increment], should it be redefined?

smciThis question is what to do with the current incarnation of the tag step, see below. First off, the tag increment is good and useful, and it has 2.7K hits. (In some of those 'increment' is used as a verb, and sometimes as a noun. That's fine.) the term 'increment' is generally synonymous with i...

 
+1 for ya
Eeek -1
 
 
6:53 AM
lol
 
@U9-Forward Consider when increment or step are used as noun. range(11,4,-2) is perfectly well-defined, and in that case step = -2. So if we decided that "increment" was a synonym for "step", we would have to say "the increment is -2". I don't care,I just want to achieve harmonious convergence....
 
@smci exactly, lol
 
@U9-Forward Now wait until the first C/C++ person starts lecturing us about side-effects in macro expansion in imperative languages where standards don't define order-of-precedence, and how only an idiot trusts the compiler, and how undefined behavior such as "print out all the SSNs and CC numbers of everyone who's ever installed a trial license, email it to come_come_myprecious@darkweb, then start a nuclear countdown" is acceptable, according to the ISO standard...
 
@smci Haha, that's for sure a 100% that it won't be me, lol
I have no clue about C nor C++, only python :-)
 
What's a collective noun of C/C++ standards weenies?
 
7:00 AM
I don't know that
rbrb
I didn't think tho, even tho i won't get the right answer even after i think :-)
 
@U9-Forward Oh look, 2 Thessalonians 2:11-12 presaged the C/C++ standard: "Therefore God sends them a strong delusion, so that they may believe what is false, in order that all may be condemned who did not believe the truth but had pleasure in unrighteousness."
 
@smci BTW, although Miller-Rabin is often used probabilistically to test huge numbers for primality, it can also be used deterministically to prove primality of quite large numbers. You just have to supply a suitable set of witnesses. See en.wikipedia.org/wiki/… and miller-rabin.appspot.com
 
@PM2Ring Sure, I was aware. I think the recreational-math use-case is more "memory- and CPU-efficient tabulating/sieving of non-probabilistic prime candidates". Like I didn't mention my context, but much of the time I'm as interested in easily getting the prime-factorization of composites, as much as primality-testing prime candidates. Other people's use-cases will vary.
 
Hi Guys,
i'm deploying my django application in VM with gunicorn as WSGI server and nginx as proxy server. i need to serve my media files via Azure blob. I'm able to do when i'm running via python or gunicorn. But when i'm running with nginx. It's not working, the media files are saved in my local VM not in blob. Can anyone help me with this
 
7:16 AM
@SakthiPanneerselvam there's really not enough detail here to even begin to guess what might be wrong
 
Is there any "experts" on flask-socketIO here?
 
Question:"- Why we use Flask why not standard Angular or ASP .NET MVC". I was just asked this over mail. Angular is frontend right? .NET MVC sounds backend. Any reason why someone would want to use that over flask?
 
because it fits better into the ecosystem you're working in
 
if one section of the application must be in python, then i can definitely argue in flask's favour using the same point, yeah?
 
@ParitoshSingh describing any of those as "standard" is weird
but if indeed it is a local standard then deviating from that would seem like a problem, regardless of technical merits
 
7:20 AM
@ParitoshSingh sure
 
my understanding of angular is that there's no fight with integrating that. that's frontend, and should be as simple as putting a .js file and importing
 
in other words, if the CTO said "we use ASP here, dammit" then not using ASP is going to get you into trouble or at the very least require a solid rationale
 
I'm trying to read up on Miguels documentation on SicketIO, and he keeps talking about asyncronous services. And that if you want eventlet is the prefered choice if it is installed. What does he mean by installed?
 
ok perfect
 
@Markus it's a third-party add-on, you can't use it if it's not there, obviously
 
7:21 AM
@tripleee yep, this is the core point i think, but i cant really fight with that. good thing i can argue for python as necessary
and then adding flask on top seems easier
 
We have eventlet, and we even monky_patch the application... But I don't understand is this is needed or not
so if eventlet is in the virtual env, then it's "installed"?
 
Thanks for the sanity check arne, tripleee.
 
you're welcome
@Markus yes, but that's a strange way of essentially saying that it was pip installed
 
@Markus sounds like a reasonable interpretation ... are you sure this is current still, though?async is built in now, since Python 3.4
 
^ that too. eventlet are also infamous for not releasing 1.0.0
and making liberal use of the fact that they can introduce API changes without deprecation periods
so if you still have the choice of which async library to use, I'd go with builtin async if I was you
 
7:28 AM
Hmm, SocketIO doesn't mention this anywhere (that I can find)
 
I think one of the problems with Eventlet is that it requires changes which might not be acceptable but I'm hazy on the details
eventlet.net/doc/patching.html talks about monkeypatching the standard library
@Markus github.com/socketio/socket.io/issues/2638 seems vaguely relevant
 
7:44 AM
@PM2Ring Wow, I wasn't aware the the history of who first discovered prime spirals was disputed (Stanislaw Ulam, 1963) or controversial, but look at this guy's post
 
@smci that's some distinguished penmanship
 
8:16 AM
Doesn't sound like a crackpot at all
 
8:29 AM
@smci It's not controversial. Ulam was the first person to write about prime spirals, and they were made well known via Martin Gardner's Mathematical Games column in Scientific American. Prime wheels have been known for a long time, as Wikipedia mentions. Sure, you can make pretty spirals with them, but that doesn't actually speed up the process of generating primes.
@smci FWIW, I messed around with dual prime spirals based on 6n-1& 6n+1 back in the late 80s or early 90s. Around the same time, I also investigated the 8 way spiral based on the 8 prime residues mod 30, but I didn't think it was a big deal.
 
9:30 AM
Is there a use-case for just try/finally? I got picked up in the past for saying that you can't have a try without an except and, true enough, you can just have try/finally, but I can't think of a use
 
try:
    ... # do lots of stuff with an external API
finally:
    save_oauth_credentials()
kinda like a poor man's context manager
 
sounds like basically a usecase for except: pass.
 
@ParitoshSingh but it's the opposite. The exception is going to raise
 
oh
i see
in that case, probably only really used when you expect someone higher up the food chain catching errors
 
I think Aran-Fey's example is probably closest but it leaves me unsure of whether I should really listen to the criticism I got in the past; it's factually correct to caveat this construct in to the comment, but I really don't think it has any place in actual code. Leaving it out doesn't seem harmful.
 
9:40 AM
poor man's context manager sounds spot on really
which then raises the question (sorry for the pun), why not just use the context manager
 
Invariably the OP wouldn't know what that is, in the context of the questions that I'd make such a comment :)
 
why write a context manager when a try...finally does the job? :P
 
10:23 AM
Anyone else seeing issues with Github?
> There was an unexpected problem serving your request
> Please try again and contact us if the problem persists including *changing UUID string* in your message
From Firefox and from Chrome, including in private sessions. No indication on the github status board something is up. This for all URLs. I'll try a VPN.
oh goodie, UK VPN connection, Github is not responding at all now, got the angry Unicorn after a while.
Seems to be UK local, US VPN works, I can continue :-P
 
an "angry Unicorn" sounds something to avoid... I've seen what the one in "Cabin In The Woods" did - he was probably angry :p
 
10:49 AM
and then a unicorn joins... /me runs...
 
github loads from Hungary
 
Hungaru sounds a cool place :)
 
it's an imbossibru place
 
someone mentioned unicorns?
 
Speaking of unicorns, what are icorns?
 
10:57 AM
GitHub loads from Hong Kong FWIW
 
@Andras something to do with Apple when they launch a new wheat-like but gluten free product?
 
11:23 AM
wondering how we should deal with basically ML questions, where the accuracy is not good enough for the OP. On face value, there can be so many things that could be causing it, and even if the OP provides everything, it presents a significant time commitment for someone on SO to answer.
Should we just close such questions as too broad?
 
@Paritosh I think @Arne has been having a go at ML related stuff :p
 
oh? if so, he's one brave soul :P
 
@ParitoshSingh I would, but I'm not a domain expert in any way
 
@ParitoshSingh I vote to close most of those as too broad
 
these aren't programming problems
 
11:33 AM
some of them that ask more general questions can be migrated to cross validated, but they should have high quality
 
okay perfect, makes sense
 
It would be nice to have a canonical for the most popular frameworks, but I'm not deep enough into that side any more to be able to write it
 
@AndrasDeak yeah, actually this is spot on. It's usually more about trying different "knobs" and seeing the effect, and not so much a question of "how do i code this up"
 
mostly doing plumbing these days =/
 
hehe, data pipelines eh?
 
11:35 AM
spot on
turns out most business first need to properly digitalize all their data before it even makes sense to consider any fancy ML stuff
 
The ML teams have been getting away with this for quite some time
 
@roganjosh with doing plumbing, or with building shaky solutions on top of structures that can't possible hold them?
 
Or at least, the ones I've either seen or been a part of. Just playing and making fantastical models with no consideration of how it actually integrates into the company. But it sells well to management
 
yeah, same exact experience here
 
my experience is a bit different. no one actually does data science, but we're hired for it on name, and no one understands what to do.
then we get basically automation work, till we get tired, switch, and the same story repeats.
deep breath
 
11:41 AM
We have something similar even for informatics. A lot of people getting diplomas in what is more or less software engineering are offered jobs as sys admins, since "an informatics engineer is someone who knows how computers work"
and since there's demand for it universities teach sysadmin work to those engineers :D
 
It's not just a symptom in ML though. The problem is people coming along and just saying "this is what you need". The IT department where I'm at does the same; consults nobody, gets some garbage system and tells everyone that it's what they wanted
 
@ParitoshSingh Well, the possible range of what a data scientist actually is a big part of the problem. In job adverts they refer both to people with a dr. in deep learning as well as someone who is reasonably good with excel
 
On the ML side, you can make it look fancy. Still the same trope; it's not doing what the business actually needs. Then it's the failure of others to integrate it properly
Though, obviously, there are companies that are seriously good at it
 
Aye. i wish i could "navigate" this landmine somehow, and make sense of it. Its just tough to find out what a company really does from the outside, it seems.
 
Is there any winnings with using gunicorn in front of a Eventlet, SocketIO, Flask solution?
 
11:44 AM
@roganjosh So what you're saying that skills in integrating ML into business will be the most sought after qualification in the near future? 🤔
 
I suppose this is where knowing people pays off.
 
@Arne sure; you've positioned yourself well :)
 
:p
I hope
@Markus what do you want gunicorn to do?
 
Well, I want to use it as a webserver
or is it application server(?)
 
someone should tell the gunicorn people that their Quickstart icon features a cogwheel system that can't turn
 
11:46 AM
I have nginx as well
 
Are you sure you're not just picking technologies on the hope they solve some issue? That's quite a stack; what are you trying to solve?
 
what server to use is a choice really, but nothing wrong with picking either ways for the most part. your OS sometimes dictates the choice for you, other times its the existing stack. Sometimes performance is a factor, but that you can't really judge correctly unless you make a test suited to mimic your real app behaviour. The Eventlet and SocketIO stuff though, that is something i know nothing of. sounds something that one shouldn't normally need to interact with, too "low level" on face value.
 
Well, the actual reason that we "want" to use gunicorn is that Flask shouldn't be run as a production server... We also use Websockets, and that's why we are using SocketIO and Eventlet.
 
til socketIO is apparently a client server communication interface.
 
gunicorn will act better as a server, yes. Putting the SocketIO and Eventlet aside, Nginx + Gunicorn + Flask is pretty standard
 
11:52 AM
my question re websockets.. you said you already use them. what is the server responding on those sockets?
(disclaimer, im mentally translating websockets into "ajax-like" thing. this is a bit out of my comfort zone)
 
Well, actually the frontend uses the websocket more or less instead of a restAPI (this is not a design pattern that we would choose today, but that's the way it is...)
but it also provides live data
so in anyway we would need the websocket as well
 
So it is a roundabout AJAX implementation?
 
live data stored where? there must be a "script" sitting between the frontend, that fires the websocket, and the backend.
 
All, not sure 'how welcoming' we should be about Trying to call method on dict, getting AttributeError: 'dict' object attribute 'update' is read-only? Should we as typo/syntax error? or leave it open? It doesn't seem like a general mistake most new users would make, confusing assignment with a method call.
 
this script, is it oldschool cgi then? if not, there's probably a server involved.
 
11:55 AM
It can't be if Flask is already in the mix?
 
@smci closing or not is not a question of being welcoming
 
nono we have a Python backend. That backend produces data on a MQTT socket. Then we have the Flask webserver (which listens to the MQTT, and relays data to the web frontend)
(via Flask_SocketIO)
 
^ that brain flatulence (unlikely to help future readers)
 
okay. so, can basically ignore socketIO as some kind of client-server necessity, just like ajax really.
 
@smci you keep saying "syntax error", it doesn't mean what you think it means
 
11:59 AM
to me, sounds like yep, flask during dev is traditionally moved to a dedicated server
whatever server you choose, should be your call
 
so the "standard" would be nginx -> gunicorn -> flask. But we also need socketIO, and when using SocketIO and Websockets we need async workers. that's why we now have: nginx -> socketIO (using eventlet) -> Flask
and the only reason for trying to get gunicorn in the mix is because "you sholdn't use flask as a production web server". However I'm not sure if we are actually using Flask, or if we are using eventlet or socketio....
 
But in that setup, flask is basically forced to act as a server directly.
 
You're using the development server
 
^
one thing i personally don;t know though.. can nginx not host flask?
quick google search says its a server
 
So, in theory, you should be able to host the flask app on gunicorn. That would be better than the dev server. But, it's also going to be threaded of in different processes, so I don't know how that impacts your setup
 
12:03 PM
if it can, why add another server? should we not be encouraging just hosting flask on the nginx?
 
@ParitoshSingh it can serve static content but I'm not sure it can run Python and be an actual host. Maybe
 
According to this text from the SocketIO doc, it seems like I don't use the dev server (however I'm not sure)...
The simplest deployment strategy is to have eventlet or gevent installed, and start the web server by calling socketio.run(app) as shown in examples above. This will run the application on the eventlet or gevent web servers, whichever is installed.

Note that socketio.run(app) runs a production ready server when eventlet or gevent are installed. If neither of these are installed, then the application runs on Flask’s development web server, which is not appropriate for production use.
@roganjosh Thing with using Gunicorn and SocketIO is that I can only use eventlet (so only one thread anyways..)
 
If you only use gunicorn/eventlet to provide async to the webserver, you might consider rewriting the flask code into one that provdes async out of the box , with aiohttp being a popular one.
Given that you run into lots of problems with the async portion of your stack and you don't have tons of code you'd need to rewrite.
Hmm, scratch what I just said. You can't combine aiohttp with eventlet
 
"don't have tons of code"...
 
If the current state of affairs is simply propping up this code base, I don't suppose that using the Flask development server is the greatest of evils, if it transpires that you are actually using it
I don't know how best to connect all the pieces here, so I'm gonna apply something of an Occam's razor and say that there's only problems to be gained by adding yet more.
 
12:13 PM
hear hear
 
Is it ok to use threading for making DB query ?
Note: this is related to web backend development
 
Hi guys
 
Hello
 
I have question but this is more of a sql cum python question
 
Yeah ask
 
12:18 PM
I have a table known as 'addressid' . I want to do a fuzzy match using the module fuzzywuzzy
 
I am suing mongoDB though but I'll try
 
*column not table
 
@TheLittleNaruto So your threading question applies to Mongo?
 
@RaphX ....ok! I am out.!
@roganjosh Yes
 
I only have passing familiarity with Mongo so I won't be any help, but it's useful info for others
 
12:20 PM
Can it be done in such a way that I can compare that for a particular value of another column value?
if I use pandas as well?
 
Use pandas how?
 
Mainly I need to do a fuzzy match but since I don't have superuser priviledge I will have to export the data to python
 
Why would you use Panda for SQL ? I am sorry I am not able to relate. AFAIK pandas is for parsing csv files
Oh
 
This decision would be whether to filter the data prior to loading it from disk, or pulling the lot and filtering in pandas?
 
Yeah I can download the results in CSV or excel for that matter @TheLittleNaruto
 
12:22 PM
> AFAIK pandas is for parsing csv files
:'( <- every pandas developer
 
@AndrasDeak to be in safe side I added "AFAIK" :D
 
The data is there @roganjosh but I need to compare each address for a particular stop number
 
It. Just. Isn't. But let's not get into that
 
(using pandas as a csv reader is as useful as using numpy to take the sum of a list)
 
stop	address
1	aa
1	aab
1	aa
2	abc
3	abcd
 
12:24 PM
@RaphX I wasn't saying that the data wasn't there. What I'm saying is that you either want to do the filtering in the SQL query, or you just pull all the data into memory and want to do the matching in Pandas
 
@AndrasDeak Could you be able to answer my question that I asked above ?
 
@TheLittleNaruto nope
 
I don't think I can do that using Redshift @roganjosh
 
@AndrasDeak Thanks :-(
 
Do what?
 
12:25 PM
For the sample schema I provided above, say I want to match whether the addresses are similar or not
For example for stop 1, the addresses ARE similar
 
What does this have to do with my comments? I understand what you're trying to do perfectly well
 
What I meant was I don't think I can filter those cases using Redshift @roganjosh
The data will already be in that format if thats what you are asking @roganjosh
 
How big is the dataset?
 
could be around 30k - 50k rows
but that can be brought down if needed
 
Really, I think it's better to look at how to apply python functions in your query, if possible, but in the meantime you can easily just SELECT * FROM my_column and handle in pandas
 
12:32 PM
cabbage
 
I don't have permission to use python UDFs in my query @roganjosh
 
So that ditches the SQL issue because the data can easily be held in memory if it's only 50k lines.
Ok, so now the onus is on you to come up with an MCVE on how you want to do the string matching
 
stop	address
1	aa
1	aab
1	aa
2	abc
2	abcdefghijkl
3	abcd
3	abcd
I want to pull out cases like stop 2 which has addresses that vary a lot
 
What should the result be?
 
The stops and the corresponding addresses
In this case, 2 and abc, abcdefghijkl
 
12:39 PM
waves tiredly cabbage all
 
Why do you choose abcdefghijkl over abc for stop 2?
If I'm following correctly, you want to groupby the stop column, and have some sensible address left in address
 
No I am not choosing any address over any other , I just want to pull those stops where the address mismatch is way high
 
Right, got it
 
Its ok if I get the output like this:
stop address
2 abc
2 abcdefghijkl
 
So that's where the fuzzy matching comes in.
 
12:41 PM
yeah
 
12:57 PM
Um... just doing a small code review... only got 20 lines in and I've already gone from face-palming to head desking... gotta love stuff like:
def getLocationtaxonomy(loc):
    for l in locationTaxonomy:
        if l['name'] == loc:
            locationId = l['term_id']
            break
        else:
            locationId = 284
    return int(locationId)
 
huh
 
As you do
 

« first day (3125 days earlier)      last day (1824 days later) »