« first day (2666 days earlier)      last day (2285 days later) » 
00:00 - 18:0018:00 - 00:00

wim
12:25 AM
oh man .... pytest 3.3 to 3.4 just broke a billion of my unit tests related to logging
ok, what the heck changed
 
cbg. I thought that this was a chat room about pythons, not pangolin, but maybe we should dedicate the room to pangolin.
 
pytest.BREAK_FOR_WIM = True; new in 3.4
@Mr.Zeus a pangolin is just a python with legs
 
XD @AndrasDeak.
Wait, what is the winter 2018 general meeting.
Well, its that time of day, rhubarb!
 
rbrb @Mr.Zeus
 
> Incompatible change: after community feedback the logging functionality has undergone some changes. Please consult the logging documentation for details. (#3013)
rhubarb
 
12:40 AM
rbrb @AndrasDeak
or maybe not you, idk... how about rbrb for me for a bit
 
wim
yup, already fixed it
I think I understand the change, they were messing with the root logger defaults when they shouldn't have been
 
they don't respect semver?
 
1:08 AM
Would it work to create a question with a bunch of keywords that are references to a subject and hammer it as a dup of a good target? "Work" meaning that it might effectively capture search traffic and redirect it to said dup target?
 
wim
1:18 AM
@FlorianMargaine apparently not
 
Hey all, I don't have much knowledge about data science but I think it's a path I might want to take. Any suggestions on where to start?
 
that is a very broad question. depends on where you are at already. look into one of the degree programs.
 
2:01 AM
I guess it was broad, I just know some people have certain resources that really helped them or are well done. I didn't want to stumble upon something similar to Learn Python the Hard Way :P
 
user7437554
2:13 AM
environment is set, i think..
 
user7437554
many problems to install pandas anyway :(
 
user3736406
2:34 AM
In Django my app cant find my index.html template. I have it in my app and have the folder structure as: mysite/templates/mysite/index.html
 
user3736406
The view is
 
user3736406
def index(request):
    return render(request, "mysite/index.html")
 
user3736406
What else is there to be done here?
 
user3736406
Nvm
 
did you setup the urls?
to pass from the main urls file to the the mysite url file
 
user3736406
3:25 AM
Can anyone assist with this issue: stackoverflow.com/questions/48575174/…
 
user3736406
@HashRocketSyntax I never added my app to the settings in the app so it was looking for the template inside the project directory
 
6:29 AM
operating on 5 hours of sleep cabbage
 
6:42 AM
5 hours... sounds like a lot
 
In that case, there must be something wrong with your ears or your sleep schedule, my friend :p
 
You are correct... the problem has a name "Parenthood"
 
@piRSquared Amen
 
Oh. I see, that explains that. Do the kids make all the sacrifices worthwhile? :)
 
Yes (-: It's pretty awesome
 
6:55 AM
And cbg
 
cbg
 
7:20 AM
obviously an opinion question: Yes or No, should I use data classes? Rather, should I spend my time right now reading about them in order to determine if I need them?
 
yes. At least skimming over the pep is reasonable I think. It's well written and gives a clear idea in which cases dataclasses can be useful.
 
I made a bet with myself that you'd be the first to respond. I won (-:
 
I joined 2 minutes ago and didn't get to idle yet :P
also, asking for opinions always gets my ears perked up
 
You and Antti joined at the same time just as I hit "return"
 
well, it's central-europe-morning o'clock
 
7:28 AM
11:30 pm and I have a big decision to make. Do I subclass pandas Series/DataFrame for real this time. Subclassing is easy, I also want to subclass the index which I'm finding difficult to find examples. So data classes are my distraction
 
Hello all. I want to use pyhive/phys2 library. I installed pyhive, but when I use it in a program I get an Import error:No sasl module found. So i pip installed sasl. Then I am getting: sasl/saslwrapper.cpp(247) : fatal error C1083: Cannot open include file: 'stdint.h': No such file or directory
error: command 'C:\Users\Imdad\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\amd64\cl.exe' failed with exit status 2
 
@ImdadulChoudhury Didn't you post the same thing here yesterday? Why not make a post on the main site then?
 
The question got unnoticed yesterday I thought. So posted it again
 
people notice, it seems there is just noone here who can answer it
 
Ok thanks. Shall post it the main site then
 
7:35 AM
good luck =)
 
@piRSquared Subclassing pd.DataFrame sounds like you're going to have a bad time...
 
Nah, that part is pretty easy for me now. https://stackoverflow.com/a/47468205/2336654

I want to create a subclass of pandas.MultiIndex whose first level is a DatetimeIndex and whose second level is another subclass of pandas.Index that takes one of several types
I say those things, but I'm not certain I even know what I'm saying
 
This begs the question "why?" :P
 
I've built a financial research platform that leverages pandas/numpy/python. I started 4 years ago when I first started learning Python. I've since learned a ton and can't stand the sight of my old code. I'm starting the whole project over and incorporating everything I've learned as well as including new things I've yet to master. This includes, proper (better) use of git, documentation, sphinx, pytest.
At the moment, I'm tackling identifier conversion. I think the trick for me to handle both multiple securities at a single point in time as well as historical points in time is to utilize the MultiIndex. This is easy enough to use, but when my functions are expecting a pandas.Series with specific characteristic indices, how should I enforce that?
All this cool typing doesn't work on a Series. I have to actually create the class to enforce that the components have the correct parts.

Hence why.
 
7:56 AM
cbg
@Rawing same here
 
cbg @AndrasDeak
 
@piRSquared A matter of opinion, and having not seen the actual code etc., I'd just assert or some such and not worry too much about it. Garbage in, garbage out.
 
I'm still very much considering that option. And honestly, I'd bet that's where I'll end up.
 
Hello, everyone. I am having trouble with array (numpy). when I try to slice a multi dimensional array i got the following error. too many indices for array
example of my array
array([ ['Shreyas Gopal is sold to Rajasthan Royals for 20 lacs.\n#IPLAuction', '0'],
['@blogeswari Which channel ?', '1'],
['8. bhansali made a film that was an adaptation of the poem. He didnt make a film that is so literally a translatio https://t.co/UsIHwKZG5T', '0'],
...,
["You think you're brave? \n\nNo, stealing meat from lions is brave.\n\nhttps://t.co/yO4gCqoixu", '0'],
['@muladhara @anavrta If you listen to Univ debates in US, now, u might think wealth is a bad word. Seems like ppl ha https://t.co/t7795w9d7J', '0'],
 
If I try to reference a 2D array with 3 indices:
a[1, 0, 0]

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-174-0602218bc6a0> in <module>()
----> 1 a[1, 0, 0]

IndexError: too many indices for array
 
8:12 AM
i am trying to do this
arr1 = arr[:, 1]
to get the second element of all the array
 
I'll make one guess. I bet you have a one dimensional array of lists. You can confirm this if arr.ndim == 1. Fix this with np.array(arr.tolist())[:, 1]
Or np.array([t[1] for t in arr])
 
ya it shows as 1d array. thanks for this information.
I am reading a csv and make an array. and slicing it to two different array. this is what i am doing
 
cbg morning
 
csv_reader = csv.reader(self.tweets_file)
tweets_array = [row for row in csv_reader]
np_array = np.array(tweets_array)
array1 = np_array[:, 0]
array2 = np_array[:, 1]
i couldn't find where i did wrong ??
 
I haven't used csv.reader so I don't know for sure. But [row for row in csv_reader] will be a list of things that come out of csv_reader. I don't know what those are. What ever it is, numpy decided to make a 1D array of them instead of a 2D array. It could also be that each row is of different length. I believe that will mess numpy up too.
 
8:25 AM
Also, numpy isn't that great a tool for lists of lists of strings of varying length to begin with, I think.
 
uhh, there it is. The good 'ol falling asleep at the keyboard to signal that should go to bed. Sleepy rhubarb!
 
Thanks @piRSquared and @IljaEverilä. I ll check my code.
 
@piRSquared rbrb!
 
@Tamizharasan you could try making use of np.genfromtxt
But the earlier points about needing rectangular data still holds
And I don't know how well genfromtxt plays with strings (i.e. strings containing delimiters)
 
8:51 AM
@AndrasDeak thanks for the suggestion. when i did, i got the error as
got 2 columns instead of 1
@AndrasDeak I think I will restructure my csv and try.
 
@Tamizharasan that just means you were trying wrong
Use whatever works, it was just a suggestion
 
@AndrasDeak Actually the problem is with csv structure. When i restructure it and run. it works fine now. though i couldn't use the np.genfromtxt.
csv_reader = csv.reader(self.tweets_file)
tweets_array = [row for row in csv_reader]
np_array = np.array(tweets_array)
array1 = np_array[:, 0]
array2 = np_array[:, 1]
this thing works fine.
 
9:44 AM
What does this do? a: 1
 
How would I go about testing whether my program outputs nice error messages? Should I hard-code the expected error message into the test case? That's horribly annoying if I ever want to make changes to the error message. Should I use some kind of regex to check if the error message looks roughly as expected? Should I not test this at all?
 
It does not register an attribute or name a, but it also doesn't fail.
 
@ArneRecknagel It's a type hint. That said, I'm not sure where that hint is actually stored.
 
ahh. probably in the module's __annotations__ then. Checking.. yes, that's exactly what happens.
 
For some reason, type hints allow arbitrary expressions that aren't types (like 1)
 
9:52 AM
Maybe since they're annotations, not type hints, though they can be used as such?
 
That makes sense, but it also sounds like a messy and terrible implementation. What good is an annotation if you don't know what it means?
 
+ 1, Also makes scoping syntax, if it ever gets included, very messy: my_scope: {code} won't work
nobody else might mind, but I care A LOT about scoping :<
 
features that can be used in different ways but really should be used for exactly 1 specified thing tend to ruin programming languages
 
the Perl effect?
 
@ArneRecknagel I don't think that's going to be a problem. You wouldn't create a new scope for a single line of code anyway, so x: {code} is a type annotation and x:<newline><indented block> is a scope
I don't know perl. I was thinking of java, actually
 
10:02 AM
Ah, my day is saved and hope is alive again =)
I wrote Perl in one uni course. It wasn't that bad, but looking up code in documentation or asking for advice were useless. Which just fostered an "i'll figure out how to do this on my own" even further
What about java, specifically? Despite everything, it didn't feel too frayed to me.
 
Well, it's not exactly the same thing (i.e. it's not a feature that can be abused), but IMO java is a language where there are a lot of established "good practices" that are supposed to make up for the language's weaknesses. An example would be the getter/setter situation. Getters and setters are so deeply rooted that there is no longer a single scenario in which people would consider it acceptable to have attributes without a getter and setter
Even if you have a DTO class, whose sole purpose is to hold values, people still expect you to write getters and setters for that class. In the end you have unreadable code like obj.setX(obj.getX() + 1) instead of obj.x++ just because java gave people too much freedom with getters and setters. They should've just made it so that obj.x automatically calls the getter, if one exists.
That's one of the many things that were fixed by Kotlin, by the way. RIP java, long live Kotlin.
 
10:24 AM
It would be quite depressing for the java team to keep a list of languages that allow you to run java code without writing it
 
11:07 AM
Cabbage!
 
Good morning people,... I am new to Tensorflow, I am looking for a step by step tutorial that shows me how to implement a simple(like explaining why we import so and so libray, here we are creating nodes and stuff like that) MLP in Tensorflow
 
cbg
 
11:28 AM
Though a common typo, apparently...
 
How is that unclear? O.o
 
too-broad had strange categories that didn't apply. but asking "this code works fine, but could it work better?" without stating a tangible problem is a reason to close, right?
 
it is unclear
Is this code review? Are they asking about the algorithm? Code style? Anyway, if it's clear it's too broad.
 
11:43 AM
Meh. He explained what he wants to do, and he showed his attempt. Just because his attempt works doesn't mean it should be closed.
 
No, just because his question is off-topic is why it should be closed
> I created following script to clean text that I scraped. Particularly, I'm interested in feedback to the following code:
[...]
So far, the script does the job, which is great. However, how could the script above be improved, or be written cleaner?
that is not a question, not a narrowly scoped one anyway
if that really is OP's question, it should be posted to Code Review with due care
 
True, it does sound like "how do I improve this" is indeed his question now that he's edited it. But IMO the first revision sounded more like he was just using the code to explain his goal, and wouldn't have minded a complete rewrite of it.
 
too much code per text won't let you post
the obvious solution is to fill it up with gibberish
 
11:56 AM
Surprisingly decent question though, all things considered.
 
How so?
 
It almost has an mcve and stuff
 
it's as if The Dude from The Big Lebowski told you about this piece of code he found
fortunately he has the code printed in his hand...
it uses wonky formatting and pushed OP's garbage into the code block...
 
12:24 PM
cbg!
I want to declare Content-Type: "text/plain;charset=ISO-8859-1" in curl command and read it in python using flask requests. Can someone please give me an example? I'm putting my trial here.
curl command: curl -F file=@./data.txt -H "Content-Type: text/plain; charset=ISO-8859-1" URL
Now, how do I capture that this charset is requested in python?
When I print(request.headers['content-type']), i am able to get the Charset, but then python is not capturing request.files['file'] anymore!
 
 
2 hours later…
2:16 PM
cbg \o
 
cbg
 
2:53 PM
Anyone know of a good solution for Git on windows? I am trying to create something and want to keep a repository but dont want it public due to the source code having internal data.
 
I don't think git makes your code public on its own. Are you thinking of Github?
 
Morning cabbage
 
I am yes. Can you use git on windows? I thought that was linux only?
 
Or are you saying "how do I avoid putting passwords and things into source control?"? You could put those passwords in a file, and exclude that file in your .gitignore
 
DSM
Morning cabbage for the apprentice (and everyone else, too.)
 
2:57 PM
Git is indeed available for Windows.
 
@ZackTarr did you try goggle?
 
Well then...
 
/me googles "goggle"
;-)
 
Ouch -_- lol
Thanks guys!
 
Even with a private repo, you should avoid putting sensitive information in source control
 
3:01 PM
@DSM hows your productivity level now ?
 
cbg
 
Noted. Its not really sensitive data just yet. Just an IP of a DB. Ill be moving it to a separate file likely.
 
DSM
Yesterday I pushed through a lot of stuff, which was good. Today I have four hours of meetings (mercifully reduced from 4.5), and yet somehow I need to finish this report. We'll see how it goes.
 
I'm told it's best to adopt best practices early
 
@ZackTarr data like that which differs between development and production should be easily configurable. For example read it from an environment variable.
 
3:03 PM
(adoot sounds very Canadian)
 
@Code-Apprentice So like in a config file? Then keep the file outside of the repo?
 
@ZackTarr that works, too
 
On one hand, we have "do it right the first time". On the other hand, we have YAGNI. with no hands left for typing, the problem is effectively solved.
Can't write bad code if you can't write code.
 
I don't get it
How are those two competing concepts?
 
Completeness vs pre-emptiveness?
 
3:11 PM
did they just announce python 4?
 
Example: you're choosing a data storage approach for your application. You need to store user data for ten users. You could keep user data in a .json file, which takes five minutes of development; or you could install a database, which takes an hour. "Do it right the first time" mandates that you use a proper DB. YAGNI dictates that the .json file is fine.
 
@Neoares ha?
 
If you choose the DB, your boss chews you out for going over time for a simple application. If you choose the .json file, your boss chews you out in six months when the user base unexpectedly grows to ten million
 
I love how you can hover over a link to see things
 
we're usually way above flatulence-related jokes in this room
 
3:14 PM
Are we ^?
 
@MooingRawr you're the typical guy that clicks and then says "haha I knew it I haven't clicked"
@AndrasDeak sorry I come from JS room
 
@Neoares what you're doing is the online equivalent of "pull my finger" and I don't find that amusing at all
 
fart jokes are on point
 
Or he's the typical guy that hovers and says "I don't think I'll click, because the url looks weird". You don't have enough evidence to distinguish between them
 
I got rolled? I literally woke up less than 5 minutes ago. But I probably would've fallen for it even after coffee
 
3:18 PM
@Kevin I don't see how "Do it right the first time" mandates that you use a proper DB.
 
"Right" is whatever your manager says it is, and you discover that during code review when he says "Do you think Facebook, Google, and Amazon store their users in a plaintext file?"
 
No, google would clearly store it in a google sheet. ;-)
 
"Right" is too ambiguous that you can give it whatever meaning you want to fit the situation.
 
Hence all the flame wars that have ever occurred in any medium where two techies can talk to each other
 
Exactly!
 
3:23 PM
@Neoares believe it or not, no i don't click random links I don't know from random people in an IRC ...
 
Anyway I am off to work. Rbrb all
 
it's not a random link
it's a python 4 one
 
DSM
Sorry, are we twelve now?
 
On second thought, no hints ;-)
 
What do we use to close a question that has an answer so simple it doesn't justify posting an answer?
 
3:39 PM
the python hammer
 
Unpopular opinion: I don't think simplicity is a good reason to close a question.
Simple questions are often typos. Simple questions are often duplicates. These should be closed. But this does not mean all simple questions should be closed.
 
I'd have to fill an answer with fluff just to make the character requirement.
Fine, I gave a simple answer.
 
Is a simple one thats for sure.
 
You could link and/or paste part of the documentation into the answer
 
3:44 PM
I like to include a quote from the documentation if I want a little more substance for my borderline-too-short answer.
Oops beaten
 
I'm doing that now
 
Moderately irritating that the docs don't explicitly say that they're the same.
... Not saying that you could have done anything about that.
 
I'm digging in source to show that is the case
 
you're risking precious upvotes that way ;)
(just joking, in reference to the vast popularity of simple RTFM answers)
 
I'm still waiting for a simple RTFM answer for my typing.Generic question :I
I had a faint hope that I'd somehow missed the relevant info in the docs, but I guess that info might actually not exist
 
3:59 PM
You guys still use cabbage to say hello?
 
DSM
#tradition
 
haha nice
Is it normal I can't ctrl+C my way out of a program when progressbar is running lol
 
no, but programs might intercept keypresses
 
Well it does stop the program, but I don't get control of the terminal again
Or, actually it doesn't sotp. But it doesn't continue either
 
you can't type reset?
 
4:02 PM
I can't type anything
 
I believe you can type reset<enter> and it will work again
 
Lemme give that a shot
 
if not, then no, the program isn't exiting.
you can check with ps axf in another terminal
 
No it isn't exiting of that I'm sure
I cannot use reset
I'm sure it's my implementation, I'll keep fiddling
 
Are we talking about the progressbar module on pypi, or a progress bar function you wrote yourself, or what
 
4:10 PM
or a progress bar in another module, such as tqdm
 
Found it -> https://github.com/pandas-dev/pandas/blob/a00154dcfe5057cb3fd86653172e74b6893e337d/pandas/core/missing.py#L80-L81
That rabbit hole was pretty deep.
 
Yeah progressbar module
 
4:38 PM
Bad sign
C:\Users\Kevin\Desktop>pip install progressbar
Collecting progressbar
  Downloading progressbar-2.3.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
<snip>
Worse sign
Whoops the first one is for progressbar2 and the second one is for progressbar
 
Worser sign, You see [build|failing] and install anyway.
 
Ok, progressbar2 actually installs and works OK on my machine. I can ctrl-C and it exits properly, doesn't freeze or anything.
 
Yeah it works fine
Sorry lemme give some more context
 
@piRSquared "I don't want to live on this planet any more" sign: you see [build|failing] and install anyway, and it works anyway
 
I'm running concurrent http requests and using this to show context, and I'm printing output while I go
 
4:44 PM
Is threading involved? Threading laughs in the face of ctrl-C
 
@Kevin that's a bad sign alright. Is there a cult for that?
 
Most "let's abandon our bodies and ride a comet to heaven" cults tend to lose all their members pretty quickly.
Maybe excepting a couple adherents that stay behind to keep their website updated
 
Here's the function pastebin.com/yrVFMsN0
If anybody wants to take a look
 
Huh, never had to solve a captcha to view a pastebin before. Weird.
 
Bope, neither have I
 
4:50 PM
At a glance, the only problem I see is that you're not supposed to use the is and is not operators when comparing to booleans
 
Okay
 
can I use google's API just with REST instead of using their library?
 
if x is False:  #bad
if x == False:  #ok
if not x:       #best (if you don't mind the implicit boolean conversion which causes objects such as [] to be considered "falsey")
 
Thanks! I don't often code in Python
 
yeah, actually most IDEs tell you to go the "not" way
 
4:52 PM
Style nitpicking aside I wouldn't expect that to make a ctrl-C signal hang
 
Oh, you know what
I wonder if it's because I'm using Cmder in Windows
 
My guess is that any time you use anything involving parallel computing, (threading, mulitprocessing, concurrent.futures), then ctrl-C will successfully terminate only the process/thread/whatever that's actually running at that time, leaving all the other process/threads/whatevers to continue running
 
I was thinking that it doesn't give progressbar the opportunity to release stdout
That I should maybe deliberately handle the KeybordInterrupt exception to release it ?
 
Try it if you want.
 
Didn't work lol
 
4:59 PM
I was about to suggest handling KeyboardInterrupt myself, but in a different fashion: every thread gets an except KeyboardInterrupt: clause that puts a "time to die" message in a Queue (or whatever inter-process communication object is available to you). The main thread periodically checks the queue and kills all its children if it sees that message.
Or, if the parallel computing module you're using doesn't have a convenient child-killing method, then every thread needs to periodically check the queue and kill themselves
 
shrug
I'll fiddle with other progress bars
 
Ok, but my expectation is that the problem will continue as long as you're using concurrent.futures
 
Oh
So I should try another method of running concurrent http requests
Or is this something that I'll have to deal with regardless, you think?
 
Clarification: my expectation is that the problem will continue as long as you try to do anything involving the word "concurrent"
 
Gotcha
I'm okay with that
I have to peruse 5500 domains so I'd rather run them in parallel and have to restart my term every time I see a problem midway than run them syncronously
 
5:06 PM
19
Q: How do you kill Futures once they have started?

Nick ChammasI am using the new concurrent.futures module (which also has a Python 2 backport) to do some simple multithreaded I/O. I am having trouble understanding how to cleanly kill tasks started using this module. Check out the following Python 2/3 script, which reproduces the behavior I'm seeing: #!/u...

> The typical workaround is to have some global state, that each thread can check to determine if they should do more work or not.
Nailed it B-)
Looking a little more into it, for thread-based implementations only the main thread needs to catch KeyboardInterrupt, because only the main thread sees ctrl-C signals.
 
@davidism: you may see notice a few more rep points..
 
Cbg
 
Hi guys, I have a quick question and hope you don't mind me leeching a bit :)
In javascript I can insert a value at a specific array-index. If the array had no values in it all preceding entries will be filled with undefined.
 
5:21 PM
If you're going to ask if python lists work the same way, they don't. But it wouldn't be super hard to write a function that does it for you.
 
let bla = []
bla[1] = 73
console.log(bla)
-> [undefined, 73]
So this is not possible in python oob? Or maybe numpy?
 
why do you want that? :P
there are probably better patterns for that in python
 
I have no experience with numpy but it wouldn't be so hard to do it. But Andras is right, what's the motivation?
 
you can allocate empty arrays in numpy, but those don't stretch for you, they have fixed size
 
Can I interest you in a defaultdict?
>>> from collections import defaultdict
>>> bla = defaultdict(lambda:None)
>>> bla[1] = 73
>>> print(bla[0])
None
>>> print(bla[1])
73
 
5:22 PM
or...just a dict
 
If you're willing to use .get in lieu of brackety indexing, yes
 
I've been called brackety before, I think
 
they didn't talk about item retrieval, FWIW
 
DSM
+1 for saying "in lieu", -1 for Kevinning me, so +0, I guess.
 
better than -0
 
5:24 PM
@Kevin this seems to do the job, ty
 
wim
@piRSquared we obviously can't answer if you should use them, but time spent learning is never a waste
 
Alternately,
class ExtendingList(list):
    def __setitem__(self, idx, val):
        if idx > len(self)-1:
            self.extend([None]*(idx - len(self) + 1))
        list.__setitem__(self, idx, val)
    def __getitem__(self, idx):
        if idx >= len(self):
            return None
        return list.__getitem__(self, idx)

bla = ExtendingList()
bla[1] = 72
print(bla)
#result: [None, 72]
print(bla[100])
#result: None
But ultimately I think trying to replicate the behavior of a type from another language will only hold you back in learning the idioms and customs of this language
 
@wim of course (-: and thx.

Today's installment of "Arguments of a 3 Year Old"

3yr Old: Mom, can I have syrup on my pancakes?
Mom: No. It's pure sugar.
3yr Old: But if I put it on my strawberries it has fiber.
 
:D
so switch to maple syrup, that's 100% Canada
 
5:40 PM
Not sure where the FDA is in deciding if Canada is healthy.
 
Hi guys. I was wondering what the most efficient way to run 10 different while True functions that take about 5% CPU each?
I tried running each on 10 separate AWS EC2 instances but that just got hard to manage.
 
multiprocessing would take advantage of however many cores you've got.
 
I also tried to do multithreading but I want all of the functions to run in parallel and not waiting for each one to go idle then start up
 
then what Kevin said
 
Threading does not take advantage of however many cores you've got.
 
5:45 PM
unless you just run 10 different independent programs in parallel
 
@Kevin so would you suggest like to get a higher number of cores and run all on a single AWS EC2 instance?
 
that's not what he suggested
 
Isn't AWS a cloud thingy? In which case it doesn't matter how many cores you have, but how many cores the cloud has.
... Or is that what you're saying.
 
@AndrasDeak Ya I do not think I want to run each script just at the same time like "python3 script.py | python3 script2.py | ..." etc
@Kevin yes, it is the cloud service
 
From my use of the highly technical term "cloud thingy" you can probably guess how useful my recommendations for non-local solutions are
 
5:48 PM
I was also thinking about creating a cluster computer out of raspberry pis but part of me is saying that it would not work to my intend purpose
@Kevin LOL I can see
what is your local suggestion?
here are the cloud options if you want to take a look: aws.amazon.com/ec2/pricing/on-demand
 
@JulianRachman running 10 instances, each with 5% CPU load seems terribly wasteful. The suggestion was to use multi_processing_ not multi_threading_
 
also something tells me that a while True loop with 5% CPU load would work fine with threading
unless the fair distribution of clock cycles wasn't a given for 10 threads
 
@roganjosh @AndrasDeak ok could you point me in the direction on what to particularly look at in multi_processing?
 
@JulianRachman launching individual processes. You haven't really been very specific in what they are doing
 
^
with the amount of detail you've given us, "something in the multiprocessing module"
 
5:59 PM
@JulianRachman by from the instance angle. In my last job we had 350 ish stores that we had to run an optimisation system for. I asked the question of why they couldn't just get an instance for each store.... my god, it's $0.2 per hour each! Well, when you work it out, that would be over half a million dollars each. The pricing of instances is a great illusion.
 
each one of the while True functions checks to see if a bitcoin transaction has been made from a particular exchange site. if so the transaction is recorded in a csv
 
each year*
 
00:00 - 18:0018:00 - 00:00

« first day (2666 days earlier)      last day (2285 days later) »