« first day (3076 days earlier)      last day (1858 days later) » 
00:00 - 18:0018:00 - 00:00

12:00 AM
Yea, I guess you could say that. Anyways do you currently use that in your current projects?
 
For me, sometimes. Depends on who the end user of the project is. Most of the time, no, because typing numpy arrays is super annoying and 90% of what I do uses numpy arrays.
 
Ah, so if it's something easy to type like int or List[bool] then you do it, but if you have the same type over and over you just ditch it?
 
No, not because it's the same type over and over, but because there isn't a specific type for numpy arrays that is easily extendable to use wherever
Partly because numpy is a little stubborn and "array-like" things that work on numpy methods are not well-defined/under constant flux
 
Oh, ok.
 
For people here, from what I see, it varies widely, for e.g. many people aren't using a modern-enough Python to use them broadly. Why are you asking?
 
12:05 AM
there's ongoing discussion to pin down some protocol of "array-likeness"
 
@AlexanderReynolds I don't think it's about "modern-enough python". A lot of people merely dislke type hints
 
Yea for sure, I meant that as a "for e.g."
 
I kind of just decided to just do it since type hinting is more readable to me personally. It also feels more natural since it's exactly like Swift's static typing.
 
@NiNisanNijackle Well not many people are going to yell at you for using them :D
 
12:07 AM
Albeit most of the time I just let the compiler determine it.
 
Use what makes your workflow, documentation, etc work
 
@NiNisanNijackle compiler?
Determine what?
 
Swift's compiler.
 
oh, OK
I thought you were still talking about python
 
I am, just kind of veered off
 
 
1 hour later…
1:09 AM
Cbg
 
 
3 hours later…
Bin
3:48 AM
how to pause currently running loop
 
4:42 AM
Hi
date_cols = ['CLOSE', 'D_MONTH', 'OPEN']
gbl = globals()
for i in date_cols:
gbl[i] = pd.to_timedelta(min(data_frame_1[i]), unit='s') + pd.datetime(1960, 1, 1)
all the outputs are in pandas._libs.tslib.Timestamp
but i need an all these outputs in a single dataframe like below
Attr Min_Date
CLOSE 2009-04-30 00:00:00
OPEN 1993-02-12 00:00:00
D_MONTH 2018-01-01 00:00:00
is it possible
 
 
2 hours later…
7:06 AM
@Mohan you asked the same question yesterday, with broken indentation then too; you seem to have missed the feedback you got?
@Bin on Unix-like platforms you can pause any process with a signal
 
I don't know pandas, but surely creating a dataframe out of 3 values should be a trivial exercise? Am I missing something?
 
zxy
7:44 AM
hello,
 
8:02 AM
pandas can be shockingly hard to work with sometimes :P
i think its more so an issue of getting used to the way rows/columns are arranged internally though, more so than anything. We naturally tend to think of groupings of rows
but it helps to think of pandas as groupings of columns instead, and then things start to make some more sense intuitively
(in my experience/opinion atleast)
 
8:22 AM
cbg
 
Good day
For creating a web based inventory system what will be an easier tool to use to deploy an inventory system: Django or Flask
 
recbg
@Pherdindy I wouldn't use either, preferring Pyramid...
... otherwise it depends on what you want to do...
django's got lots of batteries but most of them are bad anyway :D
 
I do not plan to make mobile applications at all
 
you ask which one is easier to deploy
 
My need it just to be able to access the data input by several people in the company's systems
 
8:30 AM
both of them are relatively easy to deploy
 
Yes easier to deploy
 
the problem is writing the code where the differences show.
 
I asked a question regarding having a database over the internet because I use VBA in excel only to create applications for the company
 
@Pherdindy so you're like "which one is easier to eat, banana or apple"
 
But it's unsafe so I thought using dekstop GUIs like PyQt won't solve my problem
 
8:31 AM
and then I say "it is easier to peel bananas"
and then you're like "ok so I am going to grow bananas in Alaska"
 
I just know that Django is a full stack
And flask is bits of the whole
Lol the analogies
But I get you..
 
django is like PlayStation
 
Maybe my question is something like: "What fruit will give me more energy to run a marathon while not needing to eat so much of it"
 
and flask is like an ATX form factor desktop unit. Doesn't come with a pad
@Pherdindy yes, if you go buy it in a shop.
are you procuring a django or flask solution?
 
I am not sure which one is the one I need
I do not need so much features but the ability to use the databases
Because the data retrieving is my headache using offline methods
Since opening the mysql server is not safe to the internet
But if it was I would use VBA and the excel GUI
 
Sam
8:39 AM
When deducting a set of items of a list from another: set(lista) - set(listb) .. should this process retain order? If not, is there a way I can do so
Currently its not preserving the order but I'm unsure if it should be..
 
@Sam no, sets are completely unordered
even with Python 3.6 with ordered dictionaries, sets retain no order
 
Sam
How would one do something like this in an ordered fashion?
 
exclude = set(listb)
deducted = [val for val in lista if val not in exclude]
 
8:44 AM
set(lista) also eats duplicates, so you might want to avoid it
 
Sam
@Arne Perfect thanks
 
you're welcome =)
 
9:07 AM
Hello again
I know the question is a bit general but
As of now I am quite certain I need to learn
Django and MySQL at the minimum besides the python language to deploy what I plan to do
Do I need to learn Html and CSS also?
Also what is the order I should learn them at
 
9:19 AM
you need basic HTML to put something on a web page, though templates take care of the most of the work for you
CSS is only required if you want to apply styling to the HTML and is not useful without knowledge of HTML
 
do you even want a webpage, or just a web api you can code against?
 
@tripleee Thanks I think i'll re-brush on html and MySQL first before django
@Arne Web API meaning I can extract information and put information in the database without an interface?
I would appreciate if there is a way I can send data across several locations with just using the API and Excel
I'll have 10 excel files retrieving and placing data into the database
That's the idea
So it's a crude ERP system
Nevermind lol sorry my questions are too broad i'll just keep Googling
 
@Pherdindy "just a web api" means that the service that handles the database exposes a RESTful interface that you communicate with, as opposed to getting a webpage that you render in a browser.
 
Yeah a web API would be all that I need
So that means I can do the same what the web application in the browser does
In terms of data extraction
While using desktop applications like PyQT right?
 
erm, sure
 
9:27 AM
Can I use that for excel VBA also?
 
I I don't know half the words you're using, but there should be no limit to a web API as opposed to an application
 
I see thanks i'll look it up. What do you call it?
 
@Pherdindy I'm sorry, I'm not fluent in .net
 
Yeah it's not actually VB.net also
It's visual basic for applications it's a very limited language
But since i'm a 1 man team at my company it can deploy lots of information handling with very little code
 
@Pherdindy What I'm describing are restful APIs, if that was the question
 
9:30 AM
Thanks i'll read into this
Before I even start with django
 
Sam
@Pherdindy Note also Flask has less of a learning curve than Django. And is super easy to start building APIs
 
@Sam thanks perhaps to build the minimum that I need flash is a good idea
flask*
 
Sam
It's definitely less bulk if all you want is a simple web server
 
yeah probably as simple as possible
Even if the interface looks like crap
All about functionality for me lol
I was having no problem with VBA but when the data in one program needed data from the others it became troublesome I am trying to create a real time system
 
Sam
VBA makes me shiver
 
9:40 AM
it's so easy though
But so limited lol
 
Sam
Not easy when trying to build a real time system as you say :)
 
I found a workaround to it
You can connect google sheets to VBA
So it's like an online database lol
But I can imagine how a single mistake can screw that thing over
so anything typed in google sheets can update in the excel cells
I just never tried if there was any sync issues
 
 
2 hours later…
11:28 AM
Ew
 
Corned beef and cbg (a couple of days late)
 
Sam
@AndrasDeak lol
@Pherdindy Can you write a DLL in something like C#/C++ which is pulling from a db (assuming u want to stick with VBA to handle your processing....)
 
11:44 AM
@Mohan you've asked this the third time. First, second. Don't ask a fourth time.
 
sh.t
numpy does not understand aware dates :F
numpy.issubdtype(pandas.DataFrame([{'foo': datetime.datetime.now(tz=datetime.timezone.utc)}])['foo'].dtype, numpy.integer)
 
why would numpy understand pandas dtypes? :P
 
so I need to manually replace tzinfo :F
 
how on earth did this garbage get so popular?!
 
but frak if it is aware with non-utc timezone
 
11:55 AM
Why are you using numpy for datetimes? You don't do that. Numpy is for numbers.
 
@AndrasDeak I am using "pandas"
I am not using pandas.
I am just using pandas.
 
okay?
 
actually I am using pyarrow to write parquet
 
@Aran-Fey wow, no idea. Do people routinely mess up their jsons by hand?
 
It had a really misleading title that catapulted it to the top of the google results, which explains the millions of views. But the upvotes are a mystery
 
11:58 AM
well we know most voters on SO are chimpanzees sitting in front of typewriters
also look at revision 1
that...was not this question :P
@AnttiHaapala so use pandas
 
@AndrasDeak so how do i use pandas to check if the dtype is integer?!
I need to convert all integer columns in a dataframe into floats
 
that sounds wrong :P
>>> df = pd.DataFrame([{'foo': datetime.datetime.now(tz=datetime.timezone.utc), 'bar': 42, 'baz': 1.0}])

>>> df
   bar  baz                              foo
0   42  1.0 2019-03-19 12:03:53.235977+00:00

>>> [(col, issubclass(df[col].dtype.type, np.integer)) for col in df]
[('bar', True), ('baz', False), ('foo', False)]
dunno if foolproof but looks good at first glance
Why don't you like integers?
 
@AndrasDeak discussed already. Integers cannot have nans
the datatype must be correct for parquet as several partitions form a table...
 
@AnttiHaapala you can check if it's an int by shifting to the right by 0 and seeing if it equals the original value
 
@Rick slap.
 
12:09 PM
@AnttiHaapala ah
when you set an integer to nan it will promote the column to double though :P
 
@AnttiHaapala I deserved that
should have read more of the discussion
actually, I don't see why I'm wrong.
that was good advice.
 
12:27 PM
How is that good advice?
 
@AndrasDeak because he needs to distinguish between ints and floats without changing the float.
 
Why is raising an error more helpful than a direct test?
 
I'd prefer a more deepcopy-oriented target but we can add that once this is closed
dupes from here
hmm, those are worse than I remembered
 
cbg
@AnttiHaapala what about select_dtypes?
 
12:42 PM
neat
 
@AndrasDeak it seems like his floats encode ints as well. A direct test might not reveal that.
 
@Rick he wants ints to be floats. Existing floats are irrelevant. And they will raise an exception.
 
@Arne closed
 
thx
 
@miradulo aah interesting
 
1:02 PM
btw @Aran-Fey, I wrote an answer that is at least 50% frankensteined from comments you gave, I'm trying to clean up the python-dataclasses tag a little.
I hope it's fine with you to do stuff like that
 
Anything said in comments is free game
my philosophy is "if you didn't post it as an answer, it's your own fault if someone else does it for you"
 
cool, thx again =)
 
1:24 PM
@AndrasDeak Have we learned nothing from JavaScript's Number implementation? :| I like my 64-bit integer accuracy, thank you.
 
why are you telling me this? :P
 
"he wants ints to be floats" → ಥ_ಥ
 
@amcgregor it's actually a 64-bit float
 
So, 53-bit integer accuracy.
 
No, I met this problem from leetcode.com, so import new package is not allowed!! — gery 55 mins ago
 
1:29 PM
@amcgregor it's not really an int. but 32-bit operations can be thought of as in operations int js if you were so inclined
 
Practical example, here's a Twitter post ID:

> parseInt("10765432100123456789", 10)
10765432100123458000
Silly radix.
 
you need arbitrary precision a.k.a. strings
 
Or Python-style transparent bigint support, though these IDs are "just 64-bit". <3
{"id": 10765432100123456789, "id_str": "10765432100123456789", ...} makes me want to slap someone with 10,000 printouts of the MessagePack specification.
 
\o cbg
 
2:05 PM
@amcgregor it's funny how js and python have such a one dimensional perspective on ints. in this regard, they should both move in the direction of C++
 
That integers are simple scalar values of fixed minimum and maximum ranges based on signedness and bit depth? I'm all for rolling back to BCD everything, but… that'd be a lot of silicon to reissue. ;)
 
Sam
2:28 PM
What do all these threads represent in Pycharm? (if they are threads)
 
That looks like a stack trace of a single thread.
E.g. the main thread is running the top-most line there actively, as called by the next line, as called by the next line…
 
yup, recursion in the making
 
Sam
I have a recursive function but something very odd is happening
 
It started the recursion from line 42 of tree.py, in build(). Then it started recursing between two points in _traverse: being called again from line 89 and 95
I guess there are two conditional branches
 
Sam
There are 2 points in my function where it can call itself again.. is that bad design
 
2:33 PM
not at all as long as it makes sense
what's the "very odd" that is happening?
 
Sam
I'm scraping a load of web data in said recursive function... the point in which i want to break out of the recursive function is based on a conditional statement.. But, if i step over the code and observe the point in which i expect the function to return, it first jumps to lines 89 and 95 (like in the above list) and each time pops the item from that stack trace? Does that make sense?
 
you can't "break out" of the recursive function without walking back up the stack trace and returning from every call
(without raising an exception)
 
Sam
hmmm i dont think i understand. What if im returning another call?
 
In your screenshot the bottommost _traverse call is twiddling its thumbs until it gets a result from _traverse() it's calling. That call is twiddling its thumbs until it gets a result from the _traverse() call it made. And so on.
 
Sam
ah
that makes sense
 
2:39 PM
execution can only return to tree.py:42 by returning from top to bottom in the stack that's in your screenshot
 
Sam
2:51 PM
Yeah makes sense. Thanks
 
3:15 PM
cbg
Can anyone explain this to me: Mypy wants me to structure my if/else strangely
 
Are there no type annotations for _mystr?
I find it a bit surprising that it's complaining about assigning to _mystr
And is it not weird to have an str type function argument that defaults to non-str? I don't know type hinting best practices.
 
@toonarmycaptain You've declared the type of the variable to be str (string), but provide a default that is not a string.
 
_mystr would be str or None, I just put that together to make a MCVE.
 
I can also imagine that None is an exception
@toonarmycaptain I'm asking about the MCVE specifically. Whether that exact code produces that exact error.
 
@AndrasDeak Yep. I was putting together a @property, and it kept complaining, so I put that together and got the same thing.
@amcgregor It's the same response when I put Optional[str]=None instead.
[Or Union[str, None]=None for that matter.]
 
3:24 PM
@toonarmycaptain Huh. That's getting into the realm of the weird and wonderful. I'm not sure about mypy conventions, does it perform type inference and strict adherence checks? E.g. L3 would potentially declare the type of _mystr as being str, which L5 violates?
 
3:36 PM
@amcgregor Maybe? When I change to Any[str, None] = None I don't get an issue with the MCVE, but when I do Any[Path, str, None] = None in my @property, I do get a similar traceback, unless I do the if not mypath: form.
 
@toonarmycaptain if you annotate it that way the error disappears
def incorrect(mystr: str = ''):
    _mystr: Optional[str]
    if mystr:
        _mystr = mystr
    else:
        _mystr = None
I have no clue why the second function doesn't show the exact same behaviour though
I would guess that statically analyzing if-else blocks is tricky, and depending on how exactly they are written mypy infers different kinds of information
or it's a bug ¯\_(ツ)_/¯
 
@avatar_path.setter
def avatar_path(self, avatar_path: Union[Path, str] = None):
    if avatar_path:
        self._avatar = Path(avatar_path)
    else:
        self._avatar = None
 
I think the lesson you should take away from this is that if you want type annotation to behave sanely, declare a variable before you instantiate it. Especially if your instantiation includes ifs and trys
 
Hmm. self._avatar: Union[Path, None] before the if/else worked.
I might have found that sooner if I hadn't omitted the self. initially..
 
3:53 PM
Werkzeug 0.15.0 Released. I got the backlog down from 150+ in November to 35.
6
 
4:16 PM
Hot dang. Nice!
Good job @davidism :)
 
thanks :-)
 
Working on Salt I'm hugely empathetic about reducing massive backlogs
 
@davidism Nice. Do you want feedback on a typo in the release notes here or somewhere else?
 
submit a PR fixing it
you can usually do those directly from github
 
4:28 PM
@davidism Done. :)
 
Well, there's the beginning of the end of Trello's awesomesauce
 
where?
 
> Starting today, free Teams in Trello can have up to 10 open boards, and can upgrade to Business Class or Enterprise for unlimited boards. Existing free Teams with more than 10 open boards will continue to have access to those boards, but will need to upgrade to Business Class to add more boards to the Team.
I mean, I don't begrudge them making a dollar
 
ah, pushing towards paying customers?
 
But the main reason that I used them was because they were free, if you were fine with the totally basic features
that's the first time they've actually taken something away
 
4:33 PM
I see
 
I don't know what, if any, the impact will be - but to me that looks a heckin lot like Atlassian's influcence
Which is not a good sign.
 
corporation will corporate
 
Today I spent four hours writing code that invokes the darkyst magick to contort the program's environment in impossible ways in order to modify a piece of memory that really really didn't want to be modified.
Then I looked at my chthonic work, noticed I could do the same thing by writing a perfectly ordinary three line subclass, and threw away my sinful implementation.
Only by climbing the ninety degree cliff of jagged rocks could I reach the pinnaccle and notice that there was a chairlift on the other side of the ridge. And complimentary hot chocolate.
("You're beautiful", whispers the hot chocolate. It really is complimentary.)
 
4:55 PM
@Kevin It's good to know I'm not the only one who does this ;)
 
wim
chthonic (opposite of pythonic) nice one for vocab
 
5:11 PM
Variable scoping is tricky business.

def foo():
x = x * 2
return x

x = 42
foo()

UnboundLocalError: local variable 'x' referenced before assignment
^ that, to me, seems absolutely pathological. Removing the x = x * 2 and simply return x * 2 instead works fine.
Esp. pathological given it's the local assignment that's exploding.
 
wim
no surprises here
python is lexically scoped, x can't be both global and then local because the scopes are determined at function definition time
 
@Kevin Writing my own HTTP/1.1 server, I thought, hey, there's a bunch of these out there already, one of them must have some form of fantastic header name canonicalization process (e.g. WSGI requiring all-upper-case w/ underscores instead of hyphens). So I yoinked some C code from the most popular Ruby HTTP server. ~1000 lines of C generated from a Ruby spec. 17 lines of Python were ~100x more performant, and infinitely easier to grok.
 
wim
I suppose you're relatively new to Python?
 
Me? I started with Python 2.2/2.3. That HTTP server project was where my timeit suite came from; notice the many HTTP concerns referenced therein. "Over-optimization" was the name of that game. To the point of such great success, I needed to modify the Linux kernel to even get near stressing it. (To basically eliminate TCP CLOSE_AWAIT state… my benchmark would run out of assignable outbound port numbers in under two seconds.)
(Ultimately ended up with a process pool of async main threads communicating and handing off work units to a thread pool per process executing the WSGI application. It was a beast of pure Python networking.)
 
wim
I'm surprised for someone using Python so long to still be getting confused by things such as str.format, function scope, descriptors...
 
5:21 PM
Scope determination occasionally defies intuition. My favorite is that it doesn't care about the order of statements in a block. Even if a local assignment literally never executes, it marks all previous instances of that name as local. Example:
x = 1
def f():
    return x
    x = 0
f()
x = 0 will never ever execute, but it doesn't matter, you're getting UnboundLocalError anyway
 
wim
This does not defy intuition at all.
 
if only python could solve the halting problem
 
For the same reason, earlier versions of Python allowed you to put global statements at the bottom of your block.
Or in the middle or wherever you liked.
 
wim
Python is not magic. It's static scoping. Why do you expect magic.
 
Because it tries so hard to maintain the illusion that everything is dynamic everywhere
 
wim
5:23 PM
no it doesn't
 
@Kevin Similar to being able to import things at any location, "Your scientists were so preoccupied with whether or not they could… they didn't stop to think if they should."
 
@wim Yeah huh. Yeah huh times infinity.
 
wim
it even tells you the cell vars, free vars, in f.__code__.co_cellvars etc
straight up public api
 
@wim I refute your proof on the basis of I don't know what those things are
 
wim
it's the names determined after function is defined (locals, free vars, closure vars etc)
 
5:25 PM
How about this: it makes no attempt to hide these aspects of its public api, but it also makes no attempt to advertise its public api to users who don't have a reason to think a public api exists.
 
wim
they only need to read the datamodel
nothing in it suggests dynamic scoping
who even does dynamic scoping anyway, no modern languages that i'm aware of.
the only one I could think of is lisp
>>> x = 1
... def f():
...     return x
...     x = 0
...
>>> f.__code__.co_varnames
('x',)
a local variable because there's an assignment statement. that's as simple as you can get.
it's the same way a function is a generator just by having a yield statement (even if it's obviously unreachable)
it has to be that way, because the function object itself can't exist unless the names are determined. those are needed to create the function instance.
in a similar fashion, a generator can't just decide not to be a generator at call time. because they have to guarantee no code is executed until they are iterated.
 
Clueless users like myself form a naturalistic mental model of the language, and this model may not resemble any actual language. The argument of "it's silly to assume that Python uses dynamic scoping, because implementing dynamic scoping basically requires magic" doesn't do much to change my intuition. I don't know how static scoping is implemented so both static and dynamic scoping appear equally magical to me.
 
wim
well you know about the tokenizer and the parser right?
 
Yeah.
One possible solution to this dilemma is for me to actually learn how static scoping is implemented. Then it will cease to appear magical, and will suddenly appear much more sensible than dynamic scoping.
 
I like to think of it as a form of eager optimization. Variables ultimately represent slots in an in-memory structure. Pre-calculation of that structure makes sense to me; this is strangely similar to JavaScript, in fact.
 
wim
5:33 PM
astdump is pretty readable
>>> astdump.indented("def f():\n return x\n x = 0")
Module
  FunctionDef
    arguments
    Return
      Name
        Load
    Assign
      Name
        Store
      Num
see "Name Load "
 
KevinScript has closures and such but I wrote that part with my eyes closed so I'm no longer sure how it works
 
wim
and "Name Store"
Assign Name ==> oh we have a local var name. add it to the local var names.
you can't reasonably expect it to do some static analysis and know that there was a return statement before you got there. otherwise you would be obliged to do the same static analysis if the return was inside an if True: or an if False: or an if random.random() > 0.5:
 
Clueless (AliceScript ;) allows scopes to be constructed on demand, based on indentation changes, and when a scope is closed, any assignments within are effectively thrown away / rolled back. (Scopes "overlay" each-other in a similar way to MRO.) So the "pathological" case in Python I gave would resolve as one might naturally expect, e.g. it'd work. (x = x * 2 ← LOAD x, LOAD 2, MUL, STORE x. The first LOAD would pull from the outer scope, bare assignments are to local scope.)
And yup, that means that particular class of legitimate bug won't be preemptively caught, and certain structures of code might seem non-obvious.
 
What I'm hearing is "Static scoping systems must take possibly-unreachable code into account because otherwise it must solve the Halting Problem". Sounds reasonable to me.
 
+9001 (That's over nine thousand.)
 
5:42 PM
yay :D
 
cbg. i think Andras by this point you should probably realise youre too fast for the rest of us.
Just saying :P
 
You can watch every invocation during runtime and ultimately determine the reachability of everything within, but only with statistical probability given the input you've seen so far. Hard to truly guarantee dead code is actually unreachable except in trivial cases, such as if False. ;P
 
This is consistent with my original statement that return x; x = 0 raising an UnboundLocalError is surprising, because my naturalistic mental model of Python is not statically scoped.
Magically dynamically scoped Python does not need to solve the halting problem because it doesn't try to analyze anything before it actually executes. I guess magical dynamically scoped Python doesn't have generators at all, then, since it can't see that a function has a yield until it runs that line.
 
I have a DO server hosting my website that uses weather data to model crop growth. Currently I have a map layer that allows for selection of a regional location. When the location is selected a call is made to one of two weather APIs and site info is displayed. The site is under Django framework.
I am now going to set a chrontab to run a python script that makes a call the two APIs for all sites once a day and stores the data locally so I can begin the process of keeping my own record of historical data.
My question is this: should I be storing the data in a data base or in .csv files locally that are accessed by the view and provided to the javascript chart that I am using?
 
entirely up to you. How much data is it? What is your comfort? and so on.
If you want to ask what people would generally do though, to me databases make more sense if its not the "final output" so to speak.
 
5:48 PM
In ten years I will have ten years of data met data. I have read the a traditional database is not well suited for this sort of time series data
 
and in this case, you need to use the values in the chart. i'd use databases
Eh. it can. depends on the database and how it is stored. One thing i know for sure though...
attempting to solve problems that happen 10 years down the road might not be as productive as it may initially seem
 
If you're only adding one row a day, then in ten years you'll have ~3,650 rows. Even if it's not "optimally suited", a database should be able to handle that much data with lightning speed.
 
times 30 or 45 but that is a good point
 
And if you ever realise you genuinely have a problem, solve it then. Theres distributed databases and whatnot too, no point worrying about it at this stage. YAGNI
 
@ParitoshSingh The problem that I am trying to solve ten years down the road Is that I want to have ten years locally stored met data for between 30 to 45 cities in this region. But I get your point about getting too far out ahead
 
5:53 PM
At my workplace we store things in the db unless we have a compelling reason not to. For one thing, wrangling read/write permissions for subsections of the file system is more complicated than read/update/insert/delete permissions on the database. But that may be an artifact of our corporate environment that doesn't generalize to other workplaces. idk.
In any case I think it's fine to initially choose an approach that doesn't effortlessly scale up to a zillion times your current load. As long as the approach you choose can be migrated to a more scalable approach when it becomes appropriate.
 
Its good to keep those potential issues in mind, so that when you need to deal with it, or when you expect it to crop up, you are prepared. But this is definitely "too far out ahead" territory in terms of actual implementation. A lot of similar situations crop up far too often where you end up hurting the current development as well thinking about issues that may crop up, or features.
 
Like, don't use a database that makes it impossible to export data.
 
@Dodge I have 18 years of data: chat logs, "cached" HTML content via transparent proxy (which thus includes every video I've watched ;), a copy of StackExchange, a copy of Wikipedia, …, totalling just under 40 TiB, much of it highly (xz, H.265, etc.) compressed for deep archival of rarely-accessed material. Having structured data over such timespans would be like the glorious release of death compared to wrangling what I do have.
 
Good then. I'll use a db... and it shouldn't be too hard to write some python code that writes the met data I'm pulling to the same postgres db that my site is using, right?
 
@Kevin Precisely this, you phrased it much better than me ^^
 
@Dodge Yep, python integrates with databases like a charm in general. With django, theres some OEM? ORM? thingy? that you may want to look at. Full disclaimr, i have no idea what any of those acronyms mean, and i dont know django :P
 
wim
nice ISS live stream
 
Like currently Django helps my db interactions but if the db is getting filled by a chronjob and python script I'll need to write code to interact with the db directly which may be over my head
 
I always strongly advise and recommend use of SQLAlchemy, for a framework-agnostic bundle of DAO joy.
 
wim
OEM lol
that's like when you buy cheap unbranded computer parts
 
5:59 PM
@amcgregor will SQLAlchemy work with a postgres db?
 
@Dodge store the historical data in a DB. But have the clients make their own calls and have them store their own version locally. That can take away the overhead from having to do all that data management and distribution.
 
wim
of course
 
@wim So close yet DEFINITELY so far. haha
 
@Dodge Absolutely, yes. I've used it in concert with Postgres on many previous projects.
 
00:00 - 18:0018:00 - 00:00

« first day (3076 days earlier)      last day (1858 days later) »