« first day (3615 days earlier)      last day (1548 days later) » 
10:00 - 18:0018:00 - 23:00

18:00
@Kevin During the recent PEP622 discussions I raised exactly this point, and was surprised to hear Guido object that "too many toolsets rely on the colon" until I reflected on the huge compatibility gap that would open up.
@holdenweb How can that reasonably be? Anything that C code can optimise to can only produce assembly and thus only equal to someone handwriting that same assembly.
Because people are not always the best at optimizing code
Of course, development time/portability will surely differ, and handwriting such assembly would not be simple..
@PaulMcG Sure.. but that's not so far from saying python is better than handwritten C...which is certainly possible or even often true, but entirely dependent on the writer's skill. The statement was an absolute claim "is faster" rather than an often likelyhood.
@toonarmycaptain And yet an algorithm is the world champion Go player.
@toonarmycaptain If you want to pick that kind of nit, you are on your own.
18:18
All valid points. I'm thinking of theoretical proof rather than practical usage.
"I'll just buy a new game" 38GB. Woah.
So far I've not knocked out the pub TVs. This might be pushing the free wifi a bit :P
Go for it!
@toonarmycaptain Experiment: Method: write a small but significant Python algorithm. Run it (properly warmed up) and measure the performance. Measure how long it takes you to get as fast a solution in C (or other compiled language of your choice).
@holdenweb I'm certain that's the case for specific cases, but... yeah.
cbg folks
Most of all, Python is a pragmatist's language: people are generally more interested in Getting Sh*t Done than they are in programming. A generalisation, but good enough.
18:29
cbg
curious: does anyone know if grep can be used with tail -f?
cbg, @roganjosh - how's the chaos level today?
As in "can tail -f handle piped input?"
@holdenweb I didn't get the system set up actually, but thanks for the prompt. It appears that painful circumstance means I'm gonna be stuck in the pub for a few hours now. I suppose I could start on that again :)
I have a script that takes a while to run, but keeps dumping results in the output csv file. I'd normally do tail -f output.csv | cut -d ',' -f2 to monitor progress. The problem is that sometimes, the first column has quoted text that contains commas, that cut won't know about. So I'd like to grep the output for the col2 prefix (everything in col2 has the same prefix for... reasons)
@holdenweb more like "can grep handle piped input that hasn't terminated yet?"
Yes
....I'd think
easy enough to try by echoing to a file
18:36
Since tail's file argument is optional, it should be happy to handle piped input. grep will then process the tail output forever. Interactive performance may be different because pipe buffering.
@inspectorG4dget I often do this
tail -f /tmp/latest_test.log | grep "This code should never get called"
hmm... thanks. I think then, that I'm getting stuck on pipe buffering. Any ideas how I could get around that?
Are you tailing a log file that is logrotate'ing? Do tail -F instead of tail -f
negative
python mycode.py writes iteratively/continually to staticfile.csv. tail -f staticfile.csv | cut -d ',' -f2 | grep prefix stays blank for far too long, even if tail staticfile.csv | cut -d ',' -f2 shows the prefix
Is your prefix arg a regex? Needs grep -E
18:44
Or egrep, as we used to call it?
negative. The whole thing translates to tail -f staticfile.csv | cut -d ',' -f 2 | grep "20-06-"
Check grep with and without cut on a smaller representative example
Seems pretty cut and dried.
@holdenweb just in case you know off-hand; is the fact that I'm downloading that mega file going to affect my response time on localhost response (other than the obvious system process its launched)?
I too feel cut is the culprit here
18:46
Without tail -f I mean
I feel like localhost is independent of network traffic to other sites, but maybe it's a factor
@roganjosh giga file
> "I'd like to grep the output for the col2 prefix"

Which output?
He wants to filter column 2 with a prefix
@AndrasDeak apologies. Indeed, giga file. I bet my laptop can't even play it after this ordeal :/
18:48
@roganjosh Dunno. Downloading how?
ssh pipe from remote process?
Just through a regular wifi connection
@roganjosh colour of laptop? :P
Eh, we'll see anyway. I've got requests-futures working, now on to asyncio
ok. Update: tail -f staticfile.csv | grep prefix behaves as you guys expect (i.e. does not seem slowed down, which was the cause of me bringing this up in the first place). However, tail -f staticfile.csv | grep prefix | cut -d',' -f2 does slow it back down. So I have a feeling that the problem is with cut buffering things
18:51
@holdenweb output in this case would be what tail -f staticfile would put on stdout
@AndrasDeak I'm guessing that calls to localhost don't get affected by the fact that my network adapter is busy with downloading something because 127.0.0.1 is "ficticious"
That did it! Thanks @AndrasDeak
Separately, this is broken as as answer, right?
It's throwing the following for me:
runfile('D:/requester.py', wdir='D:')
Traceback (most recent call last):

  File "D:\requester.py", line 14, in <module>
    loop.run_until_complete(main())

  File "C:\Users\jpilk\Anaconda3\lib\asyncio\base_events.py", line 566, in run_until_complete
    self.run_forever()

  File "C:\Users\jpilk\Anaconda3\lib\asyncio\base_events.py", line 521, in run_forever
    raise RuntimeError('This event loop is already running')

RuntimeError: This event loop is already running
@roganjosh Correct. The network is real, but entirely implemented by passing buffers from process to process.
You can use cpu time using that, but it won't effect throughput on other networks otherwise.
19:29
@holdenweb Thank you. I've got bogged down in async again and I think that counts for something when it comes to the ease of requests-futures. But, it's only my word/mind
19:46
How cat i eliminate the for loop from this.
    for line in (line.split(":", 1) for line in msg_lines):
        k, v = line
        headers[k.strip()] = v.strip()
    print(headers)
You shouldn't. That needs more lines, not fewer.
If I may suggest using line.partition(':') instead of line.split(':', 1). partition has the benefit of always giving you 3 elements back, whereas split(char, 1) will give 0, 1, or 2 elements depending on the string being split.
headers = {k: v for k, v in (map(str.strip, line.partition(':')[::2]) for line in msg_lines)}
^^ untested
I'd prefer a loud error in the malformed input case
20:22
cbg
I'm using tabula to read tables from one pdf file while i convert number of pages into single dataframe but i got the output not aligned in same shape.
import tabula
import pandas as pd

tables = []
for page in range(540, 550):
    table = tabula.read_pdf("bawlo.pdf", pages=page)[0]
    tables.append(table)
    # print(table)

new = pd.concat(tables, ignore_index=True)

new.to_csv("out.csv")
I'm getting the following output
I'm guessing that pd.concat is looking at the first row of each table as a header row. Try copying the header row from the first table onto each following table.
20:38
am not sure how to do that ? is by appending to the list ?
Are there 7 or 8 tables in that PDF that you are trying to scrape/merge? That's about how many records appear to be tacked onto that header.
well the header is only within the first page which is 540 and the rest of pages is just the extend of the table record
Can you just concatenate a tabula.table, so that instead of a list of tables, you build up just one consolidated table? (I don't know the tabula table API)
i tried the following
import tabula
import pandas as pd


tables = tabula.read_pdf("data.pdf", pages='540-550')

new = pd.concat(tables, ignore_index=True)

new.to_csv("out.csv")
same issue.
it's all about the header of the first dataframe.
Again, and my theory is more and more likely, every table after the first one is misinterpreting the first data row as a header row. That's why you get more columns in the resulting headers, and why you see each page's table stairstep through the resulting table. Somehow you need to convince tabula to reuse the headers from the first table when reading the 2-n'th tables. I don't know how you do this. But that appears to be what is happening.
From the tabula docs:
pandas_options (dict, optional) –
Set pandas options.

Example

{'header': None}
pandas_options is an argument you can pass to tabula.read_pdf. Use this to read the tables on pages 541-550.
But I really don't use pandas or tabula, so these are just guesses and hints. You need to take it from here.
20:52
Thanks Paul, i will give it a try now.
Thank you so much. it's works
I have a question thats not exactly python related, but I dont know where else to ask. why are signed urls for s3/google cloud storage a thing? why aren't other means of authentication used?
e.g. today I was writing a flask server to limit upload size, and google has a special header x-goog-content-length-range. why is it this way?
Flask isn't a server :'(
As for the other aspects, I have no answer sorry
22:02
@holdenweb lol. I can write a decent Fibonacci in Python...I've never written any C, so...
10:00 - 18:0018:00 - 23:00

« first day (3615 days earlier)      last day (1548 days later) »