« first day (3793 days earlier)      last day (1139 days later) » 

12:38 AM
yawns - but the DB server won't let me sleep yet... /me taps paws and wonders if it'd be unprofessional to just kill the server and hope no one notices for a few hours... cough
 
 
3 hours later…
3:16 AM
https://stackoverflow.com/q/22768976/1048539

which of these do folks here prefer for sharing config in pytest?

I'm not sure any of them are great...

We have some command line flags which people are using and also want to use them in parametrize, so global fixtures don't work great as you can't easily use a fixture in the parmatrize definitions
 
 
2 hours later…
5:40 AM
@Code-Apprentice :51717746 my two cents with trying to wrap my mind around numpy was this: always build logic for it ground-up. Existing python code will probably only get in the way if you try to use it as-is. This is because numpy doesn't make sense without vectorization. as for vectorization itself, it's like imagining a column/axis in your mind, and doing an operation that automatically works "at once"™ across the whole column.
So the trick there is, you have to really stop yourself from going for the easy "let me just loop it myself" option
 
 
2 hours later…
7:23 AM
Hey guys anybody have experience with requests_html
 
@Kwsswart you still looking at scraping stuff? :p
 
Haha yeah I have the project now but this one is scraping a website that requires executing js, I have it executing but trying to change the proxy in requests_html so as to not get blocked but having a bit of trouble changing it
 
dpaste.com/CKV3Y3QUV I have this so far but is contanty returning the origin as my ip instead of a different one the proxies() is dpaste.com/DCH7BFVRU when printing in the
terminal i get
Also have tried selenium but cant figure rotating proxies on there either
 
7:50 AM
umm... @Kwsswart you're diving in the deep-end here... my first opinion (and please don't take this the wrong way) was you were struggling with basic selection code... and now you're going into infrastructure mechanics and such?
and that's something you never need to worry about these days if you don't mind paying a few pennies to a company that you can send your GET/POST requests to and they deal with all the proxying/sessions/and re-trying requests/detecting captchas and all that for you
 
@JonClements I have found that when scraping a website like idealista.com you need to execute the js so I have found a way to execute it and select the items needed from it, but then I get hit with a captcha which I thought I could get around by changing proxy
i.e running this dpaste.com/FNLFD2382 handles the request and returns the expected links but after running a 3rd time I get blocked
 
if you're going to go the scraping route - start with simpler stuff first :)
but seriously though, look at scrapy
 
or just say, no scraping, bad people!
 
@JonClements will do
 
and if you need some javascript stuff (which sometimes you don't need if you look at the endpoints called per page - call those directly instead of having a pseudo-browser render it) - or use splash
@ParitoshSingh agreed... there's APIs for a reason :p
 
8:01 AM
hmm lol thing is when given for interview task to scrape something I may as well do it no?
 
@Kwsswart not necessarily :)
 
@Kwsswart yay free labour! - "some corporate guy, probably"
 
@JonClements I have sent a email requesting api access hope that will work lol
or else need to find a way to scrape it without getting blocked in the next 5 days
 
8:34 AM
Yes, I love design review meetings where the endings is you do what you want :D
 
9:13 AM
@tripleee done, need one more
"StackOverflow is not like US Social Services. We won't help you against your will."
 
Hey I added my model to models.py and sqlobject.py in Django I ran python manage.py makemigrations which created the correct migration then ran python manage.py migrate and it ran successfully, but I still don't see a new table in the sql db even after relogging. Is there something more that I need to do?
As I read the django migration docs that's it, the default db is also correct, but still no new table in the sql
I have:
# Generated by Django 3.1.7 on 2021-03-05 08:43

from django.db import migrations, models


class Migration(migrations.Migration):

    dependencies = [
        ('other_table', '0077_comment'),
    ]

    operations = [
        migrations.CreateModel(
            name='Config',
            fields=[
                ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
                ('timestamp', models.FloatField(db_index=True)),
                ('conf', models.TextField(null=True)),
which looks right to me
and I got.
Running migrations:
  Applying 0077_comment... OK
  Applying 0078_config... OK
 
@Hakaishin you're definitely look at the right DB? I've done that before...
 
Ah yes, I found it. Ok sqlmigrate is very useful for debugging. Our model prepends our appname to the db table, that's why I couldn't find it
which seems odd to me, because we have:
class sqlmeta:
    table = 'config'
so I would expect the table to be named that way
oh or is that even django?
4
Q: app name is appended to the table name in django

DilaniWhen I try to fetch data from the table, the app name is appended to the table name and displays an error. Following is my code. from models import open_cart class test(APIView): def get(self,request,format=None): values = open_cart.objects.get() My app name that I have defined in ...

yup, I see :)
interestingly it only does it if you don't do it. So a name like app_name_my_table will stay this way, but my_table will have app_name prepended. Is there a name for "smart" appending, I've seen it a few times and wonder if it's named
 
10:14 AM
hi guys
 
@NIKHILCHANDRAROY please don't ask for help here with fresh questions on the main site as per our rules
 
@tripleee they just edited to clarify
 
@tripleee seems clear enough
of course they rewrote the question
 
it's a very recent edit
yeah
 
now it's probably a duplicate instead but I have already spent my close vote
 
@tripleee I still have mine if you have a target
 
10:45 AM
wait, you can actually do this? ie, to me the question is clear but i didn't know you can just "add a command" by running a python script which then subsequently works
 
seems niche enough, not sure there's a dupe
 
@ParitoshSingh the equivalent of editing .bashrc surely works
@tripleee unixy question and ugh answers
 
11:45 AM
Added my own 2¢ on that question: tl;dr: virtualenvs let you implement cli commands.
 
11:58 AM
This really is doing my head in... got the scraper working but keep getting blocked
 
12:12 PM
might be an indication to wait for any api access then :)
 
12:27 PM
Can someone think of a way to make the pattern lambda x: (await somefunction(x) for _ in '_').__anext__() any less ugly? It's currently my best shot at an "async lambda".
 
presumably '_' is a placeholder for something?
 
it's just the shortest non-empty iterable I could think of.
 
12:56 PM
I am trying to share pytest config, which comes from a command line flag, with tests.

Currently, this is done via a autouse fixture and other than some janky conftest.py work its fine.

However, someone wants to use parametrize based on this config - which causes issues because you can't use a fixture outside a testcase.

Is there an approach here which is good? I've explored a few ways (setting a `pytest.runtime_config = {}` type of thing, which is the stopgap "recommended" by pytest as a replacement for their `pytest_namespace` method after its [deprecation](https://docs.pytest.org/en/la
 
@MisterMiyagi Isn't that entire expression equivalent to somefunction? __anext__ returns an awaitable, and so does somefunction. As far as I can tell, that code doesn't achieve anything
In any case, I don't quite get what your async lambda is supposed to be doing
 
@Aran-Fey erm, yes. Let's pretend I would have used a non-trivial expression like await x + 5.
 
1:22 PM
I want to make an async function call from my logging handler (emit event). This is by definition not an async function. How can I make this work?

class Handler(logging.Handler):
  def emit(self, record: logging.LogRecord):
    await myfunc(record) # obviously does not work
 
1:51 PM
do you call the logger from within a sync or async function?
 
I am just using print('my log message'). Is that stupid?
@MisterMiyagi now looking into 'aiologger'
 
@ChristophBühler no, but it's unrelated to logging and handlers.
 
I think the main thing about sync/async is forgetting that it pretty much has to be one or the other
 
I understand.
@MisterMiyagi From my understanding, I cannot create a logger based on the default logger and make an async call within 'emit', because 'emit' is synchronous. Is that assumption correct?
 
You cannot use await inside Handler.emit. That is slightly different from not making any async calls.
For example, you can use asyncio.run_coroutine_threadsafe if emit is in another thread than the event loop.
 
2:17 PM
@MisterMiyagi I feel like I am doing sth wrong. How would you solve logging in my case? I am using socket.io and want to send log messages to the client.
My frontend is written in JS if that matters
 
Can't really say how I would do it, since I don't know what "it" is.
logging can already send messages via UDP and TCP fine.
 
socket.io is a library for communication with the web socket protocol:
https://python-socketio.readthedocs.io/en/latest/server.html
Sorry, I am boring myself. Hope I don't annoy you too much :P
 
I just don't understand the practical problem. socket.io appears to have a synchronous API as well.
 
Yeah but I am using the asynchronous one
 
Why? (serious question)
 
2:29 PM
I thought maybe the program will block if there is only one thread being used?
Actually, I guess I will use the synchronous API, if I encounter such issues, I will just switch back
 
I'm not very familiar with socket.io, but network communication libraries often have non-blocking transfer methods, so it's certainly worth trying the synchronous api first
 
ok thanks a lot!
 
 
2 hours later…
4:19 PM
late morning cabbages, folks
 
@MisterMiyagi Alright. Well, if an async def is out of the question, I can't think of a nicer solution
 
4:40 PM
@MisterMiyagi I'm reading through your asyncstdlib docs, but can't find anything that converts a regular function to an async function (i.e. a function that returns an awaitable). I'm sure you have something like that, but is that not part of the public API?
 
@Aran-Fey I've pondered adding one. So far I couldn't decide how much magic it should have.
The internal one has the weird three-layer sprinkles on top.
Do you think having a bread-and-butter wrapper plus a magic-sprinkles wrapper would be too much?
 
I'm not sure what the difference between the two is :D But I don't think it's necessary to have multiple kinds of wrappers; just checking if the return value is awaitable should be enough
 
Magic sprinkles it is, then. :D
 
@ParitoshSingh yah, my forays into numpy often devolve into trying to loop myself. I still have a hard time getting my head around vectorization and broadcasting.
 
4:55 PM
If that's magic sprinkles, what's the bread-and-butter one? inspect.iscoroutinefunction? As far as I'm concerned, that's much more magical
 
Having a wrapper that only works on sync functions.
 
@Arne You mean you don't start at protons and electrons?
apparently throwing a big AWS instance at a celery cluster doesn't make it read 100k emails faster.
 
@MisterMiyagi Ah. That would be unexpected, I think. Most of the functions in your library are async neutral, so it'd be weird if that wrapper only worked with regular functions
Well, I suppose you could argue that it does "work" with async functions...
 
5:12 PM
@Code-Apprentice my only tip there is this. vectorization is a "fundamentally different" style of thinking about your problems. give it the opportunity to just "click". And here's something that will perhaps resonate with you: vectorization and broadcasting will feel like wasteful operations. The bottom line is: they are. and yet they will absolutely destroy traditional loops for number crunching
It's like an eye opener moment.
 
wasteful? how so?
anyone online that's familiar with poetry?
 
@Code-Apprentice if the poet in question is the great Neil Peart then sure :)
 
I just jumped on the room and found a conversation about vectorization!! I absolutely agree with @ParitoshSingh about vectorization. You start coming at problems differently because you need to think about how they will work in a vectorized implementation and it's am amazing feeling when it starts to click. Stick with it, it's totally worth it.
 
@Code-Apprentice say, an operation like finding a distance matrix for a list of values. you might write two loops, and have the inner loop skip checking the pairs that are already covered. but vectorization says: just dump it all, let it crunch the numbers
 
5:25 PM
I dabbled a little with numpy in my previous job. For one project, we should have used it and pandas much sooner than we did. My current job doesn't use numpy afaik. And I don't think it ever will.
 
or say, similarly, you're interested in items that are below 20 in some list for example
 
but I've only been here for a month, so I'm barely skimming the surface of this code base
 
traditional thinking says: iterate and only keep items below 20. vectorization says, construct a boolean for every value upfront
and then only subset at the end
things like that i suppose. just this idea that "dumping calculations against multiple values" is faster than trying to guard and filter things upfront
 
@ParitoshSingh interesting take on this. I've used vectorization to make a array of booleans before, but didn't really think about the wastefulness there.
 
yeah, it is a bit wasteful if you think about it, and these are just simple examples. vectorization can make you do a task in 4 steps with lots of booleans and masks that you would have thought to do in a single loop with a function
but therein lies one key truth: the way we are "pretending" that programs run line by line has..sort of..strayed far from how our modern hardware actually runs instructions
vectorization brings us closer to that notion in some sense. thats why it's fast
(i should put a disclaimer: all these are opinionated statements, just how vectorization made sense to me as i started picking it up)
 
5:34 PM
@ParitoshSingh you're right. The idea behind vectorization is that using vector operations on data already on the processor is faster than talking to memory to pull data in one at a time.
 
I like to think as vectorisation versus looping as moving one large thing versus many small things. Fiddling with many small things is often much more of a chore than brute forcing one large thing.
 
So even though you're operating on more things, that was never the bottleneck in the first place
Similar to how it's faster to copy one big file than many small ones
 
Doing things piece by piece is really only worth it when you are dealing with huge volumes. That's the domain of streaming, and frankly it seems most people just don't have to deal with that.
 
6:08 PM
@ParitoshSingh as long as you have memory
 
oh yeah, if you run into memory issues its a sad day for sure :P
 
Vectorization trades RAM for CPU time
 
Hi Team,
Wanting to see if Flask provides support for HTTP/2 protocol?
If so, how it can be enabled in my app?

I am hosting my app on a Apache server and I enabled httpd2 there, but still see requests are being served by http/1.1 protocol.
 
"Fiddling with many small things is often much more of a chore than brute forcing one large thing." kind of also applies to the GIL, this is how I remember the advantage of that :D
not that it is relevant to what is spoken here, but just throwing this out
 
It's actually pretty similar. The GIL does give a sizeable performance boost, so for the common cases of low concurrency it is just better.
 
6:28 PM
@Code-Apprentice I've used poetry. What do you need?
 
6:46 PM
@holdenweb do you know any good tools to generate pyproject.toml from either requirements.txt or from an existing virtualenv?
or is such a feature built into poetry?
 
6:57 PM
@Code-Apprentice gotta start with something that I understand myself
(implying I understand transistors)
 
@Code-Apprentice I doubt such a tool exists, because the dependencies are only a tiny part of all the metadata that goes into a pyproject.toml file. You have to fill out things like the author, the supported python versions, etc manually anyway, so is copy-pasting the dependencies from your requirements.txt into your pyproject.toml really worth automating?
 
I have the meta data part. The difficulty is that the requirements.txt has all dependencies, but I only need the top-level ones in pyproject.toml. So it's more than just a copy/paste if I want to keep my pyproject as minimal as I can
 
@Code-Apprentice No, sorry. I don't think they even use the same format for dependencies.
 
that, too
 
How many dependencies are you looking at?
 
7:03 PM
50ish top level ones it looks like
requirements.txt is 150+ lines
 
Sounds like what you really need is a tool that fixes a b0rked dependency list for you :P
 
that's why I'm going to poetry. It looks a lot easier to manage dependency versions along with a lock file to ensure reproducible builds
 
If you have the names of the top-level ones, you could script a poetry add command for each one, and let poetry work out the dependency resolution.
 
that has been the first task...trying to figure out which ones are top level
 
You could try running johnnydep on your project, it might organize your dependencies into a proper tree for you
 
7:06 PM
is that similar to pipdeptree?
 
When I search for my problem I always get SO questions like this where they suggest using get instead of filter. But this doesn't solve my question, I want to get a many-to-many relationship of each record in a queryset and that all combined in a single array or queryset, is that possible?
 
of course it's a wim project
 
@Code-Apprentice yeah, pretty much
 
what is everyone's opinion on monkey patching, say like attaching a .name property to a pandas df
 
with great power comes great responsibility.
3
 
7:16 PM
Avoid it unless you know what you're doing or there's no other option. So yeah, basically what Paritosh said
 
@Skyler as per a recentish discussion here: adding new attributes is traditionally not called monkeypatching
 
@AndrasDeak ok, so usually that's more reserved for if you add something like a method I'm guessing?
 
and pandas dataframes specifically support column access with attributes, so it might be an extra bad idea
@Skyler no, replacing existing functionality
 
So let's say there is Authors and Books tables, I have many-to-many relationship on those. And lets say I want to get all the books from authors that have the name "Jack", I try something like: Authors.objects.filter(name='Jack').books.all() but I get 'QuerySet' object has no attribute 'books'
 
@AndrasDeak if you put underscores in front of an attribute like that would that do name mangling or is that only when something was an initial member attribute/method
 
7:23 PM
Oh I should note that I am talking about Django
lol
 
@Silidrone The error is pretty clear. filter() returns a QuerySet which doesn't have a books attribute. I assume Authors has a books attribute from the M2M relationship, right?
 
@Code-Apprentice Yeah. I am not surprised as to why the error occurs, rather, I want to know how to accomplish what i initially tried but resulted in this error
do I have to use a loop?
 
the naive way is to iterate over the QuerySet, yes. There might be a better more django-ish way, though.
 
@Code-Apprentice Yeah I am looking for the django-ish way lol
 
Is there more than one Author with name=Jack?
 
7:26 PM
@Skyler Name mangling only happens if your code is inside a class definition
 
If you know that there is only one, then use get() instead of filter()
 
Yeah, actually, I gave the example with one name but in my case it is name__in=names where names is an array of author names
@Code-Apprentice Nope, sadly can't do that.
 
So, you want the set of books with a given author's name. So do the query from the books side of the relationship: Books.objects.filter(authors__name__in=names).
You can use double underscores to follow relationships. Basically authors__name is equivalent to authors.name.
 
Oh so I can use the m2m relationships while filtering too?
I'll try that now, thanks
 
yes
in general, you should start the query with the model class that you want results from...in this case Books, not Authors.
 
7:30 PM
But when you do authors__name__in, does it check for each author's name's existence in names array? I mean I expect authors__name to be an array of author names, not a single name. Can __in still be used there?
 
No, authors__name is not an array of names.
We are using keyword argument syntax here. filter() translates this into a SELECT statement.
 
Ah I see
it worked, thanks again
 
Specifically, the kwargs get turned into the WHERE clause of the SELECT. See docs.djangoproject.com/en/3.1/topics/db/queries/#field-lookups for the documentation describing this.
 
thanks, aran, andras
 
Rnj
7:41 PM
is there any way to reset my python pip packages to the clean state when my ubuntu shipped, that is remove all packages which I installed afterwards?
sorry for sudden question, but seems that am screwed :(
 
I smell XY. What's the actual problem? Why are you screwed?
 
Rnj
Asked this question explaining why am screwed askubuntu.com/questions/1321421/…
 
@Rnj sounds like youre actively taking the wrong first step, good thing the comments steered you in the right direction. dont remove the builtin python that comes on linux machines, it's usually there for dependencies
looks like this might help you
however, it also is quite possible you've already messed up the state of your system perhaps. in which case, why not just back up your data and do a clean install perhaps?
an extreme step perhaps, but if its an option, it's the best way to start over
 
Rnj
clean install ubuntu?
 
i want you to pause and "consider" the options available
as opposed to picking one because i said so
 
Rnj
7:51 PM
Yeah I guess I have to do some more experiments before doing any of above ... was thinking to start off clean if that was quick, but seems not the case...

may be I have to come back here for some other questions once I complete those experiments
 
before even that, why do you want to play around with it in the first place
what's wrong with "leaving it be" so to speak
 
Rnj
well thats what the word "experiments" in my above comment referred to
I was trying to run one notebook in windows pipenv, it gave one weird error after partially running. I tried to in several ways, but no luck.
So I thought I will try in wsl. But I was not able to install required dependencies themselves in wsl, some weird error.
So I switched to Ubuntu.
The notebook had one main dependency, which I installed, but this time, I thought not to install jupyterlab. Instead I exported notebook as py file and ran it and it worked. But graph didn't show up. So then I installed jupyterlab, but th
Even in ubuntu I tried inside pipenv.
But now was thinking to get back to clean state in Ubuntu. But seems that Ubuntu is fully screwed as I deleted those python folders.
 
so yeah, you briefly touched on it, but i'll state this in no unclear terms: do not use the default python installation to run things. it sounds like you did make virtual envs too, but yeah, exclusively use some kind of virtual env. that should hopefully keep your "experiments" self contained
 
Rnj
yeah ... its just that weird framework specific thing making me think whats going on ...
 
only one way to find out. start digging!
(well, after fixing your environments and stuff)
 
Rnj
8:01 PM
yess ... just found this comment: askubuntu.com/questions/173323/… 😂😶
I know its uncomparable, but in Windows, its simply uninstalling Python ...
 
windows does not have dependencies on python
 
Rnj
yeah ..
 
8:27 PM
@ParitoshSingh I don't think this is true? It's just a protected python version that it requires, no?
 
windows? zero python involved as far as i know, but im not sure if this will be easy to google
perhaps this article comes close
> It’s been widely known for many years that Windows is the only mainstream operating system that does not include a Python interpreter out of the box.
> While Python continues to remain completely independent from the operating system, every ...
 
Huh, maybe I stand corrected!
I could have sworn they had a "boxed-off" python installation. So you won't find it in the PATH or anything like that, but it was embedded internally. Indeed, it's hard to search but I'm tempted to take your word for it based on that quote
 
yeah im happy to be corrected if it's not the case. just not quite sure how to even begin searching for it
 
Every phrasing of my question keeps pulling up the same result first, so... maybe this one has to be laid to bed unless someone from MS happens to wander by :P
 
8:45 PM
Based on this blog I think you're probably correct. It looks like there's growing support in products but it suggests to me that it isn't used internally for Windows
 
Rnj
9:12 PM
yeah windows doesnt need python
had done several windows installations in past and installed python only when I had to develop something or needed some python tool
 
Is there an alternative to frozendict available on pypi that won't stop working with the next python version? >_>
Never mind, just discovered immutabledict
 
9:52 PM
Even oxfordenglishdict is not immutable.
 
unless it's the audiobook
no, wait, double negative
 
hey guys anybody familiar with this error:
 "Access to XMLHttpRequest at 'chrome-extension://jnhgnonknehpejjnehehllkliplmbmhn/images/icon16.png' from origin 'https://geo.captcha-delivery.com' has been blocked by CORS policy: Cross origin requests are only supported for protocol schemes:
http, data, chrome, chrome-untrusted, https.", source: geo.captcha-delivery.com/captcha/…
 
Do you know what CORS is?
 
@AndrasDeak I have use it for communication between react and flask not quite sure how to fix this within this python script
 
Does googling the error message not lead to any hints? Honest question; I don't know web stuff.
 
10:06 PM
I have googled it but doesn't lead to anything remotely related to selenium nor python tbh I have been googling for some time... may take the break now and pick it up again tomorrow.. been long day
 
I suspect selenium is irrelevant
what matters is that the captcha domain is trying to make a request at that chrome-extension:// address
Hmm, although the error says it's not supported.
people who know web stuff might be able to help (or tell you if you're doomed :P)
 
thanks mate lol
 
@Kwsswart if that means that this is not your website that's raising that error but rather you're trying to automate someone else's website, I also suspect that this is not irrelevant information in the least
I thought you were having issues configuring CORS for your own site
 
I suspect it's a legit error for a legit reason. It looks (at least to me) like you're scraping something that actively doesn't want to be scraped
 
"It's not what it looks like officer, I'm just here trying to solve this captcha with my script" :D
 
10:11 PM
Having not done scraping myself, my only impression is that you're requesting dynamically loaded content and it can see that you're doing it
 
@AndrasDeak hmmm no I am busy building a scraper and using selenium to execute the js within this file dpaste.com/9GFUEF4UU and being called here with this scraper dpaste.com/7G3UKYCT8 dpaste.com/4C92Z868N
lol as it happens its actually for practical interview project
been given the task of scraping that site
 
@Kwsswart did you say this was a legit company?
Make sure they're not asking you to do something illegal (in the sense of ToS violation)
 
got the single page scraping successfully but when automating or making the scraper go over many requests it goes dead
@AndrasDeak it is atleast its online and on linked in
 
You're not applying to a scraping factory, are you?
 
2 days ago, by Hakaishin
wow a week seems excessive. Just beware of scammers. Especially if it's for a scrapper, which is often used for scams
2 days ago, by Kwsswart
@Hakaishin seems to be a legit company they have been around for 25 years and essentially working with market research
bit more sus now
 
10:15 PM
I was actually impressed with your progress on the web-dev side from our convos over the last year or so. This seems a step-down from the skills you've developed :(
 
not a factory lol but they have a department that scrapes data for market research and then uses it for another part
 
@Kwsswart do check the ToS of what you're scraping
 
@roganjosh :/ never realized it was considered a bad thing
@AndrasDeak I will do what should i be on the look out above all?
 
if it's small enough fish it might not even think to forbid scraping explicitly, though
@Kwsswart anything that forbids automated retrieval of content
 
Will do... I never thought it was a bad thing to go into a company asking for web scraping... just trying to get into the working industry
 
10:18 PM
> 3. Using eBay

In connection with using or accessing our Services you agree to comply with this User Agreement, our policies, our terms, and all applicable laws, rules, and regulations, and you will not:
[...]
- use any robot, spider, scraper, data mining tools, data gathering and extraction tools, or other automated means to access our Services for any purpose, except with the prior express permission of eBay;
@Kwsswart like many things in life, scraping in itself is not unethical
 
@Kwsswart well, you'll surely know that you have to pay for instance sizing, and that size will depend on the request frequency. If you had a website you were paying for and had someone requesting every page they could, sequentially, just to scrape data to be a competitor and it either a) actively harmed the user experience of legit customers and b) meant you had to pay out more to handle the requests - would you be happy?
 
There are basic courtesies like respecting robots.txt, not burdening servers, using APIs instead where available, not trying to get around restrictions trying to prevent scraping. Things like that.
 
Makes sense.... I guess I will have to really think of the following steps... As I have this interview process I am not sure as to what is the best next step
 
In any case you'd be facing similar situations at the company, so if doing shady things is a dealbreaker then it's probably in your best interest to find out what's up.
 
I take a particularly dim view on scraping, so don't just take my opinion on this because I know what the industry does. My own opinion is that you could be creating new stuff with your skills, not trawling what other people have done
 
10:29 PM
Perhaos I should just curious as to how to actually ask
 
"Is this a scraping factory?" :P
(joking)
 
heh
 
I would like to roganjosh... just also want to get into the industry -,- difficult with no experience
 
Looking at the site's ToS might be the best start. If it forbids scraping that's an obvious place to start asking.
 
@Kwsswart disagree. Build a demo site
And maybe suck up the scraping in the short-term if you need that financially
 
10:31 PM
idealista.com/ayuda/articulos/legal-statement/?lang=en I have seems I need to maybe send an email to the guy just to check
 
@roganjosh scraping the bottom of the barrel...
> You may not damage third parties or idealista, break the law, use automatic mechanisms to copy or extract our content, create false contacts nor use the user access details of other people without their permission.
yyyyeah
 
yeah
@roganjosh I have been building up my portfolio on github and working towards it just difficult between working also as a teacher so applying for work at the moment
 
@Kwsswart but you've still got full-time employment and you're just looking for a career change?
 
Think I will message him in the morning just to check and yeah I ahve employment but wanting to get into the industry
 
What do you think "the industry" is?
Programming underpins pretty much every aspect of our lives these days
 
10:36 PM
I want to get into a job working with python and using it to build things... preferably using it to work between the relationship between frontend and backend and also perhaps starting to learn and understand machine learning
 
I think you would do better to take some more faith in yourself and start picking jobs that are aligned with what you want to do and your skillset. Personally, I think you have a decent shot at a BI (business intelligence) job
 
The I in BI gets translated into Hungarian as I in IQ (rather than I in CIA). Result is hilariously nonsense.
"I work in Business Smartness"
 
that's some pretty solid BS right there! ;P
 
tell me about it
to be honest my mind phases out at "business"
 
@AndrasDeak I think this is progress. Maybe the CV personal statement is just "I bring the smarts". Snappy and impactful
 
10:42 PM
same for me with "business", "smart", "advanced", "cutting-edge"
I've often debated whether my CV should read "I am the village idiot [that you want to proof against]"
 
@roganjosh Will definitely start applying along these lines... going to head off to bed and think about it all...
 
Blargh I just lost a beamer slide due to a dumb cp (dumb being me, not cp). Thank goodness for git.
 
phew! Sounds like you need to go git help :P
 
10:59 PM
What I need is to git gud
 
11:18 PM
You're all presumably aware that 'pip search' via XMLRPC is disabled and may be deprecated (and the third-party 'pip_search' is the workaround). reddit: PyPI XMLRPC search API has been disabled due to flood of requests. pip search may be deprecated.
 
yeah, "being deprecated" is a very polite way of putting "been broken for months"
 
wait...what's pip search? /me goes to read some docs
 
bit late to read that :P
 
yah, if it doesn't work currently
still looks like a useful tool
 
@Code-Apprentice When you want to find a package on pypi. (often, there can be name-confusion, multiplicity or the project and related package(s) names differ, or change over time).
 
11:24 PM
@smci I'm told there's a web search option on pypi.org that works. This is just about the command line.
 
I usually google for that info
didn't know there was a CLI for it
 
Which, granted, is mildly annoying because I prefer the command line, but still. We're not left without an official search option.
 
Related github and discussion: github.com/pypa/pip/issues/5216
@AndrasDeak Can you access pypi search via curl?
 
I can't access anything via curl.
 

« first day (3793 days earlier)      last day (1139 days later) »