last day (16 days later) » 

21:41
Hi
hi
look at the difference between the summaries returned. the first one is from an article it hadn't summarized before, so the info wasn't in the HTML: https://pastebin.com/seRCGhMd
compare to this second article which it had summarized before: https://pastebin.com/uaj6hWxm
one second
Yeah, I see.
However, I notice that even in articles that haven't been summarized before, if I manually download the web page (i.e - navigate to the link in chrome and hit control-S), the HTML summary will be there.
Let me find an example
what you want is control-U
that will show you the raw HTML, not the rendered page
the initial HTML is what will mirror your initial Fetch response
Oh, I see. One second - I'll try that.
what I would do is I would search the response text for the string 'onmouseover="sm_flash_add('
if it does not exist, the page has not been summarized yet - set a timeout, retry after 5 seconds, do the same thing
(probably, don't retry more than 3 times or so)
21:50
Ok, thanks. I will try that.
does that sound like something that would work?
I'm not sure if it will work, but I think I can implement it. I'll try it and see what happens.
never used smmry before, but that seems to be one distinguishing factor
This is odd - I just control-U'd the link I sent you, and the summary is inside the HTML source: pastebin.com/WTzZcUB8
However, I waited maybe 10 or so seconds (after the article loaded) before hitting control-U
so maybe that changed something
I'm going to try again with another URL that has not been summarized before.
I tried it again with another article that I think was not summarized (the loading bar showed up when I loaded it into smmry.com) and the summary was inside the HTML source (I used control-U). view-source:smmry.com/https://www.newyorker.com/books/this-week-in-fiction/…
maybe try doing it through your extension and looking only at what it gives you
since that'll be closer to the real world behavior
browser download might be doing something funky
compared to fetch => plain html => download text file
21:57
yeah, I'll try doing it through the extension
maybe the browser gets a new copy of the page when you view the source
Huh - I just chose two articles at random, and both of them had the summary of the webpage, even when downloaded through my extension. I thought it would be unlikely that smmry.com had already had them in its database since both were pretty recent, but it's still possible.
I'm going to find an article that doesn't contain the html source of the summary, and then re-download it a few minutes later and see if does have the summary.
I'll repeat that a few different times with a few different articles, and see if I find a consistent pattern.
You might be right - this is only with one article tested - but I found an article, where when I downloaded it through my extension, the summary was not there.
But I came back to the article after a minute or so and re-downloaded it, the summary was there.
I'll try this with a few more articles and see what happens.
Also, I haven't mentioned it yet, but thanks a lot for your help so far. It's really helpful to get live feedback and advice.
22:12
yep np
So - I tried this with another article. I used my extension, but the downloaded source did not have the summary.
I waited a bit and tried again. The downloaded source still did not have the summary.
I waited some more, but the downloaded source still did not have the summary.
Then I manually went to the URL smmry.com/breitbart.com/2018-elections/2018/04/02/… (Yes, I know it's Breitbart, but it's just for testing purposes)
then, I re-downloaded the page
with my extension
(not through my browser)
and it worked.
It seems like in the cases where the downloaded source does not have the summary, I have to manually visit smmry.com/<insert_url>. And only after I do that will the extension download the page with the summary contained inside the html
perhaps it looks at your browser headers and sees that it's not an actual person who's navigated to the page
and thus does not proceed to summarize the article
you can try forging headers
That's possible. Do you know how I can forge headers?
Just a tutorial link would be helpful
look at the headers when you make an ordinary request to the page, in your browser
open up console, go to network tab
navigate to the URL
for example, in FF, you can go to "Headers" tab and look at "Request Headers"
then look at the network tab while doing a Fetch
and see how the headers are different
Ok. I'll try that.
By the way, do you think the problem/solution could be similar to this: stackoverflow.com/questions/27652543/…
(scroll down to Daniel Butler's post)
I was looking through the output of a fetch() request
with one of the attempts that did not return a summary
and I found the following code:
        <div style="position:relative;top:10px;left:0px;font-family:Verdana, Arial, Helvetica, sans-serif;font-size:17px;font-weight:bold;background-color:#EEFA11;width:600px;">PLEASE ENABLE JAVASCRIPT FOR THIS SITE TO WORK</div>
Maybe I need a "Javascript interpreter"?
while that could be a solution, getting it operational is way more complicated than you need
pretty sure "PLEASE ENABLE JAVASCRIPT FOR THIS SITE TO WORK" is on every page
fair enough - I'll try faking the headers first
forging headers via python is actually exactly what I was thinking of, I've never done it with JS (much) before
I wasn't sure how to view the headers of the fetch() request through the FF console, but I did find out my own user agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0
so, I'll try sticking that into my fetch request
22:49
Forging the headers didn't seem to work - but it's also possible that my code is terrible.
I'll upload my code, so you can take a look.
if you want too
I've never done anything with browser extensions myself
Hmm...
I'm going to test if the problem is a lack of a javascript interpreter
it looks like the summary response comes back via XHR initially
look at your network tab when going to a new article
for example, I see a GET to smmry.com/… (with proper referrer)
Ok, one second.
I wonder where the SM_TOKEN comes from
that's probably what you need
to trigger a summary
you could also consider just getting the summary directly smmry.com/partner
22:59
One second, I'm going to try my javascript interpreter test first
if you want to run the javascript on the page, might be easier just to open the page in an invisible iframe
Hmm... my javascript test was useless, it said "please enable javascript" whether I manually downloaded the page or got it through a fetch() request
I'm going to look at the summary response
Even though - I might also try opening the page through an invisible iframe
By the way, what do you think of the following workaround: Whenever the user clicks the link, the extension actually opens up a new tab for let's say... 1 to two seconds
then closes it
then performs the fetch() request
right now - the extension seems to work if a user manually visits the smmry.com/<insert-url> link beforehand. So this could work
if you can open the tab in the background, it's OK
but an iframe would be better
disturbs the user less
yeah, that could work too, open an invisible iframe with the page, so it fully loads with javascript/simulating the user
and then do fetch()
a more elegant solution would be to only use the iframe
instead of making two requests, one after another
23:07
I'm new to coding extensions, but I'll take a shot at doing that.
First, though, I'm going to try to simulate the request header under XHR as closely as possible
oh wait, the SM_TOKEN is visible on the page itself
that makes it really easy
how so?
why is it really easy?
if you're doing the fetch route, that is. fetch the smmry page, if the summary is populated, download it. if not, search the page response for &SM_TOKEN= and save the numbers found after that - then, fetch that URL, when the response comes back, fetch the smmry page again
like I said above, the xhr was to a URL like smmry.com/…
so just substitute in your own SM_TOKEN and the appropriate news link
Ok, so I found another failed fetch() request, and found the following code URL inside of the html: smmry.com/…
let me try typing out the correct URL
with the token
I tried on my own with the following URL: smmry.com/…
but it said the token had expired
try getting the token and sending it immediately
it might reject it if it takes more than a second or something
(make sure the token is only sent once. if you navigate to the page and the JS runs, the token might get sent - you don't want that)
23:22
oh yeah - that might be what happened
Hmm.. this is hard. The way I'm doing it right now, is I'm taking this token (from when I opened up console and used the website to get a summary by hand): smmry.com/…
and substituting the SM_Token and SM_URL values
but I'm still getting token_expired. It's possible I'll have to automate this with javascript in order to be fast enough
Ok - I'm going to try experiment with this some more. Unfortunately, I'm out of time today, so can I set up a chat room with you tomorrow? You've been really helpful and I appreciate someone taking the time out of their day to help a stranger.
Thanks a lot.
maybe, later

  last day (16 days later) »