last day (51 days later) » 

7:55 PM
hello friends
 
hi! :)
are you familiar with perl?
 
no
sorry
 
@MooingDuck what look cool?
 
@user1690130 click the orange words
 
8:00 PM
i'm confused
 
8:21 PM
@MooingDuck i want the number of results. 1975
 
you can assign a regex result in array context and directly get the captures. I never use $1 like that
 
@user1690130 the code in the question doesn't appear to even be looking for the number of results, it appears to be attempting to parse individual results.
 
my @captures = $input =~ /regex/;
 
@doug65536 so delete $val = $1; ?
@doug65536 my @captures = $input =~ /regex/; put that in plce of $val = $1 ?
 
not saying thats the bug, just sayin I wouldnt do that. @captures would be an array so the $1 would refer to the @captures - meaning do $val = $captures[0] etc
 
8:24 PM
@user1690130 print "'$location',$year,$gender,$val\n"; does not look like it's trying to print the number 1975
 
@doug65536 are you saying to put my @captures = $input =~ /regex/ in place of $val = $1 ?
@MooingDuck $val\n" is supposed to be 1975
 
@user1690130 Everywhere.
 
@user1690130 mention that in the question would you?
 
@user1690130 don't let the script go full speed, their server probably isn't that dumb.
 
@doug65536 thanks. i just want it to work for now before remembering how to add the tmer
 
8:32 PM
can you give a line of the input file locations26.txt?
 
@doug65536 San Diego
 
@user1690130 I'm getting '1',1923,Male,unknown what should I be expecting?
 
@doug65536 You hould get: Sn Diego, 1923, Male, 1,975 (or 1975)
Currenntly, I get: Sn Diego, 1923, Male, unknown
 
@user1690130 are you mixing up $u and $location? - bottom part should say $u not $location
 
@doug65536 The code works for a different webite.
 
8:43 PM
oh I see nevermind
 
It issomething bout this webiste in particular
@doug65536 thnk you for your help!! :)
 
those % might be interpreted as hashrefs
but strict should have fired right?
 
@doug65536 sory, what?
 
it looks right. A %variable is a "hashref" representing a handle to all data that has that name
@user1690130 replacing chomp with $location =~ s/^(.*?)\s*$/\1/s; helped a lot in my tests
oops, edited my previous buggy chat msg
 
@doug65536 what do ou men exctly?
 
8:56 PM
@user1690130 there might be a typo in that message
 
@user1690130 I got it generating valid URLs, the pattern "of .strong" etc doesn't occur
@user1690130 this generates this link
 
@doug65536 What o you mean by "of .strong"? I did not know I was dealing with .strong ?
 
see that if near the bottom? it tries to extract the information from the html. the scraper would have to be redesigned for the other page
@user1690130 click this link and right click the page and view source. that if regex has to find and extract the data from what you see in view source
@user1690130 yeah I realized you wanted me to 100% do it for you. thought you had a bug
 
9:14 PM
@doug65536 I never said that
 
@user1690130 not mad or anything...
just realized that the regex is for an entirely different site, isn't it?
 
@doug65536 I'm not too famliar with regex.
 
I bought a book about Perl once. It had a lama on its cover. That's all I can remember right now.
 
the new site seems resistant to scraping
that simple little regex will have to be completely redesigned
and they may make it scripted and work with ajax etc so youd have to get the scrape at runtime. you probably shouldnt be trying to circumvent if they really block it
 
9:17 PM
@doug65536 is the regex the name of the code thta I wrote?
 
@FredOverflow lol. back when web programming == CGI == common gateway interface I learned it pretty extensively
 
CGI reminds me of Command & Conquer with its CGI vs. NOD story.
 
@user1690130 in this page look at line 32
/ of .strong.([0-9,]+)..strong. /
 
yes, i see the strong
my bad
 
that means, find a space followed by 'o' followed by 'f' followed by space followed by any character followed by the word strong etc
the [0-9,]+ means find one or more characters in the range '0' to '9' or a ','
the parentheses mean capture that
followed by any two characters followed by strong
am I making sense?
brb few minutes...
 
9:22 PM
yes i think so.
instead of strong, should it say results?
 
39 messages moved from Lounge<C++>
 
My output look like 'San Diego', 1923, Male,unknown
 
I tried to explain it before, they don't just hand you the data in the page. it's behind a scripting system that fetches data and modifies the page dynamically
 
I just don't understand what that means.
 
a web page can contain programs. programs on the page can run and request information from the server. it can then process it in the program and modify the page to display it
when you get the page, you are just getting some lines that tell the browser to load up javascript programs and run them
one way to hack through it is to use frames and reach into the frame and read the page afterward
thats pretty malicious though unless you are expressly authorized to do that
 
9:32 PM
@doug65536 malicious? how?
 
likely to violate their terms of use
 
@doug65536 i will only do few queries
i did not se nythingin the terms bout scraping
 
setting up a cross frame scrape hack is beyond the scope of helping someone. I'd essentially be doing the whole thing right?
 
@doug65536 why, is it relly complicated?
 
user142019
Wow a Perl room.
 
user142019
9:36 PM
A room for my favourite language! \o/
 
@Zoidberg Hi!
We were discussing this question I posted: stackoverflow.com/questions/14654288/…
@doug65536 How much more involved would the code be?
 
first test would be to see if they have countermeasures against a simple frame page
make an html frame page where you can use a script in another page to extract the data from their frame
 
@doug65536 i did not see anything in their terms to suggest that thsi is wrong to o.
 
9:52 PM
let me do a quick test
 
thnk you!! :)
 
10:05 PM
in the interim, are there other people who like perl in here?
 
10:16 PM
@doug65536 does it seem like there is countermeasure against frame?
 
yes, it is an encrypted page, just got access denied attempting to scrape
 
@doug65536 damn :(
 
being https is their protection
browser puts it pretty off limits and Im not going anywhere near malicious
try asking them!
 
who would i ask exactly?
 
did you try emailing them and asking nicely where they got the data and if there is any access
 
10:21 PM
i tried calling but you get like receptionists
 
It's worth asking them where they got the data, they might point you to a public source or some compilation (wild guess)
from the looks of the site, they are not driving traffic to it or anything, its a public service
right?
 
@doug65536 yes
they have the original census images
then they made a search engine from them
 
sometimes the data is free, you just download some enormous data file and run some long running scripts that fill in a database that you must install to host the data
 
is it illegal to scrape their website?
the data are HUGE
there are millions of people
lke 38 million
i tried calling earlier today, they are just like volunteers who do not know much
i'm not sure how to get to the software people
 
10:32 PM
what doe this stuff mean: familysearch.org/developers/support
 
well there you go
they provide an API
 
Thank you doug, but that is not the same census. that is the census for 2000 or 2010
i don't know what api mens.
 
a thing that lets you request the raw data properly
application programming interface
things you call (as in programming)
you know some programming right?
 
yes
i wrote that perl script!! :)
 
just checking, you might have 100% got it from some site and had no clue
 
10:34 PM
how could i get a broken computer code from a site?
that code works for a different website
i wrote it for ancestry.com.
 
you mean how can you get any code that isn't broken in some way - face it, every program is flawed completely. usually in ways that dont really matter
 
i don't know enough about computers or the internet to get random code.
anyhow.
what are you suggesting?
to contact somebody?
 
I have no way of knowing either way, just asking
its an API demo
do you know javascript at all? jQuery?
 
no, not relly
 
let me see if they have a lower level api
not so easy for them to use as javascript
 
10:40 PM
what is the problem? that it is illegal or that it is challenging intellectually to do this?
 
no
its fine if they provide API
you get a password and have an account
 
yes i hve n account
 
your program uses your account and makes requests
 
i had to do that with ancestry.com too
 
they have API?
 
10:44 PM
this is what i wrote: pastebin.com/wGTayf7h
that was for ancestry.com
i put that at the top of the program
 
looks like they mainly use JSON
so you could probably just use a JSON perl package (I'm betting you have one by default) and get going on it
 
@doug65536 how difficult is thta intellectually?
I never heard of JSON
 
you know the form in the sample? <===
 
no
i am not good with terminology
i just cod
code and do things but don't fully know what is going on
 
click the link in my previous msg
 
10:46 PM
like guess and check my way through computing.
which form?
 
demo / search
 
ok
yes?
do i need to add some lines to the code up top before scraping?
 
those fields are the search parameters, so your program would send a request like the example
there would be no scraping
 
ok . . .
 
JSON is a simple way to encode data that any program can decode
 
10:49 PM
how much different will the code become?
 
it's human readable
 
is this just 1 line tht gets adde at the top?
like a library?
 
I haven't read through the real API docs but this is most likely what it is if they are using jquery
 
use WebService::Simple;

# Simple use case
my $flickr = WebService::Simple->new(
base_url => "http://api.flickr.com/services/rest/",
param => { api_key => "your_api_key", }
);

# send GET request to
# http://api.flickr.com/service/rest/?api_key=your_api_key&method=flickr.test.echo&name=value
$flickr->get( { method => "flickr.test.echo", name => "value" } );

# send GET request to
# http://api.flickr.com/service/rest/extra/path?api_key=your_api_key&method=flickr.test.echo&name=value
$flickr->get( "extra/path",
thta part?
 
yeah
instead of flickr it would be whatever the api docs say to use
 
10:51 PM
and that goes at the top of the code before the scraping part?
 
this: $flickr->get( { method => "flickr.test.echo", name => "value" } ); would call some ancestry
theres no scraping
it returns real data
probably an array of associative arrays
 
it still returns the 1975?
i'm confused by the output
 
it returns what you want it to return. you figure out what API is doing the request you want, call that thing with your search parameters (location, etc) and it returns an array of info where each array entry is the fields (columns displayed on their page)
 
i'm so confused.
 
when you go to that page you tried to scrape, the page uses their api to get the data, and it makes the page update to show it
 
10:53 PM
can i get the output San Diego, 1923, Male, 1975 ?
And then do that to loop over multiple years (so not just 1923)
 
the page itself uses the api and the persons computer requests the raw data. the page then formats it into html and updates the page
 
and then multiple counties (like also San Francisco)?
 
yes whatever you were inserting into the url would be some parameter for the search call
 
will it return a .txt?
like i use .cmd
i have the perl code
i run the code in .cmd
it gives me the out26.txt file
 
look up web services (in general) and you'll get an idea. its simple, a human-readable interface to make a machine run code for you
 
10:54 PM
can i still get that?
 
yes
 
so i'm not chnging my method. just changing the script radically?
 
no guarantees. as far as I can tell without reading their documentation that is how this stuff works
 
the output or the approach i am using to scrape will not need to change. just the code itself ?
 
they make an api (functions to call) that take parameters. you use the webservices protocol to make the call, their server runs the code for the request (accessing database etc) and sends back the data as json
 
10:56 PM
i sooo apppreciate this
like i said before though, i'[m not good with terminology
like i said before, i use a a .cmd
i put the counties into a locations.txt
i get the output like san diego, 1923, male, 1975 as a .txt
will any of that change?
or is it only the perl script that changes?
 
perl has all those cpan libraries. someone did all the real work for you already. you just use this nice luxurious library to do all the real work for you :)
 
so are yo usaying that just the script changes? tht is all?
 
webservice is a way to make it extremely easy for all programming languages to access it
 
i don't know any termniology
i just want to know what mechncally is changing?
 
If I were you I would read their API documentation. they should outline the requests they are making on the web page
 
10:59 PM
before i unerstand it
is the end result going to be new .pl file and that is all?
I really appreciate this
i just don't understand the jargon. i am trying to get the big picture in plain english first.
 
go here which resource do you want to search
 
I want to search the United states 1940 census
 
people
 
yes that makes sense
 
from the looks of it you can search for people, and go beyond that and get relationships like brothers etc
 
11:03 PM
I CAN GET BROTHERS!!!
OH MY!!!
 
like, "give me a list of brothers for person id 12345"
 
that was going to be the second thing i wnate to discuss with you
FOR REAL
 
yes
 
like 2 hours ago when i was asking about perl discusion
but let us defer thta for the moment
first thing first
 
XY problem. you thought you wanted X, when you wanted Y
 
11:04 PM
NO
it is like X and X^2
i want X and X^2
 
lol
 
i thought i cannot get X^2 until I get X
and i do want X regardless
 
look how easy this is
 
if you say!
it does not make sense to me yet
 
11:06 PM
i still want to know how i go through the counties and just extract the total number of people before i get into the individual-level data
so the code i wrote for ancestry.com, the code for familysearch is even simpler?!?
 
no scrape. use the perl library to decode the json and you just use hash names to extract fields
seems so
 
because i'm not really following the big picture yet
 
this api info and this perl library and a little bit of code from you to fill in the request parameters
you may need to use persons to search for people. then each one is a person request I suppose
 
so am i just copy and pastgng the 2 codes?
# send GET request to
# http://api.flickr.com/service/rest/?api_key=your_api_key&method=flickr.test.echo&name=value
$flickr->get( { method => "flickr.test.echo", name => "value" } );
# send GET request to
# http://api.flickr.com/service/rest/extra/path?api_key=your_api_key&method=flickr.test.echo&name=value
$flickr->get( "extra/path",
{ method => "flickr.test.echo", name => "value" });
so there is request and response?
 
yes
 
11:11 PM
the code seems to be 2 part
and they provide the 2 in that link that you found?
and that is all?
 
/platform/tree/persons/12345 is the thing that gets persons
 
that would be the entire program?
GET /platform/tree/persons/12345
Accept: application/x-gedcom-v1+xml
Authorization: Bearer YOUR_ACCESS_TOKEN_HERE
Put my login info here: YOUR_ACCESS_TOKEN_HERE
 
here you "connect" to the webservice:

my $flickr = WebService::Simple->new(
base_url => "http://api.flickr.com/services/rest/",
param => { api_key => "your_api_key", }
);
 
?
so you are saying that this is all legal?
i'm sooo confused still
 
so base_url would be: whatever.com/platform/tree/persons/12345
yes
 
11:13 PM
can i still get the scraping done?
 
I think so
 
so how long is the code?
 
if your account allows you to make the calls (probably does) then yes
 
it sounds like it is easy
but totally different from the code that i currently have
like none of the code i currently have is of any use
 
WebService::Simple should do practically ALL the work for you
it gives you back actual data already "scraped"
your work is to look at the api documentation and figure out how to make it do a search and find out what the parameter names are for the search
then just pass parameters with those names with the appropriate values and it returns the data already scraped for you as real data (strings, numbers)
 
11:19 PM
i am very confused: pastebin.com/qmuNF4gQ
 
you don't know perl very well then
its HTTP protocol request headers
 
no, i don't.. i wrote this code about a year or two ago
and only know the minimum i had to do get though it
i cannot understand which parameter names are for the search because i still don't fully see the big picture
 
oops $flikr would be $fs
 
api_key is my username?
thnk you so much for writing this?
 
they sent you a mail with that information?
you might need to request a key
 
11:34 PM
!
 
usually some random looking string of letters and numbers
usually there is a testing key you can use that is restricted - that they just give out
 
testing key?
 
you can write to the database with their API
so they allow people they trust. testing key account is probably unable to write
 
how do i get one?
 
this is just pure speculation based on other times I've used webservices
you should look around that API documentation stuff
probably in there
 
11:36 PM
the API documentation on their website?
can i just telephone someone instead?
i like the telephone better
 
a "sandbox" is a place where everybody is allowed to test their programs without worrying about anything bad happening
 
so i cannot do this until i fill out that form?
 
doe it have to be a google emil?
gmail?
i thought i read that they want gmail address
 
doubt it
dont know
 
11:43 PM
@doug65536 where did you find that link?
 
@doug65536 do i have to give a good email adress? or can i give a spam one thta i use?
 
any one that you can read would be fine for the sandbox request
 
Request Access to the sandbox* (a place for test development)
then why 7 days?
are they going to do a government background search! it sounds scray
 
I don't work there
how would they do that from an email address
 
11:45 PM
can i calll an dask?
do you see number?
they ask for a phone number too
i don't want spam calls
 
ok, whatever, i'll assume they won't do a big background search and gave a secondary email
so in the meantime, what can be done?
i can't run the code until they get back to me?
 
right
 
can the code be written in the meantime?
 
you could spend some time reading the docs\
they said 7 days?
is this a freelance job or something?
 
11:55 PM
what do you mean?
like why am i doing this?
 
yeah
 
oh for a book that i'm working on
 
cool
 
or may work on if i can get it to work out
ok so you had written: pastebin.com/4XybcEZ5
for your_api_key, i'd fill in my username that i'd get?
 
that's just a rough in - the real parameters would have to be passed and the result processed
yes
 
11:57 PM
i do'nt know what pased and processed means
i am not good with buzz words
how difficult is this intellectually?
 
easy
thats what you need to search for people
the parameters are those things in the table
first time start=0 count=100
second time do start = 100 count=100
etc
q is the name, location, (whatever you know) etc formatted as described
 

  last day (51 days later) »