last day (15 days later) » 

7:35 PM
2
A: Parse JSON file in R with 'non standard format'

jlhowardtext <- '{"response time":"1075","_p":"99ae8e5f1eb64091c019e57ff0e686d3","_t":1397514301} {"_n":"searcher","version":"v4","mobile":"No","_p":"1797370","_t":1397514315} {"month":"April","_n":"shown availability","availability":"unavailable","_t":1397514320}' library(rjson) json.text <- readLines...

 
This looks extremely promising. One thing I can't work out is how to import my data from a file as opposed to text straight into a variable? In this case my file is called "testdata.json". Thanks. (btw you were right to fix the malformed data, my mistake when trying to make it more readable.)
 
Use json.text <- readLines("testdata.json")
 
OK. Getting really close! That doesn't quite work, I think because the readLines() does something slightly weird to the json data. here is a row: [1] "{\"response time\":\"1075\",\"_p\":\"99ae8e5f1eb64091c019e57ff0e686d3\",\"_t\":1397514301},"
I tried gsub() but it doesn't seem to like the "\"
 
It works for me - run the full code. Those backslashes are put in by R's print function: they are not in the data. You can see this using print(json.text,quote=FALSE).
 
I ran the full code and got the error below which is why I tried to print json.text: Error in FUN(c("{\"response time\":\"1075\",\"_p\":\"99ae8e5f1eb64091c019e57ff0e686d3\",\"_t\":1397514301},"‌​‌​, : unexpected escaped character '\3' at pos 10
As you say, it looks good with print(json.text,quote=FALSE)
REALLY appreciate your help. My email is in my profile, if you dropped me a mail I could send you some of the real data :-)
 
7:35 PM
Better to upload it somewhere (Dropbox??) and post a link.
 
7:58 PM
So when I download your file, rename it testdata.json and run the code, it works.
 
8:14 PM
Sorry! I've replaced it with the full thing. try downloading again.
Note - some of the reocrds contain a URL. Maybe this is a problem?
 
 
1 hour later…
9:47 PM
It's not the URLs. Some of your values have a backslash (open the file in Notepad++ ans look at line 175). Using unexpected.escape="keep" in the call to fromJSON(...) will deal with the backslash characters, but your JSON is still malformed (unless it's to do with the upload/download. The very last line of the file seems to be truncated. When I fix that (manually), I can read the file and convert the JSON to an R named list.
But, converting this list to a data frame with rbind.fill(...) is really slow; about 1 sec per 100 rows on my system, for so for 100,000 rows, 1000 sec or about 15 min. Also the total columns represented in your data is about 97, to the data frame will have 100,000 rows and 97 columns, or 10MM elements, may of which are long char strings. You might have memory problems.
 
10:09 PM
Thanks! Really appreciate your help. It just ran for me - the bad news is that we generate about 10m records a day so looks like it's back to the drawing board!
 

last day (15 days later) »