last day (15 days later) » 

22:36
0
Q: R - fasttext how to load output into a dataframe from command line

astelI am working on a project in R that calls fasttext from the command line, and I am not sure how to load the output that fasttext gives me as a dataframe > data.train<-data.frame(index=c(rep("__label__1",3),rep("__label__2",3)),country=c("ENGLAND","BRITAIN","UNITED KINDOM","USA","AMERICA","UNITED ...

Maybe use intern = TRUE. Have you tried using fread from "data.table" yet?
Looks like intern=TRUE does mostly what I want in that it allows me to store the output as a list in R however each row is stored as a string that I will have to parse into columns later which I think I might be able to do. I hadn't tried fread though, not sure at what point I would do that, can you explain?
fread (from data.table) should be able to read from system commands. If you dput the head of the data you managed to read in using system and show your desired output, it would be easier to help out.
Ok I edited the question to include a reproducible example and show what the end result that I want is
What I had in mind with fread would be something like: library(data.table); fread(cmd = "fasttext predict-prob model_data.bin data.test.txt 2"). Looking at the resulting string in your sample data, you'd end up with a 4-column data.table. I don't know enough about the fasttext format to say whether this is a good answer or not. For instance, will there always be the same number of labels per element of res? Will the labels always be the same?
For instance, with the reprex you've shared, the following would work: x <- fread(cmd = "fasttext predict-prob model_data.bin data.test.txt 2"); ind <- rep(c(FALSE, TRUE), length.out = ncol(x)); x <- setnames(x[, ..ind], sapply(x[, !ind, with = FALSE], [[, 1))[]; x but I'm not sure whether that solution generalizes to the fasttext data structure in general. Hope this helps!
22:36
On fasttext output the number of labels varies but is controlled by the last entry in the call. For example you could have 10 labels but in my example code I have the number 2 at the end which only shows the top 2 labels. Also the label names can vary but will always start with label. So you could have a series of labels as __label__1, __label__03 etc
@astel, then the approach I shared should work.
x <- fread(cmd = "fasttext predict-prob model_data.bin data.test.txt 2")
ind <- rep(c(FALSE, TRUE), length.out = ncol(x))
x <- setnames(x[, ..ind], sapply(x[, !ind, with = FALSE], '[[', 1))[]
Here, ind is just a way to select every alternate column. The first line would result in something ike:
x
#            V1       V2         V3       V4
# 1: __label__1 0.500768 __label__2 0.499252
# 2: __label__1 0.500768 __label__2 0.499252
I couldn’t get fread to work as it was throwing an error that UNC paths were not supported
Since you've gotten to creating res, you could also try x <- fread(text = res).
Error text= is type list but must be character
What does the output look like when you use fread?
x, as shown above.
What do you get when you do str(res)?
22:48
Data frame 2 one of 1 variable $res : factor w 1 level “__label__2 0.5 __label__1 0.498047: 11
*2 obs
So I suppose if I split each row by space and create new columns from the single res column that would give what you have in x
Ah right sorry I changed res to a dataframe and that’s not in the original q
That's strange that you get a data.frame from the output of system. I'm following the code you've posted in your question and I get a character vector.
Yeah sorry at some point I made it a df
I’ll try your suggestion on the point before I made it a df
Yeah that worked
Ahh perfect. This seems to work great
I'll go ahead and post it as an answer, though I'm not sure why you weren't able to use fread directly with cmd = .
What OS are you using?
One last question, will it still work if the labels appear in a different order? Fast text lists the label with the highest probability from left to right so each row might be in a different order
No. It would need to be modified.
22:56
Haha of course it would
I’m on windows but the directory where fast text is located is on a server
I’m not sure, it’s a work project and I’m a bit out of my element when you start talking about OS
To clarify about the label order, do you mean, for example, that row 2 of column V1 could be "__label__2" and V3 could be "__label__1" while row 1 could be in the reverse order?
Yes. Exactly
For row 1 label__1 could be the most probable label and be listed first but for row 2 label__2 could be most probable and as such it would be first
(That’s what the number in col V2 and V4 are, the probability attached to that label)
@astel, I'll be back in a couple of minutes with an example.
Great thanks
23:16
Here's how I'd approach it if the order of labels in each row might be different:
x <- fread(text = res2)
x
##            V1       V2         V3       V4
## 1: __label__1 0.500768 __label__2 0.499252
## 2: __label__2 0.500768 __label__1 0.499252
setnames(x, paste0(rep(c("var_", "val_"), length.out = ncol(x)),
                   rep(1:2, each = ncol(x)/2)))[]
##         var_1    val_1      var_2    val_2
## 1: __label__1 0.500768 __label__2 0.499252
## 2: __label__2 0.500768 __label__1 0.499252
x[, row := .I][]
##         var_1    val_1      var_2    val_2 row
## 1: __label__1 0.500768 __label__2 0.499252   1
I've shown what happens in each step, though typically these steps can be combined.
Awesome, thanks so much for all the help. I’ll accept your answer when you put it up
This should also work if any row has different labels that don't appear in other rows.
For example:
res3 <- c("__label__1 0.500768 __label__2 0.499252",
          "__label__2 0.500768 __label__1 0.499252",
          "__label__3 1")

x <- fread(text = res3, fill = TRUE)
setnames(x, paste0(rep(c("var_", "val_"), length.out = ncol(x)),
                   rep(1:2, each = ncol(x)/2)))[, row := .I][]
out <- melt(x, measure = patterns("var_", "val_"), na.rm = TRUE)[
  , dcast(.SD, row ~ value1, value.var = "value2")]
out
#    row __label__1 __label__2 __label__3
# 1:   1   0.500768   0.499252         NA
Oh yea there is potential for that if you have for example ten labels but only choose to show the top three
The top three labels won’t be the same for each row so you’ll have missing

last day (15 days later) »