last day (15 days later) » 

12:02 PM
0
A: Look for word in sentence with R, if not there: drop row

tospigAssuming I've understood your question correctly, here's a solution using a basic regex expression (which I'm sure can be improved and simplified by someone more knowledgeable): Given a data frame e t1 <- "Het aids-probleem ontstaat door mensen zonder vaste partner, legt de speciale editie uit"...

 
Or, do you want everything to the left of the word?
 
Couple of things. The lines you assigned to e$leftContext should actually be sentences. leftContext holds everything left from the word (e.g. *Vorig jaar stierven 3 miljoen mensen aan *). Second, I need this to work dynamically, i.e. for all words (derived from the file name). In other words, I can't define pattern like that. Finally, when I test this, I don't get [2] as output.
 
@BramVanroy Edited: does this work for you?
 
Please see my edit, under the horizontal line. I don't succeed in making this dynamic.
Also, wouldn't this regex be simpler? ( word |^word | word$)
 
@BramVanroy I've edited my e1 <- sub() line; I should have subsetted e on the left hand side to select only the 'leftContext' column
@BramVanroy and, I added the .* in the regex for use in the sub command later on
 
12:02 PM
As you can see in my initial post, I know how to subtract the left context (contexts <- strsplit(df$sentence, df$node)) but what isn't working is what I actually need: removing the rows that don't have the node in the sentence. I probably did something wrong when defining the pattern by using df$node.
 
I think your attempt failed because you are passing df$node into gsub(). Have you tried just using gsub("-","aids",pattern)?
df$node gives you a vector - c("aids","aids","aids","aids",...)
 
That's what I was trying to tell you. my actual data contains of more than simply the node "aids". The vector that df$node returns has many different words.
 
Ah, right. I was under the assumption you could do the substitution when reading the initial file, which would only have one 'node' value.
or is that assumption incorrect too?
 
No, you are right. Let me sketch the whole project for you:
 
I'm thinking you can do the substitution within your e <- do.call(rbind, lapply(files, function(x) {
# if SENTENCE contains WORD (case insensitive)
data.frame(fileName = x, sentence = readLines(x, encoding="UTF-8"), stringsAsFactors = FALSE)
# endif
}))
 
12:12 PM
I have one directory that contains over 15,000 files. The word you see in each file name is the NODE we are looking for. I already extracted the file name (see the beginning of my post). Each file contains one or more lines. Each line consists of a sentence. What needs to be done is: check the NODE (as found in the file name) with each sentence individually. If the node can be found in the sentence, keep it in the dataframe, if it doesn't: drop the whole row.
This is how the files look like:
 
Got it.
This is a sketch of how I *think* it should work
e <- do.call(rbind, lapply(files, function(x) {
df <- data.frame(fileName = x, sentence = readLines(x, encoding="UTF-8"), stringsAsFactors = FALSE)
pattern <- c(" - .*","^- .*"," -$")
pattern <- gsub("-",word,pattern)
pattern <- paste0(pattern, collapse="|")
}))
hit enter too soon:

e <- do.call(rbind, lapply(files, function(x) {
df <- data.frame(fileName = x, sentence = readLines(x, encoding="UTF-8"), stringsAsFactors = FALSE)
pattern <- c(" - .*","^- .*"," -$")
pattern <- gsub("-",word,pattern)
pattern <- paste0(pattern, collapse="|")

df1 <- df[grepl(pattern, df$leftContext, ignore.case=TRUE),]
df1$leftContext <- sub(pattern,"",df1$leftContext, ignore.case=TRUE)
df1
}))
Without your files it's hard to test, but I can't see why the logic of this won't work
 
Hm, isn't it strange that you are assigning elements to dataframe df, when you are acutally inside the function that is creating frame e?
but I'll try it, gimme a sec
 
But, thinking a bit more about it, there is probably a way to do this on the whole data frame after you've combined all the files too.
 
But also here the same problem exists. You haven't defined "word", and I can't just put "agressor" or any other word in there. It ought to be based on node. And node is not defined inside this function either.
 
you can get node from 'x'
e <- do.call(rbind, lapply(files, function(x) {
df <- data.frame(fileName = x, sentence = readLines(x, encoding="UTF-8"), stringsAsFactors = FALSE)

word <- sub("\\..*","",x)

pattern <- c(" - .*","^- .*"," -$")
pattern <- gsub("-",word,pattern)
pattern <- paste0(pattern, collapse="|")

df1 <- df[grepl(pattern, df$leftContext, ignore.case=TRUE),]
df1$leftContext <- sub(pattern,"",df1$leftContext, ignore.case=TRUE)
df1
}))
x is what you've passed into the function, which in this case is your list of 'files', and 'lapply' is applying the function over each item in the list
 
12:29 PM
Oh, right.
D'oh!
 
:) - is it working or are there still issues/
?
 
I tried your function, but for some reason df1 isn't echoed. What I mean is that the function completes without error
but when calling df1 I got the error
object 'df1' not found
 
what about e?
 
[1] fileName sentence leftContext
<0 rows> (or 0-length row.names)
 
oh
 
12:31 PM
I don't expect you to put too much time in this. In a couple of days I can add a bounty to the question. Maybe then you can try again? :)
 
Just looking back over your question, and I probably should have noticed before, that your 'leftContext' column is defined outside of your do.call() statement - which of course means it won't be found inside the function... d'oh too!
so that would explain why my solution isn't working.
 
Haha, indeed. But it seems to me that you don't leftContext anywhere? The sentence is defined in column "sentence"
leftContext holds everything LEFT from the node (the word), excluding the word itself. So it would never return true.
 
ah, yes, so, try it but replace df$leftcontext wtih df$sentence (and df1$leftContext with df1$sentence)?
that probably won't work because there will be something else I've overlooked.. It's getting late here :)
If it still isn't solved by (my) morning I'll have another look at it :)
 
df1 still doesn't exist but I'm guessing it's because df$sentence isn't defined anywhere.
oh never mind
it is
But when trying to call it with df$sentence, I get the error
Error in df$sentence : object of type 'closure' is not subsettable
 
12:49 PM
Where are you calling it, inside or outside the function?
I'm getting more convinced a better way to do this is to use a 'sapply()' on your final data frame 'e'.
df <- sapply(e, function (x) {
Word <- x$node
patrern .... etc


}
Something like that...
 
 
8 hours later…
9:20 PM
I've slept on it, and it should be simpler to do this on your final data frame e.

As an example

t1 <- "Het aids-probleem ontstaat door mensen zonder vaste partner, legt de speciale editie uit"
t2 <- "Vorig jaar stierven 3 miljoen mensen aan aids en raakten er 5 miljoen besmet met hiv"
t3 <- "Zuid-Afrika heeft de meeste aids-gevallen ter wereld."
t4 <- "Aids is geen pretje."
t5 <- "Veilige seks kan aids voorkomen."
t6 <- "Want naarmate de aids-epidemie in Zuid-Afrika en omliggende landen groeit"
I thnk this misses the one where 'Aids' is the first word in the sentence, so you may need to tweak the regexp
 

last day (15 days later) »