« first day (2673 days earlier)      last day (502 days later) » 

05:59
@hadley, I respect the contributions you have made to the field of data science, statistical programming, and R language. Having said this, tidyeval and its syntax as applied to interfaces being directly promoted by RStudio end up making life harder for data analysts. As an example of this kind of push, consider the version of RStudio that only had an import data wizard using readr functions and not base R, which caused tibbles to be injected into the analysis.
But, the more important part is how the DSLs for data manipulation are evolving under tidyeval. To this end, let's look at a tidyeval function written by Romain in the start of February -- c.f. twitter.com/romain_francois/status/959811892922347520. Within this function, the goal is to create multiple lags of the same variable. Is this able to be easily inferred from the contents of the function? How can a student much less an analyst read the code?
I think the major sticking point to this interface is the introduction of !!/!!!, quo/ enquo, .x, overriden ~, and more has unintentionally increased the barrier of entry for generalizing a routine when compared to the "old" NSE _ approach. To exist within this new framework, there needs to be a much deeper understanding of R's underlying system to describe the contexts for when these symbols must be used.
Add in the fact that there are inherent problems with the override of the existing negation operator !, c.f. adv-r.hadley.nz/quasiquotation.html#slicing-an-array, which is a side effect introduced within the tidyverse to solve a self-inflicted problem.
5
Under the present iteration of the tidyeval, I believe a step was taken backwards as the tidyverse team become far more fluent in R. In essence, the rungs of a 'ladder of abstraction' were shifted from concrete systems to working with abstracted artifacts.
This lead to the new design iteration lacking vision for what the simplest possible design could be, which caused a massive paradigm shift away from causal users to sophisticated package developers. Essentially, there lacks a clear intuition as to what guided the decisions in the design of this interface.
5
In short, the argument against tidyeval is:

"You cannot run before you learn to walk. Thus, we'll first show you how to walk and, then, how to run before potentially teaching you how to fly."

I say this because instead of trying to show users how to walk, tidyeval immediately skips to trying to teach users how to fly.
3
 
3 hours later…
09:12
@hadley I never said it makes life harder. Don't make things up. I said it doesn't make it easier. It doesn't make for easily readable code. And it does mean that in order to understand the inner workings of a number of packages, I have to study a whole set of other packages before I can make sense of that code. That's learning a new syntax and I don't mind. It's useful. It adds possibilities. But it's not easier. No, it's not.
2
And I didn't argue that functions shouldn't quote their arguments either. You're defending I-don't-know-what-exactly to something I never said.
2
 
3 hours later…
11:46
Right on. The gist of the argument made is "look we invented this easier second language". Apart from it constantly changing, it also means you now have to learn two languages.
3
 
3 hours later…
14:47
@coatless I obviously disagree quite strongly here - the old lazyeval system (with the _ suffixes) didn't solve many important problems and ultimately required learning just as much about the internals of R as tidy eval, but gave you much less power
You are of course free to criticise our design decisions, but I don't think it's fair to say we lack a vision of the simplest possible design. It is our strong belief that this is the simplest possible design that satisfies our goals (i.e. safe composability of quoting functions)
I would challenge you to come up with a simpler system which is internally consistent and still allows you to safely and easily wrap functions that quote their arguments
I think the main source of confusion about these functions is that our thinking, tooling, and vocabulary has undergone a lot of churn as we've iterated to improve the system. I believe that in a years time we'll have a simpler presentation of the key ideas that allows data analysts to pick up the main ideas and use them in their code.
Effectively we are arguing about a question that can only be answered empirically; only time will tell if people can pick up these tools are use them effectively
(Although again, I don't think it's necessary for most data analysts to learn these tools - you only need to learn about them later in your journey of mastery towards R)
@JorisMeys we seem to be talking at cross-purposes. I am claiming that tidy eval does not make things worse when doing data analysis, and makes things easier when doing programming. I think you are claiming that it makes things harder when programming? I am trying to understand your position.
15:15
@hadley It is fundamentally a fairly complex problem. Part of the challenge is that it seems rlang is designed to comprehensively solve all/most NSE use cases, and that is a complex set. Many people are just looking for basic NSE features, try to use rlang, and are overwhelmed by the complexity (in particular by the at first seemingly subtle distinctions between quote/!!/!!!).
Struggling myself with that complexity I tried to see if there could be a simpler alternative for the simple cases only, which ended up in oshka. Interestingly now that I look at it again it too will suffer from the "large object on call stack" performance problem that ggplot2 had.
15:29
@BrodieG there's a simple pattern that you can follow if you just want to wrap a quoting function and don't want to know all the details: quote with quo() and unquote with !!
That's something we're still working on writing up
You might also want to look at adv-r.hadley.nz/evaluation.html#base-subset — I think your implementation will only solve one of the three problems
(I also don't love that with oshka subset(df, x) is ambiguous - is x a boolean variable in the data frame or a quoted call in the environment?)
I agree that tidy eval is complex — but I have now isolated 5 key ideas that you need to understand to use it effectively, I think you can learn enough of them to be useful in in <30 minutes
It's my belief that learning those ideas pays off by making it easy to solve problems that were previously hard, and by giving you a better mental model of how R works
@hadley And I disagree that it makes things easier. I'm not claiming it makes things harder, I'm disagreeing with your claim that it make things easier. It does not make things easier when doing programming. It makes things different, and it adds for me the complexity of having to check whether my formula interface in pim still functions when someone loads the tidyverse.
I also note that you compare this to your own old lazyeval system. And in that case, this is an improvement. But if you know the internals of R, the only place where you need tidyeval is when programming with the tidyverse. And that only makes sense in specific cases.
@JorisMeys you also need it in for all the base R functions that use NSE
@hadley strange, but they've managed to do that for 18 years without tidyeval. You can use it if you want in R (which goes beyond the base package btw), but there's no need to do so.
Yes, by using 4 different approaches: adv-r.hadley.nz/quasiquotation.html#unquote-base
I claim tidy eval makes programming simpler in the same way that stringr makes data analysis simpler: it's not that there's anything you could not do before, but the consistency generally makes it much easier
@hadley That's where my hair in my neck raises itself. tidyeval does NOT give you a better mental model of how R works. It gives you a better mental model of how tidyverse can be used in programming. And again, I value your work. I just disagree with the sales pitch.
15:43
For an example that I think is particularly beautiful with unquoting see adv-r.hadley.nz/quasiquotation.html#map-reduce-to-generate-code - I think this would be harder to with base R
Another problem that I struggled with for a long time in plyr is adv-r.hadley.nz/quasiquotation.html#slicing-an-array
@JorisMeys I think it does give you a better mental model because you need to learn that code is a tree and that expressions have an associated environment. Both of these ideas are needed to program safely with NSE
@hadley the x resolves as it normally would in R. If it exists in the data frame it resolves to that, if it doesn't it keeps looking through the enclosing environments. While this does create a limitation, it also has the benefit of behaving the same way as R does in that respect.
understanding that code is a tree is particularly important because it steers you away from the "natural" approach of pasting strings together
@BrodieG IMO that means that you can't safely program with oshka (unless you specifically and carefully create unlikely variable names)
@hadley That's not natural. I never paste a single string together. So you want to say you provide tools for people that have a bad understanding of
of R
Take your coef example. What's wrong with
```
```
makeeq <- function(coef) function(x,y) coef[1] + (x*coef[2]) - (y * coef[3])
That would be a different way of making an equation where one can adjust the coefficients. And that one is not better or worse, but different.
That's a different problem - and obviously you could solve that with a function. If you don't like that motivation, how about creating a modelling formula? adv-r.hadley.nz/evaluation.html#making-formulas
(It may not be natural to you, but paste + parse is a very common technique amongst newer R users)
@hadley Feel free to browse through the pim package and see how I extend the formula interface.
15:51
@JorisMeys I don't understand what it has to do with tidy eval?
@hadley You asked "how about creating a modelling formula". Well, it's in there.
But pim doesn't depend on rlang? I don't understand the connection between pim and tidy eval and why you need to "check whether my formula interface in pim still functions when someone loads the tidyverse."
And regarding "paste + parse is a very common technique amongst newer R users": that's my point exactly: you provide tools for people with a bad understanding of R.
@hadley I disagree. It just means you have to be aware of the symbols that are defined in the environments in question. When someone uses functions like f(DATA, expression), my expectation is that the user is aware of what the symbols defined in DATA are. Also, this issue is not limited to NSE either, but a side effect of the nested evaluation model of R.
@hadley Because I learn that the tidyverse also overrides the tilde ~. And as I extend the formula interface, I need to check whether overriding the tilde in any of your packages doesn't break stuff in my package. Why? Because people use your packages and I care about stability in ALL situations.
15:54
@BrodieG tidy eval supplies pronouns to reduce ambiguity where necessary
@JorisMeys where did you learn that? it's not true
@hadley And this is useful, but also increases the complexity of tidyeval. I think most people have safely navigated the nested environment scoping rules of R for a long time, so I don't think it is critical to have the removal of the ambiguity from the same symbols being bound to multiple nested environments.
@BrodieG sometimes the symbols are evaluated and sometimes they're already quoted. That seems inconsistent to me
@BrodieG the pronouns are optional so it's not like they increase the complexity of tidy eval if you don't care about them
@hadley I'm not sure what you mean by this. Example?
@BrodieG (btw you have a typo on your readme: instal_github)
This fails: subset2(iris, quote(Sepal.Width > 4.1))
@hadley Thanks.
15:59
But this succeeds:
exp.a <- quote(Sepal.Width > 4.1)
subset2(iris, exp.a)
@hadley I was looking at Lionel's rlang. The tilde mess is removed again from tidy-eval.R inseed.
@hadley I have to run now, but that feels consistent to me. I'll follow up later.
var <- quote(Sepal.Width)
subset2(iris, var > 4.1)
subset2(iris, quote(Sepal.Width) > 4.1)
I'm still going to check though, because seeing how you manipulate environments, there's still risk the buildup of my own objects might be influenced.
(although that's not really your fault because the result of quote(Sepal.Width) > 4.1 is just bizarre)
@JorisMeys we only manipulate our own environments, not yours. There's no way for tidy eval to slip its bounds and affect your code
16:04
@hadley I hope so. I'm still going to check. I have to check whether it works with tibbles and the likes anyway. And I know you aim for making it as unintrusive as possible, but that doesn'
doesn't mean I should just trust on it and hope for the best.
@hadley quote(Sepal.Width) > 4.1 is just bizarre. Why would one want to compare a name object with a numeric? quote() returns a name (symbol if you rather have the type instead of the class). So that code makes no sense in R.
@JorisMeys right, but it returns TRUE 🙀
I think this is the way to refer indirectly to a variable with oshka: var <- quote(Sepal.width); subset2(iris, var > 4.1). We'll have to wait for @BrodieG to get back to clarify
@hadley so? if you feed it garbage, you get garbage. And if you don't read the warning on ?subset, you use it for things you shouldn't use it for.
@JorisMeys this is the sort of thing that makes me nervous when writing (base) R code: you give it input that is clearly invalid and it gives you a reasonable looking output. That sort of behaviour can lead to surprising and hard to find bugs
One could argue this is a bug in the primitive >. It should return an error, but it doesn't.
ah, see. On that we both agree wholeheartedly. There's a lot of things that can happen unnoticed in R. T and F are an atrocity that should be removed asap if you ask me, but T <- t(c(0,1)) ; isTRUE(T) doesn't return an error either.
I think we differ primarily in our response to such problems in base R — I think it's often worth attempting to fix bad behaviour at the package level (since base R does/can not change in large ways)
16:19
@hadley ofcourse it is. Just don't call it "easier", because you add complexity. I wouldn't be surprised when RStudio suddenly announces an R derivative based on tidy principles. A bit like the Python 2 - Python 3 story. Or actually more a Perl - Python story, as Python is heavily based on Perl but wanted to get rid of all the funny Perl ways.
And if you then remove all the quirky bits and replace them with something else (and don't use !! and !!! puhleaze...) then you might argue it is easier to understand. Just like Python code is easier to understand compared to much of my Perl code.
@JorisMeys RStudio would NEVER do that. If you talk to any of the RStudio leadership you will understand that would be fundamentally against our mission
@hadley I take your word for it, and I don't expect it any time soon. But I wouldn't be surprised if it happened either.
 
1 hour later…
17:28
@hadley Yes. Variables that resolve to quoted language according to (my best approximation of) the standard R symbol resolution rules are substituted.
@hadley I think this is consistent. There is an implicit quote in the substitution of subset/subset2, so the first case is really quote(quote(Sepal.Width > 4.1)).
For example, you get the same error if you do subset2(iris, quote(exp.a)) as when you do subset2(iris, quote(Sepal.Width > 4.1)).
 
4 hours later…
21:26
@BrodieG we tried that approach at one point in tidy eval but moved away from it because it felt too magical
My latest attempt to explain tidy eval: youtu.be/nERXS3ssntw (in ~5 mins)
@hadley I would have thought your bigger concern was that it is limited in what it can do. The magic feels beautiful to me (obviously beauty in the eye of the beholder...). I think it is also very close to what a naive user would expect (those that think "why can't I store this column name in a variable for subset").

« first day (2673 days earlier)      last day (502 days later) »