last day (15 days later) » 

03:16
1
A: How to use Typescript's references and build mode to compile a mono repo to a serverless bundle?

Matt McCutchenIf it works for you to generate a single output file and use a module loader, you can use outFile and prepend. If you want multiple output files, maybe it's worth filing a suggestion to ask for an option to bundle dependencies in that case; you would be the second person who has asked about this...

Tom
Tom
I'm fine with usign outFile, but lambda requires the function code to have an entry point, i.e. a module.exports = functionName. The outFile option only allows for amd or system modules, and therefore doesn't seem compatible with lambda?
In other words, Lambda expects you to provide a CommonJS module? You may be able to use TypeScript's amd output and then add in amdefine to make the result loadable as a CommonJS module. But this is a hack. I'll file a suggestion for a better way.
Tom
Tom
Exactly. Thanks for filing a suggestion! Would love to participate in the discussion
I also looked at using amdefine but this seems non-trivial, I'd have to somehow inject it (including its source since lambda requires all dependencies to be packaged) to the top of the built file.
Re amdefine: can you do that with a referenced project with allowJs and the prepend option?
Tom
Tom
It says Option 'allowJs' cannot be specified with option 'declaration'. The declaration option seems quite important, so I guess not?
03:16
I got something sort of working at github.com/mattmccutchen/serverless-mono-example/tree/amdefi‌​ne: I can require('./packages/serverless-random/dist') in the Node REPL and the function works. However, I realized another problem: there's no mechanism to copy node_modules from random to serverless-random, so you'd have to add all the dependencies of random to serverless-random, which is ugly. If you opt to instead use another tool to copy the node_modules, then the TypeScript suggestion is of questionable value, but then so is the existing prepend option.
New idea: given that you are using Yarn workspaces and Yarn makes a symlink from node_modules/@org/random to packages/random, etc., is there a way you could get your packaging process to pick up packages/random via that symlink, the same way you pick up the rest of the dependencies? Then you don't need anything special from tsc.
Tom
Tom
I don't really have a "packaging process", I thought that this is where the Typescript build feature could help. I expected Typescript to follow dependencies (hence the need to define references), such that it could bundle all deps to a single folder or file. Since this is not possible, perhaps my only way out is to resort to the old fashioned way of using webpack. There is an example here: github.com/serverless-heaven/serverless-webpack/tree/master/‌​… -- however there are several issues with this approach, hence me looking to use the new tsc -b feature.
Then what happens when you run serverless package or serverless deploy? It must be picking up node_modules somehow. Or does nobody write serverless services with Node dependencies?
Tom
Tom
To be honest
It's not actually described there.
They just describe how to deviate from the default behavior with include and exclude
But the default behavior is to look at the package.json file and include the dependencies from node_modules in that directory, I believe.
Right now my serverless-mono-example/packages/serverless-random/node_modules is empty except for .bin. Is that how it is for you?
Tom
Tom
https://serverless.com/framework/docs/providers/aws/cli-reference/package/

It's poorly documented
@MattMcCutchen yes that's because of Yarn workspaces
Yarn workspaces will hoist all dependencies to the root level yarnpkg.com/lang/en/docs/workspaces
03:26
So is serverless package picking anything up? Or do you have a special case because you don't depend on anything but your other module, @org/random?
My point was, if we have a solution for people to get their external dependencies into node_modules, it might not be hard to get symlinks to their other packages into node_modules too, which might allow serverless package to pack them up.
Tom
Tom
@MattMcCutchen let me confirm but I believe that serverless is not able to work with yarn workspaces out of the box

that's based on the reply at https://forum.serverless.com/t/using-serverless-with-yarn-workspaces/4560

so my thinking was that if typescript bundles everything to a single directory, serverless would automatically work with yarn workspaces too
If you disable hoisting all dependencies should go to the local node_modules folder. Is that what you want?
ISTM fixing the interaction between serverless and yarn workspaces is better than trying to add options to tsc. If you use nohoist as described in that thread, then do you get @org/random in your bundle?
Tom
Tom
@MattMcCutchen no you don't, because @org/random is not a real node_module

it is a `path mapping`, that's why this problem is in fact related to tsc
I am 90% sure about that. If you give me 5 mins I can confirm it
In my serverless-mono-example I have a symlink node_modules/@org/random -> ../../packages/random. If you turn on nohoist, does yarn make a similar symlink from serverless-mono-example/packages/serverless-random/node_modules/@org/random ?
Tom
Tom
@MattMcCutchen let me check
@MattMcCutchen not entirely related but strangely enough I get this after doing `yarn install` in the root folder after disabling hoisting:

```
Toms-MacBook-Pro-2:serverless-mono-example tommedema$ yarn install
yarn install v1.7.0
[1/4] 🔍 Resolving packages...
error Couldn't find package "@org/random@^1.0.0" required by "[email protected]" on the "npm" registry.
```
let me see if I can still get an answer to your question by simply removing the local package from my package.json....
03:36
I got that too. I had to fix the name in packages/random/package.json.
It said random and I changed it to @org/random.
I enabled nohoist and I am getting the symlink! The only remaining question is whether serverless package follows the symlink.
Tom
Tom
@MattMcCutchen can you push your repo?
Ah yes the symlink is there
Let me try to package with serverless
@MattMcCutchen so, here is the thing
you cannot let serverless package typescript source files
you have to build first
hence my package.json in serverless-random:
sorry, hence my serverless.yml:
plugins:
- serverless-plugin-scripts
custom:
scripts:
hooks:
'before:deploy:createDeploymentArtifacts': yarn run build
this means "run build before packaging"
so you can manually reproduce this
first run yarn run build
then whatever is build is what serverless can bundle
you can then package with ./node_modules/.bin/serverless package
or just run yarn run package
I get .serverless/org-random.zip with dist/index.js in it. Should work, right?
Tom
Tom
@MattMcCutchen I'm pretty sure that index.js has nothing related to @org/random inside :) except for the import statement..
Oh, you are right. I was misled by the name.
Tom
Tom
@MattMcCutchenso right now the dependencies are all missing
the bundle is 2kb small..
03:47
It's because you excluded ./** in serverless.yml.
If I include node_modules/** then I get a huge bundle.
Tom
Tom
@MattMcCutchen you are right, so now the challenge is to make the references inside that bundle work
Which they should using normal Node module resolution rules, given that you specify "main": "dist/index.js" in random/package.json.
Tom
Tom
@MattMcCutchen the good news is that because of `tsc -b`, @org/random is now also built

previously this was not possible
I'll try to deploy it
got an internal server error.. checking logs now
@MattMcCutchen

```
Unable to import module 'dist/serverless-random/src/index': Error
at Function.Module._resolveFilename (module.js:547:15)
at Function.Module._load (module.js:474:25)
at Module.require (module.js:596:17)
at require (internal/module.js:11:18)
```
ok this is because of my serverless.yml
handler: dist/serverless-random/src/index.fetchRandomNumber is wrong
changing it to handler: dist/index.fetchRandomNumber and redeploying
@MattMcCutchen

https://kdmmj1rit6.execute-api.us-east-1.amazonaws.com/dev/number

!!!!
So we have something that works. The next question is if the bundle really needs to be 38 MB.
Fiddling with the yarn configuration can probably improve that.
Tom
Tom
Yeah. I suppose aws-sdk is there, and redundant source files, too
@MattMcCutchen I think it's more probable that it's the serverless configuration
03:59
I think the problem may be that yarn installs the transitive dependencies all in a single directory, while serverless package only excludes the direct dev dependencies, so you end up with lots of indirect dev dependencies being packaged.
Tom
Tom
@MattMcCutchen you're absolutely right.. I just realized that too
that sounds like a big problem
Maybe you can use artifact mode (serverless.com/framework/docs/providers/aws/guide/packaging) and substitute your own command to make the zip.
E.g., make a dummy package that has a dependency on serverless-random and no dev dependencies, call yarn install on that package, zip up its node_modules, and change the handler in serverless.yml to match.
Tom
Tom
Sure but then I need a dependency graph, which is very complicated
You mean create a dummy package only for the purpose of packaging serverless-random?
Tom
Tom
That seems uglier than going back to webpack
04:04
It's up to you. You said there were bugs with webpack.
Tom
Tom
https://yarnpkg.com/blog/2018/02/15/nohoist/

what if I make the nohoist less aggressive ?
There are :'( I'd have to try to solve them
Actually github.com/serverless/serverless/issues/3827 suggests my theory is wrong and serverless package does have logic to keep only the transitive production dependencies.
Tom
Tom
by changing package.json in the root to:

```
"nohoist": [
"**/serverless-random/**"
]
```

and in serverless-random serverless.yml to:

```
package:
exclude:
- ./**
include:
- dist/**
- node_modules/**
- '!node_modules/aws-sdk/**'
```

the package size became 22MB
I'm not sure how much of the 22MB is unnecessary though
@MattMcCutchen I don't think there is a syntax error though ?
If you move the @types dependencies to the devDependencies where they belong, then the only necessary packages should be @org/random and @org/serverless-random : <1 MB.
Tom
Tom
@MattMcCutchen here https://github.com/Microsoft/types-publisher/issues/81 they say it's better to put them in dependencies

but let me try it
it's still at 22MB so I guess the deep dev dependencies are not resolved ?
04:17
Something is going wrong there, I don't know what.
Maybe github.com/serverless/serverless/pull/… describes your case too?
It might be time for a new bug against serverless. But I think the conclusion is that the yarn symlink is the right solution to this use case, not some new TypeScript option.
Tom
Tom
@MattMcCutchen perhaps. I do expect there to be more use-cases to have a single outputDir, just like typescript has outputFile
If you find any that can't be solved with yarn the way we just did, please add them to the suggestion!
Tom
Tom
thanks @MattMcCutchen

honestly right now I am still at a point where I can't really use it

curious where you think it's going wrong? is serverless even able to figure out which of these deep dependencies are dev dependencies?
That would probably take an hour or two of debugging the serverless tool to answer, which is more than I'm interested in doing.
It reminds me of debugging the Meteor tool when I was trying to structure my project in a way they didn't expect, which wasn't fun.
Tom
Tom
@MattMcCutchen

ok if you just comment out everything:

# package:
# exclude:
# - ./**
# include:
# - dist/**
# - node_modules/**
# - '!node_modules/aws-sdk/**'

the bundle is only 77Kb
04:29
Oh, maybe the exclude/include were overriding the dev dependency exclusion!
Tom
Tom
yeah...!
Does it run?
Gotta love incomplete documentation.
At some point you try to guess what you're doing differently from everyone else that's causing it not to work.
Tom
Tom
@MattMcCutchen I am always thinking this...
right now there is a small issue left
which is that I'm not sure how to exclude some redundant source files...
but that's not a huge issue
and for some weird reason it is including ts-node, which I guess is a small bug
04:41
Ooh, I found the logic that is excluding the dev dependencies: github.com/serverless/serverless/blob/…
It is taking what npm thinks are the transitive dev dependencies minus what npm thinks are the transitive prod dependencies, and excluding that set. npm doesn't think ts-node is a transitive dev dependency. Why does yarn think it is?
Ah, this is something to do with ts-node being a dev dependency of @org/random and yarn workspaces doing something funny.
Tom
Tom
Hmmm. I'm confused.
I can't believe they are hardcoding calls to npm.. I was planning to uninstall npm
@MattMcCutchen good news is that I managed to remove the source code by changing the exclusion pattern to:

```
package:
exclude:
- ./**
- '!dist/**'
- '!node_modules/**'
```

Defining it this way does not overwrite the dev dependencies exclusion
So only 2 issues left for which I can create issues at the serverless framework:

- Serverless artifacts contain a few [redundant dependencies](https://github.com/serverless/serverless/pull/3889#issuecomment-414547166)
- Yarn workspace hoisting is currently disabled for [serverless services](https://forum.serverless.com/t/using-serverless-with-yarn-workspaces/4560)
04:57
It looks like the first problem is that yarn is installing the dev dependencies of all workspaces in packages/serverless-random, not just the dev dependencies of serverless-random itself. IMO that is a yarn bug/limitation, not something I'd expect to be fixed on the serverless side. But it's up to you how you want to pursue this.
Tom
Tom
@MattMcCutchen that's because the ** makes the dependencies of dependencies also nohoisted
I don't see how yarn could solve that
Yes, but a devDependency of a dependency of serverless-random shouldn't be relevant at all.
Tom
Tom
@MattMcCutchen so your suggestion is that nohoist should only be relevant to production dependencies?
nohoist should only be applicable to the modules that would normally get installed in a folder: the transitive dependencies, plus the transitive dependencies of the direct devDependencies, and no other combinations.
I suppose you might convince the serverless folks to just keep the prod dependencies rather than doing this weird (whatever is on the filesystem - (npm dev dependencies - npm prod dependencies)). Is there a use case for people to have things in node_modules that aren't in package.json and want them in the bundle? But I definitely feel that what yarn is doing is wrong.
Anyway, we now know that my suggestion of a dummy package may not fix the problem due to this weird yarn behavior.
Tom
Tom
@MattMcCutchen but if a dependency A has a dependency B and that dependency B has a dependency C, then dependency C must also be nohoisted, even though it is 3 levels deep. Or are you not against that? You're only talking about deep dev dependencies?
05:10
That would fit the condition of "transitive dependency" that I stated.
Tom
Tom
Ok, my bad
@MattMcCutchen I think the reason they use the filesystem is just because they are too "lazy" to traverse through all depths of production dependencies in nested package.json files ;)
But that's what their "npm ls --prod=true" command is doing...
Tom
Tom
Hmmm. So the real problem is just that they don't support yarn.
If they would support yarn, there would probably not be a mismatch
As I've said, IMO the real problem is that yarn is doing something weird. If yarn did the sane thing, there would be no mismatch. But you pursue this how you like.
Tom
Tom
ok, I guess I don't really follow what yarn is doing wrong :P I'll have a deeper look
thanks for the help man
05:14
It is installing a devDependency of a dependency in the folder. That's something that never happens when you don't use yarn workspaces.
You're welcome.

last day (15 days later) »