Feature consideration - predefined loadable subgraphs

[genericallyloud]

Hi, I just watched the jsconfeu presentation. I was eager to see your approach so I could compare it to my own. I also have an optimizing module server of sorts. My team and I went down the same sort of road. First started loading files as needed. Then we concatenated everything upfront. Then we hit the 1MB download and knew we had to break it apart. I had actually already done the work of making a dependency graph in order to do the single script concatenation so that I could correctly order the files in the concat script.

This is where our approaches diverge. I had considered doing something similar to your approach. What I dislike about it was the combinatorial explosion. I had gotten used to deploying with a single static script file. There are obvious advantages. Deployment is simpler, the scripts themselves can go anywhere, and you can even use a CDN. Browser caching is potentially much better because order of interaction doesn't change the scripts loaded.

In our approach, instead of allowing for the loading of arbitrary dependencies, certain dependencies were marked as roots of a new dependency subgraph. To get a big picture on the structure, in a very large app like ours, the dependency graph becomes a tree of subgraphs. The tree structure assumes that parents are always loaded before children. Dependencies can only belong to a single subgraph, but if a dependency is needed by multiple subgraphs, they are moved to the closest parent subgraph. As opposed to a module-server, we have what we call a "module compiler". It does many other things in addition to the dependency graph stuff, but that dependency graph is the heart of it. Anyway, at the very end of module compilation, the end result is a static js file for each subgraph, using a naming convention of the path to the module separated by underscores.

Navigating around the app results in dynamic script loads that might look something like:

main.js
main_child3.js
main_child2.js
main_child2_gchild1.js
main_child2_gchild3.js
main_child1.js
main_child1_gchild4.js

Where main, childX, gchildX are just example names for the root dependency of that subgraph. Notice that parents are always loaded before children, but siblings can be loaded in any order.

In our app, this would basically be main loaded initially in the html, and childX being based on first level nav, gchildX based on 2nd level nav. That works best for us, but obviously an app with more functionality per screen and fewer screens would use it differently.

I hope this was helpful. I know its a different direction than you are currently going, but seeing as I'm not able to contribute my own code to open source ATM, I was hoping to spread a little of my own experience/knowledge this way.

[cramforce]

Hey, thanks for the email. If I understand correctly what you are saying,
this is essentially a full subset of the functionality of module server.
Module Server does not force you to make such decisions early, but it, of
course, allows you to make them. Is your request, that module server should
have a feature to enforce a certain shape of the graph?

A future version of module server will likely require predefining which
modules can be endpoints, which will enable things like closure compilers
cross module code motion and synthetic modules.

On Wed, Nov 21, 2012 at 7:26 AM, genericallyloud
notifications@github.comwrote:

Hi, I just watched the jsconfeu presentation. I was eager to see your
approach so I could compare it to my own. I also have an optimizing module
server of sorts. My team and I went down the same sort of road. First
started loading files as needed. Then we concatenated everything upfront.
Then we hit the 1MB download and knew we had to break it apart. I had
actually already done the work of making a dependency graph in order to do
the single script concatenation so that I could correctly order the files
in the concat script.

This is where our approaches diverge. I had considered doing something
similar to your approach. What I dislike about it was the combinatorial
explosion. I had gotten used to deploying with a single static script file.
There are obvious advantages. Deployment is simpler, the scripts themselves
can go anywhere, and you can even use a CDN. Browser caching is potentially
much better because order of interaction doesn't change the scripts loaded.

In our approach, instead of allowing for the loading of arbitrary
dependencies, certain dependencies were marked as roots of a new dependency
subgraph. To get a big picture on the structure, in a very large app like
ours, the dependency graph becomes a tree of subgraphs. The tree structure
assumes that parents are always loaded before children. Dependencies can
only belong to a single subgraph, but if a dependency is needed by multiple
subgraphs, they are moved to the closest parent subgraph. As opposed to a
module-server, we have what we call a "module compiler". It does many other
things in addition to the dependency graph stuff, but that dependency graph
is the heart of it. Anyway, at the very end of module compilation, the end
result is a static js file for each subgraph, using a naming convention of
the path to the module separated by underscores.

Navigating around the app results in dynamic script loads that might look
something like:

main.js
main_child3.js
main_child2.js
main_child2_gchild1.js
main_child2_gchild3.js
main_child1.js
main_child1_gchild4.js

Where main, childX, gchildX are just example names for the root dependency
of that subgraph. Notice that parents are always loaded before children,
but siblings can be loaded in any order.

In our app, this would basically be main loaded initially in the html, and
childX being based on first level nav, gchildX based on 2nd level nav. That
works best for us, but obviously an app with more functionality per screen
and fewer screens would use it differently.

I hope this was helpful. I know its a different direction than you are
currently going, but seeing as I'm not able to contribute my own code to
open source ATM, I was hoping to spread a little of my own
experience/knowledge this way.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2.

http://twitter.com/cramforce
http://nonblocking.io

[genericallyloud]

Is your request, that module server should
have a feature to enforce a certain shape of the graph?

Yes, I suppose that is my request. However, in that case, the module-server would not need to be a server as much as a build step. During development, it would probably still need to be a server so that I could do a page refresh without rebuilding (that's what I do now). I already have this working in my own code base, so I'm not really sure if its a request as much as a suggestion, but someday if there was feature parity, I might be happy to replace my custom code with module-server.

A future version of module server will likely require predefining which
modules can be endpoints, which will enable things like closure compilers
cross module code motion and synthetic modules.

Ok, I think that answers one of my remaining questions. I was going to say that the only reason module-server would not be a subset is because of the need to moving up shared dependencies, but if you're saying closure compiler already does this, I guess you're in a pretty good spot to implement this quickly.

The difference as I see it is something like this:
My example

main.js
main_child3.js
main_child2.js

Translated to module-server

/js/main.js
/js/+child3-main.js
/js/+child2-main.js

Let's say for a moment that child3 and child2 each have a dependency on "foo", but main does not. The load order of child3 and child2 would affect where foo was included in module-server, but not my module-compiler. My module-compiler would put foo in main, even though nothing in main directly depends on it. Your module-server would currently put foo in child3 if that was loaded first, or in child2 if you changed the order and loaded it before child3. I think you were saying closure compiler could handle this, but I just wanted to clarify the issue.

[cramforce]

Module server does not require any of this moving. Whatever gets loaded
first, gets the dep, the next one can use it. Why would you want to move it
up the chain? Seems wasteful.

On Wed, Nov 21, 2012 at 8:36 AM, genericallyloud
notifications@github.comwrote:

Is your request, that module server should
have a feature to enforce a certain shape of the graph?

Yes, I suppose that is my request. However, in that case, the
module-server would not need to be a server as much as a build step. During
development, it would probably still need to be a server so that I could do
a page refresh without rebuilding (that's what I do now). I already have
this working in my own code base, so I'm not really sure if its a request
as much as a suggestion, but someday if there was feature parity, I might
be happy to replace my custom code with module-server.

A future version of module server will likely require predefining which
modules can be endpoints, which will enable things like closure compilers
cross module code motion and synthetic modules.

Ok, I think that answers one of my remaining questions. I was going to say
that the only reason module-server would not be a subset is because of the
need to moving up shared dependencies, but if you're saying closure
compiler already does this, I guess you're in a pretty good spot to
implement this quickly.

The difference as I see it is something like this:
My example

main.js
main_child3.js
main_child2.js

Translated to module-server

/js/main.js
/js/+child3-main.js
/js/+child2-main.js

Let's say for a moment that child3 and child2 each have a dependency on
"foo", but main does not. The load order of child3 and child2 would affect
where foo was included in module-server, but not my module-compiler. My
module-compiler would put foo in main, even though nothing in main directly
depends on it. Your module-server would currently put foo in child3 if that
was loaded first, or in child2 if you changed the order and loaded it
before child3. I think you were saying closure compiler could handle this,
but I just wanted to clarify the issue.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2#issuecomment-10604202.

http://twitter.com/cramforce
http://nonblocking.io

[genericallyloud]

Ok, so I looked over the example I gave, and I translated it to module-server urls wrong. It should be:

/js/main.js
/js/+child3-main.js
/js/+child2-main,child3.js

I think that this is an extremely sensible and correct approach for dynamic loading, but the combinatorial explosion of + and - modules would lead to an unreasonable number of static files. Hypothetically, if I were to use module-server to generate static files, assuming only the three dependency load points main,child2,child3, I would have to generate:

main.js
main-child2.js
main-child3.js
main-child2,child3.js

child2.js
child2-main.js
child2-child3.js
child2-main,child3.js

child3.js
child3-child2.js
child3-main.js
child3-child2,main.js

I'm not a mathematician, but I believe I calculated correctly, and determined that 20 modules would result in over 10 million static files if there were no assumed load order.

Obviously, module-server does not try to make static files upfront. However, I think there are a lot of benefits, and it's what we chose to do. I suppose all I'm trying to do is point out the benefits, and say how we were able to accomplish it.

It's a balance of load order flexibility and upfront module creation. Sorry if I didn't make that clear. Module-server maintains the dependency graph at runtime, and concats on the fly based on a url. What I was saying is that there is value to creating a set of static files upfront - as a build step. So in production, my module-compiler is just a build step. It does all of the dependency graph stuff and then generates a set of static files - one for each subgraph. When we deploy to production, there is no dependency graph - on the server or on the client. The advantage is mostly 3 things:

• ease of deployment (no additional server, just static files)
• js files can go to a CDN
• browsers can cache the js modules (including appcache)

With module-server, none of those are possible, because if it is possible to go to child2 before child3 or vice-versa, the url is different, and therefore the resulting JS code is different (unless you are willing to accept the huge combinatorial overhead).

The balance that we came to is as I described previously. Every load point has a single canonical module, so when you want to load child2, you will always load main_child2 no matter what other modules have been loaded (but assuming main has been loaded first). In order to do this, the two concessions are:

1. There is a required load order around the ancestor axis (but not the sibling axis)
2. Any shared dependencies move to the nearest common parent.

Perhaps you have another approach?

[cramforce]

We used the static system before. It is a sensible approach and certainly
easier to maintain in production. Module Servers' approach, of course,
works with CDNs (proxying CDNs only) and browsers can cache the JS.

See
https://docs.google.com/spreadsheet/ccc?key=0AoIOxKkr6fGqdGJIT2FFYjJ1Q0JLbEFXUnpqSFQya1E#gid=0
for a table of trade offs between different loading strategies. Your
approach is in row 6.

On Wed, Nov 21, 2012 at 11:07 AM, genericallyloud
notifications@github.comwrote:

Ok, so I looked over the example I gave, and I translated it to
module-server urls wrong. It should be:

/js/main.js
/js/+child3-main.js
/js/+child2-main,child3.js

I think that this is an extremely sensible and correct approach for
dynamic loading, but the combinatorial explosion of + and - modules would
lead to an unreasonable number of static files. Hypothetically, if I were
to use module-server to generate static files, assuming only the three
dependency load points main,child2,child3, I would have to generate:

main.js
main-child2.js
main-child3.js
main-child2,child3.js

child2.js
child2-main.js
child2-child3.js
child2-main,child3.js

child3.js
child3-child2.js
child3-main.js
child3-child2,main.js

I'm not a mathematician, but I believe I calculated correctly, and
determined that 20 modules would result in over 10 million static files if
there were no assumed load order.

Obviously, module-server does not try to make static files upfront.
However, I think there are a lot of benefits, and it's what we chose to do.
I suppose all I'm trying to do is point out the benefits, and say how we
were able to accomplish it.

It's a balance of load order flexibility and upfront module creation.
Sorry if I didn't make that clear. Module-server maintains the dependency
graph at runtime, and concats on the fly based on a url. What I was saying
is that there is value to creating a set of static files upfront - as a
build step. So in production, my module-compiler is just a build step. It
does all of the dependency graph stuff and then generates a set of static
files - one for each subgraph. When we deploy to production, there is no
dependency graph - on the server or on the client. The advantage is
mostly 3 things:

• ease of deployment (no additional server, just static files)
• js files can go to a CDN
• browsers can cache the js modules (including appcache)

With module-server, none of those are possible, because if it is possible
to go to child2 before child3 or vice-versa, the url is different, and
therefore the resulting JS code is different (unless you are willing to
accept the huge combinatorial overhead).

The balance that we came to is as I described previously. Every load point
has a single canonical module, so when you want to load child2, you will
always load main_child2 no matter what other modules have been loaded
(but assuming main has been loaded first). In order to do this, the two
concessions are:

1. There is a required load order around the ancestor axis (but not
   the sibling axis)
2. Any shared dependencies move to the nearest common parent.

Perhaps you have another approach?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2#issuecomment-10610065.

http://twitter.com/cramforce
http://nonblocking.io

[genericallyloud]

We used the static system before. It is a sensible approach and certainly
easier to maintain in production. Module Servers' approach, of course,
works with CDNs (proxying CDNs only)

You're right, I hadn't been thinking about proxying CDNs because that's not what we use, but you're google, so I should have known better ;)

and browsers can cache the JS.

They can, but only when the load order is the same. Depending on the app, I suppose this might be more or less likely. A typical case might be something like an app with a landing screen, and lets say 4 other tabs. Assuming I want to lazy load additional code when I navigate to another tab, I would want tab1-4 to be my additional lazy load points. Depending on what order you visit the tabs, you would need to serve different files with different urls, so some of the caching would be lost.

Anyway, you clearly have a handle on it. It obviously depends on circumstances. I was merely trying to lend some experience with one approach in case it was something you eventually wanted to be able to support. If you have any interest in discussing more let me know, but it sounds like you've got it covered.

PS. I'm curious why approach 6 didn't get an X for awesome cacheability.

[cramforce]

With the dynamic approach you can simulate the static approach with
discipline, but that can fail, of course.

The X for cacheability was an oversight! Thanks!

On Wed, Nov 21, 2012 at 12:36 PM, genericallyloud
notifications@github.comwrote:

We used the static system before. It is a sensible approach and certainly
easier to maintain in production. Module Servers' approach, of course,
works with CDNs (proxying CDNs only)

You're right, I hadn't been thinking about proxying CDNs because that's
not what we use, but you're google, so I should have known better ;)

and browsers can cache the JS.

They can, but only when the load order is the same. Depending on the app,
I suppose this might be more or less likely. A typical case might be
something like an app with a landing screen, and lets say 4 other tabs.
Assuming I want to lazy load additional code when I navigate to another
tab, I would want tab1-4 to be my additional lazy load points. Depending on
what order you visit the tabs, you would need to serve different files with
different urls, so some of the caching would be lost.

Anyway, you clearly have a handle on it. It obviously depends on
circumstances. I was merely trying to lend some experience with one
approach in case it was something you eventually wanted to be able to
support. If you have any interest in discussing more let me know, but it
sounds like you've got it covered.

PS. I'm curious why approach 6 didn't get an X for awesome cacheability.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2#issuecomment-10613260.

http://twitter.com/cramforce
http://nonblocking.io