Support frontend recording

[lachrist]

Problem

This specification needs to be adapted to enable the js agent to record browser applications. That is because sources can no longer be assumed to be static files. Indeed, on browser, code comes from http responses and not static files. For instance, when the man-in-the-middle proxy intercepts the http response of GET http://localhost:8080/script.js, it has no idea to which static file this source should be linked. This static file might not even exist if the source has been dynamically generated.

Solution

Currently, the agent is breaking the spec in two places:

1. The root packages of classmaps can be url origins. For instance, when the source is associated to http://localhost:8080/directory/script.js the root package will be http://localhost:8080.
2. The location of functions can be urls -- eg: http://localhost:8080/directory/script.js. Note that the visualizer should not attempt to fetch these. This code can only be retrieved if it was inlined in the appmap.

classMap:
  - type: package
    name: "http://localhost:8080"
    children:
      - type: package
        name: directory
        children:
          - type: class
            name: script
            children:
              - type: function
                location: "http://localhost:8080/directory/script.js"

File URLs

It may be worthwhile to support file urls. That way, function locations could consistently be provided as (absolute) urls. File urls would represent static sources that can be fetched while other urls would represent dynamic sources that cannot be fetched. That would simplify the js agent and probably the visualizer as well as it helps avoiding shenanigans related to windows path.

Relative URLs

Absolute urls are not great to visualize. In some projects it makes express absolute urls relatively to a reference absolute url. There is two ways we can go about this:

1. The appmap contains only absolute urls. The reference url could be provided in appmap.yml or using a heuristic such as: if more then half of the urls start with http://localhost:8080:/public then use it as reference url. In any case, it becomes the job of the visualizer to decide how these absolute urls should be displayed and this specification remains not impacted by this consideration.
2. The appmap may contains relative urls. We should then provide the reference url inside the appmap itself. Maybe something like: metadata.home = "http://localhost:8080/public". If this field is missing we can assume that it is the directory containing the appmap.yml file.

[lachrist]

Dynamic Source Content

In the face of dynamic URLs, the visualizer does not know how the fetch the code of functions. There exists the inline field of functions but this is not optimal as it can contains many times the same code. It is has a very bad memory complexity of O(2^n) where n is the maximum nesting level. To be more compact, the content of sources should be stored only once. For instance it could be stored in the classMap at the root element of a file. Or in metadata as a mapping from source url to source content.

NB: Implementing a heuristic to link a dynamic source to a static file is not trivial. And I don't think it is a good idea. For instance the chrome debugger does not use location on disk but URLs as well:

[kgilpin]

These URLs really can't be used to do anything (like open source files) by the AppMap client tools (diagrams, CLI, etc). So I am not sure that it makes sense to even bother reporting them. What we want to do is resolve to an actual file on disk, that the user can open in their code editor or in source control (e.g. link to GitHub).

…

On Tue, Mar 28, 2023 at 12:06 PM Laurent Christophe < ***@***.***> wrote: NB: Chrome debugger does not use location on disk but URLs [image: Screenshot 2023-03-28 at 18 04 36] <https://user-images.githubusercontent.com/5468597/228300936-cd830866-90ae-4642-a58e-47e82a2c2171.png> — Reply to this email directly, view it on GitHub <#38 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAVC65RQI2DIU2FAK4XYUTW6MD7BANCNFSM6AAAAAAWKYJW3E> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[lachrist]

If you were to make me do this. I would hash every file in the user repo. Upon encountering a dynamic source I would check the presence of the content of the source in the hashed repo. If there is a hit I would replace the dynamic url of the source with a static (file) url.

I'm not sure if this is worth it. Maybe @dustinbyrne you could give us your input on whether this strategy would lead to many hits in popular web frameworks. It might be decent in production mode.

[kgilpin]

A location that can't be resolved to a local file in the filesystem is basically useless to the client tools.

…

On Tue, Mar 28, 2023 at 12:33 PM Laurent Christophe < ***@***.***> wrote: If you were to make me do this. I would hash every file in the user repo. Upon encountering a dynamic source I would check the presence of the content of the source in the hashed repo. If there is a hit I would replace the dynamic url of the source with a static (file) url. I'm not sure if this is worth it. Maybe @dustinbyrne <https://github.com/dustinbyrne> you could give us your input on whether this strategy would lead to many hits in popular web frameworks. It might be decent in production mode. — Reply to this email directly, view it on GitHub <#38 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAVC63XE3K5N2HLHHKN2UDW6MHHFANCNFSM6AAAAAAWKYJW3E> . You are receiving this because you commented.Message ID: ***@***.***>