Support external links in the navigator (#1247)

* External Render Node Support

Created an `ExternalRenderNode` structure that represents a pseudo render node out of an externally resolved value.

This structure contains the minimal information needed to hold a value that can be passed to the Navigator Index.

Fixes rdar://146123573.

Co-authored-by: Sofía Rodríguez <sofia_rodriguez@apple.com>

* External Render Node representation for the navigator

The structures `NavigatorExternalRenderNode` and `ExternalRenderNodeMetadataRepresentation` are meant to hold the information needed to add an external render node in the navigator index.

Each of these contains the information of a external render node for a specific language variant, since things like the node title change depending on the selected variant.

Fields which cannot be populated due to insufficient information have been documented.
These can have varying effect on the resulting navigation node, particularly the navigator title.

Fixes rdar://146123573.

Co-authored-by: Sofía Rodríguez <sofia_rodriguez@apple.com>

* Index external render nodes

This method mimics the index method for render nodes [1] by indexing the main node representation, and the Objective-C representation.

[1] https://github.com/swiftlang/swift-docc/blob/38ea39df14d0193f52900dbe54b7ae2be0abd856/Sources/SwiftDocC/Indexing/Navigator/NavigatorIndex.swift\#L636

Fixes rdar://146123573.

Co-authored-by: Sofía Rodríguez <sofia_rodriguez@apple.com>

* Consume external render nodes

During the convert process create the external render nodes from the context external cache and consume them to get them added into the navigator index.

Fixes rdar://146123573.

Co-authored-by: Sofía Rodríguez <sofia_rodriguez@apple.com>

* Only add external nodes to index if manually curated in Topics section

The NavigationIndex code has some logic about what to do with fallouts (i.e. nodes which are not curated anywhere). It places them at the top level of the navigator, so that they are not lost.

This works well for render nodes, but for external render nodes, we don't want them to persist at the top level if they are not referenced anywhere. We'd like them to be excluded if not referenced.

This can happen due to the fact that we consume **all** successfully resolved external entities for indexing, which includes external references in other sections of the page such as the Overview section, the abstract and the See also section.

Fixes rdar://146123573.

* Propagate isExternal value to RenderIndex

Now that the `isExternal` property is populated as part of the `NavigatorItem`s, we can now use that value to propagate whether the node `isExternal` in the final `RenderIndex`, something which we couldn't support before.

External nodes are now marked as "external": true in `index.json` as expected.

Fixes rdar://146123573.

---------

Co-authored-by: Sofía Rodríguez <sofia_rodriguez@apple.com>

Author: anferbui

Sources/SwiftDocC/Indexing/Navigator/NavigatorIndex.swift

@@ -630,6 +630,28 @@ extension NavigatorIndex {
             }
         }
 
+        /// Index a single render `ExternalRenderNode`.
+        /// - Parameter renderNode: The render node to be indexed.
+        package func index(renderNode: ExternalRenderNode, ignoringLanguage: Bool = false) throws {
+            let navigatorRenderNode = NavigatorExternalRenderNode(renderNode: renderNode)
+            _ = try index(navigatorRenderNode, traits: nil, isExternal: true)
+            guard renderNode.identifier.sourceLanguage != .objectiveC else {
+                return
+            }
+            // Check if the render node has an Objective-C representation
+            guard let objCVariantTrait = renderNode.variants?.flatMap(\.traits).first(where: { trait in
+                switch trait {
+                case .interfaceLanguage(let language):
+                    return InterfaceLanguage.from(string: language) == .objc
+                }
+            }) else {
+                return
+            }
+            // If this external render node has a variant, we create a "view" into its Objective-C specific data and index that.
+            let objVariantView = NavigatorExternalRenderNode(renderNode: renderNode, trait: objCVariantTrait)
+            _ = try index(objVariantView, traits: [objCVariantTrait], isExternal: true)
+        }
+        
         /// Index a single render `RenderNode`.
         /// - Parameter renderNode: The render node to be indexed.
         /// - Parameter ignoringLanguage: Whether language variants should be ignored when indexing this render node.
@@ -684,7 +706,7 @@ extension NavigatorIndex {
         }
 
         // The private index implementation which indexes a given render node representation
-        private func index(_ renderNode: any NavigatorIndexableRenderNodeRepresentation, traits: [RenderNode.Variant.Trait]?) throws -> InterfaceLanguage? {
+        private func index(_ renderNode: any NavigatorIndexableRenderNodeRepresentation, traits: [RenderNode.Variant.Trait]?, isExternal external: Bool = false) throws -> InterfaceLanguage? {
             guard let navigatorIndex else {
                 throw Error.navigatorIndexIsNil
             }
@@ -781,7 +803,8 @@ extension NavigatorIndex {
                 title: title,
                 platformMask: platformID,
                 availabilityID: UInt64(availabilityID),
-                icon: renderNode.icon
+                icon: renderNode.icon,
+                isExternal: external
             )
             navigationItem.path = identifierPath
 
@@ -812,7 +835,8 @@ extension NavigatorIndex {
                         languageID: language.mask,
                         title: title,
                         platformMask: platformID,
-                        availabilityID: UInt64(Self.availabilityIDWithNoAvailabilities)
+                        availabilityID: UInt64(Self.availabilityIDWithNoAvailabilities),
+                        isExternal: external
                     )
 
                     groupItem.path = identifier.path + "#" + fragment
@@ -976,7 +1000,8 @@ extension NavigatorIndex {
                 // curation, then they should not be in the navigator. In addition, treat unknown
                 // page types as symbol nodes on the assumption that an unknown page type is a
                 // symbol kind added in a future version of Swift-DocC.
-                if let node = identifierToNode[nodeID], PageType(rawValue: node.item.pageType)?.isSymbolKind == false {
+                // Finally, don't add external references to the root; if they are not referenced within the navigation tree, they should be dropped altogether.
+                if let node = identifierToNode[nodeID], PageType(rawValue: node.item.pageType)?.isSymbolKind == false , !node.item.isExternal {
 
                     // If an uncurated page has been curated in another language, don't add it to the top-level.
                     if curatedReferences.contains(where: { curatedNodeID in

Sources/SwiftDocC/Indexing/Navigator/NavigatorItem.swift

@@ -49,6 +49,11 @@ public final class NavigatorItem: Serializable, Codable, Equatable, CustomString
 
     var icon: RenderReferenceIdentifier? = nil
 
+    /// Whether the item has originated from an external reference.
+    ///
+    /// Used for determining whether stray navigation items should remain part of the final navigator.
+    var isExternal: Bool = false
+    
     /**
      Initialize a `NavigatorItem` with the given data.
 
@@ -61,14 +66,15 @@ public final class NavigatorItem: Serializable, Codable, Equatable, CustomString
         - path: The path to load the content.
         - icon: A reference to a custom image for this navigator item.
      */
-    init(pageType: UInt8, languageID: UInt8, title: String, platformMask: UInt64, availabilityID: UInt64, path: String, icon: RenderReferenceIdentifier? = nil) {
+    init(pageType: UInt8, languageID: UInt8, title: String, platformMask: UInt64, availabilityID: UInt64, path: String, icon: RenderReferenceIdentifier? = nil, isExternal: Bool = false) {
         self.pageType = pageType
         self.languageID = languageID
         self.title = title
         self.platformMask = platformMask
         self.availabilityID = availabilityID
         self.path = path
         self.icon = icon
+        self.isExternal = isExternal
     }
 
     /**
@@ -82,13 +88,14 @@ public final class NavigatorItem: Serializable, Codable, Equatable, CustomString
         - availabilityID:  The identifier of the availability information of the page.
         - icon: A reference to a custom image for this navigator item.
      */
-    public init(pageType: UInt8, languageID: UInt8, title: String, platformMask: UInt64, availabilityID: UInt64, icon: RenderReferenceIdentifier? = nil) {
+    public init(pageType: UInt8, languageID: UInt8, title: String, platformMask: UInt64, availabilityID: UInt64, icon: RenderReferenceIdentifier? = nil, isExternal: Bool = false) {
         self.pageType = pageType
         self.languageID = languageID
         self.title = title
         self.platformMask = platformMask
         self.availabilityID = availabilityID
         self.icon = icon
+        self.isExternal = isExternal
     }
 
     // MARK: - Serialization and Deserialization

Sources/SwiftDocC/Indexing/RenderIndexJSON/RenderIndex.swift

@@ -86,7 +86,15 @@ public struct RenderIndex: Codable, Equatable {
     /// - Parameter named: The name of the new root node
     public mutating func insertRoot(named: String) {
         for (languageID, nodes) in interfaceLanguages {
-            let root = Node(title: named, path: "/documentation", pageType: .framework, isDeprecated: false, children: nodes, icon: nil)
+            let root = Node(
+                title: named,
+                path: "/documentation",
+                pageType: .framework,
+                isDeprecated: false,
+                isExternal: false,
+                children: nodes,
+                icon: nil
+            )
             interfaceLanguages[languageID] = [root]
         }
     }
@@ -236,18 +244,18 @@ extension RenderIndex {
             path: String,
             pageType: NavigatorIndex.PageType?,
             isDeprecated: Bool,
+            isExternal: Bool,
             children: [Node],
             icon: RenderReferenceIdentifier?
         ) {
             self.title = title
             self.children = children.isEmpty ? nil : children
 
             self.isDeprecated = isDeprecated
+            self.isExternal = isExternal
 
-            // Currently Swift-DocC doesn't support resolving links to external DocC archives
+            // Currently Swift-DocC doesn't support marking a node as beta in the navigation index
             // so we default to `false` here.
-            self.isExternal = false
-            
             self.isBeta = false
             self.icon = icon
 
@@ -318,6 +326,7 @@ extension RenderIndex.Node {
             path: node.item.path,
             pageType: NavigatorIndex.PageType(rawValue: node.item.pageType),
             isDeprecated: isDeprecated,
+            isExternal: node.item.isExternal,
             children: node.children.map {
                 RenderIndex.Node.fromNavigatorTreeNode($0, in: navigatorIndex, with: builder)
             },

Sources/SwiftDocC/Infrastructure/ConvertActionConverter.swift

@@ -34,7 +34,7 @@ package enum ConvertActionConverter {
     package static func convert(
         bundle: DocumentationBundle,
         context: DocumentationContext,
-        outputConsumer: some ConvertOutputConsumer,
+        outputConsumer: some ConvertOutputConsumer & ExternalNodeConsumer,
         sourceRepository: SourceRepository?,
         emitDigest: Bool,
         documentationCoverageOptions: DocumentationCoverageOptions
@@ -104,6 +104,15 @@ package enum ConvertActionConverter {
         let resultsSyncQueue = DispatchQueue(label: "Convert Serial Queue", qos: .unspecified, attributes: [])
         let resultsGroup = DispatchGroup()
 
+        // Consume external links and add them into the sidebar.
+        for externalLink in context.externalCache {
+            // Here we're associating the external node with the **current** bundle's bundle ID.
+            // This is needed because nodes are only considered children if the parent and child's bundle ID match.
+            // Otherwise, the node will be considered as a separate root node and displayed separately.
+            let externalRenderNode = ExternalRenderNode(externalEntity: externalLink.value, bundleIdentifier: bundle.id)
+            try outputConsumer.consume(externalRenderNode: externalRenderNode)
+        }
+        
         let renderSignpostHandle = signposter.beginInterval("Render", id: signposter.makeSignpostID(), "Render \(context.knownPages.count) pages")
 
         var conversionProblems: [Problem] = context.knownPages.concurrentPerform { identifier, results in

Sources/SwiftDocC/Infrastructure/ConvertOutputConsumer.swift

@@ -105,3 +105,12 @@ package struct _Deprecated<Consumer: ConvertOutputConsumer>: _DeprecatedConsumeP
         try consumer.consume(problems: problems)
     }
 }
+
+/// A consumer for nodes generated from external references.
+///
+/// Types that conform to this protocol manage what to do with external references, for example index them.
+package protocol ExternalNodeConsumer {
+    /// Consumes a external render node that was generated during a conversion.
+    /// > Warning: This method might be called concurrently.
+    func consume(externalRenderNode: ExternalRenderNode) throws
+}

Sources/SwiftDocC/Infrastructure/Link Resolution/LinkResolver+NavigatorIndex.swift

@@ -0,0 +1,148 @@
+/*
+ This source file is part of the Swift.org open source project
+
+ Copyright (c) 2025 Apple Inc. and the Swift project authors
+ Licensed under Apache License v2.0 with Runtime Library Exception
+
+ See https://swift.org/LICENSE.txt for license information
+ See https://swift.org/CONTRIBUTORS.txt for Swift project authors
+*/
+
+import Foundation
+import SymbolKit
+
+/// A rendering-friendly representation of a external node.
+package struct ExternalRenderNode {
+    /// Underlying external entity backing this external node.
+    private var externalEntity: LinkResolver.ExternalEntity
+
+    /// The bundle identifier for this external node.
+    private var bundleIdentifier: DocumentationBundle.Identifier
+
+    init(externalEntity: LinkResolver.ExternalEntity, bundleIdentifier: DocumentationBundle.Identifier) {
+        self.externalEntity = externalEntity
+        self.bundleIdentifier = bundleIdentifier
+    }
+    
+    /// The identifier of the external render node.
+    package var identifier: ResolvedTopicReference {
+        ResolvedTopicReference(
+            bundleID: bundleIdentifier,
+            path: externalEntity.topicRenderReference.url,
+            sourceLanguages: externalEntity.sourceLanguages
+        )
+    }
+
+    /// The kind of this documentation node.
+    var kind: RenderNode.Kind {
+        externalEntity.topicRenderReference.kind
+    }
+    
+    /// The symbol kind of this documentation node.
+    var symbolKind: SymbolGraph.Symbol.KindIdentifier? {
+        // Symbol kind information is not available for external entities
+        return nil
+    }
+    
+    /// The additional "role" assigned to the symbol, if any
+    ///
+    /// This value is `nil` if the referenced page is not a symbol.
+    var role: String? {
+        externalEntity.topicRenderReference.role
+    }
+    
+    /// The variants of the title.
+    var titleVariants: VariantCollection<String> {
+        externalEntity.topicRenderReference.titleVariants
+    }
+    
+    /// The variants of the abbreviated declaration of the symbol to display in navigation.
+    var navigatorTitleVariants: VariantCollection<[DeclarationRenderSection.Token]?> {
+        externalEntity.topicRenderReference.navigatorTitleVariants
+    }
+    
+    /// The variants of the abbreviated declaration of the symbol to display in links.
+    var fragmentsVariants: VariantCollection<[DeclarationRenderSection.Token]?> {
+        externalEntity.topicRenderReference.fragmentsVariants
+    }
+    
+    /// Author provided images that represent this page.
+    var images: [TopicImage] {
+        externalEntity.topicRenderReference.images
+    }
+
+    /// The identifier of the external reference.
+    var externalIdentifier: RenderReferenceIdentifier {
+        externalEntity.topicRenderReference.identifier
+    }
+
+    /// List of variants of the same external node for various languages.
+    var variants: [RenderNode.Variant]? {
+        externalEntity.sourceLanguages.map {
+            RenderNode.Variant(traits: [.interfaceLanguage($0.id)], paths: [externalEntity.topicRenderReference.url])
+        }
+    }
+}
+
+/// A language specific representation of an external render node value for building a navigator index.
+struct NavigatorExternalRenderNode: NavigatorIndexableRenderNodeRepresentation {
+    var identifier: ResolvedTopicReference
+    var externalIdentifier: RenderReferenceIdentifier
+    var kind: RenderNode.Kind
+    var metadata: ExternalRenderNodeMetadataRepresentation
+    
+    // Values that don't affect how the node is rendered in the sidebar.
+    // These are needed to conform to the navigator indexable protocol.
+    var references: [String : any RenderReference] = [:]
+    var sections: [any RenderSection] = []
+    var topicSections: [TaskGroupRenderSection] = []
+    var defaultImplementationsSections: [TaskGroupRenderSection] = []
+    
+    init(renderNode: ExternalRenderNode, trait: RenderNode.Variant.Trait? = nil) {
+        // Compute the source language of the node based on the trait to know which variant to apply.
+        let traitLanguage = if case .interfaceLanguage(let id) = trait {
+            SourceLanguage(id: id)
+        } else {
+            renderNode.identifier.sourceLanguage
+        }
+        let traits = trait.map { [$0] } ?? []
+
+        self.identifier = renderNode.identifier.withSourceLanguages(Set(arrayLiteral: traitLanguage))
+        self.kind = renderNode.kind
+        self.externalIdentifier = renderNode.externalIdentifier
+        
+        self.metadata = ExternalRenderNodeMetadataRepresentation(
+            title: renderNode.titleVariants.value(for: traits),
+            navigatorTitle: renderNode.navigatorTitleVariants.value(for: traits),
+            externalID: renderNode.externalIdentifier.identifier,
+            role: renderNode.role,
+            symbolKind: renderNode.symbolKind?.identifier,
+            images: renderNode.images
+        )
+    }
+}
+
+/// A language specific representation of a render metadata value for building an external navigator index.
+struct ExternalRenderNodeMetadataRepresentation: NavigatorIndexableRenderMetadataRepresentation {
+    var title: String?
+    var navigatorTitle: [DeclarationRenderSection.Token]?
+    var externalID: String?
+    var role: String?
+    var symbolKind: String?
+    var images: [TopicImage]
+
+    // Values that we have insufficient information to derive.
+    // These are needed to conform to the navigator indexable metadata protocol.
+    //
+    // The fragments that we get as part of the external link are the full declaration fragments.
+    // These are too verbose for the navigator, so instead of using them, we rely on the title, navigator title and symbol kind instead.
+    //
+    // The role heading is used to identify Property Lists.
+    // The value being missing is used for computing the final navigator title.
+    //
+    // The platforms are used for generating the availability index,
+    // but doesn't affect how the node is rendered in the sidebar.
+    var fragments: [DeclarationRenderSection.Token]? = nil
+    var roleHeading: String? = nil
+    var platforms: [AvailabilityRenderItem]? = nil
+}

Sources/SwiftDocCUtilities/Action/Actions/Convert/ConvertFileWritingConsumer.swift

@@ -11,7 +11,7 @@
 import Foundation
 import SwiftDocC
 
-struct ConvertFileWritingConsumer: ConvertOutputConsumer {
+struct ConvertFileWritingConsumer: ConvertOutputConsumer, ExternalNodeConsumer {
     var targetFolder: URL
     var bundleRootFolder: URL?
     var fileManager: any FileManagerProtocol
@@ -68,6 +68,11 @@ struct ConvertFileWritingConsumer: ConvertOutputConsumer {
         indexer?.index(renderNode)
     }
 
+    func consume(externalRenderNode: ExternalRenderNode) throws {
+        // Index the external node, if indexing is enabled.
+        indexer?.index(externalRenderNode)
+    }
+    
     func consume(assetsInBundle bundle: DocumentationBundle) throws {
         func copyAsset(_ asset: DataAsset, to destinationFolder: URL) throws {
             for sourceURL in asset.variants.values where !sourceURL.isAbsoluteWebURL {

Sources/SwiftDocCUtilities/Action/Actions/Convert/Indexer.swift

@@ -62,6 +62,22 @@ extension ConvertAction {
             })
         }
 
+        /// Indexes the given external render node and collects any encountered problems.
+        /// - Parameter renderNode: A ``ExternalRenderNode`` value.
+        func index(_ renderNode: ExternalRenderNode) {
+            // Synchronously index the render node.
+            indexBuilder.sync({
+                do {
+                    try $0.index(renderNode: renderNode)
+                    nodeCount += 1
+                } catch {
+                    self.problems.append(error.problem(source: renderNode.identifier.url,
+                                                  severity: .warning,
+                                                  summaryPrefix: "External render node indexing process failed"))
+                }
+            })
+        }
+        
         /// Finalizes the index and writes it on disk.
         /// - Returns: Returns a list of problems if any were encountered during indexing.
         func finalize(emitJSON: Bool, emitLMDB: Bool) -> [Problem] {