Skip to content

feat: design complete fastn-p2p and fastn-context systems for distributed operations #2202

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
amitu wants to merge 9 commits into from
Closed

feat: design complete fastn-p2p and fastn-context systems for distributed operations #2202

amitu wants to merge 9 commits into from

Conversation

@amitu
Copy link
Contributor

@amitu amitu commented Sep 16, 2025 •
edited
Loading

Summary

Design and implement fastn-p2p streaming API and basic fastn-context integration needed for end-to-end remote shell functionality (rshell/rexec). Focused implementation for immediate remote access needs.

🎯 Focused Implementation Plan

Phase 1: Basic fastn-context (Minimal)

  • Create fastn-context crate with basic Context struct
  • Implement simple context tree (parent/child relationships)
  • Add basic cancellation tokens per context
  • Move #[fastn_p2p::main] to #[fastn_context::main]
  • Basic spawn() with context inheritance
  • Scope: Just enough for fastn-p2p integration, no full monitoring yet

Phase 2: fastn-p2p Streaming API

  • Add client module with connect() function
  • Implement client::Session with stdin/stdout streams
  • Add basic open_uni()/accept_uni() for stderr-like streams
  • Update server::listen() to return Session
  • Add Session -> Request conversion (into_request())
  • Preserve existing Request methods (get_input, handle)
  • Basic context integration (sessions get context)
  • Scope: Just streaming API, reuse existing call() infrastructure

Phase 3: Test and Validate

  • Update fastn-p2p-test sender/receiver to use new API
  • Test both RPC (Session->Request) and streaming patterns
  • Validate iroh connection reuse works correctly
  • Scope: End-to-end P2P streaming validation only

Phase 4: Remote Shell Integration

  • Update fastn-remote to use fastn-p2p streaming API
  • Implement rshell using client::connect() -> Session
  • Implement server-side rshell using Session streams
  • Add stderr handling via open_uni()/accept_uni()
  • Test fastn rshell and fastn rexec commands work
  • Scope: Just remote shell, no other ecosystem changes

🎯 Minimal Breaking Changes

fastn-p2p changes needed:

  • Add client::connect() function (new)
  • Add client::Session type (new)
  • Update server::listen() return type to Session
  • Add Session::into_request() method
  • Keep all existing Request

    methods intact

Not included in this PR:

  • Full fastn-context monitoring system
  • Named locks and deadlock detection
  • Global counter storage
  • System metrics monitoring
  • Status HTTP server
  • Ecosystem-wide migration (only fastn-remote updated)

📋 Success Criteria

  1. ✅ fastn-p2p-test passes with new streaming API
  2. ✅ fastn rshell works - Interactive shell over P2P
  3. ✅ fastn rexec works - Command execution with separate stderr
  4. ✅ No regressions - Existing fastn-p2p call() functionality preserved

Goal: Working remote shell over P2P with minimal API surface for immediate use.

The comprehensive monitoring and ecosystem migration will come in follow-up PRs once the core streaming foundation is proven.

🤖 Generated with Claude Code

amitu and others added 9 commits September 16, 2025 13:50
@amitu @claude
docs: design complete fastn-p2p API with client/server modules and st…
c44249e 
…reaming sessions

- Document three core patterns: client::call(), client::connect(), server::listen()
- Design unified server::Session<PROTOCOL> that handles both RPC and streaming
- Add server::Session -> server::Request conversion with into_request()
- Specify public fields for direct stream access (send, recv, stdin, stdout)
- Include complete API reference with all methods and signatures
- Use fully qualified paths throughout examples (no use statements)
- Add protocol consistency requirements (client and server must use same protocol)
- Clarify stream directions with "back to client" terminology

Key design decisions:
- server::listen() returns Session<PROTOCOL> stream
- Session can convert to Request for RPC or use streams directly
- Client Session accepts streams from server, Server Session opens streams to client
- Public fields eliminate mutability conflicts with accessor methods

Ready for implementation phase after design agreement.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
@amitu @claude
docs: add persistent global counter storage with dotted path keys to …
2676f20 
…fastn-context

- Add global counter storage system that survives context drops
- Design dotted path keys: "global.service.session.task.counter_name"
- Add path tracking to Context structure with full_path field
- Document persistent total counters vs auto-cleanup live counters
- Add counter storage examples with hierarchical path structures
- Ensure historical metrics persist beyond context lifecycles
- Enable hierarchical counter aggregation via path-based queries

Counter examples:
- "global.connections" (app-level)
- "global.remote-access.alice@bv478gen.commands" (session-level)
- "global.http-proxy.requests" (service-level)

Global HashMap storage ensures metrics survive context drops while
maintaining hierarchical structure for status display and aggregation.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
@amitu
docs: add note about iterative design documentation in fastn-context …
c7fb9f6 
…README
@amitu
cleanup: remove WIP fastn-p2p code files, keep only README designs
85ee366 
- Remove incomplete command.rs file that was accidentally committed
- Revert lib.rs to clean state without WIP command exports
- Preserve complete fastn-context README design
- Keep fastn-p2p README design for reference
- Ready to begin implementation phase with clean slate
@amitu @claude
refactor: organize fastn-context documentation and create minimal imp…
decdb72 
…lementation scope

- Move fastn-context to repo root (not v0.5) for ecosystem-wide availability
- Split comprehensive README into focused current vs future features
- Create README.md with minimal Context API needed for fastn-p2p integration
- Split future features into focused NEXT-*.md files:
  - NEXT-monitoring.md: Status trees, timing, system metrics
  - NEXT-locks.md: Named locks and deadlock detection
  - NEXT-counters.md: Global counter storage with dotted paths
  - NEXT-status-distribution.md: P2P status access
  - NEXT-complete-design.md: Full original design reference
- Add workspace integration for fastn-context crate
- Create minimal test example showing required API surface
- Add basic Cargo.toml with minimal dependencies

This focuses implementation on just what's needed for remote shell functionality:
- Basic Context tree structure
- Hierarchical cancellation
- Task spawning with inheritance
- Integration points for fastn-p2p

Comprehensive monitoring features deferred to future PRs after core functionality proven.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
@amitu @claude
docs: restore complete fastn-context README with all essential curren…
5ef6007 
…t-scope content

- Restore Context Tree Structure and Automatic Context Creation sections
- Add back complete Context API with metrics, data storage, and builder pattern
- Restore ContextBuilder fluent API with with_data() and with_metric() methods
- Include detailed main macro configuration options (logging, shutdown modes)
- Add back integration examples showing fastn-p2p usage patterns
- Restore usage patterns for simple vs detailed task spawning
- Include design benefits and debugging rationale
- Reference NEXT-*.md files for future planned features

All important current implementation details restored from original comprehensive design.
No content lost - proper categorization between current API and future enhancements.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
@amitu @claude
refactor: update fastn-context to use explicit context passing instea…
a814f97 
…d of current()

- Remove fastn_context::current() function from API to avoid hidden dependencies
- Update test example to use explicit context passing patterns
- Show two explicit patterns: context cloning and builder pattern with context parameter
- Update README examples to use explicit context instead of current()
- Remove thread-local/task-local context access from design

Explicit context approach benefits:
- Clear dependencies - functions declare context needs explicitly
- Better testability - easy to provide specific context for tests
- No magic behavior - context comes from explicit sources
- Reviewable code - easy to see context usage patterns

Will prove if explicit approach works well or if current() convenience needed later.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
@amitu @claude
feat: add spawn_child() shortcut method to fastn-context API
611e55a 
- Add spawn_child(name, |ctx|) as clean shortcut for common named task pattern
- Update README to show three spawning patterns:
  - spawn() - inherit current context (no child)
  - spawn_child(name, |ctx|) - common case shortcut
  - child(name).with_*().spawn(|ctx|) - full builder pattern
- Update test example to use spawn_child() instead of verbose cloning
- Show both shortcut and builder pattern alternatives in documentation

The spawn_child() shortcut eliminates verbose context cloning while providing
named tasks essential for debugging. Most common pattern for everyday use.

Three-tier API provides appropriate complexity levels:
- Simple inheritance for basic tasks
- Named children for debuggable tasks
- Full builder for complex initialization

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
@amitu @claude
feat: add operation tracking design and clean minimal fastn-context R…
eb135de 
…EADME

- Create NEXT-operation-tracking.md with named await/select macro design
- Design fastn_context::await!() macro for tracking stuck operations
- Design fastn_context::select!() macro for select operation tracking
- Remove all metrics and data storage from current README scope
- Clean up test examples to focus only on context trees and cancellation
- Simplify Context struct to just name and cancellation (no metrics/data)
- Update ContextBuilder to remove with_data/with_metric methods

Current README now focused only on core context functionality:
- Context tree structure with parent/child relationships
- Hierarchical cancellation with wait/cancel
- Three spawning patterns: spawn, spawn_child, child().spawn
- Integration patterns for fastn-p2p

All advanced features properly moved to NEXT-*.md files for future implementation.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
@amitu
Copy link
Contributor Author

amitu commented Sep 17, 2025 •
edited
Loading

We extracted context related changes from this PR and moved to #2203, this PR is quite messy now, closing and starting afresh. New PR: #2205.

@amitu amitu closed this Sep 17, 2025
@amitu amitu deleted the feat/fastn-p2p-command-api branch September 17, 2025 08:41
This was referenced Sep 17, 2025
Merged
Closed
Closed
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@amitu