diff --git a/README.md b/README.md index 0256804..20754ad 100644 --- a/README.md +++ b/README.md @@ -116,13 +116,20 @@ If you prefer to set up the benchmarking tool manually, follow these detailed st ``` 4. **Point miners to the following endpoints** - - For Stratum V1: + - For Stratum V1 (Public Pool): ```bash stratum+tcp://:3333 ``` 🚨 For SV1, you should use the address format `[bitcoin_address].[nickname]` as the username in your miner setup. E.g. to correctly run a CPU miner, you need to run it with: `./minerd -a sha256d -o stratum+tcp://127.0.0.1:3333 -q -D -P -u tb1qa0sm0hxzj0x25rh8gw5xlzwlsfvvyz8u96w3p8.sv2-gitgab19` + - For Stratum V1 (Solo-CKPool): + ```bash + stratum+tcp://:3335 + ``` + 🚨 For Solo-CKPool, the username MUST be a valid Bitcoin address for the target network. + E.g. to correctly run a CPU miner, you need to run it with: `./minerd -a sha256d -o stratum+tcp://127.0.0.1:3335 -q -D -P -u tb1qa0sm0hxzj0x25rh8gw5xlzwlsfvvyz8u96w3p8` + - For Stratum V2: ```bash stratum+tcp://:34255 diff --git a/SOLO_CKPOOL_MODIFICATIONS.md b/SOLO_CKPOOL_MODIFICATIONS.md new file mode 100644 index 0000000..69be0d5 --- /dev/null +++ b/SOLO_CKPOOL_MODIFICATIONS.md @@ -0,0 +1,287 @@ +# Solo-CKPool Integration for Benchmarking Tool + +This document describes the implementation of GitHub issue #23: integrating solo-ckpool as an SV1 pool option for optimization measurements and comparisons. + +## Overview + +Solo-ckpool has been successfully integrated into the benchmarking tool to provide additional SV1 pool comparison capabilities. This integration enables performance comparisons between: + +- **public-pool** (Node.js-based) vs **solo-ckpool** (C-based) +- **Pool-based solo mining** vs **True solo mining** +- **Different SV1 implementations** under identical conditions + +## Architecture + +### Network Layout + +``` +┌─────────────────┬─────────────────┬─────────────────┐ +│ Public Pool │ Solo-CKPool │ SV2 │ +│ (Node.js) │ (C) │ (Various) │ +├─────────────────┼─────────────────┼─────────────────┤ +│ 10.5.0.8:3332 │ 10.5.0.24:3333 │ 10.5.0.4:34254 │ +│ ↕ │ ↕ │ ↕ │ +│ Pool-Miner │ Solo-CKPool │ SV2 Proxies │ +│ Proxy │ Proxy │ │ +│ 10.5.0.19:3333 │ 10.5.0.25:3335 │ Multiple ports │ +│ ↕ │ ↕ │ ↕ │ +│ Miners │ Miners │ Miners │ +└─────────────────┴─────────────────┴─────────────────┘ + ↕ + Bitcoin Node (10.5.0.16) + via proxy (10.5.0.21) +``` + +### Integration Components + +1. **solo-ckpool.dockerfile**: Builds solo-ckpool from GitHub source (https://github.com/xyephy/solo-ckpool) +2. **solo-ckpool service**: Runs ckpool in BTCSOLO mode (-B flag) +3. **solo-ckpool-miner-proxy**: Monitors traffic and collects metrics +4. **Configuration files**: Pool settings and Bitcoin daemon connection +5. **Prometheus integration**: Metrics collection and monitoring + +## Configuration + +### Pool Configuration (`custom-configs/solo-ckpool/ckpool.conf`) + +```json +{ + "btcd": [{ + "url": "10.5.0.21:48330", + "auth": "username", + "pass": "password", + "notify": true + }], + "mindiff": 1, + "startdiff": 42, + "maxdiff": 0, + "serverurl": ["0.0.0.0:3333"], + "logdir": "/var/log/ckpool", + "sockdir": "/var/log/ckpool", + "loginterval": 60, + "update_interval": 30, + "zmqblock": "tcp://10.5.0.16:28332" +} +``` + +### Network Endpoints + +- **Solo-CKPool Stratum**: `stratum+tcp://:3335` +- **Public Pool Stratum**: `stratum+tcp://:3333` +- **SV2 Stratum**: `stratum+tcp://:34255` + +### Miner Setup + +```bash +# Solo-CKPool (requires valid Bitcoin address as username) +./minerd -a sha256d -o stratum+tcp://127.0.0.1:3335 \ + -u tb1qa0sm0hxzj0x25rh8gw5xlzwlsfvvyz8u96w3p8.solo-ckpool -q -D -P + +# Public Pool (standard username format) +./minerd -a sha256d -o stratum+tcp://127.0.0.1:3333 \ + -u tb1qa0sm0hxzj0x25rh8gw5xlzwlsfvvyz8u96w3p8.public-pool -q -D -P +``` + +## Key Features + +### Solo Mining Capabilities + +- **True Solo Mining**: Each miner gets individual coinbase transactions +- **Address Validation**: Usernames must be valid Bitcoin addresses +- **Direct Block Rewards**: 100% of block rewards go to solving miner +- **No Pool Accounting**: Simplified architecture without share tracking + +### Performance Characteristics + +- **C Implementation**: Optimized low-level performance +- **Memory Efficient**: Minimal RAM usage compared to Node.js pools +- **High Concurrency**: Handles large numbers of connections efficiently +- **Hardware Acceleration**: Supports optimized SHA256 (AVX2/SSE4) + +### Monitoring Integration + +All existing SV1 metrics are collected: + +- **Share Metrics**: Submitted, valid, stale shares and acceptance rates +- **Latency Metrics**: Job delivery times and block propagation +- **Bandwidth Metrics**: Farm and pool level network usage +- **Resource Metrics**: CPU, memory usage via cadvisor +- **Block Metrics**: Template values and blocks found + +## Enhanced Features + +### Fractional Difficulty Support ✅ + +**NEW**: Solo-ckpool now supports fractional difficulty values (including values < 1.0). + +**Enhanced Capabilities**: +- ✅ Support for fractional difficulty values (e.g., 0.1, 0.5, 2.5) +- ✅ Backward compatibility with integer configurations +- ✅ Suitable for low-hashrate testing scenarios +- ✅ Improved benchmarking accuracy for performance comparisons + +**Configuration Examples**: +```json +{ + "mindiff": 0.1, // Minimum difficulty of 0.1 (fractional) + "startdiff": 0.5, // Starting difficulty of 0.5 (fractional) + "maxdiff": 100.0, // Maximum difficulty of 100.0 + "shares_per_minute": 15.0 // Target 15 shares per minute for consistent benchmarking +} +``` + +**Backward Compatibility**: +- Integer values still work: `"mindiff": 1, "startdiff": 42` +- Mixed configurations supported: `"mindiff": 0.5, "startdiff": 10` +- Default shares_per_minute: 18.0 (maintains existing behavior if not specified) + +### Shares Per Minute Control ✅ + +**NEW**: Solo-ckpool now supports configurable share submission rates for standardized benchmarking. + +**Key Benefits**: +- ✅ **Consistent benchmarking**: Set identical share rates for SV1 and SV2 comparisons +- ✅ **Flexible testing scenarios**: Easily adjust load characteristics +- ✅ **Real-time adjustment**: Variable difficulty adapts to maintain target rate +- ✅ **Backward compatibility**: Default 18.0 spm maintains existing behavior + +**Configuration Examples for Different Scenarios**: +```json +// Low-latency testing +{"shares_per_minute": 5.0} // 1 share every 12 seconds + +// Balanced benchmarking +{"shares_per_minute": 15.0} // 1 share every 4 seconds + +// High-throughput testing +{"shares_per_minute": 30.0} // 1 share every 2 seconds +``` + +**Algorithm Enhancement**: +- Dynamic difficulty adjustment based on `shares_per_minute / 60.0` target rate +- Proportional hysteresis bounds to prevent oscillation +- Configurable timing thresholds based on target share rate + +## Limitations and Considerations + +### Solo Mining Requirements + +- **Valid Addresses**: Miner usernames must be valid Bitcoin addresses for the target network +- **Network Sync**: Requires fully synced Bitcoin node for optimal performance +- **Block Notifications**: Needs ZMQ or blocknotify for real-time updates + +## Usage Instructions + +### Starting the Benchmarking Tool + +```bash +# Configuration A (with local JDC) +docker compose -f docker-compose-config-a.yaml up -d + +# Configuration C (pool-managed) +docker compose -f docker-compose-config-c.yaml up -d +``` + +### Connecting Miners + +Connect miners to different endpoints for comparison: + +```bash +# Terminal 1: Solo-CKPool +./minerd -a sha256d -o stratum+tcp://127.0.0.1:3335 \ + -u .solo-ckpool -q -D -P + +# Terminal 2: Public Pool +./minerd -a sha256d -o stratum+tcp://127.0.0.1:3333 \ + -u .public-pool -q -D -P + +# Terminal 3: SV2 (via translator) +./minerd -a sha256d -o stratum+tcp://127.0.0.1:34255 -q -D -P +``` + +### Monitoring Results + +Access Grafana dashboard: `http://localhost:3000/d/64nrElFmk/sri-benchmarking-tool` + +The dashboard will show comparative metrics between: +- Public Pool vs Solo-CKPool performance +- SV1 vs SV2 protocol efficiency +- Resource usage and network characteristics + +## Expected Performance Differences + +### Solo-CKPool Advantages + +1. **Lower Memory Usage**: C implementation more memory efficient +2. **Higher Connection Capacity**: Better handling of concurrent miners +3. **Faster Processing**: Optimized protocol handling and validation +4. **Authentic Solo Mining**: True individual coinbase generation +5. **✅ NEW: Fractional Difficulty**: Support for difficulty < 1.0 with enhanced precision + +### Public Pool Advantages + +1. **Feature Rich**: More configuration options and pool features +2. **Detailed Logging**: More comprehensive debugging information +3. **Flexibility**: Easier to modify and customize +4. **Web Interface**: Built-in monitoring and management tools + +## Troubleshooting + +### Common Issues + +1. **Address Validation Errors** + - Ensure usernames are valid Bitcoin addresses for target network + - Use correct address format (testnet vs mainnet) + +2. **Connection Failures** + - Check that solo-ckpool service is running: `docker logs solo-ckpool` + - Verify Bitcoin node connectivity: `docker logs sv1-node-pool-side` + +3. **Metrics Not Appearing** + - Confirm proxy is running: `docker logs solo-ckpool-miner-proxy` + - Check Prometheus targets: `http://localhost:9090/targets` + +### Log Analysis + +```bash +# Solo-CKPool logs +docker logs solo-ckpool + +# Proxy metrics logs +docker logs solo-ckpool-miner-proxy + +# Bitcoin node logs +docker logs sv1-node-pool-side +``` + +## Future Enhancements + +### Planned Improvements + +1. **Fractional Difficulty Support**: Wrapper layer for difficulty scaling +2. **Enhanced Metrics**: Solo-mining specific performance indicators +3. **Configuration Profiles**: Different solo mining scenarios +4. **Multi-Pool Comparison**: Side-by-side analysis of multiple pools + +### Integration Opportunities + +- **ASIC Testing**: Real hardware performance comparisons +- **Mainnet Benchmarks**: Production environment analysis +- **Scalability Tests**: Large-scale miner farm simulations +- **Network Analysis**: Latency impact studies across different pools + +## Contributing + +To modify or extend the solo-ckpool integration: + +1. **Configuration Changes**: Edit `custom-configs/solo-ckpool/ckpool.conf` +2. **Docker Modifications**: Update `solo-ckpool.dockerfile` +3. **Service Changes**: Modify docker-compose files +4. **Metrics Addition**: Extend Prometheus configuration + +## References + +- [Solo-CKPool Repository](https://bitbucket.org/ckolivas/ckpool-solo) +- [Benchmarking Tool Documentation](./docs/benchmarking-tool-overview.md) +- [SV1 Protocol Specification](https://braiins.com/stratum-v1/docs) +- [GitHub Issue #23](https://github.com/stratum-mining/benchmarking-tool/issues/23) \ No newline at end of file diff --git a/custom-configs/solo-ckpool/README.md b/custom-configs/solo-ckpool/README.md new file mode 100644 index 0000000..25d53b0 --- /dev/null +++ b/custom-configs/solo-ckpool/README.md @@ -0,0 +1,57 @@ +# Solo-CKPool Configuration + +This directory contains configuration files for integrating solo-ckpool into the benchmarking tool. + +## Files + +- `ckpool.conf` - Main solo-ckpool configuration file +- `README.md` - This documentation file + +## Configuration Parameters + +### Bitcoin Daemon Connection +- **URL**: `10.5.0.21:48330` - Connects through sv1-node-pool-proxy for metrics collection +- **Auth**: Basic authentication credentials (username/password) +- **Notify**: Enables block notifications from Bitcoin daemon + +### Difficulty Settings +- **mindiff**: 1 - Minimum difficulty (solo-ckpool limitation: integers only) +- **startdiff**: 42 - Starting difficulty for new miners +- **maxdiff**: 0 - Maximum difficulty (0 = unlimited) + +### Network Settings +- **serverurl**: `0.0.0.0:3333` - Stratum server binding +- **zmqblock**: `tcp://10.5.0.16:28332` - ZMQ block notification endpoint + +### Logging +- **logdir**: `/var/log/ckpool` - Log file directory +- **sockdir**: `/var/log/ckpool` - Unix socket directory +- **loginterval**: 60 - Log interval in seconds + +## Usage Notes + +1. **Solo Mining Mode**: Pool runs with `-B` flag for BTCSOLO mode +2. **Address Validation**: Miner usernames must be valid Bitcoin addresses +3. **Fractional Difficulty**: Not supported - minimum difficulty is 1 +4. **Network Integration**: Uses existing Bitcoin node through proxy for monitoring + +## Testing + +Connect miners using: +```bash +# For testnet +./minerd -a sha256d -o stratum+tcp://:3335 -u .solo-ckpool -q -D -P + +# Example with valid testnet address +./minerd -a sha256d -o stratum+tcp://127.0.0.1:3335 -u tb1qa0sm0hxzj0x25rh8gw5xlzwlsfvvyz8u96w3p8.solo-ckpool -q -D -P +``` + +## Metrics Integration + +Solo-ckpool integrates with the existing SV1 metrics collection system: +- Share submissions, valid/stale shares +- Block template values and propagation times +- Network bandwidth usage (farm and pool level) +- Container resource usage (CPU, memory) + +All metrics are collected through the `solo-ckpool-miner-proxy` component. \ No newline at end of file diff --git a/custom-configs/solo-ckpool/ckpool.conf b/custom-configs/solo-ckpool/ckpool.conf new file mode 100644 index 0000000..2abc8d6 --- /dev/null +++ b/custom-configs/solo-ckpool/ckpool.conf @@ -0,0 +1,19 @@ +{ + "btcd": [{ + "url": "10.5.0.21:48330", + "auth": "username", + "pass": "password", + "notify": true + }], + "mindiff": 0.1, + "startdiff": 0.5, + "maxdiff": 0.0, + "shares_per_minute": 15.0, + "btcaddress": "tb1qa0sm0hxzj0x25rh8gw5xlzwlsfvvyz8u96w3p8", + "serverurl": ["0.0.0.0:3333"], + "logdir": "/var/log/ckpool", + "sockdir": "/var/log/ckpool", + "loginterval": 60, + "update_interval": 30, + "zmqblock": "tcp://10.5.0.16:28332" +} \ No newline at end of file diff --git a/docker-compose-config-a.yaml b/docker-compose-config-a.yaml index ad134a7..8d16f9d 100644 --- a/docker-compose-config-a.yaml +++ b/docker-compose-config-a.yaml @@ -58,6 +58,13 @@ services: image: pools-latency-calculator-builder-image command: echo "pools-latency-calculator build completed" + solo-ckpool-builder: + build: + dockerfile: ./solo-ckpool.dockerfile + container_name: solo-ckpool-builder + image: solo-ckpool-builder-image + command: echo "solo-ckpool build completed" + template-provider-pool-side: labels: logging: "config-a" @@ -408,6 +415,63 @@ services: sv2-net: ipv4_address: 10.5.0.21 + solo-ckpool: + image: solo-ckpool-builder-image + container_name: solo-ckpool + labels: + logging: "config-a" + command: + [ + "/bin/sh", + "-c", + "/usr/local/bin/start-ckpool.sh", + ] + ports: + - "3336:3333" + environment: + - NETWORK=${NETWORK} + volumes: + - "./custom-configs/solo-ckpool/ckpool.conf:/etc/ckpool/ckpool.conf:ro" + restart: unless-stopped + depends_on: + - solo-ckpool-builder + - sv1-node-pool-proxy + networks: + sv2-net: + ipv4_address: 10.5.0.24 + cap_add: + - NET_ADMIN + + solo-ckpool-miner-proxy: + image: sv1-custom-proxy-builder-image + labels: + logging: "config-a" + command: + [ + "/bin/sh", + "-c", + "./monitor_and_apply_latency.sh 10.5.0.24 2 & exec ./sv1-custom-proxy", + ] + ports: + - "3335:3335" + - "2346:2346" + environment: + - SERVER=10.5.0.24:3333 + - CLIENT=0.0.0.0:3335 + - PROM_ADDRESS=10.5.0.25:2346 + - PROXY_TYPE=pool-miner + - RUST_LOG=${LOG_LEVEL} + container_name: solo-ckpool-miner-proxy + depends_on: + - sv1-custom-proxy-builder + - solo-ckpool + restart: unless-stopped + networks: + sv2-net: + ipv4_address: 10.5.0.25 + cap_add: + - NET_ADMIN + monitor-traffic-tcpdump: image: inzania/network-traffic-metrics:latest network_mode: host diff --git a/docker-compose-config-c.yaml b/docker-compose-config-c.yaml index 059b113..84f998c 100644 --- a/docker-compose-config-c.yaml +++ b/docker-compose-config-c.yaml @@ -57,6 +57,13 @@ services: image: pools-latency-calculator-builder-image command: echo "pools-latency-calculator build completed" + solo-ckpool-builder: + build: + dockerfile: ./solo-ckpool.dockerfile + container_name: solo-ckpool-builder + image: solo-ckpool-builder-image + command: echo "solo-ckpool build completed" + template-provider-pool-side: labels: logging: "config-c" @@ -333,6 +340,63 @@ services: sv2-net: ipv4_address: 10.5.0.21 + solo-ckpool: + image: solo-ckpool-builder-image + container_name: solo-ckpool + labels: + logging: "config-c" + command: + [ + "/bin/sh", + "-c", + "/usr/local/bin/start-ckpool.sh", + ] + ports: + - "3336:3333" + environment: + - NETWORK=${NETWORK} + volumes: + - "./custom-configs/solo-ckpool/ckpool.conf:/etc/ckpool/ckpool.conf:ro" + restart: unless-stopped + depends_on: + - solo-ckpool-builder + - sv1-node-pool-proxy + networks: + sv2-net: + ipv4_address: 10.5.0.24 + cap_add: + - NET_ADMIN + + solo-ckpool-miner-proxy: + image: sv1-custom-proxy-builder-image + labels: + logging: "config-c" + command: + [ + "/bin/sh", + "-c", + "./monitor_and_apply_latency.sh 10.5.0.24 2 & exec ./sv1-custom-proxy", + ] + ports: + - "3335:3335" + - "2346:2346" + environment: + - SERVER=10.5.0.24:3333 + - CLIENT=0.0.0.0:3335 + - PROM_ADDRESS=10.5.0.25:2346 + - PROXY_TYPE=pool-miner + - RUST_LOG=${LOG_LEVEL} + container_name: solo-ckpool-miner-proxy + depends_on: + - sv1-custom-proxy-builder + - solo-ckpool + restart: unless-stopped + networks: + sv2-net: + ipv4_address: 10.5.0.25 + cap_add: + - NET_ADMIN + monitor-traffic-tcpdump: image: inzania/network-traffic-metrics:latest network_mode: host diff --git a/prometheus/prometheus.yml b/prometheus/prometheus.yml index a0375b8..1a0deff 100644 --- a/prometheus/prometheus.yml +++ b/prometheus/prometheus.yml @@ -109,4 +109,12 @@ scrape_configs: scrape_interval: 5s static_configs: - - targets: ['sv2-translator-miner-proxy:5676'] # The Network Traffic Metrics IP/port \ No newline at end of file + - targets: ['sv2-translator-miner-proxy:5676'] # The Network Traffic Metrics IP/port + + - job_name: 'solo-ckpool-miner-proxy' + + # Override the global default and scrape targets from this job every 5 seconds. + scrape_interval: 5s + + static_configs: + - targets: ['solo-ckpool-miner-proxy:2346'] # Solo-ckpool proxy metrics \ No newline at end of file diff --git a/solo-ckpool.dockerfile b/solo-ckpool.dockerfile new file mode 100644 index 0000000..2e34748 --- /dev/null +++ b/solo-ckpool.dockerfile @@ -0,0 +1,66 @@ +############################ +# Docker build environment # +############################ +FROM ubuntu:22.04 AS build + +# Install build dependencies +RUN apt-get update || true && \ + DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ + build-essential \ + yasm \ + libzmq3-dev \ + git \ + autotools-dev \ + autoconf \ + automake \ + pkg-config \ + libtool \ + && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* + +# Clone solo-ckpool with fractional difficulty and shares_per_minute support +WORKDIR /build +RUN git config --global http.sslverify false && \ + git clone -b solobtc https://github.com/xyephy/solo-ckpool.git ckpool-solo + +WORKDIR /build/ckpool-solo + +# Build ckpool-solo +RUN ./autogen.sh && \ + ./configure && \ + make + +############################ +# Docker runtime environment # +############################ +FROM ubuntu:22.04 + +# Install runtime dependencies +RUN apt-get update && \ + DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ + libzmq5 \ + iproute2 \ + iputils-ping \ + curl \ + jq \ + && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* + +# Copy built binaries +COPY --from=build /build/ckpool-solo/src/ckpool /usr/local/bin/ +COPY --from=build /build/ckpool-solo/src/ckpmsg /usr/local/bin/ + +# Copy the monitoring script and startup script +COPY ./pools-latency-calculator/monitor_and_apply_latency.sh /usr/local/bin/monitor_and_apply_latency.sh +COPY ./start-ckpool.sh /usr/local/bin/start-ckpool.sh +RUN chmod +x /usr/local/bin/monitor_and_apply_latency.sh /usr/local/bin/start-ckpool.sh + +# Create required directories +RUN mkdir -p /var/log/ckpool /etc/ckpool + +# Set working directory +WORKDIR /etc/ckpool + +# Expose stratum port +EXPOSE 3333 + +# Default command - will be overridden by docker-compose +CMD ["/usr/local/bin/ckpool", "-B", "-c", "/etc/ckpool/ckpool.conf"] \ No newline at end of file diff --git a/start-ckpool.sh b/start-ckpool.sh new file mode 100755 index 0000000..d969913 --- /dev/null +++ b/start-ckpool.sh @@ -0,0 +1,10 @@ +#!/bin/bash + +# Clean up any leftover PID files +rm -f /var/log/ckpool/*.pid /var/log/ckpool/*.sock + +# Start latency monitoring in background +/usr/local/bin/monitor_and_apply_latency.sh 10.5.0.25 2 & + +# Start ckpool in foreground +exec /usr/local/bin/ckpool -B -c /etc/ckpool/ckpool.conf \ No newline at end of file