Avoid expensive production outages

Self-Hosted AI Code Reviewer

Hexmos LiveReview AI-Powered Code Review
Live Review
+AI-Powered Code Review +

Gather Intel - Amass Intel | Online Free DevTools by Hexmos

Gather open-source intel with Amass Intel. Discover root domains and ASNs related to organizations and IP addresses. Free online tool, no registration required.

amass intel

+
+

Collect open source intel on an organisation like root domains and ASNs. +More information: https://github.com/owasp-amass/amass/blob/master/doc/user_guide.md#the-intel-subcommand.

+
+
    +
  • Find root domains in an IP [addr]ess range:
    • +
+

amass intel -addr {{192.168.0.1-254}}

+
    +
  • Use active recon methods:
    • +
+

amass intel -active -addr {{192.168.0.1-254}}

+
    +
  • Find root domains related to a [d]omain:
    • +
+

amass intel -whois -d {{domain_name}}

+
    +
  • Find ASNs belonging to an [org]anisation:
    • +
+

amass intel -org {{organisation_name}}

+
    +
  • Find root domains belonging to a given Autonomous System Number:
    • +
+

amass intel -asn {{asn}}

+
    +
  • Save results to a text file:
    • +
+

amass intel -o {{output_file}} -whois -d {{domain_name}}

+
    +
  • List all available data sources:
    • +
+

amass intel -list

See Also
diff --git a/frontend/metric_analisi.md b/frontend/metric_analisi.md new file mode 100644 index 0000000000..829c660e63 --- /dev/null +++ b/frontend/metric_analisi.md @@ -0,0 +1,455 @@ +# Overview + +Currently I want to find avg of db response time for tldr pages + +Already have a script serve/pmd_logs.py which was used to calculate avg response time for pmd pages. + + +now modify this script for identifying avg response time for tldr pages. + + +``` +[TLDR_DB] Initializing worker pool with 2 workers... +[TLDR_DB] Worker pool initialized in 128ms +[Auth Middleware] ENABLE_SIGNIN env value: "undefined", enabled: false, Path: /freedevtools/tldr/common/pgmtost4 +[Auth Middleware] Signin disabled, skipping auth check +[TLDR_DB][2025-12-06T14:09:57.606Z] Dispatching getPage params={"platform":"common","slug":"pgmtost4"} +[2025-12-06T14:09:57.608Z] [TLDR_DB] Worker 0 START getPage params={"platform":"common","slug":"pgmtost4"} +[2025-12-06T14:09:57.616Z] [TLDR_DB] Worker 0 END getPage finished in 8ms +[TLDR_DB][2025-12-06T14:09:57.616Z] getPage completed in 10ms +[BASE_LAYOUT] Start rendering Convert PGM to ST-4 - Format Images | Online Free DevTools by Hexmos at 2025-12-06T14:09:57.640Z +19:39:57 [200] /tldr/common/pgmtost4 7948ms + +``` +Current logs are from tldr pages. + + + + +## After Removing Body + +25-12-06T14:54:57.563548Z INFO pmdaemon::process: Process astro-4321 started with PID: 51698 +Started process 'astro-4321' with ID: 452b784c-fe1f-4d35-9934-b6a411027b68 +2025-12-06T14:54:57.564381Z INFO pmdaemon::manager: Allocated port 4322 to process astro-4322 +2025-12-06T14:54:57.564411Z INFO pmdaemon::process: Starting process: astro-4322 +2025-12-06T14:54:57.568546Z INFO pmdaemon::process: Process astro-4322 started with PID: 51700 +Started process 'astro-4322' with ID: 11213e67-1e8c-48a1-9876-1f3b7a0f7ec0 +Started 2 processes from config file +pmdaemon list +2025-12-06T14:54:59.594399Z INFO pmdaemon: PMDaemon v0.1.4 starting +2025-12-06T14:54:59.876916Z INFO pmdaemon::manager: Loaded 2 process configurations +┌──────────┬────────────┬────────┬───────┬────────┬──────────┬───────┬────────┬──────┐ +│ ID ┆ Name ┆ Status ┆ PID ┆ Uptime ┆ Restarts ┆ CPU % ┆ Memory ┆ Port │ +╞══════════╪════════════╪════════╪═══════╪════════╪══════════╪═══════╪════════╪══════╡ +│ 11213e67 ┆ astro-4322 ┆ online ┆ 51700 ┆ 0s ┆ 0 ┆ 0.0 ┆ - ┆ 4322 │ +├╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌┤ +│ 452b784c ┆ astro-4321 ┆ online ┆ 51698 ┆ 0s ┆ 0 ┆ 0.0 ┆ - ┆ 4321 │ +└──────────┴────────────┴────────┴───────┴────────┴──────────┴───────┴────────┴──────┘ +./pmd-pin.sh +Pinning PID 51698 (astro-4321) to CPU 0 +pid 51698's current affinity list: 0-3 +pid 51698's new affinity list: 0 +Pinning PID 51700 (astro-4322) to CPU 1 +pid 51700's current affinity list: 0-3 +pid 51700's new affinity list: 1 +./pmd-verifypin.sh +Checking CPU pinning for PMDaemon apps: astro-4321 astro-4322 +-------------------------------------- +App: astro-4321 + PID: 51698 | pid 51698's current affinity list: 0 + +App: astro-4322 + PID: 51700 | pid 51700's current affinity list: 1 + +-------------------------------------- +Done. +⬇️ Server up ----------- +⬇️ Warming up server logs will be flushed for the below requests, ignore the below curls ----------- +hey -z 10s -host hexmos-local.com http://127.0.0.1/freedevtools/tldr/adb/adb + +Summary: + Total: 10.0773 secs + Slowest: 0.4417 secs + Fastest: 0.0196 secs + Average: 0.1151 secs + Requests/sec: 431.7625 + + +Response time histogram: + 0.020 [1] | + 0.062 [621] |■■■■■■■■■■■■■■■ + 0.104 [1614] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ + 0.146 [1179] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ + 0.188 [508] |■■■■■■■■■■■■■ + 0.231 [190] |■■■■■ + 0.273 [89] |■■ + 0.315 [75] |■■ + 0.357 [33] |■ + 0.399 [37] |■ + 0.442 [4] | + + +Latency distribution: + 10% in 0.0573 secs + 25% in 0.0735 secs + 50% in 0.1024 secs + 75% in 0.1389 secs + 90% in 0.1877 secs + 95% in 0.2377 secs + 99% in 0.3527 secs + +Details (average, fastest, slowest): + DNS+dialup: 0.0001 secs, 0.0196 secs, 0.4417 secs + DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0000 secs + req write: 0.0001 secs, 0.0000 secs, 0.0381 secs + resp wait: 0.1141 secs, 0.0151 secs, 0.4414 secs + resp read: 0.0007 secs, 0.0001 secs, 0.0496 secs + +Status code distribution: + [200] 4351 responses + + + +Peeking at 4 log files in /home/gk/.pmdaemon/logs... + +Request count table: ++------------+------------+--------------+---------------+----------+ +| Process | Requests | Dispatches | Worker Logs | Errors | ++============+============+==============+===============+==========+ +| astro-4321 | 0 | 2167 | 4334 | 0 | ++------------+------------+--------------+---------------+----------+ +| astro-4322 | 0 | 2184 | 4368 | 0 | ++------------+------------+--------------+---------------+----------+ + +Total aggregated: + total lines: 26112 + requests: 0 + dispatches: 4351 + worker logs: 8702 + errors: 0 + +Dispatches represent each time the worker pool received a query and handed it off to a worker thread (logged via [SVG_ICONS_DB|PNG_ICONS_DB|EMOJI_DB|TLDR_DB] Dispatching ). + +Query Count Details process wise: ++------------+-----------+--------------------------------------------------+ +| Process | getPage | getPage params={"platform":"adb","slug":"adb"} | ++============+===========+==================================================+ +| astro-4321 | 2167 | 2167 | ++------------+-----------+--------------------------------------------------+ +| astro-4322 | 2184 | 2184 | ++------------+-----------+--------------------------------------------------+ + +Query Duration Details process wise: ++---------+---------+----------+----------+-----------+ +| Query | Count | Avg ms | Max ms | Sum min | ++=========+=========+==========+==========+===========+ +| getPage | 4348 | 28.1 | 172 | 2.04 | ++---------+---------+----------+----------+-----------+ + +Overall window: 2025-12-06T14:55:04.533000+00:00 → 2025-12-06T14:55:14.353000+00:00 +Total coverage: 0:00:09.820000 +⬇️ Requesting tldr 10sec +hey -z 10s -host hexmos-local.com http://127.0.0.1/freedevtools/tldr/adb + +Summary: + Total: 10.4494 secs + Slowest: 1.7798 secs + Fastest: 0.1264 secs + Average: 0.6972 secs + Requests/sec: 70.1477 + + +Response time histogram: + 0.126 [1] | + 0.292 [24] |■■■■ + 0.457 [101] |■■■■■■■■■■■■■■■■■■ + 0.622 [154] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■ + 0.788 [224] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ + 0.953 [126] |■■■■■■■■■■■■■■■■■■■■■■■ + 1.118 [58] |■■■■■■■■■■ + 1.284 [25] |■■■■ + 1.449 [12] |■■ + 1.614 [5] |■ + 1.780 [3] |■ + + +Latency distribution: + 10% in 0.3748 secs + 25% in 0.5263 secs + 50% in 0.6778 secs + 75% in 0.8459 secs + 90% in 1.0186 secs + 95% in 1.1422 secs + 99% in 1.4638 secs + +Details (average, fastest, slowest): + DNS+dialup: 0.0003 secs, 0.1264 secs, 1.7798 secs + DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0000 secs + req write: 0.0003 secs, 0.0000 secs, 0.0123 secs + resp wait: 0.6899 secs, 0.1229 secs, 1.7779 secs + resp read: 0.0059 secs, 0.0016 secs, 0.0564 secs + +Status code distribution: + [200] 733 responses + + + +Peeking at 4 log files in /home/gk/.pmdaemon/logs... + +Request count table: ++------------+------------+--------------+---------------+----------+ +| Process | Requests | Dispatches | Worker Logs | Errors | ++============+============+==============+===============+==========+ +| astro-4321 | 0 | 2546 | 5092 | 0 | ++------------+------------+--------------+---------------+----------+ +| astro-4322 | 0 | 2538 | 5076 | 0 | ++------------+------------+--------------+---------------+----------+ + +Total aggregated: + total lines: 31976 + requests: 0 + dispatches: 5084 + worker logs: 10168 + errors: 0 + +Dispatches represent each time the worker pool received a query and handed it off to a worker thread (logged via [SVG_ICONS_DB|PNG_ICONS_DB|EMOJI_DB|TLDR_DB] Dispatching ). + +Query Count Details process wise: ++------------+---------------+--------------------------------------------------+-----------+--------------------------------------------------+ +| Process | getMainPage | getMainPage params={"platform":"adb","page":1} | getPage | getPage params={"platform":"adb","slug":"adb"} | ++============+===============+==================================================+===========+==================================================+ +| astro-4321 | 379 | 379 | 2167 | 2167 | ++------------+---------------+--------------------------------------------------+-----------+--------------------------------------------------+ +| astro-4322 | 354 | 354 | 2184 | 2184 | ++------------+---------------+--------------------------------------------------+-----------+--------------------------------------------------+ + +Query Duration Details process wise: ++-------------+---------+----------+----------+-----------+ +| Query | Count | Avg ms | Max ms | Sum min | ++=============+=========+==========+==========+===========+ +| getPage | 4348 | 28.1 | 172 | 2.04 | ++-------------+---------+----------+----------+-----------+ +| getMainPage | 733 | 313.9 | 1137 | 3.84 | ++-------------+---------+----------+----------+-----------+ + +Overall window: 2025-12-06T14:55:04.533000+00:00 → 2025-12-06T14:55:27.777000+00:00 +Total coverage: 0:00:23.244000 +10s Requesting tldr main +hey -z 10s -host hexmos-local.com http://127.0.0.1/freedevtools/tldr + +Summary: + Total: 11.4074 secs + Slowest: 1.9906 secs + Fastest: 0.0813 secs + Average: 0.7989 secs + Requests/sec: 58.3833 + + +Response time histogram: + 0.081 [1] | + 0.272 [22] |■■■■■ + 0.463 [63] |■■■■■■■■■■■■■ + 0.654 [138] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ + 0.845 [193] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ + 1.036 [119] |■■■■■■■■■■■■■■■■■■■■■■■■■ + 1.227 [66] |■■■■■■■■■■■■■■ + 1.418 [34] |■■■■■■■ + 1.609 [12] |■■ + 1.800 [12] |■■ + 1.991 [6] |■ + + +Latency distribution: + 10% in 0.4215 secs + 25% in 0.5787 secs + 50% in 0.7709 secs + 75% in 0.9753 secs + 90% in 1.2127 secs + 95% in 1.4018 secs + 99% in 1.8291 secs + +Details (average, fastest, slowest): + DNS+dialup: 0.0018 secs, 0.0813 secs, 1.9906 secs + DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0000 secs + req write: 0.0002 secs, 0.0000 secs, 0.0312 secs + resp wait: 0.7901 secs, 0.0773 secs, 1.9882 secs + resp read: 0.0066 secs, 0.0019 secs, 0.0508 secs + +Status code distribution: + [200] 666 responses + + + +Peeking at 4 log files in /home/gk/.pmdaemon/logs... + +Request count table: ++------------+------------+--------------+---------------+----------+ +| Process | Requests | Dispatches | Worker Logs | Errors | ++============+============+==============+===============+==========+ +| astro-4321 | 0 | 2906 | 5812 | 0 | ++------------+------------+--------------+---------------+----------+ +| astro-4322 | 0 | 2844 | 5688 | 0 | ++------------+------------+--------------+---------------+----------+ + +Total aggregated: + total lines: 36638 + requests: 0 + dispatches: 5750 + worker logs: 11500 + errors: 0 + +Dispatches represent each time the worker pool received a query and handed it off to a worker thread (logged via [SVG_ICONS_DB|PNG_ICONS_DB|EMOJI_DB|TLDR_DB] Dispatching ). + +Query Count Details process wise: ++------------+---------------+--------------------------------------------------+----------------------------------------------------+-----------+--------------------------------------------------+ +| Process | getMainPage | getMainPage params={"platform":"adb","page":1} | getMainPage params={"platform":"index","page":1} | getPage | getPage params={"platform":"adb","slug":"adb"} | ++============+===============+==================================================+====================================================+===========+==================================================+ +| astro-4321 | 739 | 379 | 360 | 2167 | 2167 | ++------------+---------------+--------------------------------------------------+----------------------------------------------------+-----------+--------------------------------------------------+ +| astro-4322 | 660 | 354 | 306 | 2184 | 2184 | ++------------+---------------+--------------------------------------------------+----------------------------------------------------+-----------+--------------------------------------------------+ + +Query Duration Details process wise: ++-------------+---------+----------+----------+-----------+ +| Query | Count | Avg ms | Max ms | Sum min | ++=============+=========+==========+==========+===========+ +| getPage | 4348 | 28.1 | 172 | 2.04 | ++-------------+---------+----------+----------+-----------+ +| getMainPage | 1399 | 320.2 | 1137 | 7.47 | ++-------------+---------+----------+----------+-----------+ + +Overall window: 2025-12-06T14:55:04.533000+00:00 → 2025-12-06T14:55:42.117000+00:00 +Total coverage: 0:00:37.584000 +1min Requesting tldr 1min +hey -z 1m -host hexmos-local.com http://127.0.0.1/freedevtools/tldr/npm/npm-fund/ +^[[F +Summary: + Total: 60.1223 secs + Slowest: 0.5615 secs + Fastest: 0.0149 secs + Average: 0.0942 secs + Requests/sec: 530.2025 + + +Response time histogram: + 0.015 [1] | + 0.070 [9937] |■■■■■■■■■■■■■■■■■■■■■■■■■ + 0.124 [16046] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ + 0.179 [4520] |■■■■■■■■■■■ + 0.234 [949] |■■ + 0.288 [298] |■ + 0.343 [90] | + 0.398 [16] | + 0.452 [16] | + 0.507 [1] | + 0.562 [3] | + + +Latency distribution: + 10% in 0.0499 secs + 25% in 0.0643 secs + 50% in 0.0860 secs + 75% in 0.1136 secs + 90% in 0.1460 secs + 95% in 0.1731 secs + 99% in 0.2471 secs + +Details (average, fastest, slowest): + DNS+dialup: 0.0000 secs, 0.0149 secs, 0.5615 secs + DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0000 secs + req write: 0.0001 secs, 0.0000 secs, 0.0277 secs + resp wait: 0.0933 secs, 0.0074 secs, 0.5612 secs + resp read: 0.0007 secs, 0.0001 secs, 0.0437 secs + +Status code distribution: + [200] 31877 responses + + + +Peeking at 4 log files in /home/gk/.pmdaemon/logs... + +Request count table: ++------------+------------+--------------+---------------+----------+ +| Process | Requests | Dispatches | Worker Logs | Errors | ++============+============+==============+===============+==========+ +| astro-4321 | 0 | 18973 | 37946 | 0 | ++------------+------------+--------------+---------------+----------+ +| astro-4322 | 0 | 18654 | 37308 | 0 | ++------------+------------+--------------+---------------+----------+ + +Total aggregated: + total lines: 227900 + requests: 0 + dispatches: 37627 + worker logs: 75254 + errors: 0 + +Dispatches represent each time the worker pool received a query and handed it off to a worker thread (logged via [SVG_ICONS_DB|PNG_ICONS_DB|EMOJI_DB|TLDR_DB] Dispatching ). + +Query Count Details process wise: ++------------+---------------+--------------------------------------------------+----------------------------------------------------+-----------+--------------------------------------------------+-------------------------------------------------------+ +| Process | getMainPage | getMainPage params={"platform":"adb","page":1} | getMainPage params={"platform":"index","page":1} | getPage | getPage params={"platform":"adb","slug":"adb"} | getPage params={"platform":"npm","slug":"npm-fund"} | ++============+===============+==================================================+====================================================+===========+==================================================+=======================================================+ +| astro-4321 | 739 | 379 | 360 | 18234 | 2167 | 16067 | ++------------+---------------+--------------------------------------------------+----------------------------------------------------+-----------+--------------------------------------------------+-------------------------------------------------------+ +| astro-4322 | 660 | 354 | 306 | 17994 | 2184 | 15810 | ++------------+---------------+--------------------------------------------------+----------------------------------------------------+-----------+--------------------------------------------------+-------------------------------------------------------+ + +Query Duration Details process wise: ++-------------+---------+----------+----------+-----------+ +| Query | Count | Avg ms | Max ms | Sum min | ++=============+=========+==========+==========+===========+ +| getPage | 36217 | 21.8 | 172 | 13.15 | ++-------------+---------+----------+----------+-----------+ +| getMainPage | 1399 | 320.2 | 1137 | 7.47 | ++-------------+---------+----------+----------+-----------+ + +Overall window: 2025-12-06T14:55:04.533000+00:00 → 2025-12-06T14:56:46.257000+00:00 +Total coverage: 0:01:41.724000 +5min Requesting tldr 5min +hey -z 5m -host hexmos-local.com http://127.0.0.1/freedevtools/tldr/npm/npm-fund/ +^C +Summary: + Total: 137.1782 secs + Slowest: 0.4614 secs + Fastest: 0.0061 secs + Average: 0.0946 secs + Requests/sec: 528.5095 + + +Response time histogram: + 0.006 [1] | + 0.052 [9227] |■■■■■■■■■■ + 0.097 [35898] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ + 0.143 [18525] |■■■■■■■■■■■■■■■■■■■■■ + 0.188 [5496] |■■■■■■ + 0.234 [2113] |■■ + 0.279 [856] |■ + 0.325 [238] | + 0.370 [99] | + 0.416 [32] | + 0.461 [15] | + + +Latency distribution: + 10% in 0.0485 secs + 25% in 0.0633 secs + 50% in 0.0849 secs + 75% in 0.1137 secs + 90% in 0.1515 secs + 95% in 0.1841 secs + 99% in 0.2560 secs + +Details (average, fastest, slowest): + DNS+dialup: 0.0000 secs, 0.0061 secs, 0.4614 secs + DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0000 secs + req write: 0.0001 secs, 0.0000 secs, 0.0599 secs + resp wait: 0.0935 secs, 0.0057 secs, 0.4612 secs + resp read: 0.0008 secs, 0.0001 secs, 0.1262 secs + +Status code distribution: + [200] 72500 responses + + +## After Adding Body + diff --git a/frontend/serve/Makefile b/frontend/serve/Makefile index 53883fe0f4..b53492e343 100644 --- a/frontend/serve/Makefile +++ b/frontend/serve/Makefile @@ -114,18 +114,23 @@ test-tldr-req-pmd: @sleep 2 @echo "⬇️ Warming up server logs will be flushed for the below requests, ignore the below curls -----------" hey -z 10s -host hexmos-local.com http://127.0.0.1/freedevtools/tldr/adb/adb + @python3 pmd_logs.py --peek @sleep 2 @echo "⬇️ Requesting tldr 10sec" hey -z 10s -host hexmos-local.com http://127.0.0.1/freedevtools/tldr/adb + @python3 pmd_logs.py --peek @sleep 2 @echo "10s Requesting tldr main" hey -z 10s -host hexmos-local.com http://127.0.0.1/freedevtools/tldr + @python3 pmd_logs.py --peek @sleep 2 @echo "1min Requesting tldr 1min" hey -z 1m -host hexmos-local.com http://127.0.0.1/freedevtools/tldr/npm/npm-fund/ + @python3 pmd_logs.py --peek @sleep 2 @echo "5min Requesting tldr 5min" hey -z 5m -host hexmos-local.com http://127.0.0.1/freedevtools/tldr/npm/npm-fund/ + @python3 pmd_logs.py --peek @sleep 2 @echo "⬇️ Showing logs -----------" # @LOG_FILES=$$(ls -1t ~/.pmdaemon/logs/astro-*-out.log 2>/dev/null | head -n2); \ diff --git a/frontend/serve/pmd_logs.py b/frontend/serve/pmd_logs.py index 9576151788..ca4573a4b8 100755 --- a/frontend/serve/pmd_logs.py +++ b/frontend/serve/pmd_logs.py @@ -79,9 +79,9 @@ def summarize_file(path): "dispatches": 0, "worker_events": 0, } - dispatch_pattern = r"\[(SVG_ICONS_DB|PNG_ICONS_DB|EMOJI_DB)\]" - request_pattern = r"\[(SVG_ICONS|PNG_ICONS|EMOJI)\] Request reached|Request reached server:" - worker_pattern = r"\[(SVG_ICONS_DB|PNG_ICONS_DB|EMOJI_DB)\]\[" + dispatch_pattern = r"\[(SVG_ICONS_DB|PNG_ICONS_DB|EMOJI_DB|TLDR_DB)\]" + request_pattern = r"\[(SVG_ICONS|PNG_ICONS|EMOJI|TLDR)\] Request reached|Request reached server:" + worker_pattern = r"\[(SVG_ICONS_DB|PNG_ICONS_DB|EMOJI_DB|TLDR_DB)\]\[" timestamp_re = re.compile(r"\[([0-9T:\-\.]+Z)\]") durations = {} timestamps = [] @@ -310,38 +310,50 @@ def main(): parser = argparse.ArgumentParser( description="Copy PMDaemon astro logs into logs/
-
-
- +
From 58af848fade1497712258db0a269cf30fe78ba34 Mon Sep 17 00:00:00 2001 From: Ganesh Kumar Date: Sat, 6 Dec 2025 21:14:40 +0530 Subject: [PATCH 197/234] added-body --- frontend/src/pages/tldr/[platform]/[slug].astro | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/frontend/src/pages/tldr/[platform]/[slug].astro b/frontend/src/pages/tldr/[platform]/[slug].astro index 6a9fdfa5da..7fcc6857ed 100644 --- a/frontend/src/pages/tldr/[platform]/[slug].astro +++ b/frontend/src/pages/tldr/[platform]/[slug].astro @@ -188,7 +188,12 @@ if (pageNumber !== null) { breadcrumbItems={commandData.breadcrumbItems} /> - +
+
+
From 62c91d13dd62dc8b5d1791d9504f074bcb3ec254 Mon Sep 17 00:00:00 2001 From: Ganesh Kumar Date: Sat, 6 Dec 2025 21:15:13 +0530 Subject: [PATCH 198/234] removed-comment --- frontend/src/content.config.ts | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/frontend/src/content.config.ts b/frontend/src/content.config.ts index d6e0b7adc4..d10aefa8ed 100644 --- a/frontend/src/content.config.ts +++ b/frontend/src/content.config.ts @@ -153,7 +153,7 @@ const pngIconsMetadata = defineCollection({ export const collections = { tldr, - // mcpMetadata, - // mcpCategoryData, - // pngIconsMetadata, + mcpMetadata, + mcpCategoryData, + pngIconsMetadata, }; From e37a4ae9430b38d346ef69e055b3a196ab496277 Mon Sep 17 00:00:00 2001 From: Ganesh Kumar Date: Sat, 6 Dec 2025 21:20:17 +0530 Subject: [PATCH 199/234] added-hey-logs --- frontend/metric_analisi.md | 432 ++++++++++++++++++++++++++++++++++--- 1 file changed, 404 insertions(+), 28 deletions(-) diff --git a/frontend/metric_analisi.md b/frontend/metric_analisi.md index 829c660e63..fa7cfa6edf 100644 --- a/frontend/metric_analisi.md +++ b/frontend/metric_analisi.md @@ -1,31 +1,3 @@ -# Overview - -Currently I want to find avg of db response time for tldr pages - -Already have a script serve/pmd_logs.py which was used to calculate avg response time for pmd pages. - - -now modify this script for identifying avg response time for tldr pages. - - -``` -[TLDR_DB] Initializing worker pool with 2 workers... -[TLDR_DB] Worker pool initialized in 128ms -[Auth Middleware] ENABLE_SIGNIN env value: "undefined", enabled: false, Path: /freedevtools/tldr/common/pgmtost4 -[Auth Middleware] Signin disabled, skipping auth check -[TLDR_DB][2025-12-06T14:09:57.606Z] Dispatching getPage params={"platform":"common","slug":"pgmtost4"} -[2025-12-06T14:09:57.608Z] [TLDR_DB] Worker 0 START getPage params={"platform":"common","slug":"pgmtost4"} -[2025-12-06T14:09:57.616Z] [TLDR_DB] Worker 0 END getPage finished in 8ms -[TLDR_DB][2025-12-06T14:09:57.616Z] getPage completed in 10ms -[BASE_LAYOUT] Start rendering Convert PGM to ST-4 - Format Images | Online Free DevTools by Hexmos at 2025-12-06T14:09:57.640Z -19:39:57 [200] /tldr/common/pgmtost4 7948ms - -``` -Current logs are from tldr pages. - - - - ## After Removing Body 25-12-06T14:54:57.563548Z INFO pmdaemon::process: Process astro-4321 started with PID: 51698 @@ -453,3 +425,407 @@ Status code distribution: ## After Adding Body +./pmd-pin.sh +Pinning PID 64886 (astro-4321) to CPU 0 +pid 64886's current affinity list: 0-3 +pid 64886's new affinity list: 0 +Pinning PID 64887 (astro-4322) to CPU 1 +pid 64887's current affinity list: 0-3 +pid 64887's new affinity list: 1 +./pmd-verifypin.sh +Checking CPU pinning for PMDaemon apps: astro-4321 astro-4322 +-------------------------------------- +App: astro-4321 + PID: 64886 | pid 64886's current affinity list: 0 + +App: astro-4322 + PID: 64887 | pid 64887's current affinity list: 1 + +-------------------------------------- +Done. +⬇️ Server up ----------- +⬇️ Warming up server logs will be flushed for the below requests, ignore the below curls ----------- +hey -z 10s -host hexmos-local.com http://127.0.0.1/freedevtools/tldr/adb/adb + +Summary: + Total: 10.4520 secs + Slowest: 1.7567 secs + Fastest: 0.1498 secs + Average: 0.6029 secs + Requests/sec: 80.7500 + + +Response time histogram: + 0.150 [1] | + 0.311 [51] |■■■■■■■■ + 0.471 [268] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ + 0.632 [218] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ + 0.793 [148] |■■■■■■■■■■■■■■■■■■■■■■ + 0.953 [67] |■■■■■■■■■■ + 1.114 [37] |■■■■■■ + 1.275 [26] |■■■■ + 1.435 [17] |■■■ + 1.596 [5] |■ + 1.757 [6] |■ + + +Latency distribution: + 10% in 0.3486 secs + 25% in 0.4049 secs + 50% in 0.5324 secs + 75% in 0.7229 secs + 90% in 0.9819 secs + 95% in 1.1733 secs + 99% in 1.5549 secs + +Details (average, fastest, slowest): + DNS+dialup: 0.0009 secs, 0.1498 secs, 1.7567 secs + DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0000 secs + req write: 0.0005 secs, 0.0000 secs, 0.0371 secs + resp wait: 0.5955 secs, 0.1480 secs, 1.6928 secs + resp read: 0.0049 secs, 0.0015 secs, 0.0367 secs + +Status code distribution: + [200] 844 responses + + + +Peeking at 4 log files in /home/gk/.pmdaemon/logs... + +Request count table: ++------------+------------+--------------+---------------+----------+ +| Process | Requests | Dispatches | Worker Logs | Errors | ++============+============+==============+===============+==========+ +| astro-4321 | 0 | 437 | 874 | 0 | ++------------+------------+--------------+---------------+----------+ +| astro-4322 | 0 | 407 | 814 | 0 | ++------------+------------+--------------+---------------+----------+ + +Total aggregated: + total lines: 5914 + requests: 0 + dispatches: 844 + worker logs: 1688 + errors: 0 + +Dispatches represent each time the worker pool received a query and handed it off to a worker thread (logged via [SVG_ICONS_DB|PNG_ICONS_DB|EMOJI_DB|TLDR_DB] Dispatching ). + +Query Count Details process wise: ++------------+-----------+--------------------------------------------------+ +| Process | getPage | getPage params={"platform":"adb","slug":"adb"} | ++============+===========+==================================================+ +| astro-4321 | 437 | 437 | ++------------+-----------+--------------------------------------------------+ +| astro-4322 | 407 | 407 | ++------------+-----------+--------------------------------------------------+ + +Query Duration Details process wise: ++---------+---------+----------+----------+-----------+ +| Query | Count | Avg ms | Max ms | Sum min | ++=========+=========+==========+==========+===========+ +| getPage | 843 | 255.6 | 1373 | 3.59 | ++---------+---------+----------+----------+-----------+ + +Overall window: 2025-12-06T15:30:44.519000+00:00 → 2025-12-06T15:30:54.588000+00:00 +Total coverage: 0:00:10.069000 +⬇️ Requesting tldr 10sec +hey -z 10s -host hexmos-local.com http://127.0.0.1/freedevtools/tldr/adb + +Summary: + Total: 10.4763 secs + Slowest: 0.7539 secs + Fastest: 0.0799 secs + Average: 0.4250 secs + Requests/sec: 114.5440 + + +Response time histogram: + 0.080 [1] | + 0.147 [17] |■■ + 0.215 [21] |■■ + 0.282 [46] |■■■■■ + 0.350 [159] |■■■■■■■■■■■■■■■■■■ + 0.417 [351] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ + 0.484 [294] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ + 0.552 [162] |■■■■■■■■■■■■■■■■■■ + 0.619 [101] |■■■■■■■■■■■■ + 0.687 [36] |■■■■ + 0.754 [12] |■ + + +Latency distribution: + 10% in 0.3083 secs + 25% in 0.3618 secs + 50% in 0.4180 secs + 75% in 0.4872 secs + 90% in 0.5651 secs + 95% in 0.6098 secs + 99% in 0.6914 secs + +Details (average, fastest, slowest): + DNS+dialup: 0.0002 secs, 0.0799 secs, 0.7539 secs + DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0000 secs + req write: 0.0002 secs, 0.0000 secs, 0.0060 secs + resp wait: 0.4201 secs, 0.0757 secs, 0.7521 secs + resp read: 0.0045 secs, 0.0016 secs, 0.0319 secs + +Status code distribution: + [200] 1200 responses + + + +Peeking at 4 log files in /home/gk/.pmdaemon/logs... + +Request count table: ++------------+------------+--------------+---------------+----------+ +| Process | Requests | Dispatches | Worker Logs | Errors | ++============+============+==============+===============+==========+ +| astro-4321 | 0 | 1032 | 2064 | 0 | ++------------+------------+--------------+---------------+----------+ +| astro-4322 | 0 | 1012 | 2024 | 0 | ++------------+------------+--------------+---------------+----------+ + +Total aggregated: + total lines: 14314 + requests: 0 + dispatches: 2044 + worker logs: 4088 + errors: 0 + +Dispatches represent each time the worker pool received a query and handed it off to a worker thread (logged via [SVG_ICONS_DB|PNG_ICONS_DB|EMOJI_DB|TLDR_DB] Dispatching ). + +Query Count Details process wise: ++------------+---------------+--------------------------------------------------+-----------+--------------------------------------------------+ +| Process | getMainPage | getMainPage params={"platform":"adb","page":1} | getPage | getPage params={"platform":"adb","slug":"adb"} | ++============+===============+==================================================+===========+==================================================+ +| astro-4321 | 595 | 595 | 437 | 437 | ++------------+---------------+--------------------------------------------------+-----------+--------------------------------------------------+ +| astro-4322 | 605 | 605 | 407 | 407 | ++------------+---------------+--------------------------------------------------+-----------+--------------------------------------------------+ + +Query Duration Details process wise: ++-------------+---------+----------+----------+-----------+ +| Query | Count | Avg ms | Max ms | Sum min | ++=============+=========+==========+==========+===========+ +| getPage | 843 | 255.6 | 1373 | 3.59 | ++-------------+---------+----------+----------+-----------+ +| getMainPage | 1200 | 167.2 | 493 | 3.34 | ++-------------+---------+----------+----------+-----------+ + +Overall window: 2025-12-06T15:30:44.519000+00:00 → 2025-12-06T15:31:07.329000+00:00 +Total coverage: 0:00:22.810000 +10s Requesting tldr main +hey -z 10s -host hexmos-local.com http://127.0.0.1/freedevtools/tldr + +Summary: + Total: 10.4226 secs + Slowest: 1.0527 secs + Fastest: 0.0418 secs + Average: 0.4193 secs + Requests/sec: 116.8619 + + +Response time histogram: + 0.042 [1] | + 0.143 [29] |■■■ + 0.244 [73] |■■■■■■■■ + 0.345 [351] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ + 0.446 [357] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ + 0.547 [180] |■■■■■■■■■■■■■■■■■■■■ + 0.648 [108] |■■■■■■■■■■■■ + 0.749 [54] |■■■■■■ + 0.851 [23] |■■■ + 0.952 [23] |■■■ + 1.053 [19] |■■ + + +Latency distribution: + 10% in 0.2592 secs + 25% in 0.3152 secs + 50% in 0.3743 secs + 75% in 0.5005 secs + 90% in 0.6468 secs + 95% in 0.7638 secs + 99% in 0.9848 secs + +Details (average, fastest, slowest): + DNS+dialup: 0.0002 secs, 0.0418 secs, 1.0527 secs + DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0000 secs + req write: 0.0001 secs, 0.0000 secs, 0.0155 secs + resp wait: 0.4140 secs, 0.0365 secs, 1.0500 secs + resp read: 0.0048 secs, 0.0018 secs, 0.0322 secs + +Status code distribution: + [200] 1218 responses + + + +Peeking at 4 log files in /home/gk/.pmdaemon/logs... + +Request count table: ++------------+------------+--------------+---------------+----------+ +| Process | Requests | Dispatches | Worker Logs | Errors | ++============+============+==============+===============+==========+ +| astro-4321 | 0 | 1653 | 3306 | 0 | ++------------+------------+--------------+---------------+----------+ +| astro-4322 | 0 | 1609 | 3218 | 0 | ++------------+------------+--------------+---------------+----------+ + +Total aggregated: + total lines: 21622 + requests: 0 + dispatches: 3262 + worker logs: 6524 + errors: 0 + +Dispatches represent each time the worker pool received a query and handed it off to a worker thread (logged via [SVG_ICONS_DB|PNG_ICONS_DB|EMOJI_DB|TLDR_DB] Dispatching ). + +Query Count Details process wise: ++------------+---------------+--------------------------------------------------+----------------------------------------------------+-----------+--------------------------------------------------+ +| Process | getMainPage | getMainPage params={"platform":"adb","page":1} | getMainPage params={"platform":"index","page":1} | getPage | getPage params={"platform":"adb","slug":"adb"} | ++============+===============+==================================================+====================================================+===========+==================================================+ +| astro-4321 | 1216 | 595 | 621 | 437 | 437 | ++------------+---------------+--------------------------------------------------+----------------------------------------------------+-----------+--------------------------------------------------+ +| astro-4322 | 1202 | 605 | 597 | 407 | 407 | ++------------+---------------+--------------------------------------------------+----------------------------------------------------+-----------+--------------------------------------------------+ + +Query Duration Details process wise: ++-------------+---------+----------+----------+-----------+ +| Query | Count | Avg ms | Max ms | Sum min | ++=============+=========+==========+==========+===========+ +| getPage | 843 | 255.6 | 1373 | 3.59 | ++-------------+---------+----------+----------+-----------+ +| getMainPage | 2418 | 162.6 | 639 | 6.55 | ++-------------+---------+----------+----------+-----------+ + +Overall window: 2025-12-06T15:30:44.519000+00:00 → 2025-12-06T15:31:20.147000+00:00 +Total coverage: 0:00:35.628000 +1min Requesting tldr 1min +hey -z 1m -host hexmos-local.com http://127.0.0.1/freedevtools/tldr/npm/npm-fund/ + +Summary: + Total: 60.2830 secs + Slowest: 0.9633 secs + Fastest: 0.0354 secs + Average: 0.2889 secs + Requests/sec: 172.6026 + + +Response time histogram: + 0.035 [1] | + 0.128 [157] |■ + 0.221 [1498] |■■■■■■■■■■■ + 0.314 [5651] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ + 0.407 [2383] |■■■■■■■■■■■■■■■■■ + 0.499 [491] |■■■ + 0.592 [113] |■ + 0.685 [52] | + 0.778 [26] | + 0.871 [24] | + 0.963 [9] | + + +Latency distribution: + 10% in 0.2043 secs + 25% in 0.2374 secs + 50% in 0.2784 secs + 75% in 0.3256 secs + 90% in 0.3830 secs + 95% in 0.4294 secs + 99% in 0.5985 secs + +Details (average, fastest, slowest): + DNS+dialup: 0.0000 secs, 0.0354 secs, 0.9633 secs + DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0000 secs + req write: 0.0001 secs, 0.0000 secs, 0.0173 secs + resp wait: 0.2850 secs, 0.0299 secs, 0.9565 secs + resp read: 0.0037 secs, 0.0012 secs, 0.0668 secs + +Status code distribution: + [200] 10405 responses + + + +Peeking at 4 log files in /home/gk/.pmdaemon/logs... + +Request count table: ++------------+------------+--------------+---------------+----------+ +| Process | Requests | Dispatches | Worker Logs | Errors | ++============+============+==============+===============+==========+ +| astro-4321 | 0 | 6865 | 13730 | 0 | ++------------+------------+--------------+---------------+----------+ +| astro-4322 | 0 | 6802 | 13604 | 0 | ++------------+------------+--------------+---------------+----------+ + +Total aggregated: + total lines: 94457 + requests: 0 + dispatches: 13667 + worker logs: 27334 + errors: 0 + +Dispatches represent each time the worker pool received a query and handed it off to a worker thread (logged via [SVG_ICONS_DB|PNG_ICONS_DB|EMOJI_DB|TLDR_DB] Dispatching ). + +Query Count Details process wise: ++------------+---------------+--------------------------------------------------+----------------------------------------------------+-----------+--------------------------------------------------+-------------------------------------------------------+ +| Process | getMainPage | getMainPage params={"platform":"adb","page":1} | getMainPage params={"platform":"index","page":1} | getPage | getPage params={"platform":"adb","slug":"adb"} | getPage params={"platform":"npm","slug":"npm-fund"} | ++============+===============+==================================================+====================================================+===========+==================================================+=======================================================+ +| astro-4321 | 1216 | 595 | 621 | 5649 | 437 | 5212 | ++------------+---------------+--------------------------------------------------+----------------------------------------------------+-----------+--------------------------------------------------+-------------------------------------------------------+ +| astro-4322 | 1202 | 605 | 597 | 5600 | 407 | 5193 | ++------------+---------------+--------------------------------------------------+----------------------------------------------------+-----------+--------------------------------------------------+-------------------------------------------------------+ + +Query Duration Details process wise: ++-------------+---------+----------+----------+-----------+ +| Query | Count | Avg ms | Max ms | Sum min | ++=============+=========+==========+==========+===========+ +| getPage | 11242 | 126.7 | 1373 | 23.75 | ++-------------+---------+----------+----------+-----------+ +| getMainPage | 2418 | 162.6 | 639 | 6.55 | ++-------------+---------+----------+----------+-----------+ + +Overall window: 2025-12-06T15:30:44.519000+00:00 → 2025-12-06T15:32:22.965000+00:00 +Total coverage: 0:01:38.446000 +5min Requesting tldr 5min +hey -z 5m -host hexmos-local.com http://127.0.0.1/freedevtools/tldr/npm/npm-fund/ +^C +Summary: + Total: 162.4016 secs + Slowest: 2.0663 secs + Fastest: 0.0312 secs + Average: 0.4522 secs + Requests/sec: 110.4484 + + +Response time histogram: + 0.031 [1] | + 0.235 [3399] |■■■■■■■■■■■■■■■■■■ + 0.438 [7598] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ + 0.642 [3444] |■■■■■■■■■■■■■■■■■■ + 0.845 [1814] |■■■■■■■■■■ + 1.049 [913] |■■■■■ + 1.252 [454] |■■ + 1.456 [197] |■ + 1.659 [81] | + 1.863 [26] | + 2.066 [10] | + + +Latency distribution: + 10% in 0.2039 secs + 25% in 0.2577 secs + 50% in 0.3705 secs + 75% in 0.5644 secs + 90% in 0.8283 secs + 95% in 1.0118 secs + 99% in 1.3745 secs + +Details (average, fastest, slowest): + DNS+dialup: 0.0000 secs, 0.0312 secs, 2.0663 secs + DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0000 secs + req write: 0.0001 secs, 0.0000 secs, 0.0647 secs + resp wait: 0.4477 secs, 0.0230 secs, 2.0513 secs + resp read: 0.0042 secs, 0.0012 secs, 0.1001 secs + +Status code distribution: + [200] 17937 responses \ No newline at end of file From c9948eb2019d66427dab4c55e56bd6eb7a6eeccc Mon Sep 17 00:00:00 2001 From: lovestaco Date: Sun, 7 Dec 2025 10:17:25 +0530 Subject: [PATCH 200/234] fix: readme --- frontend/Makefile | 6 +++--- frontend/README.md | 19 +++++++++++++++++++ 2 files changed, 22 insertions(+), 3 deletions(-) diff --git a/frontend/Makefile b/frontend/Makefile index 0e7447ecd2..6489b4de97 100644 --- a/frontend/Makefile +++ b/frontend/Makefile @@ -1,4 +1,4 @@ -.PHONY: build deploy run clean install gen banners check-pagespeed deploy-no-pagespeed local-env sync-db update-db-to-b2 index-deploy serve prod-serve prod-stop +.PHONY: build deploy run clean install gen banners check-pagespeed deploy-no-pagespeed local-env sync-db update-db-to-b2 index-deploy serve start-prod stop-prod deploy-staging-ssr # Run development server run: @@ -19,14 +19,14 @@ build: serve: bun run serve-ssr -serve-prod: +start-prod: cd serve && pmdaemon --config pmd.config.json start && bash pmd-pin.sh stop-prod: cd serve && pmdaemon delete all --force && bash kill-ports.sh && bash kill-ports.sh -deploy-ssr: +deploy-staging-ssr: @echo "⬆️ Deploying SSR build to server..." @rsync -rvz --delete --info=progress2 --no-perms --no-owner --no-group --no-times \ diff --git a/frontend/README.md b/frontend/README.md index 24462abe3a..99b6abca78 100644 --- a/frontend/README.md +++ b/frontend/README.md @@ -109,3 +109,22 @@ To add or update a database (DB) file for a category: 4. **Referencing in Code** - Always reference database files in your code using the path: `db/all_dbs/your-db.db` + +## Deploying to Staging + +### Step 1: local: + +``` + +make run bulid:icons +make deploy-staging-ssr +``` + +### Step 2: nats03: + +``` +cd FreeDevTools +git pull +make stop-prod +make start-prod +``` From e0657f134cb326c472e00dff34264f3908b602da Mon Sep 17 00:00:00 2001 From: lovestaco Date: Sun, 7 Dec 2025 10:21:11 +0530 Subject: [PATCH 201/234] fix: readme --- frontend/README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/frontend/README.md b/frontend/README.md index 99b6abca78..a434f7744b 100644 --- a/frontend/README.md +++ b/frontend/README.md @@ -115,7 +115,7 @@ To add or update a database (DB) file for a category: ### Step 1: local: ``` - +make sync-db-to-local make run bulid:icons make deploy-staging-ssr ``` @@ -125,6 +125,7 @@ make deploy-staging-ssr ``` cd FreeDevTools git pull +make sync-db-to-local make stop-prod make start-prod ``` From db57c1e08afb95cb688d7231fdb30c1f8268d073 Mon Sep 17 00:00:00 2001 From: Ganesh Kumar Date: Sun, 7 Dec 2025 11:23:45 +0530 Subject: [PATCH 202/234] feat: Implement tldr sitemap generation and dynamic serving via new database table and Astro routes. --- frontend/db/tldrs/tldr-utils.ts | 4 + frontend/db/tldrs/tldr-worker-pool.ts | 1 + frontend/db/tldrs/tldr-worker.ts | 17 +++ frontend/scripts/tldr/tldr_to_db.go | 123 ++++++++++++++--- frontend/src/pages/tldr/sitemap-[page].xml.ts | 49 +++++++ frontend/src/pages/tldr/sitemap.xml.ts | 127 ++++-------------- 6 files changed, 201 insertions(+), 120 deletions(-) create mode 100644 frontend/src/pages/tldr/sitemap-[page].xml.ts diff --git a/frontend/db/tldrs/tldr-utils.ts b/frontend/db/tldrs/tldr-utils.ts index 622d4461e7..e998d698ab 100644 --- a/frontend/db/tldrs/tldr-utils.ts +++ b/frontend/db/tldrs/tldr-utils.ts @@ -25,4 +25,8 @@ export async function getTldrPage(platform: string, slug: string) { export async function getTldrOverview() { return await query.getOverview(); +} + +export async function getTldrSitemap(url: string) { + return await query.getSitemap(url); } \ No newline at end of file diff --git a/frontend/db/tldrs/tldr-worker-pool.ts b/frontend/db/tldrs/tldr-worker-pool.ts index 4808c1e0c2..d567838eb5 100644 --- a/frontend/db/tldrs/tldr-worker-pool.ts +++ b/frontend/db/tldrs/tldr-worker-pool.ts @@ -241,6 +241,7 @@ export const query = { getOverview: () => executeQuery('getOverview', {}), getMainPage: (platform: string, page: number) => executeQuery('getMainPage', { platform, page }), getPage: (platform: string, slug: string) => executeQuery('getPage', { platform, slug }), + getSitemap: (url: string) => executeQuery('getSitemap', { url }), }; void initWorkers().catch((err) => { diff --git a/frontend/db/tldrs/tldr-worker.ts b/frontend/db/tldrs/tldr-worker.ts index 867e2dbe66..bac250cbee 100644 --- a/frontend/db/tldrs/tldr-worker.ts +++ b/frontend/db/tldrs/tldr-worker.ts @@ -49,6 +49,10 @@ const statements = { getPage: db.prepare( `SELECT html_content, metadata FROM pages WHERE url_hash = ?` ), + + getSitemap: db.prepare( + `SELECT data FROM sitemap WHERE hash = ?` + ), }; // clusterPreviews is taking 0.5 seconds need to improve db structure for this @@ -114,6 +118,19 @@ parentPort?.on('message', (message: QueryMessage) => { break; } + case 'getSitemap': { + const { url } = params; + const hashKey = `sitemap/${url}`; + const hash = hashUrlToKey(hashKey); + const row = statements.getSitemap.get(hash) as { data: string } | undefined; + if (row) { + result = JSON.parse(row.data); + } else { + result = null; + } + break; + } + default: throw new Error(`Unknown query type: ${type}`); } diff --git a/frontend/scripts/tldr/tldr_to_db.go b/frontend/scripts/tldr/tldr_to_db.go index e4b3e7e05f..cb18699b98 100644 --- a/frontend/scripts/tldr/tldr_to_db.go +++ b/frontend/scripts/tldr/tldr_to_db.go @@ -42,6 +42,7 @@ type MainPage struct { Hash int64 Data string // JSON TotalCount int + Url string } // Intermediate struct for processing @@ -193,6 +194,7 @@ func ensureSchema(db *sql.DB) error { _, err = db.Exec(` CREATE TABLE IF NOT EXISTS main_pages ( hash INTEGER PRIMARY KEY, + url TEXT NOT NULL, data TEXT DEFAULT '{}', -- JSON total_count INTEGER NOT NULL ) WITHOUT ROWID; @@ -201,12 +203,13 @@ func ensureSchema(db *sql.DB) error { return err } - // Overview table + // Sitemap table _, err = db.Exec(` - CREATE TABLE IF NOT EXISTS overview ( - id INTEGER PRIMARY KEY CHECK(id = 1), - total_count INTEGER NOT NULL - ); + CREATE TABLE IF NOT EXISTS sitemap ( + hash INTEGER PRIMARY KEY, + url TEXT NOT NULL, + data TEXT DEFAULT '[]' -- JSON list of URLs + ) WITHOUT ROWID; `) return err } @@ -232,7 +235,8 @@ func main() { // 1. Parse all files into memory fmt.Printf("Scanning %s...\n", DataDir) var allPages []*ProcessedPage - + var allUrls []string + err = filepath.WalkDir(DataDir, func(path string, d fs.DirEntry, err error) error { if err != nil { return err @@ -273,11 +277,12 @@ func main() { Features: p.Features, } metaJson, _ := json.Marshal(meta) - + _, err = stmt.Exec(p.UrlHash, p.Url, p.HtmlContent, string(metaJson)) if err != nil { log.Printf("Error inserting page %s: %v", p.Name, err) } + allUrls = append(allUrls, p.Url) } tx.Commit() @@ -292,16 +297,16 @@ func main() { if err != nil { log.Fatal(err) } - stmt, err = tx.Prepare("INSERT INTO main_pages (hash, data, total_count) VALUES (?, ?, ?)") + stmt, err = tx.Prepare("INSERT INTO main_pages (hash, url, data, total_count) VALUES (?, ?, ?, ?)") if err != nil { log.Fatal(err) } itemsPerPage := 30 - + // For Index Page (List of Platforms) var platforms []map[string]interface{} - + for cluster, pages := range pagesByCluster { // Sort pages by name sort.Slice(pages, func(i, j int) bool { @@ -325,13 +330,15 @@ func main() { }) } + clusterUrl := fmt.Sprintf("/freedevtools/tldr/%s/", cluster) platformData := map[string]interface{}{ "name": cluster, "count": totalCount, - "url": fmt.Sprintf("/freedevtools/tldr/%s/", cluster), + "url": clusterUrl, "commands": commandPreviews, } platforms = append(platforms, platformData) + allUrls = append(allUrls, clusterUrl) // Generate paginated lists for this cluster for i := 0; i < totalPages; i++ { @@ -365,8 +372,17 @@ func main() { // Hash: cluster/page (e.g., common/1) hashKey := fmt.Sprintf("%s/%d", cluster, pageNum) hash := get8Bytes(hashString(hashKey)) + + // Construct URL for this page + var pageUrl string + if pageNum == 1 { + pageUrl = fmt.Sprintf("/freedevtools/tldr/%s/", cluster) + } else { + pageUrl = fmt.Sprintf("/freedevtools/tldr/%s/%d/", cluster, pageNum) + allUrls = append(allUrls, pageUrl) + } - _, err = stmt.Exec(hash, string(dataJson), totalCount) + _, err = stmt.Exec(hash, pageUrl, string(dataJson), totalCount) if err != nil { log.Printf("Error inserting cluster page %s: %v", hashKey, err) } @@ -404,19 +420,92 @@ func main() { hashKey := fmt.Sprintf("index/%d", pageNum) hash := get8Bytes(hashString(hashKey)) - _, err = stmt.Exec(hash, string(dataJson), totalPlatforms) + // Construct URL for this page + var pageUrl string + if pageNum == 1 { + pageUrl = "/freedevtools/tldr/" + allUrls = append(allUrls, pageUrl) + } else { + pageUrl = fmt.Sprintf("/freedevtools/tldr/%d/", pageNum) + allUrls = append(allUrls, pageUrl) + } + + _, err = stmt.Exec(hash, pageUrl, string(dataJson), totalPlatforms) if err != nil { log.Printf("Error inserting index page %s: %v", hashKey, err) } } tx.Commit() - // 5. Overview - if _, err := db.Exec("INSERT INTO overview (id, total_count) VALUES (1, ?)", len(allPages)); err != nil { + // 5. Generate Sitemap + fmt.Println("Generating sitemap...") + tx, err = db.Begin() + if err != nil { + log.Fatal(err) + } + stmt, err = tx.Prepare("INSERT INTO sitemap (hash, url, data) VALUES (?, ?, ?)") + if err != nil { log.Fatal(err) } + defer stmt.Close() + + // Sort all URLs to ensure deterministic output + sort.Strings(allUrls) + + // Filter out duplicates if any + uniqueUrls := make([]string, 0, len(allUrls)) + seenUrls := make(map[string]bool) + for _, u := range allUrls { + if !seenUrls[u] { + seenUrls[u] = true + uniqueUrls = append(uniqueUrls, u) + } + } + allUrls = uniqueUrls + + sitemapChunkSize := 5000 + totalSitemapChunks := int(math.Ceil(float64(len(allUrls)) / float64(sitemapChunkSize))) + var sitemapIndexUrls []string + + for i := 0; i < totalSitemapChunks; i++ { + chunkNum := i + 1 + startIdx := i * sitemapChunkSize + endIdx := startIdx + sitemapChunkSize + if endIdx > len(allUrls) { + endIdx = len(allUrls) + } + + chunkUrls := allUrls[startIdx:endIdx] + chunkJson, _ := json.Marshal(chunkUrls) + + sitemapName := fmt.Sprintf("sitemap-%d.xml", chunkNum) + sitemapUrl := fmt.Sprintf("/freedevtools/tldr/%s", sitemapName) + sitemapIndexUrls = append(sitemapIndexUrls, sitemapUrl) + + // Hash: sitemap/sitemap-N.xml + hashKey := fmt.Sprintf("sitemap/%s", sitemapName) + hash := get8Bytes(hashString(hashKey)) + + _, err = stmt.Exec(hash, sitemapName, string(chunkJson)) + if err != nil { + log.Printf("Error inserting sitemap chunk %s: %v", sitemapName, err) + } + } + + // Insert Sitemap Index + indexJson, _ := json.Marshal(sitemapIndexUrls) + indexName := "sitemap.xml" + // Hash: sitemap/sitemap.xml + hashKey := fmt.Sprintf("sitemap/%s", indexName) + hash := get8Bytes(hashString(hashKey)) + + _, err = stmt.Exec(hash, indexName, string(indexJson)) + if err != nil { + log.Printf("Error inserting sitemap index: %v", err) + } + + tx.Commit() elapsed := time.Since(start) - fmt.Printf("Finished! Processed %d pages in %s.\n", len(allPages), elapsed) + fmt.Printf("Finished! Processed %d pages and %d URLs in %s.\n", len(allPages), len(allUrls), elapsed) } - diff --git a/frontend/src/pages/tldr/sitemap-[page].xml.ts b/frontend/src/pages/tldr/sitemap-[page].xml.ts new file mode 100644 index 0000000000..2b8f0104c5 --- /dev/null +++ b/frontend/src/pages/tldr/sitemap-[page].xml.ts @@ -0,0 +1,49 @@ +import type { APIRoute } from 'astro'; +import { getTldrSitemap } from '../../../db/tldrs/tldr-utils'; + +export const GET: APIRoute = async ({ params, site }) => { + const { page } = params; + const now = new Date().toISOString(); + + // Use site from .env file (SITE variable) or astro.config.mjs + const envSite = process.env.SITE; + const siteStr = site?.toString() || ''; + const siteUrl = envSite || siteStr || 'http://localhost:4321/freedevtools'; + + if (!page) { + return new Response('Page not found', { status: 404 }); + } + + const sitemapName = `sitemap-${page}.xml`; + const urls = await getTldrSitemap(sitemapName); + + if (!urls) { + return new Response('Sitemap chunk not found', { status: 404 }); + } + + const xml = ` + + +${urls.map((url: string) => ` + ${siteUrl.replace(/\/freedevtools$/, '')}${url} + ${now} + daily + 0.8 + `).join('\n')} +`; + + return new Response(xml, { + headers: { + 'Content-Type': 'application/xml', + 'Cache-Control': 'public, max-age=3600', + }, + }); +}; + +export function getStaticPaths() { + // This is a dynamic route, but we don't know the paths at build time easily without querying DB. + // However, since we are using SSR (server: 'server' in astro config likely), we don't strictly need getStaticPaths + // unless prerender is true. + // Assuming SSR mode for these dynamic sitemaps. + return []; +} diff --git a/frontend/src/pages/tldr/sitemap.xml.ts b/frontend/src/pages/tldr/sitemap.xml.ts index 841202a46d..32eb76e780 100644 --- a/frontend/src/pages/tldr/sitemap.xml.ts +++ b/frontend/src/pages/tldr/sitemap.xml.ts @@ -1,111 +1,32 @@ - import type { APIRoute } from 'astro'; -import { - getAllTldrPlatforms, - getTldrPlatformCommandsPaginated, -} from '../../../db/tldrs/tldr-utils'; - -async function getAllData() { - const platforms: any[] = []; - const commandsByPlatform: Record = {}; - const platformPaginationCounts: Record = {}; - - // 1. Fetch all platforms - const firstPage = await getAllTldrPlatforms(1); - if (!firstPage) return { platforms, commandsByPlatform, totalIndexPages: 0, platformPaginationCounts }; - - const totalIndexPages = firstPage.total_pages; - platforms.push(...firstPage.platforms); - - for (let i = 2; i <= totalIndexPages; i++) { - const page = await getAllTldrPlatforms(i); - if (page) { - platforms.push(...page.platforms); - } - } - - // 2. Fetch all commands for each platform - for (const platform of platforms) { - const platformName = platform.name; - commandsByPlatform[platformName] = []; - - const firstCmdPage = await getTldrPlatformCommandsPaginated(platformName, 1); - if (firstCmdPage) { - platformPaginationCounts[platformName] = firstCmdPage.total_pages; - commandsByPlatform[platformName].push(...firstCmdPage.commands); - - for (let i = 2; i <= firstCmdPage.total_pages; i++) { - const cmdPage = await getTldrPlatformCommandsPaginated(platformName, i); - if (cmdPage) { - commandsByPlatform[platformName].push(...cmdPage.commands); - } - } - } - } - - return { platforms, commandsByPlatform, totalIndexPages, platformPaginationCounts }; -} +import { getTldrSitemap } from '../../../db/tldrs/tldr-utils'; export const GET: APIRoute = async ({ site }) => { const now = new Date().toISOString(); - const { platforms, commandsByPlatform, totalIndexPages, platformPaginationCounts } = await getAllData(); - - if (!site) { - throw new Error('Site is not defined'); - } - - const urls: string[] = []; - // Category landing - urls.push( - ` \n < loc > ${ site } /tldr/ < /loc>\n ${now}\n < changefreq > daily < /changefreq>\n 0.9\n ` - ); - -// Platform pages -for (const platform of platforms) { - urls.push( - ` \n ${site}/tldr/${platform.name}/\n ${now}\n daily\n 0.8\n ` - ); -} - -// Main TLDR pagination pages -for (let i = 2; i <= totalIndexPages; i++) { - urls.push( - ` \n ${site}/tldr/${i}/\n ${now}\n daily\n 0.8\n ` - ); -} - -// Platform pagination pages -for (const [platformName, totalPages] of Object.entries(platformPaginationCounts)) { - for (let i = 2; i <= totalPages; i++) { - urls.push( - ` \n ${site}/tldr/${platformName}/${i}/\n ${now}\n daily\n 0.8\n ` - ); - } -} - -// Individual command pages -for (const [platformName, commands] of Object.entries(commandsByPlatform)) { - for (const cmd of commands) { - urls.push( - ` \n ${site}/tldr/${platformName}/${cmd.name}/\n ${now}\n daily\n 0.8\n ` - ); + // Use site from .env file (SITE variable) or astro.config.mjs + const envSite = process.env.SITE; + const siteStr = site?.toString() || ''; + const siteUrl = envSite || siteStr || 'http://localhost:4321/freedevtools'; + + const sitemapUrls = await getTldrSitemap('sitemap.xml'); + + if (!sitemapUrls) { + return new Response('Sitemap index not found', { status: 404 }); } -} - -// Sort URLs -urls.sort((a, b) => { - const urlA = a.match(/(.*?)<\/loc>/)?.[1] || ''; - const urlB = b.match(/(.*?)<\/loc>/)?.[1] || ''; - return urlA.localeCompare(urlB); -}); - -const xml = `\n\n\n${urls.join('\n')}\n`; -return new Response(xml, { - headers: { - 'Content-Type': 'application/xml', - 'Cache-Control': 'public, max-age=3600', - }, -}); + const xml = ` + +${sitemapUrls.map((url: string) => ` + ${siteUrl.replace(/\/freedevtools$/, '')}${url} + ${now} + `).join('\n')} +`; + + return new Response(xml, { + headers: { + 'Content-Type': 'application/xml', + 'Cache-Control': 'public, max-age=3600', + }, + }); }; \ No newline at end of file From e14ab35042599a59bfe0a4aee30c059b044a4f4a Mon Sep 17 00:00:00 2001 From: Ganesh Kumar Date: Sun, 7 Dec 2025 11:31:24 +0530 Subject: [PATCH 203/234] refactor: Restructure frontend pages and tools by migrating to `_` prefixed directories and updating the server script. --- frontend/scripts/tldr/tldr_to_db.go | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/frontend/scripts/tldr/tldr_to_db.go b/frontend/scripts/tldr/tldr_to_db.go index cb18699b98..dde6699953 100644 --- a/frontend/scripts/tldr/tldr_to_db.go +++ b/frontend/scripts/tldr/tldr_to_db.go @@ -211,6 +211,17 @@ func ensureSchema(db *sql.DB) error { data TEXT DEFAULT '[]' -- JSON list of URLs ) WITHOUT ROWID; `) + if err != nil { + return err + } + + // Overview table + _, err = db.Exec(` + CREATE TABLE IF NOT EXISTS overview ( + id INTEGER PRIMARY KEY CHECK(id = 1), + total_count INTEGER NOT NULL + ); + `) return err } @@ -506,6 +517,11 @@ func main() { tx.Commit() + // 6. Overview + if _, err := db.Exec("INSERT INTO overview (id, total_count) VALUES (1, ?)", len(allPages)); err != nil { + log.Fatal(err) + } + elapsed := time.Since(start) fmt.Printf("Finished! Processed %d pages and %d URLs in %s.\n", len(allPages), len(allUrls), elapsed) } From cd4caef9af2b135159309c0dc97f1b509de7dc14 Mon Sep 17 00:00:00 2001 From: Ganesh Kumar Date: Sun, 7 Dec 2025 11:36:44 +0530 Subject: [PATCH 204/234] refactor: Remove unused `getStaticPaths` function and reduce bun's max old space size. --- frontend/src/pages/tldr/sitemap-[page].xml.ts | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/frontend/src/pages/tldr/sitemap-[page].xml.ts b/frontend/src/pages/tldr/sitemap-[page].xml.ts index 2b8f0104c5..a36e74cf88 100644 --- a/frontend/src/pages/tldr/sitemap-[page].xml.ts +++ b/frontend/src/pages/tldr/sitemap-[page].xml.ts @@ -40,10 +40,4 @@ ${urls.map((url: string) => ` }); }; -export function getStaticPaths() { - // This is a dynamic route, but we don't know the paths at build time easily without querying DB. - // However, since we are using SSR (server: 'server' in astro config likely), we don't strictly need getStaticPaths - // unless prerender is true. - // Assuming SSR mode for these dynamic sitemaps. - return []; -} + From 00ea2895c316f44ff581d1cce9bdc8c8277886d6 Mon Sep 17 00:00:00 2001 From: Ganesh Kumar Date: Sun, 7 Dec 2025 11:49:58 +0530 Subject: [PATCH 205/234] feat: link XSL stylesheet to sitemap and reduce Bun server's max old space size. --- frontend/src/pages/tldr/sitemap.xml.ts | 1 + 1 file changed, 1 insertion(+) diff --git a/frontend/src/pages/tldr/sitemap.xml.ts b/frontend/src/pages/tldr/sitemap.xml.ts index 32eb76e780..0b02cda7f4 100644 --- a/frontend/src/pages/tldr/sitemap.xml.ts +++ b/frontend/src/pages/tldr/sitemap.xml.ts @@ -16,6 +16,7 @@ export const GET: APIRoute = async ({ site }) => { } const xml = ` + ${sitemapUrls.map((url: string) => ` ${siteUrl.replace(/\/freedevtools$/, '')}${url} From 3a569b308e9ed4d38ddb82ff74fd100121c70e58 Mon Sep 17 00:00:00 2001 From: Ganesh Kumar Date: Sun, 7 Dec 2025 13:11:48 +0530 Subject: [PATCH 206/234] chore: Reduce Bun server memory, ensure TLDR page URLs have trailing slashes, and ignore the `tldr_to_db` executable. --- frontend/scripts/tldr/tldr_to_db.go | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/frontend/scripts/tldr/tldr_to_db.go b/frontend/scripts/tldr/tldr_to_db.go index dde6699953..95538426ef 100644 --- a/frontend/scripts/tldr/tldr_to_db.go +++ b/frontend/scripts/tldr/tldr_to_db.go @@ -154,9 +154,12 @@ func parseTldrFile(path string) (*ProcessedPage, error) { name := strings.TrimSuffix(filepath.Base(path), filepath.Ext(path)) fullHash := createFullHash(cluster, name) urlHash := get8Bytes(fullHash) - pathUrl := strings.TrimSuffix(fm.Path, "/") + pathUrl := fm.Path if pathUrl == "" { - pathUrl = fmt.Sprintf("/freedevtools/tldr/%s/%s", cluster, name) + pathUrl = fmt.Sprintf("/freedevtools/tldr/%s/%s/", cluster, name) + } + if !strings.HasSuffix(pathUrl, "/") { + pathUrl += "/" } return &ProcessedPage{ From bd12577764bb24bdb808edfcc73a21c0081662bc Mon Sep 17 00:00:00 2001 From: Ganesh Kumar Date: Sun, 7 Dec 2025 13:12:55 +0530 Subject: [PATCH 207/234] chore: Ignore `tldr_to_db` script and reduce frontend server max old space size. --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index f6cef511ec..8af70fff50 100644 --- a/.gitignore +++ b/.gitignore @@ -162,3 +162,4 @@ frontend/env.local frontend/env.prod frontend/env.staging frontend/bun.lockb +frontend/scripts/tldr/tldr_to_db From 28ebcee1802a66b8c674ae6e1374a792163fef5f Mon Sep 17 00:00:00 2001 From: lovestaco Date: Sun, 7 Dec 2025 15:20:04 +0530 Subject: [PATCH 208/234] fix: use 8gb --- frontend/scripts/builds/emojis.js | 4 ++-- frontend/scripts/builds/icons.js | 2 +- frontend/scripts/builds/index.js | 2 +- frontend/scripts/builds/man-pages.js | 4 ++-- frontend/scripts/builds/mcp.js | 4 ++-- frontend/scripts/builds/tldr.js | 4 ++-- frontend/serve/start-server.sh | 2 +- 7 files changed, 11 insertions(+), 11 deletions(-) diff --git a/frontend/scripts/builds/emojis.js b/frontend/scripts/builds/emojis.js index c461c1d9a5..6613e9bba7 100644 --- a/frontend/scripts/builds/emojis.js +++ b/frontend/scripts/builds/emojis.js @@ -113,7 +113,7 @@ function installDependencies() { stdio: 'inherit', env: { ...process.env, - NODE_OPTIONS: '--max-old-space-size=16384', // 14GB for 16GB system + NODE_OPTIONS: '--max-old-space-size=8192', // 8GB for 8GB system UV_THREADPOOL_SIZE: '64' // 4x cores for I/O operations } }); @@ -133,7 +133,7 @@ function buildProject() { stdio: 'inherit', env: { ...process.env, - NODE_OPTIONS: '--max-old-space-size=16384', // 14GB for 16GB system + NODE_OPTIONS: '--max-old-space-size=8192', // 8GB for 8GB system UV_THREADPOOL_SIZE: '64', // 4x cores for I/O operations NODE_OPTIONS_EXTRA: '--experimental-loader' // Enable experimental features for better performance } diff --git a/frontend/scripts/builds/icons.js b/frontend/scripts/builds/icons.js index 5f5443454d..2c790f562a 100644 --- a/frontend/scripts/builds/icons.js +++ b/frontend/scripts/builds/icons.js @@ -124,7 +124,7 @@ function buildProject() { stdio: 'inherit', env: { ...process.env, - NODE_OPTIONS: '--max-old-space-size=16384', // 14GB for 16GB system + NODE_OPTIONS: '--max-old-space-size=8192', // 8GB for 8GB system UV_THREADPOOL_SIZE: '64', // 4x cores for I/O operations NODE_OPTIONS_EXTRA: '--experimental-loader', // Enable experimental features for better performance }, diff --git a/frontend/scripts/builds/index.js b/frontend/scripts/builds/index.js index 0e447812c6..01c8d4bd63 100755 --- a/frontend/scripts/builds/index.js +++ b/frontend/scripts/builds/index.js @@ -112,7 +112,7 @@ function buildProject() { stdio: 'inherit', env: { ...process.env, - NODE_OPTIONS: '--max-old-space-size=16384', // 14GB for 16GB system + NODE_OPTIONS: '--max-old-space-size=8192', // 8GB for 8GB system UV_THREADPOOL_SIZE: '64', // 4x cores for I/O operations NODE_OPTIONS_EXTRA: '--experimental-loader', // Enable experimental features for better performance }, diff --git a/frontend/scripts/builds/man-pages.js b/frontend/scripts/builds/man-pages.js index 7ee0d728fe..16eeb9bbea 100755 --- a/frontend/scripts/builds/man-pages.js +++ b/frontend/scripts/builds/man-pages.js @@ -113,7 +113,7 @@ function installDependencies() { stdio: 'inherit', env: { ...process.env, - NODE_OPTIONS: '--max-old-space-size=16384', // 14GB for 16GB system + NODE_OPTIONS: '--max-old-space-size=8192', // 8GB for 8GB system UV_THREADPOOL_SIZE: '64' // 4x cores for I/O operations } }); @@ -133,7 +133,7 @@ function buildProject() { stdio: 'inherit', env: { ...process.env, - NODE_OPTIONS: '--max-old-space-size=16384', // 14GB for 16GB system + NODE_OPTIONS: '--max-old-space-size=8192', // 8GB for 8GB system UV_THREADPOOL_SIZE: '64', // 4x cores for I/O operations NODE_OPTIONS_EXTRA: '--experimental-loader' // Enable experimental features for better performance } diff --git a/frontend/scripts/builds/mcp.js b/frontend/scripts/builds/mcp.js index e51abe1695..c6108a34cc 100644 --- a/frontend/scripts/builds/mcp.js +++ b/frontend/scripts/builds/mcp.js @@ -113,7 +113,7 @@ function installDependencies() { stdio: 'inherit', env: { ...process.env, - NODE_OPTIONS: '--max-old-space-size=16384', // 14GB for 16GB system + NODE_OPTIONS: '--max-old-space-size=8192', // 8GB for 8GB system UV_THREADPOOL_SIZE: '64' // 4x cores for I/O operations } }); @@ -133,7 +133,7 @@ function buildProject() { stdio: 'inherit', env: { ...process.env, - NODE_OPTIONS: '--max-old-space-size=16384', // 14GB for 16GB system + NODE_OPTIONS: '--max-old-space-size=8192', // 8GB for 8GB system UV_THREADPOOL_SIZE: '64', // 4x cores for I/O operations NODE_OPTIONS_EXTRA: '--experimental-loader' // Enable experimental features for better performance } diff --git a/frontend/scripts/builds/tldr.js b/frontend/scripts/builds/tldr.js index 888724b2a8..143efda4a5 100644 --- a/frontend/scripts/builds/tldr.js +++ b/frontend/scripts/builds/tldr.js @@ -113,7 +113,7 @@ function installDependencies() { stdio: 'inherit', env: { ...process.env, - NODE_OPTIONS: '--max-old-space-size=16384', // 14GB for 16GB system + NODE_OPTIONS: '--max-old-space-size=8192', // 8GB for 8GB system UV_THREADPOOL_SIZE: '64' // 4x cores for I/O operations } }); @@ -133,7 +133,7 @@ function buildProject() { stdio: 'inherit', env: { ...process.env, - NODE_OPTIONS: '--max-old-space-size=16384', // 14GB for 16GB system + NODE_OPTIONS: '--max-old-space-size=8192', // 8GB for 8GB system UV_THREADPOOL_SIZE: '64', // 4x cores for I/O operations NODE_OPTIONS_EXTRA: '--experimental-loader' // Enable experimental features for better performance } diff --git a/frontend/serve/start-server.sh b/frontend/serve/start-server.sh index 773be0d98f..9b631cd16d 100755 --- a/frontend/serve/start-server.sh +++ b/frontend/serve/start-server.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash export UV_THREADPOOL_SIZE=64 export HOST=127.0.0.1 -bun --max-old-space-size=16384 ./dist/server/entry.mjs --port ${PORT:-4321} +bun --max-old-space-size=8192 ./dist/server/entry.mjs --port ${PORT:-4321} From 170c77c21167eaa1f06becf9f865e9c7b5a1b473 Mon Sep 17 00:00:00 2001 From: Ganesh Kumar Date: Sun, 7 Dec 2025 15:22:02 +0530 Subject: [PATCH 209/234] feat: Refactor TLDR schema to use a cluster table, removing main pages and sitemap tables, and updating data generation and frontend pages. --- frontend/db/tldrs/tldr-schema.ts | 35 +- frontend/db/tldrs/tldr-utils.ts | 35 +- frontend/db/tldrs/tldr-worker-pool.ts | 7 +- frontend/db/tldrs/tldr-worker.ts | 100 ++- frontend/metric_analisi.md | 831 ------------------ frontend/scripts/tldr/tldr_to_db.go | 213 +---- .../src/pages/tldr/[platform]/[slug].astro | 18 +- .../src/pages/tldr/[platform]/index.astro | 69 +- frontend/src/pages/tldr/index.astro | 27 +- frontend/src/pages/tldr/sitemap-[page].xml.ts | 70 +- frontend/src/pages/tldr/sitemap.xml.ts | 34 +- 11 files changed, 272 insertions(+), 1167 deletions(-) delete mode 100644 frontend/metric_analisi.md diff --git a/frontend/db/tldrs/tldr-schema.ts b/frontend/db/tldrs/tldr-schema.ts index dc969aeba7..746e941cdb 100644 --- a/frontend/db/tldrs/tldr-schema.ts +++ b/frontend/db/tldrs/tldr-schema.ts @@ -10,37 +10,32 @@ export interface Page { metadata: PageMetadata; } -export interface CommandSummary { +export interface PreviewCommand { name: string; url: string; +} + +export interface Command extends PreviewCommand { description: string; - category: string; features: string[]; } -export interface PlatformSummary { +export interface Cluster { name: string; count: number; - url: string; -} - -export interface MainPageData { - commands?: CommandSummary[]; - platforms?: PlatformSummary[]; - total: number; - page: number; - total_pages: number; -} - -export interface MainPage { - hash: string; - data: MainPageData; - total_count: number; + preview_commands: PreviewCommand[]; } export interface RawClusterRow { + hash: number; name: string; - hash_name: string; count: number; - description: string; + preview_commands_json: string; +} + +export interface RawPageRow { + url_hash: number; + url: string; + html_content: string; + metadata: string; } \ No newline at end of file diff --git a/frontend/db/tldrs/tldr-utils.ts b/frontend/db/tldrs/tldr-utils.ts index e998d698ab..890733e050 100644 --- a/frontend/db/tldrs/tldr-utils.ts +++ b/frontend/db/tldrs/tldr-utils.ts @@ -1,21 +1,24 @@ +import type { Cluster, Command, Page } from './tldr-schema'; import { query } from './tldr-worker-pool'; +export async function getTldrCluster(cluster: string): Promise { + return await query.getMainPage(cluster); +} - -export async function getTldrPlatformCommandsPaginated( - platform: string, - page: number = 1 -) { - const result = await query.getMainPage(platform, page); - return result; +export async function getTldrCommandsByClusterPaginated( + cluster: string, + page: number = 1, + limit: number = 30 +): Promise { + const offset = (page - 1) * limit; + return await query.getCommandsByClusterPaginated(cluster, limit, offset); } -export async function getAllTldrPlatforms(page: number = 1) { - const result = await query.getMainPage('index', page); - return result; +export async function getAllTldrClusters(): Promise { + return await query.getAllClusters(); } -export async function getTldrPage(platform: string, slug: string) { +export async function getTldrPage(platform: string, slug: string): Promise { const page = await query.getPage(platform, slug); if (!page) return null; @@ -23,10 +26,14 @@ export async function getTldrPage(platform: string, slug: string) { return page; } -export async function getTldrOverview() { +export async function getTldrOverview(): Promise { return await query.getOverview(); } -export async function getTldrSitemap(url: string) { - return await query.getSitemap(url); +export async function getTldrSitemapUrls(limit: number, offset: number): Promise { + return await query.getSitemapUrls(limit, offset); +} + +export async function getTldrSitemapCount(): Promise { + return await query.getSitemapCount(); } \ No newline at end of file diff --git a/frontend/db/tldrs/tldr-worker-pool.ts b/frontend/db/tldrs/tldr-worker-pool.ts index d567838eb5..3e71f976cf 100644 --- a/frontend/db/tldrs/tldr-worker-pool.ts +++ b/frontend/db/tldrs/tldr-worker-pool.ts @@ -239,9 +239,12 @@ export function cleanupWorkers(): Promise { // Export query functions export const query = { getOverview: () => executeQuery('getOverview', {}), - getMainPage: (platform: string, page: number) => executeQuery('getMainPage', { platform, page }), + getMainPage: (platform: string) => executeQuery('getMainPage', { platform }), getPage: (platform: string, slug: string) => executeQuery('getPage', { platform, slug }), - getSitemap: (url: string) => executeQuery('getSitemap', { url }), + getAllClusters: () => executeQuery('getAllClusters', {}), + getCommandsByClusterPaginated: (cluster: string, limit: number, offset: number) => executeQuery('getCommandsByClusterPaginated', { cluster, limit, offset }), + getSitemapUrls: (limit: number, offset: number) => executeQuery('getSitemapUrls', { limit, offset }), + getSitemapCount: () => executeQuery('getSitemapCount', {}), }; void initWorkers().catch((err) => { diff --git a/frontend/db/tldrs/tldr-worker.ts b/frontend/db/tldrs/tldr-worker.ts index bac250cbee..be73a6d2fb 100644 --- a/frontend/db/tldrs/tldr-worker.ts +++ b/frontend/db/tldrs/tldr-worker.ts @@ -43,15 +43,46 @@ const statements = { getOverview: db.prepare('SELECT total_count FROM overview WHERE id = 1'), getMainPage: db.prepare( - `SELECT data, total_count FROM main_pages WHERE hash = ?` + `SELECT name, count, preview_commands_json FROM cluster WHERE hash = ?` ), getPage: db.prepare( `SELECT html_content, metadata FROM pages WHERE url_hash = ?` ), - getSitemap: db.prepare( - `SELECT data FROM sitemap WHERE hash = ?` + // New query for paginated commands in a cluster + getCommandsByClusterPaginated: db.prepare( + `SELECT url, metadata FROM pages + WHERE url LIKE ? + ORDER BY url + LIMIT ? OFFSET ?` + ), + + // New query for all clusters (for index) + getAllClusters: db.prepare( + `SELECT name, count, preview_commands_json FROM cluster ORDER BY name` + ), + + // Sitemap query: Union of all URLs + getSitemapUrls: db.prepare( + `SELECT url FROM ( + SELECT '/freedevtools/tldr/' as url + UNION ALL + SELECT '/freedevtools/tldr/' || name || '/' as url FROM cluster + UNION ALL + SELECT url FROM pages + ) ORDER BY url LIMIT ? OFFSET ?` + ), + + // Count total URLs for sitemap index + getSitemapCount: db.prepare( + `SELECT COUNT(*) as count FROM ( + SELECT '/freedevtools/tldr/' as url + UNION ALL + SELECT name FROM cluster + UNION ALL + SELECT url FROM pages + )` ), }; @@ -87,14 +118,15 @@ parentPort?.on('message', (message: QueryMessage) => { } case 'getMainPage': { - const { platform, page } = params; - const hashKey = `${platform}/${page}`; - const hash = hashUrlToKey(hashKey); - const row = statements.getMainPage.get(hash) as { data: string; total_count: number } | undefined; + const { platform } = params; + // Hash of cluster name + const hash = hashUrlToKey(platform); + const row = statements.getMainPage.get(hash) as { name: string; count: number; preview_commands_json: string } | undefined; if (row) { result = { - ...JSON.parse(row.data), - total_count: row.total_count + name: row.name, + count: row.count, + preview_commands: JSON.parse(row.preview_commands_json) }; } else { result = null; @@ -118,16 +150,46 @@ parentPort?.on('message', (message: QueryMessage) => { break; } - case 'getSitemap': { - const { url } = params; - const hashKey = `sitemap/${url}`; - const hash = hashUrlToKey(hashKey); - const row = statements.getSitemap.get(hash) as { data: string } | undefined; - if (row) { - result = JSON.parse(row.data); - } else { - result = null; - } + case 'getCommandsByClusterPaginated': { + const { cluster, limit, offset } = params; + // URL pattern: /freedevtools/tldr/cluster/% + const urlPattern = `/freedevtools/tldr/${cluster}/%`; + const rows = statements.getCommandsByClusterPaginated.all(urlPattern, limit, offset) as { url: string; metadata: string }[]; + + result = rows.map(row => { + const meta = JSON.parse(row.metadata); + // Extract name from URL or metadata + const name = row.url.split('/').filter(Boolean).pop() || ''; + return { + name, + url: row.url, + description: meta.description, + features: meta.features + }; + }); + break; + } + + case 'getAllClusters': { + const rows = statements.getAllClusters.all() as { name: string; count: number; preview_commands_json: string }[]; + result = rows.map(row => ({ + name: row.name, + count: row.count, + preview_commands: JSON.parse(row.preview_commands_json) + })); + break; + } + + case 'getSitemapUrls': { + const { limit, offset } = params; + const rows = statements.getSitemapUrls.all(limit, offset) as { url: string }[]; + result = rows.map(row => row.url); + break; + } + + case 'getSitemapCount': { + const row = statements.getSitemapCount.get() as { count: number }; + result = row.count; break; } diff --git a/frontend/metric_analisi.md b/frontend/metric_analisi.md deleted file mode 100644 index fa7cfa6edf..0000000000 --- a/frontend/metric_analisi.md +++ /dev/null @@ -1,831 +0,0 @@ -## After Removing Body - -25-12-06T14:54:57.563548Z INFO pmdaemon::process: Process astro-4321 started with PID: 51698 -Started process 'astro-4321' with ID: 452b784c-fe1f-4d35-9934-b6a411027b68 -2025-12-06T14:54:57.564381Z INFO pmdaemon::manager: Allocated port 4322 to process astro-4322 -2025-12-06T14:54:57.564411Z INFO pmdaemon::process: Starting process: astro-4322 -2025-12-06T14:54:57.568546Z INFO pmdaemon::process: Process astro-4322 started with PID: 51700 -Started process 'astro-4322' with ID: 11213e67-1e8c-48a1-9876-1f3b7a0f7ec0 -Started 2 processes from config file -pmdaemon list -2025-12-06T14:54:59.594399Z INFO pmdaemon: PMDaemon v0.1.4 starting -2025-12-06T14:54:59.876916Z INFO pmdaemon::manager: Loaded 2 process configurations -┌──────────┬────────────┬────────┬───────┬────────┬──────────┬───────┬────────┬──────┐ -│ ID ┆ Name ┆ Status ┆ PID ┆ Uptime ┆ Restarts ┆ CPU % ┆ Memory ┆ Port │ -╞══════════╪════════════╪════════╪═══════╪════════╪══════════╪═══════╪════════╪══════╡ -│ 11213e67 ┆ astro-4322 ┆ online ┆ 51700 ┆ 0s ┆ 0 ┆ 0.0 ┆ - ┆ 4322 │ -├╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌┤ -│ 452b784c ┆ astro-4321 ┆ online ┆ 51698 ┆ 0s ┆ 0 ┆ 0.0 ┆ - ┆ 4321 │ -└──────────┴────────────┴────────┴───────┴────────┴──────────┴───────┴────────┴──────┘ -./pmd-pin.sh -Pinning PID 51698 (astro-4321) to CPU 0 -pid 51698's current affinity list: 0-3 -pid 51698's new affinity list: 0 -Pinning PID 51700 (astro-4322) to CPU 1 -pid 51700's current affinity list: 0-3 -pid 51700's new affinity list: 1 -./pmd-verifypin.sh -Checking CPU pinning for PMDaemon apps: astro-4321 astro-4322 --------------------------------------- -App: astro-4321 - PID: 51698 | pid 51698's current affinity list: 0 - -App: astro-4322 - PID: 51700 | pid 51700's current affinity list: 1 - --------------------------------------- -Done. -⬇️ Server up ----------- -⬇️ Warming up server logs will be flushed for the below requests, ignore the below curls ----------- -hey -z 10s -host hexmos-local.com http://127.0.0.1/freedevtools/tldr/adb/adb - -Summary: - Total: 10.0773 secs - Slowest: 0.4417 secs - Fastest: 0.0196 secs - Average: 0.1151 secs - Requests/sec: 431.7625 - - -Response time histogram: - 0.020 [1] | - 0.062 [621] |■■■■■■■■■■■■■■■ - 0.104 [1614] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ - 0.146 [1179] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ - 0.188 [508] |■■■■■■■■■■■■■ - 0.231 [190] |■■■■■ - 0.273 [89] |■■ - 0.315 [75] |■■ - 0.357 [33] |■ - 0.399 [37] |■ - 0.442 [4] | - - -Latency distribution: - 10% in 0.0573 secs - 25% in 0.0735 secs - 50% in 0.1024 secs - 75% in 0.1389 secs - 90% in 0.1877 secs - 95% in 0.2377 secs - 99% in 0.3527 secs - -Details (average, fastest, slowest): - DNS+dialup: 0.0001 secs, 0.0196 secs, 0.4417 secs - DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0000 secs - req write: 0.0001 secs, 0.0000 secs, 0.0381 secs - resp wait: 0.1141 secs, 0.0151 secs, 0.4414 secs - resp read: 0.0007 secs, 0.0001 secs, 0.0496 secs - -Status code distribution: - [200] 4351 responses - - - -Peeking at 4 log files in /home/gk/.pmdaemon/logs... - -Request count table: -+------------+------------+--------------+---------------+----------+ -| Process | Requests | Dispatches | Worker Logs | Errors | -+============+============+==============+===============+==========+ -| astro-4321 | 0 | 2167 | 4334 | 0 | -+------------+------------+--------------+---------------+----------+ -| astro-4322 | 0 | 2184 | 4368 | 0 | -+------------+------------+--------------+---------------+----------+ - -Total aggregated: - total lines: 26112 - requests: 0 - dispatches: 4351 - worker logs: 8702 - errors: 0 - -Dispatches represent each time the worker pool received a query and handed it off to a worker thread (logged via [SVG_ICONS_DB|PNG_ICONS_DB|EMOJI_DB|TLDR_DB] Dispatching ). - -Query Count Details process wise: -+------------+-----------+--------------------------------------------------+ -| Process | getPage | getPage params={"platform":"adb","slug":"adb"} | -+============+===========+==================================================+ -| astro-4321 | 2167 | 2167 | -+------------+-----------+--------------------------------------------------+ -| astro-4322 | 2184 | 2184 | -+------------+-----------+--------------------------------------------------+ - -Query Duration Details process wise: -+---------+---------+----------+----------+-----------+ -| Query | Count | Avg ms | Max ms | Sum min | -+=========+=========+==========+==========+===========+ -| getPage | 4348 | 28.1 | 172 | 2.04 | -+---------+---------+----------+----------+-----------+ - -Overall window: 2025-12-06T14:55:04.533000+00:00 → 2025-12-06T14:55:14.353000+00:00 -Total coverage: 0:00:09.820000 -⬇️ Requesting tldr 10sec -hey -z 10s -host hexmos-local.com http://127.0.0.1/freedevtools/tldr/adb - -Summary: - Total: 10.4494 secs - Slowest: 1.7798 secs - Fastest: 0.1264 secs - Average: 0.6972 secs - Requests/sec: 70.1477 - - -Response time histogram: - 0.126 [1] | - 0.292 [24] |■■■■ - 0.457 [101] |■■■■■■■■■■■■■■■■■■ - 0.622 [154] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■ - 0.788 [224] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ - 0.953 [126] |■■■■■■■■■■■■■■■■■■■■■■■ - 1.118 [58] |■■■■■■■■■■ - 1.284 [25] |■■■■ - 1.449 [12] |■■ - 1.614 [5] |■ - 1.780 [3] |■ - - -Latency distribution: - 10% in 0.3748 secs - 25% in 0.5263 secs - 50% in 0.6778 secs - 75% in 0.8459 secs - 90% in 1.0186 secs - 95% in 1.1422 secs - 99% in 1.4638 secs - -Details (average, fastest, slowest): - DNS+dialup: 0.0003 secs, 0.1264 secs, 1.7798 secs - DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0000 secs - req write: 0.0003 secs, 0.0000 secs, 0.0123 secs - resp wait: 0.6899 secs, 0.1229 secs, 1.7779 secs - resp read: 0.0059 secs, 0.0016 secs, 0.0564 secs - -Status code distribution: - [200] 733 responses - - - -Peeking at 4 log files in /home/gk/.pmdaemon/logs... - -Request count table: -+------------+------------+--------------+---------------+----------+ -| Process | Requests | Dispatches | Worker Logs | Errors | -+============+============+==============+===============+==========+ -| astro-4321 | 0 | 2546 | 5092 | 0 | -+------------+------------+--------------+---------------+----------+ -| astro-4322 | 0 | 2538 | 5076 | 0 | -+------------+------------+--------------+---------------+----------+ - -Total aggregated: - total lines: 31976 - requests: 0 - dispatches: 5084 - worker logs: 10168 - errors: 0 - -Dispatches represent each time the worker pool received a query and handed it off to a worker thread (logged via [SVG_ICONS_DB|PNG_ICONS_DB|EMOJI_DB|TLDR_DB] Dispatching ). - -Query Count Details process wise: -+------------+---------------+--------------------------------------------------+-----------+--------------------------------------------------+ -| Process | getMainPage | getMainPage params={"platform":"adb","page":1} | getPage | getPage params={"platform":"adb","slug":"adb"} | -+============+===============+==================================================+===========+==================================================+ -| astro-4321 | 379 | 379 | 2167 | 2167 | -+------------+---------------+--------------------------------------------------+-----------+--------------------------------------------------+ -| astro-4322 | 354 | 354 | 2184 | 2184 | -+------------+---------------+--------------------------------------------------+-----------+--------------------------------------------------+ - -Query Duration Details process wise: -+-------------+---------+----------+----------+-----------+ -| Query | Count | Avg ms | Max ms | Sum min | -+=============+=========+==========+==========+===========+ -| getPage | 4348 | 28.1 | 172 | 2.04 | -+-------------+---------+----------+----------+-----------+ -| getMainPage | 733 | 313.9 | 1137 | 3.84 | -+-------------+---------+----------+----------+-----------+ - -Overall window: 2025-12-06T14:55:04.533000+00:00 → 2025-12-06T14:55:27.777000+00:00 -Total coverage: 0:00:23.244000 -10s Requesting tldr main -hey -z 10s -host hexmos-local.com http://127.0.0.1/freedevtools/tldr - -Summary: - Total: 11.4074 secs - Slowest: 1.9906 secs - Fastest: 0.0813 secs - Average: 0.7989 secs - Requests/sec: 58.3833 - - -Response time histogram: - 0.081 [1] | - 0.272 [22] |■■■■■ - 0.463 [63] |■■■■■■■■■■■■■ - 0.654 [138] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ - 0.845 [193] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ - 1.036 [119] |■■■■■■■■■■■■■■■■■■■■■■■■■ - 1.227 [66] |■■■■■■■■■■■■■■ - 1.418 [34] |■■■■■■■ - 1.609 [12] |■■ - 1.800 [12] |■■ - 1.991 [6] |■ - - -Latency distribution: - 10% in 0.4215 secs - 25% in 0.5787 secs - 50% in 0.7709 secs - 75% in 0.9753 secs - 90% in 1.2127 secs - 95% in 1.4018 secs - 99% in 1.8291 secs - -Details (average, fastest, slowest): - DNS+dialup: 0.0018 secs, 0.0813 secs, 1.9906 secs - DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0000 secs - req write: 0.0002 secs, 0.0000 secs, 0.0312 secs - resp wait: 0.7901 secs, 0.0773 secs, 1.9882 secs - resp read: 0.0066 secs, 0.0019 secs, 0.0508 secs - -Status code distribution: - [200] 666 responses - - - -Peeking at 4 log files in /home/gk/.pmdaemon/logs... - -Request count table: -+------------+------------+--------------+---------------+----------+ -| Process | Requests | Dispatches | Worker Logs | Errors | -+============+============+==============+===============+==========+ -| astro-4321 | 0 | 2906 | 5812 | 0 | -+------------+------------+--------------+---------------+----------+ -| astro-4322 | 0 | 2844 | 5688 | 0 | -+------------+------------+--------------+---------------+----------+ - -Total aggregated: - total lines: 36638 - requests: 0 - dispatches: 5750 - worker logs: 11500 - errors: 0 - -Dispatches represent each time the worker pool received a query and handed it off to a worker thread (logged via [SVG_ICONS_DB|PNG_ICONS_DB|EMOJI_DB|TLDR_DB] Dispatching ). - -Query Count Details process wise: -+------------+---------------+--------------------------------------------------+----------------------------------------------------+-----------+--------------------------------------------------+ -| Process | getMainPage | getMainPage params={"platform":"adb","page":1} | getMainPage params={"platform":"index","page":1} | getPage | getPage params={"platform":"adb","slug":"adb"} | -+============+===============+==================================================+====================================================+===========+==================================================+ -| astro-4321 | 739 | 379 | 360 | 2167 | 2167 | -+------------+---------------+--------------------------------------------------+----------------------------------------------------+-----------+--------------------------------------------------+ -| astro-4322 | 660 | 354 | 306 | 2184 | 2184 | -+------------+---------------+--------------------------------------------------+----------------------------------------------------+-----------+--------------------------------------------------+ - -Query Duration Details process wise: -+-------------+---------+----------+----------+-----------+ -| Query | Count | Avg ms | Max ms | Sum min | -+=============+=========+==========+==========+===========+ -| getPage | 4348 | 28.1 | 172 | 2.04 | -+-------------+---------+----------+----------+-----------+ -| getMainPage | 1399 | 320.2 | 1137 | 7.47 | -+-------------+---------+----------+----------+-----------+ - -Overall window: 2025-12-06T14:55:04.533000+00:00 → 2025-12-06T14:55:42.117000+00:00 -Total coverage: 0:00:37.584000 -1min Requesting tldr 1min -hey -z 1m -host hexmos-local.com http://127.0.0.1/freedevtools/tldr/npm/npm-fund/ -^[[F -Summary: - Total: 60.1223 secs - Slowest: 0.5615 secs - Fastest: 0.0149 secs - Average: 0.0942 secs - Requests/sec: 530.2025 - - -Response time histogram: - 0.015 [1] | - 0.070 [9937] |■■■■■■■■■■■■■■■■■■■■■■■■■ - 0.124 [16046] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ - 0.179 [4520] |■■■■■■■■■■■ - 0.234 [949] |■■ - 0.288 [298] |■ - 0.343 [90] | - 0.398 [16] | - 0.452 [16] | - 0.507 [1] | - 0.562 [3] | - - -Latency distribution: - 10% in 0.0499 secs - 25% in 0.0643 secs - 50% in 0.0860 secs - 75% in 0.1136 secs - 90% in 0.1460 secs - 95% in 0.1731 secs - 99% in 0.2471 secs - -Details (average, fastest, slowest): - DNS+dialup: 0.0000 secs, 0.0149 secs, 0.5615 secs - DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0000 secs - req write: 0.0001 secs, 0.0000 secs, 0.0277 secs - resp wait: 0.0933 secs, 0.0074 secs, 0.5612 secs - resp read: 0.0007 secs, 0.0001 secs, 0.0437 secs - -Status code distribution: - [200] 31877 responses - - - -Peeking at 4 log files in /home/gk/.pmdaemon/logs... - -Request count table: -+------------+------------+--------------+---------------+----------+ -| Process | Requests | Dispatches | Worker Logs | Errors | -+============+============+==============+===============+==========+ -| astro-4321 | 0 | 18973 | 37946 | 0 | -+------------+------------+--------------+---------------+----------+ -| astro-4322 | 0 | 18654 | 37308 | 0 | -+------------+------------+--------------+---------------+----------+ - -Total aggregated: - total lines: 227900 - requests: 0 - dispatches: 37627 - worker logs: 75254 - errors: 0 - -Dispatches represent each time the worker pool received a query and handed it off to a worker thread (logged via [SVG_ICONS_DB|PNG_ICONS_DB|EMOJI_DB|TLDR_DB] Dispatching ). - -Query Count Details process wise: -+------------+---------------+--------------------------------------------------+----------------------------------------------------+-----------+--------------------------------------------------+-------------------------------------------------------+ -| Process | getMainPage | getMainPage params={"platform":"adb","page":1} | getMainPage params={"platform":"index","page":1} | getPage | getPage params={"platform":"adb","slug":"adb"} | getPage params={"platform":"npm","slug":"npm-fund"} | -+============+===============+==================================================+====================================================+===========+==================================================+=======================================================+ -| astro-4321 | 739 | 379 | 360 | 18234 | 2167 | 16067 | -+------------+---------------+--------------------------------------------------+----------------------------------------------------+-----------+--------------------------------------------------+-------------------------------------------------------+ -| astro-4322 | 660 | 354 | 306 | 17994 | 2184 | 15810 | -+------------+---------------+--------------------------------------------------+----------------------------------------------------+-----------+--------------------------------------------------+-------------------------------------------------------+ - -Query Duration Details process wise: -+-------------+---------+----------+----------+-----------+ -| Query | Count | Avg ms | Max ms | Sum min | -+=============+=========+==========+==========+===========+ -| getPage | 36217 | 21.8 | 172 | 13.15 | -+-------------+---------+----------+----------+-----------+ -| getMainPage | 1399 | 320.2 | 1137 | 7.47 | -+-------------+---------+----------+----------+-----------+ - -Overall window: 2025-12-06T14:55:04.533000+00:00 → 2025-12-06T14:56:46.257000+00:00 -Total coverage: 0:01:41.724000 -5min Requesting tldr 5min -hey -z 5m -host hexmos-local.com http://127.0.0.1/freedevtools/tldr/npm/npm-fund/ -^C -Summary: - Total: 137.1782 secs - Slowest: 0.4614 secs - Fastest: 0.0061 secs - Average: 0.0946 secs - Requests/sec: 528.5095 - - -Response time histogram: - 0.006 [1] | - 0.052 [9227] |■■■■■■■■■■ - 0.097 [35898] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ - 0.143 [18525] |■■■■■■■■■■■■■■■■■■■■■ - 0.188 [5496] |■■■■■■ - 0.234 [2113] |■■ - 0.279 [856] |■ - 0.325 [238] | - 0.370 [99] | - 0.416 [32] | - 0.461 [15] | - - -Latency distribution: - 10% in 0.0485 secs - 25% in 0.0633 secs - 50% in 0.0849 secs - 75% in 0.1137 secs - 90% in 0.1515 secs - 95% in 0.1841 secs - 99% in 0.2560 secs - -Details (average, fastest, slowest): - DNS+dialup: 0.0000 secs, 0.0061 secs, 0.4614 secs - DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0000 secs - req write: 0.0001 secs, 0.0000 secs, 0.0599 secs - resp wait: 0.0935 secs, 0.0057 secs, 0.4612 secs - resp read: 0.0008 secs, 0.0001 secs, 0.1262 secs - -Status code distribution: - [200] 72500 responses - - -## After Adding Body - -./pmd-pin.sh -Pinning PID 64886 (astro-4321) to CPU 0 -pid 64886's current affinity list: 0-3 -pid 64886's new affinity list: 0 -Pinning PID 64887 (astro-4322) to CPU 1 -pid 64887's current affinity list: 0-3 -pid 64887's new affinity list: 1 -./pmd-verifypin.sh -Checking CPU pinning for PMDaemon apps: astro-4321 astro-4322 --------------------------------------- -App: astro-4321 - PID: 64886 | pid 64886's current affinity list: 0 - -App: astro-4322 - PID: 64887 | pid 64887's current affinity list: 1 - --------------------------------------- -Done. -⬇️ Server up ----------- -⬇️ Warming up server logs will be flushed for the below requests, ignore the below curls ----------- -hey -z 10s -host hexmos-local.com http://127.0.0.1/freedevtools/tldr/adb/adb - -Summary: - Total: 10.4520 secs - Slowest: 1.7567 secs - Fastest: 0.1498 secs - Average: 0.6029 secs - Requests/sec: 80.7500 - - -Response time histogram: - 0.150 [1] | - 0.311 [51] |■■■■■■■■ - 0.471 [268] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ - 0.632 [218] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ - 0.793 [148] |■■■■■■■■■■■■■■■■■■■■■■ - 0.953 [67] |■■■■■■■■■■ - 1.114 [37] |■■■■■■ - 1.275 [26] |■■■■ - 1.435 [17] |■■■ - 1.596 [5] |■ - 1.757 [6] |■ - - -Latency distribution: - 10% in 0.3486 secs - 25% in 0.4049 secs - 50% in 0.5324 secs - 75% in 0.7229 secs - 90% in 0.9819 secs - 95% in 1.1733 secs - 99% in 1.5549 secs - -Details (average, fastest, slowest): - DNS+dialup: 0.0009 secs, 0.1498 secs, 1.7567 secs - DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0000 secs - req write: 0.0005 secs, 0.0000 secs, 0.0371 secs - resp wait: 0.5955 secs, 0.1480 secs, 1.6928 secs - resp read: 0.0049 secs, 0.0015 secs, 0.0367 secs - -Status code distribution: - [200] 844 responses - - - -Peeking at 4 log files in /home/gk/.pmdaemon/logs... - -Request count table: -+------------+------------+--------------+---------------+----------+ -| Process | Requests | Dispatches | Worker Logs | Errors | -+============+============+==============+===============+==========+ -| astro-4321 | 0 | 437 | 874 | 0 | -+------------+------------+--------------+---------------+----------+ -| astro-4322 | 0 | 407 | 814 | 0 | -+------------+------------+--------------+---------------+----------+ - -Total aggregated: - total lines: 5914 - requests: 0 - dispatches: 844 - worker logs: 1688 - errors: 0 - -Dispatches represent each time the worker pool received a query and handed it off to a worker thread (logged via [SVG_ICONS_DB|PNG_ICONS_DB|EMOJI_DB|TLDR_DB] Dispatching ). - -Query Count Details process wise: -+------------+-----------+--------------------------------------------------+ -| Process | getPage | getPage params={"platform":"adb","slug":"adb"} | -+============+===========+==================================================+ -| astro-4321 | 437 | 437 | -+------------+-----------+--------------------------------------------------+ -| astro-4322 | 407 | 407 | -+------------+-----------+--------------------------------------------------+ - -Query Duration Details process wise: -+---------+---------+----------+----------+-----------+ -| Query | Count | Avg ms | Max ms | Sum min | -+=========+=========+==========+==========+===========+ -| getPage | 843 | 255.6 | 1373 | 3.59 | -+---------+---------+----------+----------+-----------+ - -Overall window: 2025-12-06T15:30:44.519000+00:00 → 2025-12-06T15:30:54.588000+00:00 -Total coverage: 0:00:10.069000 -⬇️ Requesting tldr 10sec -hey -z 10s -host hexmos-local.com http://127.0.0.1/freedevtools/tldr/adb - -Summary: - Total: 10.4763 secs - Slowest: 0.7539 secs - Fastest: 0.0799 secs - Average: 0.4250 secs - Requests/sec: 114.5440 - - -Response time histogram: - 0.080 [1] | - 0.147 [17] |■■ - 0.215 [21] |■■ - 0.282 [46] |■■■■■ - 0.350 [159] |■■■■■■■■■■■■■■■■■■ - 0.417 [351] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ - 0.484 [294] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ - 0.552 [162] |■■■■■■■■■■■■■■■■■■ - 0.619 [101] |■■■■■■■■■■■■ - 0.687 [36] |■■■■ - 0.754 [12] |■ - - -Latency distribution: - 10% in 0.3083 secs - 25% in 0.3618 secs - 50% in 0.4180 secs - 75% in 0.4872 secs - 90% in 0.5651 secs - 95% in 0.6098 secs - 99% in 0.6914 secs - -Details (average, fastest, slowest): - DNS+dialup: 0.0002 secs, 0.0799 secs, 0.7539 secs - DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0000 secs - req write: 0.0002 secs, 0.0000 secs, 0.0060 secs - resp wait: 0.4201 secs, 0.0757 secs, 0.7521 secs - resp read: 0.0045 secs, 0.0016 secs, 0.0319 secs - -Status code distribution: - [200] 1200 responses - - - -Peeking at 4 log files in /home/gk/.pmdaemon/logs... - -Request count table: -+------------+------------+--------------+---------------+----------+ -| Process | Requests | Dispatches | Worker Logs | Errors | -+============+============+==============+===============+==========+ -| astro-4321 | 0 | 1032 | 2064 | 0 | -+------------+------------+--------------+---------------+----------+ -| astro-4322 | 0 | 1012 | 2024 | 0 | -+------------+------------+--------------+---------------+----------+ - -Total aggregated: - total lines: 14314 - requests: 0 - dispatches: 2044 - worker logs: 4088 - errors: 0 - -Dispatches represent each time the worker pool received a query and handed it off to a worker thread (logged via [SVG_ICONS_DB|PNG_ICONS_DB|EMOJI_DB|TLDR_DB] Dispatching ). - -Query Count Details process wise: -+------------+---------------+--------------------------------------------------+-----------+--------------------------------------------------+ -| Process | getMainPage | getMainPage params={"platform":"adb","page":1} | getPage | getPage params={"platform":"adb","slug":"adb"} | -+============+===============+==================================================+===========+==================================================+ -| astro-4321 | 595 | 595 | 437 | 437 | -+------------+---------------+--------------------------------------------------+-----------+--------------------------------------------------+ -| astro-4322 | 605 | 605 | 407 | 407 | -+------------+---------------+--------------------------------------------------+-----------+--------------------------------------------------+ - -Query Duration Details process wise: -+-------------+---------+----------+----------+-----------+ -| Query | Count | Avg ms | Max ms | Sum min | -+=============+=========+==========+==========+===========+ -| getPage | 843 | 255.6 | 1373 | 3.59 | -+-------------+---------+----------+----------+-----------+ -| getMainPage | 1200 | 167.2 | 493 | 3.34 | -+-------------+---------+----------+----------+-----------+ - -Overall window: 2025-12-06T15:30:44.519000+00:00 → 2025-12-06T15:31:07.329000+00:00 -Total coverage: 0:00:22.810000 -10s Requesting tldr main -hey -z 10s -host hexmos-local.com http://127.0.0.1/freedevtools/tldr - -Summary: - Total: 10.4226 secs - Slowest: 1.0527 secs - Fastest: 0.0418 secs - Average: 0.4193 secs - Requests/sec: 116.8619 - - -Response time histogram: - 0.042 [1] | - 0.143 [29] |■■■ - 0.244 [73] |■■■■■■■■ - 0.345 [351] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ - 0.446 [357] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ - 0.547 [180] |■■■■■■■■■■■■■■■■■■■■ - 0.648 [108] |■■■■■■■■■■■■ - 0.749 [54] |■■■■■■ - 0.851 [23] |■■■ - 0.952 [23] |■■■ - 1.053 [19] |■■ - - -Latency distribution: - 10% in 0.2592 secs - 25% in 0.3152 secs - 50% in 0.3743 secs - 75% in 0.5005 secs - 90% in 0.6468 secs - 95% in 0.7638 secs - 99% in 0.9848 secs - -Details (average, fastest, slowest): - DNS+dialup: 0.0002 secs, 0.0418 secs, 1.0527 secs - DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0000 secs - req write: 0.0001 secs, 0.0000 secs, 0.0155 secs - resp wait: 0.4140 secs, 0.0365 secs, 1.0500 secs - resp read: 0.0048 secs, 0.0018 secs, 0.0322 secs - -Status code distribution: - [200] 1218 responses - - - -Peeking at 4 log files in /home/gk/.pmdaemon/logs... - -Request count table: -+------------+------------+--------------+---------------+----------+ -| Process | Requests | Dispatches | Worker Logs | Errors | -+============+============+==============+===============+==========+ -| astro-4321 | 0 | 1653 | 3306 | 0 | -+------------+------------+--------------+---------------+----------+ -| astro-4322 | 0 | 1609 | 3218 | 0 | -+------------+------------+--------------+---------------+----------+ - -Total aggregated: - total lines: 21622 - requests: 0 - dispatches: 3262 - worker logs: 6524 - errors: 0 - -Dispatches represent each time the worker pool received a query and handed it off to a worker thread (logged via [SVG_ICONS_DB|PNG_ICONS_DB|EMOJI_DB|TLDR_DB] Dispatching ). - -Query Count Details process wise: -+------------+---------------+--------------------------------------------------+----------------------------------------------------+-----------+--------------------------------------------------+ -| Process | getMainPage | getMainPage params={"platform":"adb","page":1} | getMainPage params={"platform":"index","page":1} | getPage | getPage params={"platform":"adb","slug":"adb"} | -+============+===============+==================================================+====================================================+===========+==================================================+ -| astro-4321 | 1216 | 595 | 621 | 437 | 437 | -+------------+---------------+--------------------------------------------------+----------------------------------------------------+-----------+--------------------------------------------------+ -| astro-4322 | 1202 | 605 | 597 | 407 | 407 | -+------------+---------------+--------------------------------------------------+----------------------------------------------------+-----------+--------------------------------------------------+ - -Query Duration Details process wise: -+-------------+---------+----------+----------+-----------+ -| Query | Count | Avg ms | Max ms | Sum min | -+=============+=========+==========+==========+===========+ -| getPage | 843 | 255.6 | 1373 | 3.59 | -+-------------+---------+----------+----------+-----------+ -| getMainPage | 2418 | 162.6 | 639 | 6.55 | -+-------------+---------+----------+----------+-----------+ - -Overall window: 2025-12-06T15:30:44.519000+00:00 → 2025-12-06T15:31:20.147000+00:00 -Total coverage: 0:00:35.628000 -1min Requesting tldr 1min -hey -z 1m -host hexmos-local.com http://127.0.0.1/freedevtools/tldr/npm/npm-fund/ - -Summary: - Total: 60.2830 secs - Slowest: 0.9633 secs - Fastest: 0.0354 secs - Average: 0.2889 secs - Requests/sec: 172.6026 - - -Response time histogram: - 0.035 [1] | - 0.128 [157] |■ - 0.221 [1498] |■■■■■■■■■■■ - 0.314 [5651] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ - 0.407 [2383] |■■■■■■■■■■■■■■■■■ - 0.499 [491] |■■■ - 0.592 [113] |■ - 0.685 [52] | - 0.778 [26] | - 0.871 [24] | - 0.963 [9] | - - -Latency distribution: - 10% in 0.2043 secs - 25% in 0.2374 secs - 50% in 0.2784 secs - 75% in 0.3256 secs - 90% in 0.3830 secs - 95% in 0.4294 secs - 99% in 0.5985 secs - -Details (average, fastest, slowest): - DNS+dialup: 0.0000 secs, 0.0354 secs, 0.9633 secs - DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0000 secs - req write: 0.0001 secs, 0.0000 secs, 0.0173 secs - resp wait: 0.2850 secs, 0.0299 secs, 0.9565 secs - resp read: 0.0037 secs, 0.0012 secs, 0.0668 secs - -Status code distribution: - [200] 10405 responses - - - -Peeking at 4 log files in /home/gk/.pmdaemon/logs... - -Request count table: -+------------+------------+--------------+---------------+----------+ -| Process | Requests | Dispatches | Worker Logs | Errors | -+============+============+==============+===============+==========+ -| astro-4321 | 0 | 6865 | 13730 | 0 | -+------------+------------+--------------+---------------+----------+ -| astro-4322 | 0 | 6802 | 13604 | 0 | -+------------+------------+--------------+---------------+----------+ - -Total aggregated: - total lines: 94457 - requests: 0 - dispatches: 13667 - worker logs: 27334 - errors: 0 - -Dispatches represent each time the worker pool received a query and handed it off to a worker thread (logged via [SVG_ICONS_DB|PNG_ICONS_DB|EMOJI_DB|TLDR_DB] Dispatching ). - -Query Count Details process wise: -+------------+---------------+--------------------------------------------------+----------------------------------------------------+-----------+--------------------------------------------------+-------------------------------------------------------+ -| Process | getMainPage | getMainPage params={"platform":"adb","page":1} | getMainPage params={"platform":"index","page":1} | getPage | getPage params={"platform":"adb","slug":"adb"} | getPage params={"platform":"npm","slug":"npm-fund"} | -+============+===============+==================================================+====================================================+===========+==================================================+=======================================================+ -| astro-4321 | 1216 | 595 | 621 | 5649 | 437 | 5212 | -+------------+---------------+--------------------------------------------------+----------------------------------------------------+-----------+--------------------------------------------------+-------------------------------------------------------+ -| astro-4322 | 1202 | 605 | 597 | 5600 | 407 | 5193 | -+------------+---------------+--------------------------------------------------+----------------------------------------------------+-----------+--------------------------------------------------+-------------------------------------------------------+ - -Query Duration Details process wise: -+-------------+---------+----------+----------+-----------+ -| Query | Count | Avg ms | Max ms | Sum min | -+=============+=========+==========+==========+===========+ -| getPage | 11242 | 126.7 | 1373 | 23.75 | -+-------------+---------+----------+----------+-----------+ -| getMainPage | 2418 | 162.6 | 639 | 6.55 | -+-------------+---------+----------+----------+-----------+ - -Overall window: 2025-12-06T15:30:44.519000+00:00 → 2025-12-06T15:32:22.965000+00:00 -Total coverage: 0:01:38.446000 -5min Requesting tldr 5min -hey -z 5m -host hexmos-local.com http://127.0.0.1/freedevtools/tldr/npm/npm-fund/ -^C -Summary: - Total: 162.4016 secs - Slowest: 2.0663 secs - Fastest: 0.0312 secs - Average: 0.4522 secs - Requests/sec: 110.4484 - - -Response time histogram: - 0.031 [1] | - 0.235 [3399] |■■■■■■■■■■■■■■■■■■ - 0.438 [7598] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ - 0.642 [3444] |■■■■■■■■■■■■■■■■■■ - 0.845 [1814] |■■■■■■■■■■ - 1.049 [913] |■■■■■ - 1.252 [454] |■■ - 1.456 [197] |■ - 1.659 [81] | - 1.863 [26] | - 2.066 [10] | - - -Latency distribution: - 10% in 0.2039 secs - 25% in 0.2577 secs - 50% in 0.3705 secs - 75% in 0.5644 secs - 90% in 0.8283 secs - 95% in 1.0118 secs - 99% in 1.3745 secs - -Details (average, fastest, slowest): - DNS+dialup: 0.0000 secs, 0.0312 secs, 2.0663 secs - DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0000 secs - req write: 0.0001 secs, 0.0000 secs, 0.0647 secs - resp wait: 0.4477 secs, 0.0230 secs, 2.0513 secs - resp read: 0.0042 secs, 0.0012 secs, 0.1001 secs - -Status code distribution: - [200] 17937 responses \ No newline at end of file diff --git a/frontend/scripts/tldr/tldr_to_db.go b/frontend/scripts/tldr/tldr_to_db.go index 95538426ef..cb6107e3b5 100644 --- a/frontend/scripts/tldr/tldr_to_db.go +++ b/frontend/scripts/tldr/tldr_to_db.go @@ -9,7 +9,6 @@ import ( "fmt" "io/fs" "log" - "math" "os" "path/filepath" "regexp" @@ -193,25 +192,13 @@ func ensureSchema(db *sql.DB) error { return err } - // MainPages table - Pre-calculated lists + // Cluster table - Cluster Metadata (similar to emojis category) _, err = db.Exec(` - CREATE TABLE IF NOT EXISTS main_pages ( + CREATE TABLE IF NOT EXISTS cluster ( hash INTEGER PRIMARY KEY, - url TEXT NOT NULL, - data TEXT DEFAULT '{}', -- JSON - total_count INTEGER NOT NULL - ) WITHOUT ROWID; - `) - if err != nil { - return err - } - - // Sitemap table - _, err = db.Exec(` - CREATE TABLE IF NOT EXISTS sitemap ( - hash INTEGER PRIMARY KEY, - url TEXT NOT NULL, - data TEXT DEFAULT '[]' -- JSON list of URLs + name TEXT NOT NULL, + count INTEGER NOT NULL, + preview_commands_json TEXT DEFAULT '[]' -- JSON list of preview commands ) WITHOUT ROWID; `) if err != nil { @@ -300,8 +287,8 @@ func main() { } tx.Commit() - // 3. Generate MainPages (Cluster Lists) - fmt.Println("Generating cluster lists...") + // 3. Generate Cluster (Cluster Metadata) + fmt.Println("Generating cluster metadata...") pagesByCluster := make(map[string][]*ProcessedPage) for _, p := range allPages { pagesByCluster[p.Cluster] = append(pagesByCluster[p.Cluster], p) @@ -311,16 +298,11 @@ func main() { if err != nil { log.Fatal(err) } - stmt, err = tx.Prepare("INSERT INTO main_pages (hash, url, data, total_count) VALUES (?, ?, ?, ?)") + stmt, err = tx.Prepare("INSERT INTO cluster (hash, name, count, preview_commands_json) VALUES (?, ?, ?, ?)") if err != nil { log.Fatal(err) } - itemsPerPage := 30 - - // For Index Page (List of Platforms) - var platforms []map[string]interface{} - for cluster, pages := range pagesByCluster { // Sort pages by name sort.Slice(pages, func(i, j int) bool { @@ -328,9 +310,7 @@ func main() { }) totalCount := len(pages) - totalPages := int(math.Ceil(float64(totalCount) / float64(itemsPerPage))) - // Add to platforms list for index // Get top 5 commands for preview var commandPreviews []map[string]string previewCount := 5 @@ -343,184 +323,19 @@ func main() { "url": fmt.Sprintf("/freedevtools/tldr/%s/%s/", cluster, pages[k].Name), }) } + previewJson, _ := json.Marshal(commandPreviews) - clusterUrl := fmt.Sprintf("/freedevtools/tldr/%s/", cluster) - platformData := map[string]interface{}{ - "name": cluster, - "count": totalCount, - "url": clusterUrl, - "commands": commandPreviews, - } - platforms = append(platforms, platformData) - allUrls = append(allUrls, clusterUrl) - - // Generate paginated lists for this cluster - for i := 0; i < totalPages; i++ { - pageNum := i + 1 - startIdx := i * itemsPerPage - endIdx := startIdx + itemsPerPage - if endIdx > totalCount { - endIdx = totalCount - } - - chunk := pages[startIdx:endIdx] - var commands []map[string]interface{} - for _, p := range chunk { - commands = append(commands, map[string]interface{}{ - "name": p.Name, - "url": p.Path, - "description": p.Description, - "category": p.Platform, - "features": p.Features, - }) - } - - data := map[string]interface{}{ - "commands": commands, - "total": totalCount, - "page": pageNum, - "total_pages": totalPages, - } - dataJson, _ := json.Marshal(data) - - // Hash: cluster/page (e.g., common/1) - hashKey := fmt.Sprintf("%s/%d", cluster, pageNum) - hash := get8Bytes(hashString(hashKey)) - - // Construct URL for this page - var pageUrl string - if pageNum == 1 { - pageUrl = fmt.Sprintf("/freedevtools/tldr/%s/", cluster) - } else { - pageUrl = fmt.Sprintf("/freedevtools/tldr/%s/%d/", cluster, pageNum) - allUrls = append(allUrls, pageUrl) - } - - _, err = stmt.Exec(hash, pageUrl, string(dataJson), totalCount) - if err != nil { - log.Printf("Error inserting cluster page %s: %v", hashKey, err) - } - } - } - - // 4. Generate MainPages (Index) - fmt.Println("Generating index...") - sort.Slice(platforms, func(i, j int) bool { - return platforms[i]["name"].(string) < platforms[j]["name"].(string) - }) - - totalPlatforms := len(platforms) - totalIndexPages := int(math.Ceil(float64(totalPlatforms) / float64(itemsPerPage))) - - for i := 0; i < totalIndexPages; i++ { - pageNum := i + 1 - startIdx := i * itemsPerPage - endIdx := startIdx + itemsPerPage - if endIdx > totalPlatforms { - endIdx = totalPlatforms - } - - chunk := platforms[startIdx:endIdx] - data := map[string]interface{}{ - "platforms": chunk, - "total": totalPlatforms, - "page": pageNum, - "total_pages": totalIndexPages, - "total_commands": len(allPages), - } - dataJson, _ := json.Marshal(data) - - // Hash: index/page (e.g., index/1) - hashKey := fmt.Sprintf("index/%d", pageNum) - hash := get8Bytes(hashString(hashKey)) - - // Construct URL for this page - var pageUrl string - if pageNum == 1 { - pageUrl = "/freedevtools/tldr/" - allUrls = append(allUrls, pageUrl) - } else { - pageUrl = fmt.Sprintf("/freedevtools/tldr/%d/", pageNum) - allUrls = append(allUrls, pageUrl) - } + // Hash: cluster (e.g., common) + hash := get8Bytes(hashString(cluster)) - _, err = stmt.Exec(hash, pageUrl, string(dataJson), totalPlatforms) + _, err = stmt.Exec(hash, cluster, totalCount, string(previewJson)) if err != nil { - log.Printf("Error inserting index page %s: %v", hashKey, err) + log.Printf("Error inserting cluster %s: %v", cluster, err) } } tx.Commit() - // 5. Generate Sitemap - fmt.Println("Generating sitemap...") - tx, err = db.Begin() - if err != nil { - log.Fatal(err) - } - stmt, err = tx.Prepare("INSERT INTO sitemap (hash, url, data) VALUES (?, ?, ?)") - if err != nil { - log.Fatal(err) - } - defer stmt.Close() - - // Sort all URLs to ensure deterministic output - sort.Strings(allUrls) - - // Filter out duplicates if any - uniqueUrls := make([]string, 0, len(allUrls)) - seenUrls := make(map[string]bool) - for _, u := range allUrls { - if !seenUrls[u] { - seenUrls[u] = true - uniqueUrls = append(uniqueUrls, u) - } - } - allUrls = uniqueUrls - - sitemapChunkSize := 5000 - totalSitemapChunks := int(math.Ceil(float64(len(allUrls)) / float64(sitemapChunkSize))) - var sitemapIndexUrls []string - - for i := 0; i < totalSitemapChunks; i++ { - chunkNum := i + 1 - startIdx := i * sitemapChunkSize - endIdx := startIdx + sitemapChunkSize - if endIdx > len(allUrls) { - endIdx = len(allUrls) - } - - chunkUrls := allUrls[startIdx:endIdx] - chunkJson, _ := json.Marshal(chunkUrls) - - sitemapName := fmt.Sprintf("sitemap-%d.xml", chunkNum) - sitemapUrl := fmt.Sprintf("/freedevtools/tldr/%s", sitemapName) - sitemapIndexUrls = append(sitemapIndexUrls, sitemapUrl) - - // Hash: sitemap/sitemap-N.xml - hashKey := fmt.Sprintf("sitemap/%s", sitemapName) - hash := get8Bytes(hashString(hashKey)) - - _, err = stmt.Exec(hash, sitemapName, string(chunkJson)) - if err != nil { - log.Printf("Error inserting sitemap chunk %s: %v", sitemapName, err) - } - } - - // Insert Sitemap Index - indexJson, _ := json.Marshal(sitemapIndexUrls) - indexName := "sitemap.xml" - // Hash: sitemap/sitemap.xml - hashKey := fmt.Sprintf("sitemap/%s", indexName) - hash := get8Bytes(hashString(hashKey)) - - _, err = stmt.Exec(hash, indexName, string(indexJson)) - if err != nil { - log.Printf("Error inserting sitemap index: %v", err) - } - - tx.Commit() - - // 6. Overview + // 4. Overview if _, err := db.Exec("INSERT INTO overview (id, total_count) VALUES (1, ?)", len(allPages)); err != nil { log.Fatal(err) } diff --git a/frontend/src/pages/tldr/[platform]/[slug].astro b/frontend/src/pages/tldr/[platform]/[slug].astro index 7fcc6857ed..9e4dbc681c 100644 --- a/frontend/src/pages/tldr/[platform]/[slug].astro +++ b/frontend/src/pages/tldr/[platform]/[slug].astro @@ -1,5 +1,5 @@ --- -import { getTldrPage, getTldrPlatformCommandsPaginated } from '../../../../db/tldrs/tldr-utils'; +import { getTldrCluster, getTldrCommandsByClusterPaginated, getTldrPage } from '../../../../db/tldrs/tldr-utils'; import AdBanner from '../../../components/banner/AdBanner.astro'; import Banner from '../../../components/banner/BannerIndex.astro'; import CreditsButton from '../../../components/buttons/CreditsButton'; @@ -34,19 +34,29 @@ if (pageNumber !== null) { return Astro.redirect(`/freedevtools/tldr/${platform}/`); } - const result = await getTldrPlatformCommandsPaginated(platform, pageNumber); + // Get cluster metadata first + const clusterData = await getTldrCluster(platform); - if (!result) { + if (!clusterData) { return Astro.redirect('/404'); } - const { commands: currentPageCommands, total: totalCommands, total_pages: totalPages, page: currentPage } = result; + const itemsPerPage = 30; + const totalCommands = clusterData.count; + const totalPages = Math.ceil(totalCommands / itemsPerPage); + const currentPage = pageNumber; // Validate page number if (currentPage > totalPages || currentPage < 1) { return Astro.redirect('/404'); } + const currentPageCommands = await getTldrCommandsByClusterPaginated( + platform, + currentPage, + itemsPerPage + ); + // Breadcrumb data const breadcrumbItems = [ { label: 'Free DevTools', href: '/freedevtools/' }, diff --git a/frontend/src/pages/tldr/[platform]/index.astro b/frontend/src/pages/tldr/[platform]/index.astro index 8b76e38511..f454cf3db9 100644 --- a/frontend/src/pages/tldr/[platform]/index.astro +++ b/frontend/src/pages/tldr/[platform]/index.astro @@ -1,7 +1,10 @@ --- +import { + getAllTldrClusters, + getTldrCluster, + getTldrCommandsByClusterPaginated, +} from '../../../../db/tldrs/tldr-utils'; import BaseLayout from '../../../layouts/BaseLayout.astro'; -import { getAllTldrPlatforms } from '../../../../db/tldrs/tldr-utils'; -import { getTldrPlatformCommandsPaginated } from '../../../../db/tldrs/tldr-utils'; import Tldr from '../_Tldr.astro'; import TldrPlatform from '../_TldrPlatform.astro'; @@ -26,31 +29,38 @@ if (/^\d+$/.test(platform)) { return Astro.redirect('/freedevtools/tldr/'); } - // Fetch paginated platforms data + // Fetch all clusters to paginate manually const start = Date.now(); - const result = await getAllTldrPlatforms(currentPage); + const allClusters = await getAllTldrClusters(); console.log( - `[TLDR_PLATFORM_PAGE] getAllTldrPlatforms(${currentPage}) finished in ${Date.now() - start}ms` + `[TLDR_PLATFORM_PAGE] getAllTldrClusters() finished in ${Date.now() - start}ms` ); - if (!result) { + if (!allClusters) { return Astro.redirect('/404'); } - const { - platforms: currentPagePlatforms, - total: totalPlatforms, - total_pages: totalPages, - page, - total_commands: totalCommands, - } = result; + const itemsPerPage = 30; + const totalPlatforms = allClusters.length; + const totalPages = Math.ceil(totalPlatforms / itemsPerPage); // Validate page number - if (page > totalPages || page < 1) { + if (currentPage > totalPages || currentPage < 1) { return Astro.redirect('/404'); } - const itemsPerPage = 30; + const startIndex = (currentPage - 1) * itemsPerPage; + const endIndex = startIndex + itemsPerPage; + const currentPagePlatforms = allClusters + .slice(startIndex, endIndex) + .map((c) => ({ + name: c.name, + count: c.count, + url: `/freedevtools/tldr/${c.name}/`, + commands: c.preview_commands, + })); + + const totalCommands = allClusters.reduce((acc, c) => acc + c.count, 0); // Breadcrumb data const breadcrumbItems = [ @@ -91,22 +101,29 @@ if (/^\d+$/.test(platform)) { } else { // Get commands for this platform (Page 1) const start = Date.now(); - const result = await getTldrPlatformCommandsPaginated(platform, 1); - console.log( - `[TLDR_PLATFORM_PAGE] getTldrPlatformCommandsPaginated(${platform}, 1) finished in ${Date.now() - start}ms` - ); - if (!result) { + // Get cluster metadata first + const clusterData = await getTldrCluster(platform); + + if (!clusterData) { return Astro.redirect('/404'); } - const { - commands, - total: totalCommands, - total_pages: totalPages, - page: currentPage, - } = result; const itemsPerPage = 30; + const totalCommands = clusterData.count; + const totalPages = Math.ceil(totalCommands / itemsPerPage); + const currentPage = 1; + + // Get commands for page 1 + const commands = await getTldrCommandsByClusterPaginated( + platform, + currentPage, + itemsPerPage + ); + + console.log( + `[TLDR_PLATFORM_PAGE] getTldrCommandsByClusterPaginated(${platform}, 1) finished in ${Date.now() - start}ms` + ); // Breadcrumb data const breadcrumbItems = [ diff --git a/frontend/src/pages/tldr/index.astro b/frontend/src/pages/tldr/index.astro index 95afd85eda..a5e889808d 100644 --- a/frontend/src/pages/tldr/index.astro +++ b/frontend/src/pages/tldr/index.astro @@ -1,28 +1,33 @@ --- +import { getAllTldrClusters } from '../../../db/tldrs/tldr-utils'; import BaseLayout from '../../layouts/BaseLayout.astro'; -import { getAllTldrPlatforms } from '../../../db/tldrs/tldr-utils'; import { formatNumber } from '../../lib/utils'; import Tldr from './_Tldr.astro'; // Get page 1 of platforms const start = Date.now(); -const result = await getAllTldrPlatforms(1); +const allClusters = await getAllTldrClusters(); // console.log( -// `[TLDR_PAGE] getAllTldrPlatforms(1) finished in ${Date.now() - start}ms` +// `[TLDR_PAGE] getAllTldrClusters() finished in ${Date.now() - start}ms` // ); -if (!result) { +if (!allClusters) { return Astro.redirect('/404'); } -const { - platforms, - total: totalPlatforms, - total_pages: totalPages, - page: currentPage, - total_commands: totalCommands, -} = result; const itemsPerPage = 30; +const totalPlatforms = allClusters.length; +const totalPages = Math.ceil(totalPlatforms / itemsPerPage); +const currentPage = 1; + +const platforms = allClusters.slice(0, itemsPerPage).map((c) => ({ + name: c.name, + count: c.count, + url: `/freedevtools/tldr/${c.name}/`, + commands: c.preview_commands, +})); + +const totalCommands = allClusters.reduce((acc, c) => acc + c.count, 0); // Breadcrumb data const breadcrumbItems = [ diff --git a/frontend/src/pages/tldr/sitemap-[page].xml.ts b/frontend/src/pages/tldr/sitemap-[page].xml.ts index a36e74cf88..4723d73b37 100644 --- a/frontend/src/pages/tldr/sitemap-[page].xml.ts +++ b/frontend/src/pages/tldr/sitemap-[page].xml.ts @@ -1,8 +1,23 @@ -import type { APIRoute } from 'astro'; -import { getTldrSitemap } from '../../../db/tldrs/tldr-utils'; +import type { APIRoute } from "astro"; +import { getTldrSitemapUrls } from "db/tldrs/tldr-utils"; export const GET: APIRoute = async ({ params, site }) => { const { page } = params; + + if (!page || !/^\d+$/.test(page)) { + return new Response(null, { status: 404 }); + } + + const pageNum = parseInt(page, 10); + const chunkSize = 5000; + const offset = (pageNum - 1) * chunkSize; + + const urls = await getTldrSitemapUrls(chunkSize, offset); + + if (!urls || urls.length === 0) { + return new Response(null, { status: 404 }); + } + const now = new Date().toISOString(); // Use site from .env file (SITE variable) or astro.config.mjs @@ -10,34 +25,37 @@ export const GET: APIRoute = async ({ params, site }) => { const siteStr = site?.toString() || ''; const siteUrl = envSite || siteStr || 'http://localhost:4321/freedevtools'; - if (!page) { - return new Response('Page not found', { status: 404 }); - } - - const sitemapName = `sitemap-${page}.xml`; - const urls = await getTldrSitemap(sitemapName); - - if (!urls) { - return new Response('Sitemap chunk not found', { status: 404 }); - } + const urlEntries = urls.map(url => { + // Ensure URL starts with /freedevtools if not already (it should be from DB) + // But DB stores relative paths like /freedevtools/tldr/... + // We need to prepend siteUrl (which might include /freedevtools if configured that way, but usually is domain) + // siteUrl in this context is usually http://localhost:4321/freedevtools + + // If DB url starts with /freedevtools, and siteUrl ends with /freedevtools, we might double up. + // Let's assume siteUrl is the base domain + base path. + // Actually, siteUrl constructed above includes /freedevtools. + // And DB urls start with /freedevtools. + // So we should strip /freedevtools from one of them or handle it carefully. + + // DB URL: /freedevtools/tldr/common/tar/ + // Site URL: http://localhost:4321/freedevtools + // Result should be: http://localhost:4321/freedevtools/tldr/common/tar/ + + // Let's strip /freedevtools from the start of DB URL if present + const relativeUrl = url.startsWith('/freedevtools') ? url.substring(13) : url; + const fullUrl = `${siteUrl}${relativeUrl}`; + + return ` \n ${fullUrl}\n ${now}\n daily\n 0.8\n `; + }); - const xml = ` - - -${urls.map((url: string) => ` - ${siteUrl.replace(/\/freedevtools$/, '')}${url} - ${now} - daily - 0.8 - `).join('\n')} -`; + const xml = `\n\n${urlEntries.join( + "\n" + )}\n`; return new Response(xml, { headers: { - 'Content-Type': 'application/xml', - 'Cache-Control': 'public, max-age=3600', + "Content-Type": "application/xml", + "Cache-Control": "public, max-age=3600", }, }); }; - - diff --git a/frontend/src/pages/tldr/sitemap.xml.ts b/frontend/src/pages/tldr/sitemap.xml.ts index 0b02cda7f4..5423c6f575 100644 --- a/frontend/src/pages/tldr/sitemap.xml.ts +++ b/frontend/src/pages/tldr/sitemap.xml.ts @@ -1,5 +1,5 @@ -import type { APIRoute } from 'astro'; -import { getTldrSitemap } from '../../../db/tldrs/tldr-utils'; +import type { APIRoute } from "astro"; +import { getTldrSitemapCount } from "db/tldrs/tldr-utils"; export const GET: APIRoute = async ({ site }) => { const now = new Date().toISOString(); @@ -9,25 +9,29 @@ export const GET: APIRoute = async ({ site }) => { const siteStr = site?.toString() || ''; const siteUrl = envSite || siteStr || 'http://localhost:4321/freedevtools'; - const sitemapUrls = await getTldrSitemap('sitemap.xml'); + const totalUrls = await getTldrSitemapCount(); + const chunkSize = 5000; + const totalChunks = Math.ceil(totalUrls / chunkSize); - if (!sitemapUrls) { - return new Response('Sitemap index not found', { status: 404 }); + const sitemaps: string[] = []; + + for (let i = 1; i <= totalChunks; i++) { + sitemaps.push( + ` + ${siteUrl}/tldr/sitemap-${i}.xml + ${now} + ` + ); } - const xml = ` - - -${sitemapUrls.map((url: string) => ` - ${siteUrl.replace(/\/freedevtools$/, '')}${url} - ${now} - `).join('\n')} -`; + const xml = `\n\n${sitemaps.join( + "\n" + )}\n`; return new Response(xml, { headers: { - 'Content-Type': 'application/xml', - 'Cache-Control': 'public, max-age=3600', + "Content-Type": "application/xml", + "Cache-Control": "public, max-age=3600", }, }); }; \ No newline at end of file From a0aa4f71c773c237604b798b67fdc4ed45657e8b Mon Sep 17 00:00:00 2001 From: Ganesh Kumar Date: Sun, 7 Dec 2025 15:45:08 +0530 Subject: [PATCH 210/234] feat: Introduce `cluster_hash` to the `pages` table for efficient cluster-based data retrieval, updating the Go script for data population and the TypeScript worker for query logic. --- frontend/db/tldrs/README.md | 66 +++++++++++++++++++++++++++++ frontend/db/tldrs/tldr-worker.ts | 23 +++++----- frontend/scripts/tldr/README.md | 48 +++++++++++++++++++++ frontend/scripts/tldr/tldr_to_db.go | 8 +++- 4 files changed, 132 insertions(+), 13 deletions(-) create mode 100644 frontend/db/tldrs/README.md diff --git a/frontend/db/tldrs/README.md b/frontend/db/tldrs/README.md new file mode 100644 index 0000000000..fff9c6616f --- /dev/null +++ b/frontend/db/tldrs/README.md @@ -0,0 +1,66 @@ +# TLDR Database Layer + +This directory contains the database interaction logic for the TLDR feature. It uses a **Worker Pool** architecture to handle SQLite queries efficiently and non-blocking. + +## Architecture + +The system is designed to offload database operations to background threads, preventing the main event loop from being blocked by synchronous SQLite calls. + +### Components + +1. **`tldr-worker-pool.ts`**: + * Manages a pool of worker threads (default: 2). + * Handles round-robin distribution of queries to workers. + * Manages worker lifecycle (initialization, termination). + * Exposes the `query` object with methods for each database operation. + +2. **`tldr-worker.ts`**: + * The code running inside each worker thread. + * Opens a read-only connection to the SQLite database (`tldr-db-v1.db`). + * Prepares SQL statements for performance. + * Executes queries based on messages received from the pool. + +3. **`tldr-utils.ts`**: + * A high-level wrapper around the worker pool. + * Provides typed functions for the frontend to consume. + * Imports interfaces from `tldr-schema.ts`. + +4. **`tldr-schema.ts`**: + * TypeScript interfaces defining the shape of data returned by the database. + +## Query Design + +The queries are optimized for specific frontend use cases: + +### 1. Fetching a Cluster (`getTldrCluster`) +* **Goal**: Get metadata for a platform (e.g., "common"). +* **Query**: Selects from the `cluster` table by hash. +* **Returns**: Name, total count, and a small list of preview commands. + +### 2. Paginated Commands (`getTldrCommandsByClusterPaginated`) +* **Goal**: Get a specific page of commands for a platform. +* **Query**: + ```sql + SELECT url, metadata FROM pages + WHERE url LIKE ? + ORDER BY url + LIMIT ? OFFSET ? + ``` +* **Logic**: Uses SQL `LIMIT` and `OFFSET` to fetch only the requested chunk of records, avoiding loading thousands of commands into memory. + +### 3. Fetching a Single Page (`getTldrPage`) +* **Goal**: Get the content for a specific command. +* **Query**: Selects from `pages` by `url_hash`. +* **Returns**: Rendered HTML content and full metadata. + +### 4. Sitemap Generation +* **`getTldrSitemapCount`**: Counts total URLs for the sitemap index. +* **`getTldrSitemapUrls`**: Fetches a chunk of 5000 URLs for individual sitemap files using `LIMIT` and `OFFSET`. + +## Data Flow + +1. **Frontend** calls a function in `tldr-utils.ts` (e.g., `getTldrCluster('common')`). +2. **Utils** calls `query.getMainPage('common')` in `tldr-worker-pool.ts`. +3. **Pool** selects a worker and sends a message. +4. **Worker** receives the message, executes the prepared statement, and sends the result back. +5. **Pool** resolves the promise, returning data to the frontend. diff --git a/frontend/db/tldrs/tldr-worker.ts b/frontend/db/tldrs/tldr-worker.ts index be73a6d2fb..679da0a3db 100644 --- a/frontend/db/tldrs/tldr-worker.ts +++ b/frontend/db/tldrs/tldr-worker.ts @@ -7,7 +7,7 @@ import { Database } from 'bun:sqlite'; import path from 'path'; import { fileURLToPath } from 'url'; import { parentPort, workerData } from 'worker_threads'; -import { hashUrlToKey } from '../../src/lib/hash-utils'; +import { hashNameToKey } from '../../src/lib/hash-utils'; const logColors = { reset: '\u001b[0m', @@ -21,6 +21,8 @@ const { dbPath, workerId } = workerData; const __filename = fileURLToPath(import.meta.url); const __dirname = path.dirname(__filename); + + // Open database connection with aggressive read optimizations const db = new Database(dbPath, { readonly: true }); @@ -53,7 +55,7 @@ const statements = { // New query for paginated commands in a cluster getCommandsByClusterPaginated: db.prepare( `SELECT url, metadata FROM pages - WHERE url LIKE ? + WHERE cluster_hash = ? ORDER BY url LIMIT ? OFFSET ?` ), @@ -86,10 +88,6 @@ const statements = { ), }; -// clusterPreviews is taking 0.5 seconds need to improve db structure for this -// pageByClusterAndName is taking 1 ms - - // Signal ready parentPort?.postMessage({ ready: true }); @@ -120,7 +118,7 @@ parentPort?.on('message', (message: QueryMessage) => { case 'getMainPage': { const { platform } = params; // Hash of cluster name - const hash = hashUrlToKey(platform); + const hash = hashNameToKey(platform); const row = statements.getMainPage.get(hash) as { name: string; count: number; preview_commands_json: string } | undefined; if (row) { result = { @@ -137,7 +135,7 @@ parentPort?.on('message', (message: QueryMessage) => { case 'getPage': { const { platform, slug } = params; const hashKey = `${platform}/${slug}`; - const hash = hashUrlToKey(hashKey); + const hash = hashNameToKey(hashKey); const row = statements.getPage.get(hash) as { html_content: string; metadata: string } | undefined; if (row) { result = { @@ -152,9 +150,12 @@ parentPort?.on('message', (message: QueryMessage) => { case 'getCommandsByClusterPaginated': { const { cluster, limit, offset } = params; - // URL pattern: /freedevtools/tldr/cluster/% - const urlPattern = `/freedevtools/tldr/${cluster}/%`; - const rows = statements.getCommandsByClusterPaginated.all(urlPattern, limit, offset) as { url: string; metadata: string }[]; + const clusterHash = hashNameToKey(cluster); + const rows = statements.getCommandsByClusterPaginated.all( + clusterHash, + limit, + offset + ) as { url: string; metadata: string }[]; result = rows.map(row => { const meta = JSON.parse(row.metadata); diff --git a/frontend/scripts/tldr/README.md b/frontend/scripts/tldr/README.md index ac5a27559f..2d6941b7f6 100644 --- a/frontend/scripts/tldr/README.md +++ b/frontend/scripts/tldr/README.md @@ -17,11 +17,13 @@ The script expects the following directory structure relative to its location: ## Setup 1. Navigate to the script directory: + ```bash cd scripts/tldr ``` 2. Install dependencies: + ```bash go mod tidy ``` @@ -35,6 +37,7 @@ go run tldr_to_db.go ``` The script will: + 1. Create/Reset the database at `../../db/all_dbs/tldr-db.db`. 2. Walk through the `../../data/tldr` directory. 3. Parse each markdown file (frontmatter, content, examples). @@ -44,3 +47,48 @@ The script will: ## Output The script outputs progress to the console and prints the total time taken upon completion. + +## How it Works + +The `tldr_to_db.go` script performs the following steps to transform raw markdown files into a structured SQLite database: + +### 1. Database Initialization + +- Creates the database file at `../../db/all_dbs/tldr-db-v1.db`. +- Defines the schema with three main tables: + - **`pages`**: Stores individual command pages. + - `url_hash` (PK): Integer hash of the URL. + - `url`: The full path (e.g., `/freedevtools/tldr/common/tar/`). + - `html_content`: Rendered HTML from markdown. + - `metadata`: JSON string containing title, description, keywords, etc. + - **`cluster`**: Stores metadata for command groups (platforms like `common`, `linux`). + - `hash` (PK): Integer hash of the cluster name. + - `name`: Cluster name (e.g., `common`). + - `count`: Total number of commands in the cluster. + - `preview_commands_json`: JSON array of the first 5 commands for preview. + - **`overview`**: Stores global statistics. + - `total_count`: Total number of commands across all platforms. + +### 2. File Parsing + +- Walks through the `../../data/tldr` directory. +- Parses each `.md` file: + - Extracts **Frontmatter** (YAML) for metadata like title and description. + - Renders **Markdown Body** to HTML using `gomarkdown`. + - Generates a unique **URL Hash** based on the category and filename. + +### 3. Data Insertion + +- **Pages**: Inserts every parsed page into the `pages` table. +- **Clusters**: + - Groups pages by their platform (cluster). + - Sorts pages alphabetically within each cluster. + - Generates a preview JSON for the first 5 commands. + - Inserts the aggregated data into the `cluster` table. +- **Overview**: Inserts the total count of processed pages into the `overview` table. + +This approach ensures that the frontend can efficiently fetch: + +- Individual pages by URL hash. +- Cluster summaries (for lists and pagination) by cluster hash. +- Global stats from the overview table. diff --git a/frontend/scripts/tldr/tldr_to_db.go b/frontend/scripts/tldr/tldr_to_db.go index cb6107e3b5..db52b2b9d9 100644 --- a/frontend/scripts/tldr/tldr_to_db.go +++ b/frontend/scripts/tldr/tldr_to_db.go @@ -184,6 +184,7 @@ func ensureSchema(db *sql.DB) error { CREATE TABLE IF NOT EXISTS pages ( url_hash INTEGER PRIMARY KEY, url TEXT NOT NULL, -- Kept for reference + cluster_hash INTEGER NOT NULL, html_content TEXT DEFAULT '', metadata TEXT DEFAULT '{}' -- JSON ) WITHOUT ROWID; @@ -264,7 +265,7 @@ func main() { if err != nil { log.Fatal(err) } - stmt, err := tx.Prepare("INSERT INTO pages (url_hash, url, html_content, metadata) VALUES (?, ?, ?, ?)") + stmt, err := tx.Prepare("INSERT INTO pages (url_hash, url, cluster_hash, html_content, metadata) VALUES (?, ?, ?, ?, ?)") if err != nil { log.Fatal(err) } @@ -279,7 +280,10 @@ func main() { } metaJson, _ := json.Marshal(meta) - _, err = stmt.Exec(p.UrlHash, p.Url, p.HtmlContent, string(metaJson)) + // Calculate cluster hash + clusterHash := get8Bytes(hashString(p.Cluster)) + + _, err = stmt.Exec(p.UrlHash, p.Url, clusterHash, p.HtmlContent, string(metaJson)) if err != nil { log.Printf("Error inserting page %s: %v", p.Name, err) } From 555e72e9e00ddf6930f32f3fe33ab4c92ac6dc84 Mon Sep 17 00:00:00 2001 From: lovestaco Date: Sun, 7 Dec 2025 15:57:49 +0530 Subject: [PATCH 211/234] fix: md --- frontend/md/eventListnerIssue.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/frontend/md/eventListnerIssue.md b/frontend/md/eventListnerIssue.md index 33556952fd..51f54cc446 100644 --- a/frontend/md/eventListnerIssue.md +++ b/frontend/md/eventListnerIssue.md @@ -1,3 +1,4 @@ +``` [stdout] [Auth Middleware] JWT found, allowing request to proceed [stdout] [2025-12-06T10:01:18.634Z] [EMOJI_DB] Worker 1 completed getCategoriesWithPreviewEmojis in 25ms [stdout] [2025-12-06T10:01:18.635Z] [EMOJI_DB] Worker 1 handling getTotalEmojis @@ -72,6 +73,7 @@ [stdout] [EMOJI_DB][2025-12-06T10:01:18.837Z] Dispatching getEmojiImages [stdout] [EMOJI_DB][2025-12-06T10:01:18.837Z] getEmojiBySlug completed in 631ms [stdout] [EMOJI_DB]2025-12-06T10:01: +``` once this max listner thing is hit DB query time is exploding From 15d1235b493b36e24bb50b1aaac13f802cea05a8 Mon Sep 17 00:00:00 2001 From: Ganesh Kumar Date: Sun, 7 Dec 2025 16:10:39 +0530 Subject: [PATCH 212/234] feat: Optimize TLDR sitemap generation with paginated queries, dynamic URL priorities, and XML stylesheet integration. --- frontend/db/tldrs/tldr-worker.ts | 71 ++++++++++++++----- frontend/scripts/tldr/test_sitemap.ts | 31 ++++++++ frontend/src/pages/tldr/sitemap-[page].xml.ts | 27 ++++--- frontend/src/pages/tldr/sitemap.xml.ts | 2 +- 4 files changed, 97 insertions(+), 34 deletions(-) create mode 100644 frontend/scripts/tldr/test_sitemap.ts diff --git a/frontend/db/tldrs/tldr-worker.ts b/frontend/db/tldrs/tldr-worker.ts index 679da0a3db..bee175f9d1 100644 --- a/frontend/db/tldrs/tldr-worker.ts +++ b/frontend/db/tldrs/tldr-worker.ts @@ -65,26 +65,20 @@ const statements = { `SELECT name, count, preview_commands_json FROM cluster ORDER BY name` ), - // Sitemap query: Union of all URLs - getSitemapUrls: db.prepare( - `SELECT url FROM ( - SELECT '/freedevtools/tldr/' as url - UNION ALL - SELECT '/freedevtools/tldr/' || name || '/' as url FROM cluster - UNION ALL - SELECT url FROM pages - ) ORDER BY url LIMIT ? OFFSET ?` + // Optimized sitemap queries + getClusterCount: db.prepare('SELECT COUNT(*) as count FROM cluster'), + + getClustersPaginated: db.prepare( + 'SELECT name FROM cluster ORDER BY name LIMIT ? OFFSET ?' + ), + + getPagesPaginated: db.prepare( + 'SELECT url FROM pages ORDER BY url_hash LIMIT ? OFFSET ?' ), - // Count total URLs for sitemap index + // Count total URLs for sitemap index (1 for root + clusters + pages) getSitemapCount: db.prepare( - `SELECT COUNT(*) as count FROM ( - SELECT '/freedevtools/tldr/' as url - UNION ALL - SELECT name FROM cluster - UNION ALL - SELECT url FROM pages - )` + `SELECT (SELECT COUNT(*) FROM cluster) + (SELECT COUNT(*) FROM pages) + 1 as count` ), }; @@ -183,8 +177,47 @@ parentPort?.on('message', (message: QueryMessage) => { case 'getSitemapUrls': { const { limit, offset } = params; - const rows = statements.getSitemapUrls.all(limit, offset) as { url: string }[]; - result = rows.map(row => row.url); + const urls: string[] = []; + let currentOffset = offset; + let remainingLimit = limit; + + // 1. Root URL + if (currentOffset === 0 && remainingLimit > 0) { + urls.push('/freedevtools/tldr/'); + remainingLimit--; + } else if (currentOffset > 0) { + currentOffset--; // Consumed by root + } + + // 2. Cluster URLs + if (remainingLimit > 0) { + const clusterCountRow = statements.getClusterCount.get() as { count: number }; + const clusterCount = clusterCountRow.count; + + if (currentOffset < clusterCount) { + const fetchCount = Math.min(remainingLimit, clusterCount - currentOffset); + const rows = statements.getClustersPaginated.all(fetchCount, currentOffset) as { name: string }[]; + + for (const row of rows) { + urls.push(`/freedevtools/tldr/${row.name}/`); + } + + remainingLimit -= rows.length; + currentOffset = 0; // Reset offset for next section + } else { + currentOffset -= clusterCount; // Skip all clusters + } + } + + // 3. Page URLs + if (remainingLimit > 0) { + const rows = statements.getPagesPaginated.all(remainingLimit, currentOffset) as { url: string }[]; + for (const row of rows) { + urls.push(row.url); + } + } + + result = urls; break; } diff --git a/frontend/scripts/tldr/test_sitemap.ts b/frontend/scripts/tldr/test_sitemap.ts new file mode 100644 index 0000000000..2bb5771c4a --- /dev/null +++ b/frontend/scripts/tldr/test_sitemap.ts @@ -0,0 +1,31 @@ + +import path from 'path'; +import { Worker } from 'worker_threads'; + +const workerPath = path.resolve('../../db/tldrs/tldr-worker.ts'); +const dbPath = path.resolve('../../db/all_dbs/tldr-db-v1.db'); + +const worker = new Worker(workerPath, { + workerData: { + dbPath, + workerId: 1, + }, + execArgv: ['--loader', 'ts-node/esm'], // Use ts-node loader if needed, or just run with bun +}); + +worker.on('message', (message) => { + if (message.ready) { + console.log('Worker ready'); + worker.postMessage({ + id: '1', + type: 'getSitemapUrls', + params: { limit: 10, offset: 0 }, + }); + } else if (message.id === '1') { + console.log('Sitemap URLs:', message.result); + worker.terminate(); + } else if (message.error) { + console.error('Error:', message.error); + worker.terminate(); + } +}); diff --git a/frontend/src/pages/tldr/sitemap-[page].xml.ts b/frontend/src/pages/tldr/sitemap-[page].xml.ts index 4723d73b37..56feca99e5 100644 --- a/frontend/src/pages/tldr/sitemap-[page].xml.ts +++ b/frontend/src/pages/tldr/sitemap-[page].xml.ts @@ -26,29 +26,28 @@ export const GET: APIRoute = async ({ params, site }) => { const siteUrl = envSite || siteStr || 'http://localhost:4321/freedevtools'; const urlEntries = urls.map(url => { - // Ensure URL starts with /freedevtools if not already (it should be from DB) - // But DB stores relative paths like /freedevtools/tldr/... - // We need to prepend siteUrl (which might include /freedevtools if configured that way, but usually is domain) - // siteUrl in this context is usually http://localhost:4321/freedevtools - - // If DB url starts with /freedevtools, and siteUrl ends with /freedevtools, we might double up. - // Let's assume siteUrl is the base domain + base path. - // Actually, siteUrl constructed above includes /freedevtools. - // And DB urls start with /freedevtools. - // So we should strip /freedevtools from one of them or handle it carefully. - // DB URL: /freedevtools/tldr/common/tar/ // Site URL: http://localhost:4321/freedevtools // Result should be: http://localhost:4321/freedevtools/tldr/common/tar/ - // Let's strip /freedevtools from the start of DB URL if present + // Strip /freedevtools from the start of DB URL if present const relativeUrl = url.startsWith('/freedevtools') ? url.substring(13) : url; const fullUrl = `${siteUrl}${relativeUrl}`; - return ` \n ${fullUrl}\n ${now}\n daily\n 0.8\n `; + // Determine priority based on URL depth/type + let priority = '0.8'; + if (relativeUrl === '/tldr/') { + priority = '0.9'; // Landing page + } else if (relativeUrl.split('/').filter(Boolean).length === 2) { + priority = '0.8'; // Cluster page (e.g. /tldr/common/) + } else { + priority = '0.8'; // Command page + } + + return ` \n ${fullUrl}\n ${now}\n daily\n ${priority}\n `; }); - const xml = `\n\n${urlEntries.join( + const xml = `\n\n\n${urlEntries.join( "\n" )}\n`; diff --git a/frontend/src/pages/tldr/sitemap.xml.ts b/frontend/src/pages/tldr/sitemap.xml.ts index 5423c6f575..4d9098974c 100644 --- a/frontend/src/pages/tldr/sitemap.xml.ts +++ b/frontend/src/pages/tldr/sitemap.xml.ts @@ -24,7 +24,7 @@ export const GET: APIRoute = async ({ site }) => { ); } - const xml = `\n\n${sitemaps.join( + const xml = `\n\n\n${sitemaps.join( "\n" )}\n`; From 74ec6ada9d9608515e80361947d3db8cab8f2aa5 Mon Sep 17 00:00:00 2001 From: lovestaco Date: Sun, 7 Dec 2025 20:48:40 +0530 Subject: [PATCH 213/234] fix: make --- frontend/serve/Makefile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/frontend/serve/Makefile b/frontend/serve/Makefile index b53492e343..2b464d337c 100644 --- a/frontend/serve/Makefile +++ b/frontend/serve/Makefile @@ -70,10 +70,10 @@ test-svg-req-pmd: @echo "⬇️ Server up -----------" @sleep 2 @echo "⬇️ Warming up server logs will be flushed for the below requests, ignore the below curls -----------" - hey -z 10s -host hexmos.com http://127.0.0.1/freedevtools/svg_icons/8bit/sharp-solid-alien-8bit/ + hey -z 10s -host hexmos-local.com http://127.0.0.1/freedevtools/svg_icons/8bit/sharp-solid-alien-8bit/ @sleep 2 @echo "⬇️ Requesting SVG icon -----------" - hey -z 1m -host hexmos.com http://127.0.0.1/freedevtools/svg_icons/8bit/sharp-solid-alien-8bit/ + hey -z 1m -host hexmos-local.com http://127.0.0.1/freedevtools/svg_icons/8bit/sharp-solid-alien-8bit/ @sleep 2 @echo "⬇️ Showing logs -----------" @LOG_FILES=$$(ls -1t ~/.pmdaemon/logs/astro-*-out.log 2>/dev/null | head -n2); \ From fc554c04494470de0b693ef3f22ed5d178c065c7 Mon Sep 17 00:00:00 2001 From: lovestaco Date: Sun, 7 Dec 2025 21:17:17 +0530 Subject: [PATCH 214/234] fix: double cache size and tweak other params --- frontend/db/svg_icons/svg-worker.ts | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/frontend/db/svg_icons/svg-worker.ts b/frontend/db/svg_icons/svg-worker.ts index 3a79e16fd9..e3cd80914d 100644 --- a/frontend/db/svg_icons/svg-worker.ts +++ b/frontend/db/svg_icons/svg-worker.ts @@ -34,12 +34,13 @@ const setPragma = (pragma: string) => { } }; -setPragma('PRAGMA cache_size = -64000'); // 64MB cache per connection +setPragma('PRAGMA cache_size = -1048576'); // 1GB cache per connection setPragma('PRAGMA temp_store = MEMORY'); -setPragma('PRAGMA mmap_size = 268435456'); // 256MB memory-mapped I/O +setPragma('PRAGMA mmap_size = 1073741824'); // 1GB memory-mapped I/O setPragma('PRAGMA journal_mode = WAL'); // WAL mode for better concurrent read performance setPragma('PRAGMA query_only = ON'); // Read-only mode -setPragma('PRAGMA page_size = 4096'); // Optimal page size +setPragma('PRAGMA page_size = 8192'); // Optimal page size +setPragma('PRAGMA synchronous = OFF'); // Disable synchronous writes for better performance const statements = { totalIcons: db.prepare('SELECT total_count FROM overview WHERE id = 1'), From 229e1396284b2758528f5cbc2890cfc4cf04a1fd Mon Sep 17 00:00:00 2001 From: Lince Mathew Date: Sun, 7 Dec 2025 21:37:33 +0530 Subject: [PATCH 215/234] feat: Implement paginated man page display and sitemap generation with new database queries and Astro components. --- frontend/db/man_pages/man-pages-schema.ts | 3 +- frontend/db/man_pages/man-pages-utils.ts | 92 +---- .../db/man_pages/man-pages-worker-pool.ts | 30 +- frontend/db/man_pages/man-pages-worker.ts | 347 ++++-------------- .../scripts/man-pages/migrate_manpages_db.py | 109 ++++-- frontend/src/lib/man-pages-utils.ts | 0 .../[category]/[subcategory]/[slug].astro | 280 +------------- .../[subcategory]/_CategoryPagination.astro | 176 +++++++++ .../[category]/[subcategory]/_Page.astro | 147 ++++++++ .../[subcategory]/_SubCategory.astro | 156 ++++++++ .../_SubCategoryPagination.astro | 139 +++++++ .../[category]/[subcategory]/index.astro | 288 +-------------- .../pages/man-pages/[category]/index.astro | 32 +- frontend/src/pages/man-pages/index.astro | 4 +- .../pages/man-pages/sitemap-[index].xml.ts | 115 ++---- frontend/src/pages/man-pages/sitemap.xml.ts | 132 ++----- .../man-pages_pages/sitemap-[index].xml.ts | 294 +++++++-------- .../src/pages/man-pages_pages/sitemap.xml.ts | 318 ++++++++-------- 18 files changed, 1172 insertions(+), 1490 deletions(-) delete mode 100644 frontend/src/lib/man-pages-utils.ts create mode 100644 frontend/src/pages/man-pages/[category]/[subcategory]/_CategoryPagination.astro create mode 100644 frontend/src/pages/man-pages/[category]/[subcategory]/_Page.astro create mode 100644 frontend/src/pages/man-pages/[category]/[subcategory]/_SubCategory.astro create mode 100644 frontend/src/pages/man-pages/[category]/[subcategory]/_SubCategoryPagination.astro diff --git a/frontend/db/man_pages/man-pages-schema.ts b/frontend/db/man_pages/man-pages-schema.ts index a603cfb3c5..f72aeed60d 100644 --- a/frontend/db/man_pages/man-pages-schema.ts +++ b/frontend/db/man_pages/man-pages-schema.ts @@ -9,9 +9,10 @@ export interface ManPage { } export interface ManPageCategory { - category: string; + name: string; count: number; description: string; + path: string; } diff --git a/frontend/db/man_pages/man-pages-utils.ts b/frontend/db/man_pages/man-pages-utils.ts index a10682d5a3..079ebf128e 100644 --- a/frontend/db/man_pages/man-pages-utils.ts +++ b/frontend/db/man_pages/man-pages-utils.ts @@ -1,10 +1,8 @@ import { query } from './man-pages-worker-pool'; import type { - Category, ManPage, ManPageCategory, Overview, - SubCategory, } from './man-pages-schema'; // Get all man page categories with their descriptions @@ -12,68 +10,16 @@ export async function getManPageCategories(): Promise { return query.getManPageCategories(); } -// Get categories from the category table -export async function getCategories(): Promise { - return query.getCategories(); -} - -// Get subcategories from the sub_category table -export async function getSubCategories(): Promise { - return query.getSubCategories(); -} - // Get overview data from the overview table export async function getOverview(): Promise { return query.getOverview(); } -export async function getSubCategoriesByMainCategory( - mainCategory: string -): Promise { - return query.getSubCategoriesByMainCategory(mainCategory); -} - -// Get man pages by category -export async function getManPagesByCategory(category: string): Promise { - return query.getManPagesByCategory(category); -} - -// Get man pages by category and subcategory -export async function getManPagesBySubcategory( - category: string, - subcategory: string -): Promise { - return query.getManPagesBySubcategory(category, subcategory); -} - // Get single man page by Hash ID export async function getManPageByHashId(hashId: bigint | string): Promise { return query.getManPageByHashId(hashId); } -// Generate static paths for all man pages -export async function generateManPageStaticPaths() { - return query.generateManPageStaticPaths(); -} - -// Generate static paths for categories -export async function generateCategoryStaticPaths() { - return query.generateCategoryStaticPaths(); -} - -// Generate static paths for subcategories -export async function generateSubcategoryStaticPaths() { - return query.generateSubcategoryStaticPaths(); -} - -// Get man page by category, subcategory and filename -export async function getManPageByPath( - category: string, - subcategory: string, - filename: string -): Promise { - return query.getManPageByPath(category, subcategory, filename); -} // Get man page by command name (first part of title) export async function getManPageByCommandName( @@ -93,49 +39,41 @@ export async function getManPageBySlug( return getManPageByCommandName(category, subcategory, slug); } -// Generate static paths for individual man pages using command parameter -export async function generateCommandStaticPaths(): Promise> { - return query.generateCommandStaticPaths(); -} - -// Efficient paginated queries for better performance - -export async function getSubCategoriesCountByMainCategory( - mainCategory: string -): Promise { - return query.getSubCategoriesCountByMainCategory(mainCategory); -} - export async function getSubCategoriesByMainCategoryPaginated( mainCategory: string, limit: number, offset: number -): Promise { +): Promise<{ name: string, description: string, count: number }[]> { return query.getSubCategoriesByMainCategoryPaginated(mainCategory, limit, offset); } -export async function getTotalManPagesCountByMainCategory( +export async function getTotalSubCategoriesManPagesCount( mainCategory: string -): Promise { - return query.getTotalManPagesCountByMainCategory(mainCategory); +): Promise<{ man_pages_count: number, sub_category_count: number }> { + return query.getTotalSubCategoriesManPagesCount(mainCategory); } -export async function getManPagesBySubcategoryPaginated( +export async function getManPagesList( mainCategory: string, subCategory: string, limit: number, offset: number ): Promise { - return query.getManPagesBySubcategoryPaginated(mainCategory, subCategory, limit, offset); + return query.getManPagesList(mainCategory, subCategory, limit, offset); } -export async function getManPagesCountBySubcategory( +export async function getManPagesCountInSubCategory( mainCategory: string, subCategory: string ): Promise { - return query.getManPagesCountBySubcategory(mainCategory, subCategory); + return query.getManPagesCountInSubCategory(mainCategory, subCategory); +} + +export async function getAllManPagesPaginated( + limit: number, + offset: number +): Promise<{ main_category: string; sub_category: string; slug: string }[]> { + return query.getAllManPagesPaginated(limit, offset); } // Re-export types for convenience diff --git a/frontend/db/man_pages/man-pages-worker-pool.ts b/frontend/db/man_pages/man-pages-worker-pool.ts index e134cde101..9fb947711b 100644 --- a/frontend/db/man_pages/man-pages-worker-pool.ts +++ b/frontend/db/man_pages/man-pages-worker-pool.ts @@ -218,35 +218,21 @@ export function cleanupWorkers(): Promise { // Export query functions export const query = { getManPageCategories: () => executeQuery('getManPageCategories', {}), - getCategories: () => executeQuery('getCategories', {}), - getSubCategories: () => executeQuery('getSubCategories', {}), getOverview: () => executeQuery('getOverview', {}), - getSubCategoriesByMainCategory: (mainCategory: string) => - executeQuery('getSubCategoriesByMainCategory', { mainCategory }), - getManPagesByCategory: (category: string) => - executeQuery('getManPagesByCategory', { category }), - getManPagesBySubcategory: (category: string, subcategory: string) => - executeQuery('getManPagesBySubcategory', { category, subcategory }), getManPageByHashId: (hashId: bigint | string) => executeQuery('getManPageByHashId', { hashId }), - generateManPageStaticPaths: () => executeQuery('generateManPageStaticPaths', {}), - generateCategoryStaticPaths: () => executeQuery('generateCategoryStaticPaths', {}), - generateSubcategoryStaticPaths: () => executeQuery('generateSubcategoryStaticPaths', {}), - getManPageByPath: (category: string, subcategory: string, filename: string) => - executeQuery('getManPageByPath', { category, subcategory, filename }), getManPageByCommandName: (category: string, subcategory: string, commandName: string) => executeQuery('getManPageByCommandName', { category, subcategory, commandName }), - generateCommandStaticPaths: () => executeQuery('generateCommandStaticPaths', {}), - getSubCategoriesCountByMainCategory: (mainCategory: string) => - executeQuery('getSubCategoriesCountByMainCategory', { mainCategory }), getSubCategoriesByMainCategoryPaginated: (mainCategory: string, limit: number, offset: number) => executeQuery('getSubCategoriesByMainCategoryPaginated', { mainCategory, limit, offset }), - getTotalManPagesCountByMainCategory: (mainCategory: string) => - executeQuery('getTotalManPagesCountByMainCategory', { mainCategory }), - getManPagesBySubcategoryPaginated: (mainCategory: string, subCategory: string, limit: number, offset: number) => - executeQuery('getManPagesBySubcategoryPaginated', { mainCategory, subCategory, limit, offset }), - getManPagesCountBySubcategory: (mainCategory: string, subCategory: string) => - executeQuery('getManPagesCountBySubcategory', { mainCategory, subCategory }), + getTotalSubCategoriesManPagesCount: (mainCategory: string) => + executeQuery('getTotalSubCategoriesManPagesCount', { mainCategory }), + getManPagesList: (mainCategory: string, subCategory: string, limit: number, offset: number) => + executeQuery('getManPagesList', { mainCategory, subCategory, limit, offset }), + getManPagesCountInSubCategory: (mainCategory: string, subCategory: string) => + executeQuery('getManPagesCountInSubCategory', { mainCategory, subCategory }), + getAllManPagesPaginated: (limit: number, offset: number) => + executeQuery('getAllManPagesPaginated', { limit, offset }), }; void initWorkers().catch((err) => { diff --git a/frontend/db/man_pages/man-pages-worker.ts b/frontend/db/man_pages/man-pages-worker.ts index 351877bca4..5b0cb9043c 100644 --- a/frontend/db/man_pages/man-pages-worker.ts +++ b/frontend/db/man_pages/man-pages-worker.ts @@ -40,107 +40,66 @@ setPragma('PRAGMA query_only = ON'); // Read-only mode setPragma('PRAGMA page_size = 4096'); // Optimal page size const statements = { + // get all categories under man page + // url: /man-pages/ manPageCategories: db.prepare(` - SELECT name, count, description + SELECT name, count, description, path FROM category ORDER BY name `), - categories: db.prepare(` - SELECT name, count, description, - json(keywords) as keywords, path - FROM category - ORDER BY name - `), - subCategories: db.prepare(` - SELECT hash_id, main_category, name, count, description, - json(keywords) as keywords, path - FROM sub_category - ORDER BY name - `), + // get total manpage count + // url: /man-pages/ overview: db.prepare(` SELECT id, total_count FROM overview WHERE id = 1 - `), - subCategoriesByMain: db.prepare(` - SELECT hash_id, main_category, name, count, description, - json(keywords) as keywords, path - FROM sub_category - WHERE main_category = ? - ORDER BY name - `), - manPagesByCategory: db.prepare(` - SELECT hash_id, main_category, sub_category, title, slug, filename, - json(content) as content - FROM man_pages - WHERE main_category = ? - ORDER BY title - `), - manPagesBySubcategory: db.prepare(` - SELECT hash_id, main_category, sub_category, title, slug, filename, - json(content) as content - FROM man_pages - WHERE main_category = ? AND sub_category = ? - ORDER BY title - `), + `), + // get man page by hash id + // url: /man-pages/[category]/[subcategory]/[command] manPageByHashId: db.prepare(` - SELECT hash_id, main_category, sub_category, title, slug, filename, + SELECT title, slug, filename, json(content) as content FROM man_pages WHERE hash_id = ? `), - manPageStaticPaths: db.prepare(` - SELECT hash_id, main_category, sub_category - FROM man_pages - `), - categoryStaticPaths: db.prepare(` - SELECT name - FROM category - `), - subcategoryStaticPaths: db.prepare(` - SELECT main_category, name as sub_category - FROM sub_category - `), - manPageByPath: db.prepare(` - SELECT hash_id, main_category, sub_category, title, filename, - json(content) as content - FROM man_pages - WHERE main_category = ? AND sub_category = ? AND filename = ? - `), - commandStaticPaths: db.prepare(` - SELECT main_category, sub_category, slug - FROM man_pages - ORDER BY main_category, sub_category, slug - `), - subCategoriesCountByMain: db.prepare(` - SELECT COUNT(*) as count - FROM sub_category - WHERE main_category = ? - `), + // for fetching sub categories + // for page man-pages/[category]/2 subCategoriesByMainPaginated: db.prepare(` - SELECT hash_id, main_category, name, count, description, - json(keywords) as keywords, path + SELECT name, description, count FROM sub_category - WHERE main_category = ? + WHERE main_category_hash = ? ORDER BY name LIMIT ? OFFSET ? `), - totalManPagesCountByMain: db.prepare(` - SELECT COUNT(*) as count - FROM man_pages - WHERE main_category = ? + // for fetching total man pages, sub categories count + // for page man-pages/[category]/[subcategory]/ + totalSubCategoriesManPagesCount: db.prepare(` + SELECT count,sub_category_count + FROM category + WHERE hash_id = ? `), + + // for fetching man pages by subcategory + // used in page: [category]/[subcategory]/index.astro manPagesBySubcategoryPaginated: db.prepare(` - SELECT hash_id, main_category, sub_category, title, slug, filename, content + SELECT title, slug FROM man_pages - WHERE main_category = ? AND sub_category = ? - ORDER BY title + WHERE category_hash = ? LIMIT ? OFFSET ? `), + // for fetching total man pages count by subcategory + // used in page: [category]/[subcategory]/index.astro manPagesCountBySubcategory: db.prepare(` - SELECT COUNT(*) as count - FROM man_pages - WHERE main_category = ? AND sub_category = ? + SELECT count from sub_category + WHERE hash_id = ? + `), + // for fetching all man pages paginated + // used in sitemap generation + getAllManPagesPaginated: db.prepare(` + SELECT main_category, sub_category, slug + FROM man_pages + ORDER BY hash_id + LIMIT ? OFFSET ? `), }; @@ -177,46 +136,12 @@ parentPort?.on('message', (message: QueryMessage) => { name: string; count: number; description: string; - }>; - result = rows.map((row) => ({ - category: row.name, - count: row.count, - description: row.description, - })); - break; - } - - case 'getCategories': { - const rows = statements.categories.all() as Array<{ - name: string; - count: number; - description: string; - keywords: string; path: string; }>; - result = rows.map((row) => ({ - ...row, - keywords: JSON.parse(row.keywords || '[]'), - })); + result = rows; break; } - case 'getSubCategories': { - const rows = statements.subCategories.all() as Array<{ - hash_id: bigint; - main_category: string; - name: string; - count: number; - description: string; - keywords: string; - path: string; - }>; - result = rows.map((row) => ({ - ...row, - keywords: JSON.parse(row.keywords || '[]'), - })); - break; - } case 'getOverview': { const row = statements.overview.get() as { id: number; total_count: number } | undefined; @@ -231,62 +156,7 @@ parentPort?.on('message', (message: QueryMessage) => { break; } - case 'getSubCategoriesByMainCategory': { - const { mainCategory } = params; - const rows = statements.subCategoriesByMain.all(mainCategory) as Array<{ - hash_id: bigint; - main_category: string; - name: string; - count: number; - description: string; - keywords: string; - path: string; - }>; - result = rows.map((row) => ({ - name: row.name, - count: row.count, - description: row.description, - keywords: JSON.parse(row.keywords || '[]'), - path: row.path, - })); - break; - } - case 'getManPagesByCategory': { - const { category } = params; - const rows = statements.manPagesByCategory.all(category) as Array<{ - hash_id: bigint; - main_category: string; - sub_category: string; - title: string; - slug: string; - filename: string; - content: string; - }>; - result = rows.map((row) => ({ - ...row, - content: JSON.parse(row.content || '{}'), - })); - break; - } - - case 'getManPagesBySubcategory': { - const { category, subcategory } = params; - const rows = statements.manPagesBySubcategory.all(category, subcategory) as Array<{ - hash_id: bigint; - main_category: string; - sub_category: string; - title: string; - slug: string; - filename: string; - content: string; - }>; - result = rows.map((row) => ({ - ...row, - content: JSON.parse(row.content || '{}'), - })); - break; - } case 'getManPageByHashId': { const { hashId } = params; @@ -311,68 +181,6 @@ parentPort?.on('message', (message: QueryMessage) => { break; } - case 'generateManPageStaticPaths': { - const rows = statements.manPageStaticPaths.all() as Array<{ - hash_id: bigint; - main_category: string; - sub_category: string; - }>; - result = rows.map((row) => ({ - params: { - category: row.main_category, - subcategory: row.sub_category, - page: row.hash_id.toString(), - }, - })); - break; - } - - case 'generateCategoryStaticPaths': { - const rows = statements.categoryStaticPaths.all() as Array<{ name: string }>; - result = rows.map((row) => ({ - params: { - category: row.name, - }, - })); - break; - } - - case 'generateSubcategoryStaticPaths': { - const rows = statements.subcategoryStaticPaths.all() as Array<{ - main_category: string; - sub_category: string; - }>; - result = rows.map((row) => ({ - params: { - category: row.main_category, - subcategory: row.sub_category, - }, - })); - break; - } - - case 'getManPageByPath': { - const { category, subcategory, filename } = params; - const row = statements.manPageByPath.get(category, subcategory, filename) as { - hash_id: bigint; - main_category: string; - sub_category: string; - title: string; - filename: string; - content: string; - } | undefined; - - if (!row) { - result = null; - } else { - result = { - ...row, - content: JSON.parse(row.content || '{}'), - }; - } - break; - } - case 'getManPageByCommandName': { const { category, subcategory, commandName } = params; const hashId = hashUrlToKey(category, subcategory, commandName); @@ -397,92 +205,65 @@ parentPort?.on('message', (message: QueryMessage) => { break; } - case 'generateCommandStaticPaths': { - const rows = statements.commandStaticPaths.all() as Array<{ - main_category: string; - sub_category: string; - slug: string; - }>; - console.log(`📊 Found ${rows.length} total man pages in database`); - result = rows.map((manPage) => ({ - params: { - category: manPage.main_category, - subcategory: manPage.sub_category, - slug: manPage.slug, - }, - })); - if (result.length > 0) { - console.log('🔍 Sample generated paths:', result.slice(0, 5)); - } - break; - } - - case 'getSubCategoriesCountByMainCategory': { - const { mainCategory } = params; - const row = statements.subCategoriesCountByMain.get(mainCategory) as { count: number } | undefined; - result = row?.count || 0; - break; - } - case 'getSubCategoriesByMainCategoryPaginated': { const { mainCategory, limit, offset } = params; + const categoryHashId = hashUrlToKey(mainCategory, '', ''); const limitInt = Math.floor(limit); const offsetInt = Math.floor(offset); - const rows = statements.subCategoriesByMainPaginated.all(mainCategory, limitInt, offsetInt) as Array<{ - hash_id: bigint; - main_category: string; + const rows = statements.subCategoriesByMainPaginated.all(categoryHashId, limitInt, offsetInt) as Array<{ name: string; - count: number; description: string; - keywords: string; - path: string; + count: number; }>; - result = rows.map((row) => ({ - ...row, - keywords: JSON.parse(row.keywords || '[]'), - })); + result = rows; break; } - case 'getTotalManPagesCountByMainCategory': { + case 'getTotalSubCategoriesManPagesCount': { const { mainCategory } = params; - const row = statements.totalManPagesCountByMain.get(mainCategory) as { count: number } | undefined; - result = row?.count || 0; + const categoryHashId = hashUrlToKey(mainCategory, '', ''); + const row = statements.totalSubCategoriesManPagesCount.get(categoryHashId) as { count: number, sub_category_count: number } | undefined; + result = { man_pages_count: row?.count || 0, sub_category_count: row?.sub_category_count || 0 }; break; } - case 'getManPagesBySubcategoryPaginated': { + case 'getManPagesList': { const { mainCategory, subCategory, limit, offset } = params; + const categoryHashId = hashUrlToKey(mainCategory, subCategory, ''); const limitInt = Math.floor(limit); const offsetInt = Math.floor(offset); - const rows = statements.manPagesBySubcategoryPaginated.all(mainCategory, subCategory, limitInt, offsetInt) as Array<{ - hash_id: bigint; - main_category: string; - sub_category: string; + const rows = statements.manPagesBySubcategoryPaginated.all(categoryHashId, limitInt, offsetInt) as Array<{ title: string; slug: string; - filename: string; - content: string; }>; result = rows.map((row) => ({ - hash_id: row.hash_id, - main_category: row.main_category, - sub_category: row.sub_category, title: row.title, slug: row.slug, - filename: row.filename, - content: JSON.parse(row.content), })); break; } - case 'getManPagesCountBySubcategory': { + case 'getManPagesCountInSubCategory': { const { mainCategory, subCategory } = params; - const row = statements.manPagesCountBySubcategory.get(mainCategory, subCategory) as { count: number } | undefined; + const categoryHashId = hashUrlToKey(mainCategory, subCategory, ''); + const row = statements.manPagesCountBySubcategory.get(categoryHashId) as { count: number } | undefined; result = row?.count || 0; break; } + case 'getAllManPagesPaginated': { + const { limit, offset } = params; + const limitInt = Math.floor(limit); + const offsetInt = Math.floor(offset); + const rows = statements.getAllManPagesPaginated.all(limitInt, offsetInt) as Array<{ + main_category: string; + sub_category: string; + slug: string; + }>; + result = rows; + break; + } + default: throw new Error(`Unknown query type: ${type}`); } diff --git a/frontend/scripts/man-pages/migrate_manpages_db.py b/frontend/scripts/man-pages/migrate_manpages_db.py index 74fea3b8c1..3833f76432 100644 --- a/frontend/scripts/man-pages/migrate_manpages_db.py +++ b/frontend/scripts/man-pages/migrate_manpages_db.py @@ -6,20 +6,21 @@ Destination: db/all_dbs/man-pages-new-db.db Changes: -- man_pages table: - - Remove 'id' (AUTOINCREMENT) - - Add 'hash_id' (INTEGER PRIMARY KEY) generated from main_category + sub_category + slug - - Enable WITHOUT ROWID +1. Create new schema with hash_id for man_pages and sub_category. +2. man_pages table uses main_category_hash and sub_category_hash (integers) instead of text. +3. Migrate data from old DB to new DB. +4. Add 'sub_category_count' to 'category' table and populate it. +5. Add 'category_hash_id' to 'sub_category' and 'hash_id' to 'category' tables. +6. Populate hash IDs. """ import sqlite3 import hashlib -import shutil from pathlib import Path BASE_DIR = Path(__file__).parent.parent.parent -OLD_DB_PATH = BASE_DIR / "db" / "all_dbs" / "man-pages-db.db" -NEW_DB_PATH = BASE_DIR / "db" / "all_dbs" / "man-pages-new-db-1.db" +OLD_DB_PATH = BASE_DIR / "db" / "all_dbs" / "man-pages-db_old.db" +NEW_DB_PATH = BASE_DIR / "db" / "all_dbs" / "man-pages-db.db" def generate_hash_id(main_category: str, sub_category: str, slug: str) -> int: """ @@ -27,17 +28,25 @@ def generate_hash_id(main_category: str, sub_category: str, slug: str) -> int: """ combined = f"{main_category}{sub_category}{slug}" hash_bytes = hashlib.sha256(combined.encode('utf-8')).digest() - # Take first 8 bytes, interpret as big-endian signed integer return int.from_bytes(hash_bytes[:8], byteorder='big', signed=True) -def generate_subcategory_hash_id(main_category: str, sub_category: str) -> int: +def generate_subcategory_pk_hash(main_category: str, sub_category: str) -> int: """ Generate a 64-bit signed integer hash from main_category and sub_category. + Used for sub_category table Primary Key. """ combined = f"{main_category}{sub_category}" hash_bytes = hashlib.sha256(combined.encode('utf-8')).digest() return int.from_bytes(hash_bytes[:8], byteorder='big', signed=True) +def generate_simple_hash(text: str) -> int: + """ + Generate a 64-bit signed integer hash from a single string. + Used for main_category_hash and sub_category_hash columns. + """ + hash_bytes = hashlib.sha256(text.encode('utf-8')).digest() + return int.from_bytes(hash_bytes[:8], byteorder='big', signed=True) + def migrate_db(): if not OLD_DB_PATH.exists(): print(f"Source DB not found: {OLD_DB_PATH}") @@ -53,7 +62,7 @@ def migrate_db(): old_cur = old_conn.cursor() new_cur = new_conn.cursor() - # 1. Setup New Schema + # --- 1. Setup New Schema --- # Performance PRAGMAs new_cur.execute("PRAGMA journal_mode = WAL;") new_cur.execute("PRAGMA synchronous = OFF;") @@ -61,11 +70,12 @@ def migrate_db(): new_cur.execute("PRAGMA temp_store = MEMORY;") new_cur.execute("PRAGMA mmap_size = 536870912;") - # Create 'man_pages' table with hash_id + # Create 'man_pages' table with hash_id and hashed categories new_cur.execute( """ CREATE TABLE man_pages ( hash_id INTEGER PRIMARY KEY, + category_hash INTEGER NOT NULL, main_category TEXT NOT NULL, sub_category TEXT NOT NULL, title TEXT NOT NULL, @@ -76,35 +86,37 @@ def migrate_db(): """ ) - # Create 'category' table (Copy schema) + # Create 'category' table new_cur.execute( """ CREATE TABLE category ( - name TEXT PRIMARY KEY, + name TEXT, count INTEGER NOT NULL DEFAULT 0, description TEXT DEFAULT '', keywords TEXT DEFAULT '[]', - path TEXT DEFAULT '' - ); + path TEXT DEFAULT '', + sub_category_count INTEGER DEFAULT 0, + hash_id INTEGER PRIMARY KEY + ) WITHOUT ROWID; """ ) - # Create 'sub_category' table with hash_id + # Create 'sub_category' table new_cur.execute( """ CREATE TABLE sub_category ( hash_id INTEGER PRIMARY KEY, - main_category TEXT NOT NULL, name TEXT NOT NULL, count INTEGER NOT NULL DEFAULT 0, description TEXT DEFAULT '', keywords TEXT DEFAULT '[]', - path TEXT DEFAULT '' + path TEXT DEFAULT '', + main_category_hash INTEGER ) WITHOUT ROWID; """ ) - # Create 'overview' table (Copy schema) + # Create 'overview' table new_cur.execute( """ CREATE TABLE overview ( @@ -114,18 +126,11 @@ def migrate_db(): """ ) - # Create Indexes for Performance - print("Creating indexes...") - # Index for fetching man pages by subcategory (e.g. /man-pages/linux/commands/) - new_cur.execute("CREATE INDEX idx_man_pages_main_sub_cat ON man_pages(main_category, sub_category);") - # Index for fetching subcategories by main category (e.g. /man-pages/linux/) - new_cur.execute("CREATE INDEX idx_sub_category_main_cat ON sub_category(main_category);") - - # 2. Migrate Data + + # --- 2. Migrate Data --- # Migrate 'man_pages' - # Migrate 'man_pages' - print("Migrating man_pages (LIMIT 100)...") + print("Migrating man_pages...") old_cur.execute("SELECT main_category, sub_category, title, slug, filename, content FROM man_pages") rows = old_cur.fetchall() @@ -133,14 +138,16 @@ def migrate_db(): for row in rows: main_cat, sub_cat, title, slug, filename, content = row hash_id = generate_hash_id(main_cat, sub_cat, slug) + # category_hash in man_pages corresponds to hash_id in sub_category (main + sub) + category_hash = generate_subcategory_pk_hash(main_cat, sub_cat) try: new_cur.execute( """ - INSERT INTO man_pages (hash_id, main_category, sub_category, title, slug, filename, content) - VALUES (?, ?, ?, ?, ?, ?, ?) + INSERT INTO man_pages (hash_id, category_hash, main_category, sub_category, title, slug, filename, content) + VALUES (?, ?, ?, ?, ?, ?, ?, ?) """, - (hash_id, main_cat, sub_cat, title, slug, filename, content) + (hash_id, category_hash, main_cat, sub_cat, title, slug, filename, content) ) inserted_count += 1 except sqlite3.IntegrityError as e: @@ -150,14 +157,36 @@ def migrate_db(): # Migrate 'category' print("Migrating category...") - old_cur.execute("SELECT * FROM category") + old_cur.execute("SELECT name, count, description, keywords, path FROM category") categories = old_cur.fetchall() - new_cur.executemany("INSERT INTO category VALUES (?, ?, ?, ?, ?)", categories) - print(f"Migrated {len(categories)} categories.") + + print("Calculating sub_category counts...") + old_cur.execute(""" + SELECT main_category, COUNT(DISTINCT sub_category) + FROM man_pages + GROUP BY main_category + """) + cat_counts = dict(old_cur.fetchall()) + + cat_inserted = 0 + for row in categories: + name, count, description, keywords, path = row + sub_cat_count = cat_counts.get(name, 0) + hash_id = generate_simple_hash(name) + + new_cur.execute( + """ + INSERT INTO category (name, count, description, keywords, path, sub_category_count, hash_id) + VALUES (?, ?, ?, ?, ?, ?, ?) + """, + (name, count, description, keywords, path, sub_cat_count, hash_id) + ) + cat_inserted += 1 + + print(f"Migrated {cat_inserted} categories.") # Migrate 'sub_category' print("Migrating sub_category...") - # Calculate count dynamically from man_pages and join with old sub_category table for metadata old_cur.execute(""" SELECT m.main_category, @@ -175,21 +204,21 @@ def migrate_db(): sub_cat_inserted = 0 for row in sub_categories: main_cat, sub_cat_name, count, description, keywords, path = row - # Handle nulls from left join if any count = count or 0 description = description or '' keywords = keywords or '[]' path = path or '' - hash_id = generate_subcategory_hash_id(main_cat, sub_cat_name) + hash_id = generate_subcategory_pk_hash(main_cat, sub_cat_name) + category_hash_id = generate_simple_hash(main_cat) try: new_cur.execute( """ - INSERT INTO sub_category (hash_id, main_category, name, count, description, keywords, path) + INSERT INTO sub_category (hash_id, name, count, description, keywords, path, main_category_hash) VALUES (?, ?, ?, ?, ?, ?, ?) """, - (hash_id, main_cat, sub_cat_name, count, description, keywords, path) + (hash_id, sub_cat_name, count, description, keywords, path, category_hash_id) ) sub_cat_inserted += 1 except sqlite3.IntegrityError as e: diff --git a/frontend/src/lib/man-pages-utils.ts b/frontend/src/lib/man-pages-utils.ts deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/frontend/src/pages/man-pages/[category]/[subcategory]/[slug].astro b/frontend/src/pages/man-pages/[category]/[subcategory]/[slug].astro index f71d2a46bd..2b09e89f65 100644 --- a/frontend/src/pages/man-pages/[category]/[subcategory]/[slug].astro +++ b/frontend/src/pages/man-pages/[category]/[subcategory]/[slug].astro @@ -1,16 +1,6 @@ --- -import AdBanner from '@/components/banner/AdBanner.astro'; -import CreditsButton from '@/components/buttons/CreditsButton'; -import Pagination from '@/components/PaginationComponent.astro'; -import SeeAlsoIndex from '@/components/seealso/SeeAlsoIndex.astro'; -import ToolContainer from '@/components/tool/ToolContainer'; -import ToolHead from '@/components/tool/ToolHead'; -import BaseLayout from '@/layouts/BaseLayout.astro'; -import { - getManPageBySlug, - getManPagesBySubcategoryPaginated, - getManPagesCountBySubcategory, -} from 'db/man_pages/man-pages-utils'; +import PaginationView from './_SubCategoryPagination.astro'; +import PageView from './_Page.astro'; export const prerender = false; @@ -23,264 +13,12 @@ if (!category || !subcategory || !slug) { // Check if slug is numeric (pagination) const isNumericPage = /^\d+$/.test(slug); - -// --- VARIABLES FOR BOTH VIEWS --- -let pageTitle = ''; -let pageDescription = ''; -let breadcrumbItems: any[] = []; -let currentPage = 1; -let totalPages = 1; -let isPaginationView = false; - -// Variables specific to pagination view -let currentPageManPages: any[] = []; -let totalManPagesCount = 0; - -// Variables specific to man page view -let manPage: any = null; -let tocSections: any[] = []; - -if (isNumericPage) { - // --- PAGINATION LOGIC (formerly [page].astro) --- - isPaginationView = true; - currentPage = parseInt(slug, 10); - - // Get total count of man pages for this subcategory to determine pagination - totalManPagesCount = await getManPagesCountBySubcategory( - category, - subcategory - ); - const itemsPerPage = 20; - totalPages = Math.ceil(totalManPagesCount / itemsPerPage); - - // Redirect if page is out of range or invalid (optional but good practice) - if (currentPage < 1 || (totalPages > 0 && currentPage > totalPages)) { - return Astro.redirect( - `/freedevtools/man-pages/${category}/${subcategory}/` - ); - } - - // Calculate pagination - const offset = (currentPage - 1) * itemsPerPage; - - // Get ONLY the man pages for current page from database (efficient!) - currentPageManPages = await getManPagesBySubcategoryPaginated( - category, - subcategory, - itemsPerPage, - offset - ); - - const categoryTitle = category.replace('-', ' ').charAt(0).toUpperCase() + category.replace('-', ' ').slice(1); - const subcategoryTitle = subcategory.replace('-', ' ').charAt(0).toUpperCase() + subcategory.replace('-', ' ').slice(1); - - pageTitle = `${subcategoryTitle} Man Pages - Page ${currentPage}`; - pageDescription = `Browse ${subcategoryTitle} manual pages - Page ${currentPage} of ${totalPages}. Find detailed documentation and system references.`; - - breadcrumbItems = [ - { label: 'Free DevTools', href: '/freedevtools/' }, - { label: 'Man Pages', href: '/freedevtools/man-pages/' }, - { label: categoryTitle, href: `/freedevtools/man-pages/${category}/` }, - { label: subcategoryTitle, href: `/freedevtools/man-pages/${category}/${subcategory}/` }, - { label: `Page ${currentPage}` }, - ]; - -} else { - // --- MAN PAGE LOGIC (formerly [slug].astro) --- - isPaginationView = false; - - // Get man page from database by slug - manPage = await getManPageBySlug(category, subcategory, slug); - - if (!manPage) { - // Redirect to man pages index if not found - return Astro.redirect('/freedevtools/man-pages/'); - } - - // Generate table of contents from sections - tocSections = Object.keys(manPage.content).map((section) => ({ - id: section.toLowerCase(), - label: section, - })); - - pageTitle = `${manPage.title} - Manual Page`; - pageDescription = `${manPage.title} manual page. Learn about ${manPage.title.split(' — ')[1] || manPage.title.split(' — ')[0]} in systems.`; - - breadcrumbItems = [ - { label: 'Free DevTools', href: '/freedevtools/' }, - { label: 'Man Pages', href: '/freedevtools/man-pages/' }, - { - label: manPage.main_category.replace('-', ' ').charAt(0).toUpperCase() + manPage.main_category.replace('-', ' ').slice(1), - href: `/freedevtools/man-pages/${manPage.main_category}/`, - }, - { - label: manPage.sub_category.replace('-', ' ').charAt(0).toUpperCase() + manPage.sub_category.replace('-', ' ').slice(1), - href: `/freedevtools/man-pages/${manPage.main_category}/${manPage.sub_category}/`, - }, - { - label: manPage.slug, - href: `/freedevtools/man-pages/${manPage.main_category}/${manPage.sub_category}/${manPage.slug}/`, - }, - ]; -} - -const subcategoryTitle = subcategory.replace('-', ' ').charAt(0).toUpperCase() + subcategory.replace('-', ' ').slice(1); -const categoryTitle = category.replace('-', ' ').charAt(0).toUpperCase() + category.replace('-', ' ').slice(1); --- - - -
- -
- - - {isPaginationView ? ( - <> - -
-
- Showing {currentPageManPages.length} of {totalManPagesCount} man pages (Page {currentPage} of {totalPages}) -
-
- - -
-
- {currentPageManPages.map((page) => ( - -

- {page.title.length > 60 - ? page.title.slice(0, 57) + '...' - : page.title} -

- - ))} -
-
- - - - - ) : ( - <> - -
-

Contents

- -
- - -
- -
- {Object.entries(manPage.content).map(([section, content]) => ( -
-

- {section} -

-
-
- ))} -
-
- - - - )} - - -
-
- - View {categoryTitle} Category - - {isPaginationView ? ( - - ← Back to {subcategoryTitle} - - ) : ( - - ← Back to {manPage.sub_category.replace('-', ' ').charAt(0).toUpperCase() + manPage.sub_category.replace('-', ' ').slice(1)} - - )} - -
-
- - - - +{isNumericPage ? ( + // If numeric, it's a pagination page eg: /man-pages/linux/commands/2/ + +) : ( + // If not numeric, it's a man page eg: /man-pages/linux/commands/ddttrf/ + +)} diff --git a/frontend/src/pages/man-pages/[category]/[subcategory]/_CategoryPagination.astro b/frontend/src/pages/man-pages/[category]/[subcategory]/_CategoryPagination.astro new file mode 100644 index 0000000000..b4f4f1bc99 --- /dev/null +++ b/frontend/src/pages/man-pages/[category]/[subcategory]/_CategoryPagination.astro @@ -0,0 +1,176 @@ +--- +import AdBanner from '@/components/banner/AdBanner.astro'; +import CreditsButton from '@/components/buttons/CreditsButton'; +import Pagination from '@/components/PaginationComponent.astro'; +import ToolContainer from '@/components/tool/ToolContainer'; +import ToolHead from '@/components/tool/ToolHead'; +import BaseLayout from '@/layouts/BaseLayout.astro'; +import { + getTotalSubCategoriesManPagesCount, + getSubCategoriesByMainCategoryPaginated, +} from 'db/man_pages/man-pages-utils'; +import { formatName } from '@/lib/utils'; + +const { category, page } = Astro.props; + + +// This handles /man-pages/user-commands/2/ (Page 2 of subcategories list) + +const currentPage = parseInt(page, 10); + +// Get subcategories count for this category to determine pagination +const {man_pages_count, sub_category_count} = await getTotalSubCategoriesManPagesCount(category); + + +const itemsPerPage = 12; +const totalPages = Math.ceil(sub_category_count / itemsPerPage); +// Redirect if page is out of range or invalid +if (currentPage < 1 || (totalPages > 0 && currentPage > totalPages)) { + return Astro.redirect(`/freedevtools/man-pages/${category}/`); +} + +// Calculate pagination +const offset = (currentPage - 1) * itemsPerPage; + +// Get ONLY the subcategories for current page from database (efficient!) +const subcategories = await getSubCategoriesByMainCategoryPaginated( + category, + itemsPerPage, + offset +); + +const categoryTitle = formatName(category); + +const pageTitle = `${categoryTitle} Man Pages - Page ${currentPage}`; +const pageDescription = `Browse ${categoryTitle} manual page subcategories - Page ${currentPage} of ${totalPages}. Find detailed documentation and system references.`; + +const breadcrumbItems = [ + { label: 'Free DevTools', href: '/freedevtools/' }, + { label: 'Man Pages', href: '/freedevtools/man-pages/' }, + { label: categoryTitle, href: `/freedevtools/man-pages/${category}/` }, + { label: `Page ${currentPage}` }, +]; +--- + + + +
+ +
+ + + +
+
+
+
+ { + sub_category_count >= 1000 + ? Math.round(sub_category_count / 1000) + 'k' + : sub_category_count + } +
+
+ Subcategories +
+
+
+
+ {man_pages_count >= 1000 + ? Math.round(man_pages_count / 1000) + 'k' + : man_pages_count} +
+
+ Man Pages +
+
+
+
+ + +
+
+ Showing {subcategories.length} of {sub_category_count} subcategories (Page {currentPage} of {totalPages}) +
+
+ + +
+
+ { + subcategories.map((subcategory) => ( + +
+

+ {subcategory.name.replace('-', ' ')} +

+ + {subcategory.count} pages + +
+ {subcategory.description && ( +

+ {subcategory.description} +

+ )} + + )) + } +
+
+ + + { + sub_category_count === 0 && ( +
+

+ No subcategories found. +

+
+ ) + } + + + + + +
+
+ + View {categoryTitle} Category + + +
+
+ + diff --git a/frontend/src/pages/man-pages/[category]/[subcategory]/_Page.astro b/frontend/src/pages/man-pages/[category]/[subcategory]/_Page.astro new file mode 100644 index 0000000000..40b06c3a06 --- /dev/null +++ b/frontend/src/pages/man-pages/[category]/[subcategory]/_Page.astro @@ -0,0 +1,147 @@ +--- +import AdBanner from '@/components/banner/AdBanner.astro'; +import CreditsButton from '@/components/buttons/CreditsButton'; +import SeeAlsoIndex from '@/components/seealso/SeeAlsoIndex.astro'; +import ToolContainer from '@/components/tool/ToolContainer'; +import ToolHead from '@/components/tool/ToolHead'; +import BaseLayout from '@/layouts/BaseLayout.astro'; +import { getManPageBySlug } from 'db/man_pages/man-pages-utils'; +import { formatName } from '@/lib/utils'; + + + +const { category, subcategory, slug } = Astro.props; + +const manPage = await getManPageBySlug(category, subcategory, slug); + +if (!manPage) { + // Redirect to man pages index if not found + return Astro.redirect('/freedevtools/man-pages/'); +} + +// Table of contents sections +const tocSections = Object.keys(manPage.content).map((section) => ({ + id: section.toLowerCase(), + label: section, +})); + +const pageDescription = `${manPage.title} manual page. Learn about ${manPage.title.split(' — ')[1]}`; + +const breadcrumbItems = [ + { label: 'Free DevTools', href: '/freedevtools/' }, + { label: 'Man Pages', href: '/freedevtools/man-pages/' }, + { + label: formatName(category), + href: `/freedevtools/man-pages/${category}/`, + }, + { + label: formatName(subcategory), + href: `/freedevtools/man-pages/${category}/${subcategory}/`, + }, + { + label: manPage.slug, + href: `/freedevtools/man-pages/${manPage.main_category}/${manPage.sub_category}/${manPage.slug}/`, + }, +]; + +--- + + + +
+ +
+ + + +
+

Contents

+ +
+ + +
+ +
+ {Object.entries(manPage.content).map(([section, content]) => ( +
+

+ {section} +

+
+
+ ))} +
+
+ + + + +
+
+ + View {formatName(category)} Category + + + ← Back to {formatName(subcategory)} + + +
+
+ + + + diff --git a/frontend/src/pages/man-pages/[category]/[subcategory]/_SubCategory.astro b/frontend/src/pages/man-pages/[category]/[subcategory]/_SubCategory.astro new file mode 100644 index 0000000000..3207d4d0f6 --- /dev/null +++ b/frontend/src/pages/man-pages/[category]/[subcategory]/_SubCategory.astro @@ -0,0 +1,156 @@ +--- +import AdBanner from '@/components/banner/AdBanner.astro'; +import CreditsButton from '@/components/buttons/CreditsButton'; +import Pagination from '@/components/PaginationComponent.astro'; +import ToolContainer from '@/components/tool/ToolContainer'; +import ToolHead from '@/components/tool/ToolHead'; +import BaseLayout from '@/layouts/BaseLayout.astro'; +import { + getManPagesList, + getManPagesCountInSubCategory, +} from 'db/man_pages/man-pages-utils'; +import { formatName } from '@/lib/utils'; + +const { category, subcategory } = Astro.props; + +// This handles /man-pages/linux/commands/ (Page 1 of man pages list) +const itemsPerPage = 20; +const currentPage = 1; +const offset = (currentPage - 1) * itemsPerPage; + + +const items = await getManPagesList( + category, + subcategory, + itemsPerPage, + offset +); +const totalManPagesCount = await getManPagesCountInSubCategory( + category, + subcategory +); +const totalCount = totalManPagesCount; +const totalPages = Math.ceil(totalManPagesCount / itemsPerPage); + +const categoryTitle = formatName(category); +const subcategoryTitle = formatName(subcategory); + +const pageTitle = `${subcategoryTitle} Man Pages`; +const pageDescription = `Browse ${subcategoryTitle} manual pages in ${categoryTitle}. Find detailed documentation and system references.`; + +const breadcrumbItems = [ + { label: 'Free DevTools', href: '/freedevtools/' }, + { label: 'Man Pages', href: '/freedevtools/man-pages/' }, + { label: categoryTitle, href: `/freedevtools/man-pages/${category}/` }, + { label: subcategoryTitle }, +]; +--- + + + +
+ +
+ + + +
+
+
+
+ { + totalCount >= 1000 + ? Math.round(totalCount / 1000) + 'k' + : totalCount + } +
+
+ Man Pages in {subcategoryTitle} +
+
+
+
+ + +
+
+ Showing {items.length} of {totalCount} man pages (Page {currentPage} of {totalPages}) +
+
+ + +
+
+ { + items.map((item) => ( + +

+ {item.title.length > 60 + ? item.title.slice(0, 57) + '...' + : item.title} +

+ + )) + } +
+
+ + + { + totalCount === 0 && ( +
+

+ No man pages found. +

+
+ ) + } + + + + + +
+
+ + View {categoryTitle} Category + + +
+
+ + diff --git a/frontend/src/pages/man-pages/[category]/[subcategory]/_SubCategoryPagination.astro b/frontend/src/pages/man-pages/[category]/[subcategory]/_SubCategoryPagination.astro new file mode 100644 index 0000000000..f71fcf2927 --- /dev/null +++ b/frontend/src/pages/man-pages/[category]/[subcategory]/_SubCategoryPagination.astro @@ -0,0 +1,139 @@ +--- +import AdBanner from '@/components/banner/AdBanner.astro'; +import CreditsButton from '@/components/buttons/CreditsButton'; +import Pagination from '@/components/PaginationComponent.astro'; +import ToolContainer from '@/components/tool/ToolContainer'; +import ToolHead from '@/components/tool/ToolHead'; +import BaseLayout from '@/layouts/BaseLayout.astro'; +import { + getManPagesList, + getManPagesCountInSubCategory, +} from 'db/man_pages/man-pages-utils'; +import { formatName } from '@/lib/utils'; + +const { category, subcategory, page } = Astro.props; + +const currentPage = parseInt(page, 10); + +// Get total count of man pages for this subcategory to determine pagination +const totalManPagesCount = await getManPagesCountInSubCategory( + category, + subcategory +); +const itemsPerPage = 20; +const totalPages = Math.ceil(totalManPagesCount / itemsPerPage); + +// Redirect if page is out of range or invalid (optional but good practice) +if (currentPage < 1 || (totalPages > 0 && currentPage > totalPages)) { + return Astro.redirect( + `/freedevtools/man-pages/${category}/${subcategory}/` + ); +} + +// Calculate pagination +const offset = (currentPage - 1) * itemsPerPage; + +// Get ONLY the man pages for current page from database (efficient!) +const currentPageManPages = await getManPagesList( + category, + subcategory, + itemsPerPage, + offset +); + +const categoryTitle = formatName(category); +const subcategoryTitle = formatName(subcategory); + +const pageTitle = `${subcategoryTitle} Man Pages - Page ${currentPage}`; +const pageDescription = `Browse ${subcategoryTitle} manual pages - Page ${currentPage} of ${totalPages}. Find detailed documentation and system references.`; + +const breadcrumbItems = [ + { label: 'Free DevTools', href: '/freedevtools/' }, + { label: 'Man Pages', href: '/freedevtools/man-pages/' }, + { label: categoryTitle, href: `/freedevtools/man-pages/${category}/` }, + { label: subcategoryTitle, href: `/freedevtools/man-pages/${category}/${subcategory}/` }, + { label: `Page ${currentPage}` }, +]; +--- + + + +
+ +
+ + + +
+
+ Showing {currentPageManPages.length} of {totalManPagesCount} man pages (Page {currentPage} of {totalPages}) +
+
+ + +
+
+ {currentPageManPages.map((page) => ( + +

+ {page.title.length > 60 + ? page.title.slice(0, 57) + '...' + : page.title} +

+ + ))} +
+
+ + + + + +
+
+ + View {categoryTitle} Category + + + ← Back to {subcategoryTitle} + + +
+
+ + diff --git a/frontend/src/pages/man-pages/[category]/[subcategory]/index.astro b/frontend/src/pages/man-pages/[category]/[subcategory]/index.astro index d27657d4bd..ae51234639 100644 --- a/frontend/src/pages/man-pages/[category]/[subcategory]/index.astro +++ b/frontend/src/pages/man-pages/[category]/[subcategory]/index.astro @@ -1,17 +1,6 @@ --- -import AdBanner from '@/components/banner/AdBanner.astro'; -import CreditsButton from '@/components/buttons/CreditsButton'; -import Pagination from '@/components/PaginationComponent.astro'; -import ToolContainer from '@/components/tool/ToolContainer'; -import ToolHead from '@/components/tool/ToolHead'; -import BaseLayout from '@/layouts/BaseLayout.astro'; -import { - getManPagesBySubcategoryPaginated, - getManPagesCountBySubcategory, - getSubCategoriesByMainCategoryPaginated, - getSubCategoriesCountByMainCategory, - getTotalManPagesCountByMainCategory, -} from 'db/man_pages/man-pages-utils'; +import CategoryPagination from './_CategoryPagination.astro'; +import SubCategory from './_SubCategory.astro'; export const prerender = false; @@ -21,270 +10,13 @@ if (!category || !subcategory) { return Astro.redirect('/freedevtools/man-pages/'); } -// Check if subcategory is numeric (this means it's actually a PAGE number for the CATEGORY) -// e.g. /man-pages/linux/2/ -> category="linux", subcategory="2" (page 2 of linux category) -const isNumericPage = /^\d+$/.test(subcategory); - -// --- VARIABLES FOR BOTH VIEWS --- -let pageTitle = ''; -let pageDescription = ''; -let breadcrumbItems: any[] = []; -let currentPage = 1; -let totalPages = 1; -let items: any[] = []; -let totalCount = 0; -let isCategoryPagination = false; - -// Variables specific to category pagination -let totalManPagesInCategory = 0; - -if (isNumericPage) { - // --- CATEGORY PAGINATION LOGIC (formerly [page].astro) --- - isCategoryPagination = true; - currentPage = parseInt(subcategory, 10); - - // Get subcategories count for this category to determine pagination - const totalSubcategoriesCount = await getSubCategoriesCountByMainCategory(category); - totalCount = totalSubcategoriesCount; - const itemsPerPage = 12; - totalPages = Math.ceil(totalSubcategoriesCount / itemsPerPage); - - // Redirect if page is out of range or invalid - if (currentPage < 1 || (totalPages > 0 && currentPage > totalPages)) { - return Astro.redirect(`/freedevtools/man-pages/${category}/`); - } - - // Calculate pagination - const offset = (currentPage - 1) * itemsPerPage; - - // Get ONLY the subcategories for current page from database (efficient!) - items = await getSubCategoriesByMainCategoryPaginated( - category, - itemsPerPage, - offset - ); - totalManPagesInCategory = await getTotalManPagesCountByMainCategory(category); - - const categoryTitle = - category.replace('-', ' ').charAt(0).toUpperCase() + - category.replace('-', ' ').slice(1); - - pageTitle = `${categoryTitle} Man Pages - Page ${currentPage}`; - pageDescription = `Browse ${categoryTitle} manual page subcategories - Page ${currentPage} of ${totalPages}. Find detailed documentation and system references.`; - - breadcrumbItems = [ - { label: 'Free DevTools', href: '/freedevtools/' }, - { label: 'Man Pages', href: '/freedevtools/man-pages/' }, - { label: categoryTitle, href: `/freedevtools/man-pages/${category}/` }, - { label: `Page ${currentPage}` }, - ]; -} else { - // --- SUBCATEGORY INDEX LOGIC (formerly [subcategory]/index.astro) --- - isCategoryPagination = false; - - // Efficient pagination for page 1 - const itemsPerPage = 20; - currentPage = 1; - const offset = (currentPage - 1) * itemsPerPage; // 0 for page 1 - - // Get ONLY the man pages for page 1 from database (efficient!) - items = await getManPagesBySubcategoryPaginated( - category, - subcategory, - itemsPerPage, - offset - ); - const totalManPagesCount = await getManPagesCountBySubcategory( - category, - subcategory - ); - totalCount = totalManPagesCount; - totalPages = Math.ceil(totalManPagesCount / itemsPerPage); - const categoryTitle = - category.replace('-', ' ').charAt(0).toUpperCase() + - category.replace('-', ' ').slice(1); - const subcategoryTitle = - subcategory.replace('-', ' ').charAt(0).toUpperCase() + - subcategory.replace('-', ' ').slice(1); - - pageTitle = `${subcategoryTitle} Man Pages`; - pageDescription = `Browse ${subcategoryTitle} manual pages in ${categoryTitle}. Find detailed documentation and system references.`; - - breadcrumbItems = [ - { label: 'Free DevTools', href: '/freedevtools/' }, - { label: 'Man Pages', href: '/freedevtools/man-pages/' }, - { label: categoryTitle, href: `/freedevtools/man-pages/${category}/` }, - { label: subcategoryTitle }, - ]; -} - -const categoryTitle = - category.replace('-', ' ').charAt(0).toUpperCase() + - category.replace('-', ' ').slice(1); -const subcategoryTitle = - subcategory.replace('-', ' ').charAt(0).toUpperCase() + - subcategory.replace('-', ' ').slice(1); +const isNumericPage = /^\d+$/.test(subcategory); --- - - - -
- -
- - - -
-
-
-
- { - totalCount >= 1000 - ? Math.round(totalCount / 1000) + 'k' - : totalCount - } -
-
- { - isCategoryPagination - ? 'Subcategories' - : `Man Pages in ${subcategoryTitle}` - } -
-
- { - isCategoryPagination && ( -
-
- {totalManPagesInCategory >= 1000 - ? Math.round(totalManPagesInCategory / 1000) + 'k' - : totalManPagesInCategory} -
-
- Man Pages -
-
- ) - } -
-
- - -
-
- Showing {items.length} of {totalCount} - {isCategoryPagination ? 'subcategories' : 'man pages'} (Page { - currentPage - } of {totalPages}) -
-
- - -
-
- { - items.map((item) => ( - - {isCategoryPagination ? ( - <> -
-

- {item.name.replace('-', ' ')} -

- - {item.count} pages - -
- {item.description && ( -

- {item.description} -

- )} - - ) : ( -

- {item.title.length > 60 - ? item.title.slice(0, 57) + '...' - : item.title} -

- )} - - )) - } -
-
- - - { - totalCount === 0 && ( -
-

- No {isCategoryPagination ? 'subcategories' : 'man pages'} found. -

-
- ) - } - - - - - -
-
- - View {categoryTitle} Category - - -
-
- - + +{isNumericPage ? ( + +) : ( + // If subcategory is not numeric, it's a subcategory eg: /man-pages/user-commands/ + +)} diff --git a/frontend/src/pages/man-pages/[category]/index.astro b/frontend/src/pages/man-pages/[category]/index.astro index 60c99f9557..0d0e9d3327 100644 --- a/frontend/src/pages/man-pages/[category]/index.astro +++ b/frontend/src/pages/man-pages/[category]/index.astro @@ -7,9 +7,9 @@ import ToolHead from '@/components/tool/ToolHead'; import BaseLayout from '@/layouts/BaseLayout.astro'; import { getSubCategoriesByMainCategoryPaginated, - getSubCategoriesCountByMainCategory, - getTotalManPagesCountByMainCategory, + getTotalSubCategoriesManPagesCount, } from 'db/man_pages/man-pages-utils'; +import { formatName } from '@/lib/utils'; export const prerender = false; @@ -24,23 +24,21 @@ const itemsPerPage = 12; const currentPage = 1; const offset = (currentPage - 1) * itemsPerPage; // 0 for page 1 -// Get ONLY the subcategories for page 1 from database (efficient!) -const currentPageSubcategories = await getSubCategoriesByMainCategoryPaginated( + +const subcategories = await getSubCategoriesByMainCategoryPaginated( category, itemsPerPage, offset ); -const totalSubcategoriesCount = await getSubCategoriesCountByMainCategory(category); -const totalManPagesInCategory = await getTotalManPagesCountByMainCategory(category); -const totalPages = Math.ceil(totalSubcategoriesCount / itemsPerPage); +const {man_pages_count, sub_category_count} = await getTotalSubCategoriesManPagesCount(category); +const totalPages = Math.ceil(sub_category_count / itemsPerPage); const breadcrumbItems = [ { label: 'Free DevTools', href: '/freedevtools/' }, { label: 'Man Pages', href: '/freedevtools/man-pages/' }, { label: - category.replace('-', ' ').charAt(0).toUpperCase() + - category.replace('-', ' ').slice(1), + formatName(category), }, ]; @@ -77,9 +75,9 @@ const categoryTitle =
{ - totalSubcategoriesCount >= 1000 - ? Math.round(totalSubcategoriesCount / 1000) + 'k' - : totalSubcategoriesCount + sub_category_count >= 1000 + ? Math.round(sub_category_count / 1000) + 'k' + : sub_category_count }
@@ -89,9 +87,9 @@ const categoryTitle =
{ - totalManPagesInCategory >= 1000 - ? Math.round(totalManPagesInCategory / 1000) + 'k' - : totalManPagesInCategory + man_pages_count >= 1000 + ? Math.round(man_pages_count / 1000) + 'k' + : man_pages_count }
Man Pages
@@ -105,7 +103,7 @@ const categoryTitle = class="bg-slate-50 dark:bg-slate-800 rounded-lg p-4 mb-6 mt-8" >
- Showing {currentPageSubcategories.length} of {totalSubcategoriesCount} subcategories + Showing {subcategories.length} of {sub_category_count} subcategories (Page {currentPage} of {totalPages})
@@ -114,7 +112,7 @@ const categoryTitle =
{ - currentPageSubcategories.map((subcategory) => ( + subcategories.map((subcategory) => ( (

- {category.category.replace('-', ' ')} + {category.name.replace('-', ' ')}

{category.count >= 1000 diff --git a/frontend/src/pages/man-pages/sitemap-[index].xml.ts b/frontend/src/pages/man-pages/sitemap-[index].xml.ts index 5973060a7f..936ae0f6a3 100644 --- a/frontend/src/pages/man-pages/sitemap-[index].xml.ts +++ b/frontend/src/pages/man-pages/sitemap-[index].xml.ts @@ -1,11 +1,10 @@ -// src/pages/man-pages/sitemap-[index].xml.ts -import { generateCommandStaticPaths } from '../../../db/man_pages/man-pages-utils'; +import { getAllManPagesPaginated } from 'db/man_pages/man-pages-utils'; import type { APIRoute } from 'astro'; const MAX_URLS = 5000; -const PRODUCTION_SITE = 'https://hexmos.com/freedevtools'; -// Escape XML special characters +export const prerender = false; + function escapeXml(unsafe: string): string { return unsafe .replace(/&/g, '&') @@ -15,94 +14,41 @@ function escapeXml(unsafe: string): string { .replace(/'/g, '''); } -// Loader function for sitemap URLs - extracted to work in both SSG and SSR -async function loadUrls() { - const now = new Date().toISOString(); - - // Get all man pages from database - const manPages = await generateCommandStaticPaths(); - - // Build URLs with placeholder for site - const urls = manPages.map((manPage) => { - const escapedCategory = escapeXml(manPage.params.category); - const escapedSubCategory = escapeXml(manPage.params.subcategory); - const escapedSlug = escapeXml(manPage.params.slug); - return ` - - __SITE__/man-pages/${escapedCategory}/${escapedSubCategory}/${escapedSlug}/ - ${now} - daily - 0.8 - `; - }); - - // Include the main landing page - urls.unshift(` - - __SITE__/man-pages/ - ${now} - daily - 0.9 - `); - - // Add category index pages - const { getManPageCategories, getSubCategories } = await import('../../../db/man_pages/man-pages-utils'); - - const categories = await getManPageCategories(); - categories.forEach(({ category }) => { - const escapedCategory = escapeXml(category); - urls.push(` - - __SITE__/man-pages/${escapedCategory}/ - ${now} - daily - 0.7 - `); - }); - - const subcategories = await getSubCategories(); - subcategories.forEach(({ main_category, name }) => { - const escapedCategory = escapeXml(main_category); - const escapedSubCategory = escapeXml(name); - urls.push(` - - __SITE__/man-pages/${escapedCategory}/${escapedSubCategory}/ - ${now} - daily - 0.6 - `); - }); +export const GET: APIRoute = async ({ params, site }) => { + const index = parseInt(params.index || '1', 10); + if (isNaN(index) || index < 1) { + return new Response('Invalid index', { status: 400 }); + } - return urls; -} + const offset = (index - 1) * MAX_URLS; + const limit = MAX_URLS; -export const prerender = false; + const manPages = await getAllManPagesPaginated(limit, offset); -export const GET: APIRoute = async ({ site, params }) => { - // SSR mode: call loadUrls directly - let urls = await loadUrls(); + if (!manPages || manPages.length === 0) { + return new Response('Not Found', { status: 404 }); + } - // Replace placeholder with actual site - use production site if localhost or undefined - const siteUrl = - site && !String(site).includes('localhost') - ? String(site) - : PRODUCTION_SITE; - urls = urls.map((u) => u.replace(/__SITE__/g, siteUrl)); + const now = new Date().toISOString(); - // Split into chunks - const sitemapChunks: string[][] = []; - for (let i = 0; i < urls.length; i += MAX_URLS) { - sitemapChunks.push(urls.slice(i, i + MAX_URLS)); - } + const siteUrl = site?.toString().replace(/\/$/, '') || 'https://hexmos.com/freedevtools'; - const index = parseInt(params?.index || '1', 10) - 1; - const chunk = sitemapChunks[index]; + const urls = manPages.map((page) => { + const escapedCategory = escapeXml(page.main_category); + const escapedSubCategory = escapeXml(page.sub_category); + const escapedSlug = escapeXml(page.slug); - if (!chunk) return new Response('Not Found', { status: 404 }); + return ` + + ${siteUrl}/man-pages/${escapedCategory}/${escapedSubCategory}/${escapedSlug}/ + ${now} + monthly + `; + }); const xml = ` - ${chunk.join('\n')} + ${urls.join('\n')} `; return new Response(xml, { @@ -112,3 +58,8 @@ export const GET: APIRoute = async ({ site, params }) => { }, }); }; + +export function getStaticPaths() { + return []; +} + diff --git a/frontend/src/pages/man-pages/sitemap.xml.ts b/frontend/src/pages/man-pages/sitemap.xml.ts index c33df975d6..7cc171a447 100644 --- a/frontend/src/pages/man-pages/sitemap.xml.ts +++ b/frontend/src/pages/man-pages/sitemap.xml.ts @@ -1,127 +1,37 @@ -import { generateCommandStaticPaths } from '../../../db/man_pages/man-pages-utils'; +import { getOverview } from 'db/man_pages/man-pages-utils'; import type { APIRoute } from 'astro'; -const PRODUCTION_SITE = 'https://hexmos.com/freedevtools'; - -// Escape XML special characters -function escapeXml(unsafe: string): string { - return unsafe - .replace(/&/g, '&') - .replace(//g, '>') - .replace(/"/g, '"') - .replace(/'/g, '''); -} +const MAX_URLS = 5000; export const prerender = false; export const GET: APIRoute = async ({ site }) => { const now = new Date().toISOString(); - const MAX_URLS = 5000; - - // Use production site if localhost or undefined - const siteUrl = - site && !String(site).includes('localhost') - ? String(site) - : PRODUCTION_SITE; - - // Get all man pages from database - const manPages = await generateCommandStaticPaths(); - - // Map man pages to sitemap URLs - const urls = manPages.map((manPage) => { - const escapedCategory = escapeXml(manPage.params.category); - const escapedSubCategory = escapeXml(manPage.params.subcategory); - const escapedSlug = escapeXml(manPage.params.slug); - return ` - - ${siteUrl}/man-pages/${escapedCategory}/${escapedSubCategory}/${escapedSlug}/ - ${now} - daily - 0.8 - `; - }); - - // Include the main landing page - urls.unshift(` - - ${siteUrl}/man-pages/ - ${now} - daily - 0.9 - `); - // Add category index pages (we need to fetch these if we want them, but for now let's rely on what we have or add a new query) - // Since we don't have a direct "getAllCategories" exposed for sitemap in the worker yet, - // we can either add it or just skip for now if the user didn't ask for full sitemap fidelity, - // BUT the original code had it. - // Let's use the existing utils we have. + const overview = await getOverview(); + const totalCount = overview?.total_count || 0; - // Note: The original code used direct DB access. We should use the worker functions. - // However, we don't have a "getAllCategories" that returns just names for sitemap. - // We have `getManPageCategories` which returns ManPageCategory[]. + const totalSitemaps = Math.ceil(totalCount / MAX_URLS); - const { getManPageCategories, getSubCategories } = await import('../../../db/man_pages/man-pages-utils'); - - const categories = await getManPageCategories(); - categories.forEach(({ category }) => { - const escapedCategory = escapeXml(category); - urls.push(` - - ${siteUrl}/man-pages/${escapedCategory}/ - ${now} - daily - 0.7 - `); - }); - - const subcategories = await getSubCategories(); - subcategories.forEach(({ main_category, name }) => { - const escapedCategory = escapeXml(main_category); - const escapedSubCategory = escapeXml(name); - urls.push(` - - ${siteUrl}/man-pages/${escapedCategory}/${escapedSubCategory}/ - ${now} - daily - 0.6 - `); - }); - - // If total URLs <= MAX_URLS, return the single sitemap - if (urls.length <= MAX_URLS) { - const xml = ` - - ${urls.join('\n')} -`; - - return new Response(xml, { - headers: { - 'Content-Type': 'application/xml', - 'Cache-Control': 'public, max-age=3600', - }, - }); - } - - // Otherwise, split URLs into chunks and return a sitemap index - const sitemapChunks: string[][] = []; - for (let i = 0; i < urls.length; i += MAX_URLS) { - sitemapChunks.push(urls.slice(i, i + MAX_URLS)); + const sitemapUrls: string[] = []; + for (let i = 1; i <= totalSitemaps; i++) { + sitemapUrls.push(`${site}/man-pages/sitemap-${i}.xml`); } const indexXml = ` - - ${sitemapChunks - .map( - (_, i) => ` - - ${siteUrl}/man-pages/sitemap-${i + 1}.xml - ${now} - ` - ) - .join('\n')} -`; - + + + ${sitemapUrls + .map( + (url) => ` + + ${url} + ${now} + ` + ) + .join('\n')} + `; + return new Response(indexXml, { headers: { 'Content-Type': 'application/xml', diff --git a/frontend/src/pages/man-pages_pages/sitemap-[index].xml.ts b/frontend/src/pages/man-pages_pages/sitemap-[index].xml.ts index 3bacd57688..b2560d3c67 100644 --- a/frontend/src/pages/man-pages_pages/sitemap-[index].xml.ts +++ b/frontend/src/pages/man-pages_pages/sitemap-[index].xml.ts @@ -1,147 +1,147 @@ -import { - getManPageCategories, - getSubCategoriesCountByMainCategory, - getSubCategoriesByMainCategory, - getManPagesCountBySubcategory, -} from 'db/man_pages/man-pages-utils'; -import type { APIRoute } from 'astro'; - -const MAX_URLS = 5000; -const PRODUCTION_SITE = 'https://hexmos.com/freedevtools'; - -// Escape XML special characters -function escapeXml(unsafe: string): string { - return unsafe - .replace(/&/g, '&') - .replace(//g, '>') - .replace(/"/g, '"') - .replace(/'/g, '''); -} - -// Loader function for sitemap URLs -async function loadUrls() { - const now = new Date().toISOString(); - const urls: string[] = []; - - // Root man-pages page - urls.push( - ` - __SITE__/man-pages/ - ${now} - daily - 0.9 - ` - ); - - // Get all categories - const categories = await getManPageCategories(); - - // Category pagination (12 items per page) - const categoryItemsPerPage = 12; - for (const { category: main_category } of categories) { - // Get subcategories count for this category - const subcategoryCount = await getSubCategoriesCountByMainCategory(main_category); - const totalCategoryPages = Math.ceil( - subcategoryCount / categoryItemsPerPage - ); - - // Add category index page (page 1 is the same as the category root) - const escapedCategory = escapeXml(main_category); - urls.push( - ` - __SITE__/man-pages/${escapedCategory}/ - ${now} - daily - 0.7 - ` - ); - - // Pagination pages for category (skip page 1 as it's the same as the root) - for (let i = 2; i <= totalCategoryPages; i++) { - urls.push( - ` - __SITE__/man-pages/${escapedCategory}/${i}/ - ${now} - daily - 0.6 - ` - ); - } - - // Get all subcategories for this category - const subcategories = await getSubCategoriesByMainCategory(main_category); - - // Subcategory pagination (20 items per page) - const subcategoryItemsPerPage = 20; - for (const { name: sub_category } of subcategories) { - // Get man pages count for this subcategory - const manPagesCount = await getManPagesCountBySubcategory(main_category, sub_category); - const totalSubcategoryPages = Math.ceil( - manPagesCount / subcategoryItemsPerPage - ); - - // Add subcategory index page (page 1 is the same as the subcategory root) - const escapedSubCategory = escapeXml(sub_category); - urls.push( - ` - __SITE__/man-pages/${escapedCategory}/${escapedSubCategory}/ - ${now} - daily - 0.6 - ` - ); - - // Pagination pages for subcategory (skip page 1 as it's the same as the root) - for (let i = 2; i <= totalSubcategoryPages; i++) { - urls.push( - ` - __SITE__/man-pages/${escapedCategory}/${escapedSubCategory}/${i}/ - ${now} - daily - 0.5 - ` - ); - } - } - } - - return urls; -} - -export const prerender = false; - -export const GET: APIRoute = async ({ site, params }) => { - // SSR mode: call loadUrls directly - let urls = await loadUrls(); - - // Replace placeholder with actual site - use production site if localhost or undefined - const siteUrl = - site && !String(site).includes('localhost') - ? String(site) - : PRODUCTION_SITE; - urls = urls.map((u) => u.replace(/__SITE__/g, siteUrl)); - - // Split into chunks - const sitemapChunks: string[][] = []; - for (let i = 0; i < urls.length; i += MAX_URLS) { - sitemapChunks.push(urls.slice(i, i + MAX_URLS)); - } - - const index = parseInt(params?.index || '1', 10) - 1; - const chunk = sitemapChunks[index]; - - if (!chunk) return new Response('Not Found', { status: 404 }); - - const xml = ` - - ${chunk.join('\n')} -`; - - return new Response(xml, { - headers: { - 'Content-Type': 'application/xml', - 'Cache-Control': 'public, max-age=3600', - }, - }); -}; +// import { +// getManPageCategories, +// getSubCategoriesCountByMainCategory, +// getSubCategoriesByMainCategory, +// getManPagesCountInSubCategory, +// } from 'db/man_pages/man-pages-utils'; +// import type { APIRoute } from 'astro'; + +// const MAX_URLS = 5000; +// const PRODUCTION_SITE = 'https://hexmos.com/freedevtools'; + +// // Escape XML special characters +// function escapeXml(unsafe: string): string { +// return unsafe +// .replace(/&/g, '&') +// .replace(//g, '>') +// .replace(/"/g, '"') +// .replace(/'/g, '''); +// } + +// // Loader function for sitemap URLs +// async function loadUrls() { +// const now = new Date().toISOString(); +// const urls: string[] = []; + +// // Root man-pages page +// urls.push( +// ` +// __SITE__/man-pages/ +// ${now} +// daily +// 0.9 +// ` +// ); + +// // Get all categories +// const categories = await getManPageCategories(); + +// // Category pagination (12 items per page) +// const categoryItemsPerPage = 12; +// for (const { category: main_category } of categories) { +// // Get subcategories count for this category +// const subcategoryCount = await getSubCategoriesCountByMainCategory(main_category); +// const totalCategoryPages = Math.ceil( +// subcategoryCount / categoryItemsPerPage +// ); + +// // Add category index page (page 1 is the same as the category root) +// const escapedCategory = escapeXml(main_category); +// urls.push( +// ` +// __SITE__/man-pages/${escapedCategory}/ +// ${now} +// daily +// 0.7 +// ` +// ); + +// // Pagination pages for category (skip page 1 as it's the same as the root) +// for (let i = 2; i <= totalCategoryPages; i++) { +// urls.push( +// ` +// __SITE__/man-pages/${escapedCategory}/${i}/ +// ${now} +// daily +// 0.6 +// ` +// ); +// } + +// // Get all subcategories for this category +// const subcategories = await getSubCategoriesByMainCategory(main_category); + +// // Subcategory pagination (20 items per page) +// const subcategoryItemsPerPage = 20; +// for (const { name: sub_category } of subcategories) { +// // Get man pages count for this subcategory +// const manPagesCount = await getManPagesCountInSubCategory(main_category, sub_category); +// const totalSubcategoryPages = Math.ceil( +// manPagesCount / subcategoryItemsPerPage +// ); + +// // Add subcategory index page (page 1 is the same as the subcategory root) +// const escapedSubCategory = escapeXml(sub_category); +// urls.push( +// ` +// __SITE__/man-pages/${escapedCategory}/${escapedSubCategory}/ +// ${now} +// daily +// 0.6 +// ` +// ); + +// // Pagination pages for subcategory (skip page 1 as it's the same as the root) +// for (let i = 2; i <= totalSubcategoryPages; i++) { +// urls.push( +// ` +// __SITE__/man-pages/${escapedCategory}/${escapedSubCategory}/${i}/ +// ${now} +// daily +// 0.5 +// ` +// ); +// } +// } +// } + +// return urls; +// } + +// export const prerender = false; + +// export const GET: APIRoute = async ({ site, params }) => { +// // SSR mode: call loadUrls directly +// let urls = await loadUrls(); + +// // Replace placeholder with actual site - use production site if localhost or undefined +// const siteUrl = +// site && !String(site).includes('localhost') +// ? String(site) +// : PRODUCTION_SITE; +// urls = urls.map((u) => u.replace(/__SITE__/g, siteUrl)); + +// // Split into chunks +// const sitemapChunks: string[][] = []; +// for (let i = 0; i < urls.length; i += MAX_URLS) { +// sitemapChunks.push(urls.slice(i, i + MAX_URLS)); +// } + +// const index = parseInt(params?.index || '1', 10) - 1; +// const chunk = sitemapChunks[index]; + +// if (!chunk) return new Response('Not Found', { status: 404 }); + +// const xml = ` +// +// ${chunk.join('\n')} +// `; + +// return new Response(xml, { +// headers: { +// 'Content-Type': 'application/xml', +// 'Cache-Control': 'public, max-age=3600', +// }, +// }); +// }; diff --git a/frontend/src/pages/man-pages_pages/sitemap.xml.ts b/frontend/src/pages/man-pages_pages/sitemap.xml.ts index 2d53077657..462d5787a9 100644 --- a/frontend/src/pages/man-pages_pages/sitemap.xml.ts +++ b/frontend/src/pages/man-pages_pages/sitemap.xml.ts @@ -1,159 +1,159 @@ -import { - getManPageCategories, - getSubCategories, - getSubCategoriesCountByMainCategory, - getManPagesCountBySubcategory, -} from 'db/man_pages/man-pages-utils'; -import type { APIRoute } from 'astro'; - -const PRODUCTION_SITE = 'https://hexmos.com/freedevtools'; -const MAX_URLS = 5000; - -// Escape XML special characters -function escapeXml(unsafe: string): string { - return unsafe - .replace(/&/g, '&') - .replace(//g, '>') - .replace(/"/g, '"') - .replace(/'/g, '''); -} - -export const prerender = false; - -export const GET: APIRoute = async ({ site }) => { - const now = new Date().toISOString(); - - // Use production site if localhost or undefined - const siteUrl = - site && !String(site).includes('localhost') - ? String(site) - : PRODUCTION_SITE; - - const urls: string[] = []; - - // Root man-pages page - urls.push( - ` - ${siteUrl}/man-pages/ - ${now} - daily - 0.9 - ` - ); - - // Get all categories - const categories = await getManPageCategories(); - - // Category pagination (12 items per page) - const categoryItemsPerPage = 12; - for (const { category: main_category } of categories) { - // Get subcategories count for this category - const subcategoryCount = await getSubCategoriesCountByMainCategory(main_category); - const totalCategoryPages = Math.ceil( - subcategoryCount / categoryItemsPerPage - ); - - // Add category index page (page 1 is the same as the category root) - const escapedCategory = escapeXml(main_category); - urls.push( - ` - ${siteUrl}/man-pages/${escapedCategory}/ - ${now} - daily - 0.7 - ` - ); - - // Pagination pages for category (skip page 1 as it's the same as the root) - for (let i = 2; i <= totalCategoryPages; i++) { - urls.push( - ` - ${siteUrl}/man-pages/${escapedCategory}/${i}/ - ${now} - daily - 0.6 - ` - ); - } - - // Get all subcategories for this category - // Note: getSubCategoriesByMainCategory returns all subcategories - const { getSubCategoriesByMainCategory } = await import('db/man_pages/man-pages-utils'); - const subcategories = await getSubCategoriesByMainCategory(main_category); - - // Subcategory pagination (20 items per page) - const subcategoryItemsPerPage = 20; - for (const { name: sub_category } of subcategories) { - // Get man pages count for this subcategory - const manPagesCount = await getManPagesCountBySubcategory(main_category, sub_category); - const totalSubcategoryPages = Math.ceil( - manPagesCount / subcategoryItemsPerPage - ); - - // Add subcategory index page (page 1 is the same as the subcategory root) - const escapedSubCategory = escapeXml(sub_category); - urls.push( - ` - ${siteUrl}/man-pages/${escapedCategory}/${escapedSubCategory}/ - ${now} - daily - 0.6 - ` - ); - - // Pagination pages for subcategory (skip page 1 as it's the same as the root) - for (let i = 2; i <= totalSubcategoryPages; i++) { - urls.push( - ` - ${siteUrl}/man-pages/${escapedCategory}/${escapedSubCategory}/${i}/ - ${now} - daily - 0.5 - ` - ); - } - } - } - - // If total URLs <= MAX_URLS, return the single sitemap - if (urls.length <= MAX_URLS) { - const xml = ` - -${urls.join('\n')} -`; - - return new Response(xml, { - headers: { - 'Content-Type': 'application/xml', - 'Cache-Control': 'public, max-age=3600', - }, - }); - } - - // Otherwise, split URLs into chunks and return a sitemap index - const sitemapChunks: string[][] = []; - for (let i = 0; i < urls.length; i += MAX_URLS) { - sitemapChunks.push(urls.slice(i, i + MAX_URLS)); - } - - const indexXml = ` - - ${sitemapChunks - .map( - (_, i) => ` - - ${siteUrl}/man-pages_pages/sitemap-${i + 1}.xml - ${now} - ` - ) - .join('\n')} -`; - - return new Response(indexXml, { - headers: { - 'Content-Type': 'application/xml', - 'Cache-Control': 'public, max-age=3600', - }, - }); -}; +// import { +// getManPageCategories, +// getSubCategories, +// getSubCategoriesCountByMainCategory, +// getManPagesCountInSubCategory, +// } from 'db/man_pages/man-pages-utils'; +// import type { APIRoute } from 'astro'; + +// const PRODUCTION_SITE = 'https://hexmos.com/freedevtools'; +// const MAX_URLS = 5000; + +// // Escape XML special characters +// function escapeXml(unsafe: string): string { +// return unsafe +// .replace(/&/g, '&') +// .replace(//g, '>') +// .replace(/"/g, '"') +// .replace(/'/g, '''); +// } + +// export const prerender = false; + +// export const GET: APIRoute = async ({ site }) => { +// const now = new Date().toISOString(); + +// // Use production site if localhost or undefined +// const siteUrl = +// site && !String(site).includes('localhost') +// ? String(site) +// : PRODUCTION_SITE; + +// const urls: string[] = []; + +// // Root man-pages page +// urls.push( +// ` +// ${siteUrl}/man-pages/ +// ${now} +// daily +// 0.9 +// ` +// ); + +// // Get all categories +// const categories = await getManPageCategories(); + +// // Category pagination (12 items per page) +// const categoryItemsPerPage = 12; +// for (const { category: main_category } of categories) { +// // Get subcategories count for this category +// const subcategoryCount = await getSubCategoriesCountByMainCategory(main_category); +// const totalCategoryPages = Math.ceil( +// subcategoryCount / categoryItemsPerPage +// ); + +// // Add category index page (page 1 is the same as the category root) +// const escapedCategory = escapeXml(main_category); +// urls.push( +// ` +// ${siteUrl}/man-pages/${escapedCategory}/ +// ${now} +// daily +// 0.7 +// ` +// ); + +// // Pagination pages for category (skip page 1 as it's the same as the root) +// for (let i = 2; i <= totalCategoryPages; i++) { +// urls.push( +// ` +// ${siteUrl}/man-pages/${escapedCategory}/${i}/ +// ${now} +// daily +// 0.6 +// ` +// ); +// } + +// // Get all subcategories for this category +// // Note: getSubCategoriesByMainCategory returns all subcategories +// const { getSubCategoriesByMainCategory } = await import('db/man_pages/man-pages-utils'); +// const subcategories = await getSubCategoriesByMainCategory(main_category); + +// // Subcategory pagination (20 items per page) +// const subcategoryItemsPerPage = 20; +// for (const { name: sub_category } of subcategories) { +// // Get man pages count for this subcategory +// const manPagesCount = await getManPagesCountInSubCategory(main_category, sub_category); +// const totalSubcategoryPages = Math.ceil( +// manPagesCount / subcategoryItemsPerPage +// ); + +// // Add subcategory index page (page 1 is the same as the subcategory root) +// const escapedSubCategory = escapeXml(sub_category); +// urls.push( +// ` +// ${siteUrl}/man-pages/${escapedCategory}/${escapedSubCategory}/ +// ${now} +// daily +// 0.6 +// ` +// ); + +// // Pagination pages for subcategory (skip page 1 as it's the same as the root) +// for (let i = 2; i <= totalSubcategoryPages; i++) { +// urls.push( +// ` +// ${siteUrl}/man-pages/${escapedCategory}/${escapedSubCategory}/${i}/ +// ${now} +// daily +// 0.5 +// ` +// ); +// } +// } +// } + +// // If total URLs <= MAX_URLS, return the single sitemap +// if (urls.length <= MAX_URLS) { +// const xml = ` +// +// ${urls.join('\n')} +// `; + +// return new Response(xml, { +// headers: { +// 'Content-Type': 'application/xml', +// 'Cache-Control': 'public, max-age=3600', +// }, +// }); +// } + +// // Otherwise, split URLs into chunks and return a sitemap index +// const sitemapChunks: string[][] = []; +// for (let i = 0; i < urls.length; i += MAX_URLS) { +// sitemapChunks.push(urls.slice(i, i + MAX_URLS)); +// } + +// const indexXml = ` +// +// ${sitemapChunks +// .map( +// (_, i) => ` +// +// ${siteUrl}/man-pages_pages/sitemap-${i + 1}.xml +// ${now} +// ` +// ) +// .join('\n')} +// `; + +// return new Response(indexXml, { +// headers: { +// 'Content-Type': 'application/xml', +// 'Cache-Control': 'public, max-age=3600', +// }, +// }); +// }; From 1595a330d18f31450cbe4adbc8b82cd2eec3f807 Mon Sep 17 00:00:00 2001 From: lovestaco Date: Sun, 7 Dec 2025 21:39:58 +0530 Subject: [PATCH 216/234] fix: try cahcing few queries data --- frontend/db/svg_icons/svg-icons-utils.ts | 123 +++++++++++++++++++++-- 1 file changed, 116 insertions(+), 7 deletions(-) diff --git a/frontend/db/svg_icons/svg-icons-utils.ts b/frontend/db/svg_icons/svg-icons-utils.ts index 42f599c60b..56f7bb54f7 100644 --- a/frontend/db/svg_icons/svg-icons-utils.ts +++ b/frontend/db/svg_icons/svg-icons-utils.ts @@ -1,13 +1,80 @@ -import { buildIconUrl, hashUrlToKey, hashNameToKey } from '../../src/lib/hash-utils'; +import { buildIconUrl, hashNameToKey, hashUrlToKey } from '../../src/lib/hash-utils'; import type { Cluster, ClusterWithPreviewIcons, Icon } from './svg-icons-schema'; import { query } from './svg-worker-pool'; +// In-memory cache with TTL +interface CacheEntry { + data: T; + expiresAt: number; +} + +const cache = new Map>(); + +// Cache TTLs (in milliseconds) +const CACHE_TTL = { + TOTAL_ICONS: 5 * 60 * 1000, // 5 minutes - rarely changes + TOTAL_CLUSTERS: 5 * 60 * 1000, // 5 minutes - rarely changes + CLUSTERS_WITH_PREVIEW: 2 * 60 * 1000, // 2 minutes - paginated results + ICONS_BY_CLUSTER: 3 * 60 * 1000, // 3 minutes - category pages + CLUSTER_BY_NAME: 10 * 60 * 1000, // 10 minutes - rarely changes + CLUSTERS: 5 * 60 * 1000, // 5 minutes - rarely changes +}; + +function getCacheKey(type: string, params?: any): string { + return params ? `${type}:${JSON.stringify(params)}` : type; +} + +function getCached(key: string): T | null { + const entry = cache.get(key); + if (!entry) return null; + + if (Date.now() > entry.expiresAt) { + cache.delete(key); + return null; + } + + return entry.data as T; +} + +function setCache(key: string, data: T, ttl: number): void { + cache.set(key, { + data, + expiresAt: Date.now() + ttl, + }); +} + +// Clean up expired entries periodically (every 5 minutes) +setInterval(() => { + const now = Date.now(); + for (const [key, entry] of cache.entries()) { + if (now > entry.expiresAt) { + cache.delete(key); + } + } +}, 5 * 60 * 1000); + export async function getTotalIcons(): Promise { - return query.getTotalIcons(); + const cacheKey = getCacheKey('getTotalIcons'); + const cached = getCached(cacheKey); + if (cached !== null) { + return cached; + } + + const result = await query.getTotalIcons(); + setCache(cacheKey, result, CACHE_TTL.TOTAL_ICONS); + return result; } export async function getTotalClusters(): Promise { - return query.getTotalClusters(); + const cacheKey = getCacheKey('getTotalClusters'); + const cached = getCached(cacheKey); + if (cached !== null) { + return cached; + } + + const result = await query.getTotalClusters(); + setCache(cacheKey, result, CACHE_TTL.TOTAL_CLUSTERS); + return result; } export interface IconWithMetadata extends Icon { @@ -33,7 +100,15 @@ export async function getIconsByCluster( cluster: string, categoryName?: string ): Promise { - return query.getIconsByCluster(cluster, categoryName); + const cacheKey = getCacheKey('getIconsByCluster', { cluster, categoryName }); + const cached = getCached(cacheKey); + if (cached !== null) { + return cached; + } + + const result = await query.getIconsByCluster(cluster, categoryName); + setCache(cacheKey, result, CACHE_TTL.ICONS_BY_CLUSTER); + return result; } export async function getClustersWithPreviewIcons( @@ -42,16 +117,50 @@ export async function getClustersWithPreviewIcons( previewIconsPerCluster: number = 6, transform: boolean = false ): Promise { - return query.getClustersWithPreviewIcons(page, itemsPerPage, previewIconsPerCluster, transform); + const cacheKey = getCacheKey('getClustersWithPreviewIcons', { + page, + itemsPerPage, + previewIconsPerCluster, + transform, + }); + const cached = getCached(cacheKey); + if (cached !== null) { + return cached; + } + + const result = await query.getClustersWithPreviewIcons( + page, + itemsPerPage, + previewIconsPerCluster, + transform + ); + setCache(cacheKey, result, CACHE_TTL.CLUSTERS_WITH_PREVIEW); + return result; } export async function getClusterByName(name: string): Promise { const hashName = hashNameToKey(name); - return query.getClusterByName(hashName); + const cacheKey = getCacheKey('getClusterByName', { hashName }); + const cached = getCached(cacheKey); + if (cached !== null) { + return cached; + } + + const result = await query.getClusterByName(hashName); + setCache(cacheKey, result, CACHE_TTL.CLUSTER_BY_NAME); + return result; } export async function getClusters(): Promise { - return query.getClusters(); + const cacheKey = getCacheKey('getClusters'); + const cached = getCached(cacheKey); + if (cached !== null) { + return cached; + } + + const result = await query.getClusters(); + setCache(cacheKey, result, CACHE_TTL.CLUSTERS); + return result; } // Get icon by category (cluster display name) and icon name (without .svg extension) From a0856660add1aff08b976784c5f8b21a94c55045 Mon Sep 17 00:00:00 2001 From: Lince Mathew Date: Sun, 7 Dec 2025 21:41:03 +0530 Subject: [PATCH 217/234] sitemap, schema fixes --- frontend/db/man_pages/man-pages-schema.ts | 52 ------------------- frontend/db/man_pages/man-pages-utils.ts | 2 - .../pages/man-pages/sitemap-[index].xml.ts | 6 ++- frontend/src/pages/man-pages/sitemap.xml.ts | 10 ++-- 4 files changed, 9 insertions(+), 61 deletions(-) diff --git a/frontend/db/man_pages/man-pages-schema.ts b/frontend/db/man_pages/man-pages-schema.ts index f72aeed60d..a2926e5ba4 100644 --- a/frontend/db/man_pages/man-pages-schema.ts +++ b/frontend/db/man_pages/man-pages-schema.ts @@ -37,59 +37,7 @@ export interface ManPageContent { [key: string]: string | undefined; // Allow any other section } -export interface Category { - name: string; - count: number; - description: string; - keywords: string[]; // JSON array - path: string; -} - -export interface SubCategory { - hash_id: bigint; - main_category: string; - name: string; - count: number; - description: string; - keywords: string[]; // JSON array - path: string; -} - export interface Overview { id: number; total_count: number; -} - -// Raw database row types (before JSON parsing) -export interface RawManPageRow { - hash_id: bigint; - main_category: string; - sub_category: string; - title: string; - slug: string; - filename: string; - content: string; // JSON string before parsing -} - -export interface RawCategoryRow { - name: string; - count: number; - description: string; - keywords: string; // JSON string before parsing - path: string; -} - -export interface RawSubCategoryRow { - hash_id: bigint; - main_category: string; - name: string; - count: number; - description: string; - keywords: string; // JSON string before parsing - path: string; -} - -export interface RawOverviewRow { - id: number; - total_count: number; } \ No newline at end of file diff --git a/frontend/db/man_pages/man-pages-utils.ts b/frontend/db/man_pages/man-pages-utils.ts index 079ebf128e..ee79fb41e7 100644 --- a/frontend/db/man_pages/man-pages-utils.ts +++ b/frontend/db/man_pages/man-pages-utils.ts @@ -78,7 +78,5 @@ export async function getAllManPagesPaginated( // Re-export types for convenience export type { - Category, ManPage, - SubCategory, } from './man-pages-schema'; diff --git a/frontend/src/pages/man-pages/sitemap-[index].xml.ts b/frontend/src/pages/man-pages/sitemap-[index].xml.ts index 936ae0f6a3..18b2bd99e6 100644 --- a/frontend/src/pages/man-pages/sitemap-[index].xml.ts +++ b/frontend/src/pages/man-pages/sitemap-[index].xml.ts @@ -43,13 +43,15 @@ export const GET: APIRoute = async ({ params, site }) => { ${siteUrl}/man-pages/${escapedCategory}/${escapedSubCategory}/${escapedSlug}/ ${now} monthly + 0.8 `; }); const xml = ` - + + ${urls.join('\n')} -`; + `; return new Response(xml, { headers: { diff --git a/frontend/src/pages/man-pages/sitemap.xml.ts b/frontend/src/pages/man-pages/sitemap.xml.ts index 7cc171a447..28a83a3a4a 100644 --- a/frontend/src/pages/man-pages/sitemap.xml.ts +++ b/frontend/src/pages/man-pages/sitemap.xml.ts @@ -22,16 +22,16 @@ export const GET: APIRoute = async ({ site }) => { ${sitemapUrls - .map( - (url) => ` + .map( + (url) => ` ${url} ${now} ` - ) - .join('\n')} + ) + .join('\n')} `; - + return new Response(indexXml, { headers: { 'Content-Type': 'application/xml', From 9d4f868b1470f2ef519ce5dedc30d097408127f8 Mon Sep 17 00:00:00 2001 From: lovestaco Date: Sun, 7 Dec 2025 21:52:29 +0530 Subject: [PATCH 218/234] fix: dont refresh cache --- frontend/db/svg_icons/svg-icons-utils.ts | 49 +++++++++++++----------- 1 file changed, 27 insertions(+), 22 deletions(-) diff --git a/frontend/db/svg_icons/svg-icons-utils.ts b/frontend/db/svg_icons/svg-icons-utils.ts index 56f7bb54f7..9ee1813971 100644 --- a/frontend/db/svg_icons/svg-icons-utils.ts +++ b/frontend/db/svg_icons/svg-icons-utils.ts @@ -1,5 +1,13 @@ -import { buildIconUrl, hashNameToKey, hashUrlToKey } from '../../src/lib/hash-utils'; -import type { Cluster, ClusterWithPreviewIcons, Icon } from './svg-icons-schema'; +import { + buildIconUrl, + hashNameToKey, + hashUrlToKey, +} from '../../src/lib/hash-utils'; +import type { + Cluster, + ClusterWithPreviewIcons, + Icon, +} from './svg-icons-schema'; import { query } from './svg-worker-pool'; // In-memory cache with TTL @@ -27,12 +35,12 @@ function getCacheKey(type: string, params?: any): string { function getCached(key: string): T | null { const entry = cache.get(key); if (!entry) return null; - + if (Date.now() > entry.expiresAt) { cache.delete(key); return null; } - + return entry.data as T; } @@ -43,23 +51,13 @@ function setCache(key: string, data: T, ttl: number): void { }); } -// Clean up expired entries periodically (every 5 minutes) -setInterval(() => { - const now = Date.now(); - for (const [key, entry] of cache.entries()) { - if (now > entry.expiresAt) { - cache.delete(key); - } - } -}, 5 * 60 * 1000); - export async function getTotalIcons(): Promise { const cacheKey = getCacheKey('getTotalIcons'); const cached = getCached(cacheKey); if (cached !== null) { return cached; } - + const result = await query.getTotalIcons(); setCache(cacheKey, result, CACHE_TTL.TOTAL_ICONS); return result; @@ -71,7 +69,7 @@ export async function getTotalClusters(): Promise { if (cached !== null) { return cached; } - + const result = await query.getTotalClusters(); setCache(cacheKey, result, CACHE_TTL.TOTAL_CLUSTERS); return result; @@ -93,7 +91,12 @@ export interface ClusterTransformed { url: string; keywords: string[]; features: string[]; - previewIcons: Array<{ id: number; name: string; base64: string; img_alt: string }>; + previewIcons: Array<{ + id: number; + name: string; + base64: string; + img_alt: string; + }>; } export async function getIconsByCluster( @@ -105,7 +108,7 @@ export async function getIconsByCluster( if (cached !== null) { return cached; } - + const result = await query.getIconsByCluster(cluster, categoryName); setCache(cacheKey, result, CACHE_TTL.ICONS_BY_CLUSTER); return result; @@ -123,11 +126,13 @@ export async function getClustersWithPreviewIcons( previewIconsPerCluster, transform, }); - const cached = getCached(cacheKey); + const cached = getCached( + cacheKey + ); if (cached !== null) { return cached; } - + const result = await query.getClustersWithPreviewIcons( page, itemsPerPage, @@ -145,7 +150,7 @@ export async function getClusterByName(name: string): Promise { if (cached !== null) { return cached; } - + const result = await query.getClusterByName(hashName); setCache(cacheKey, result, CACHE_TTL.CLUSTER_BY_NAME); return result; @@ -157,7 +162,7 @@ export async function getClusters(): Promise { if (cached !== null) { return cached; } - + const result = await query.getClusters(); setCache(cacheKey, result, CACHE_TTL.CLUSTERS); return result; From c8f8e1ec02f81a88d86221fc64b5d205b0973c19 Mon Sep 17 00:00:00 2001 From: lovestaco Date: Sun, 7 Dec 2025 21:53:17 +0530 Subject: [PATCH 219/234] fix: rm all pragma options --- frontend/db/svg_icons/svg-worker.ts | 18 +----------------- 1 file changed, 1 insertion(+), 17 deletions(-) diff --git a/frontend/db/svg_icons/svg-worker.ts b/frontend/db/svg_icons/svg-worker.ts index e3cd80914d..4f5b16f803 100644 --- a/frontend/db/svg_icons/svg-worker.ts +++ b/frontend/db/svg_icons/svg-worker.ts @@ -24,23 +24,7 @@ const __dirname = path.dirname(__filename); // Open database connection with aggressive read optimizations const db = new Database(dbPath, { readonly: true }); -// Wrap all PRAGMAs in try-catch to avoid database locking issues with multiple processes -// Even readonly databases can have locking conflicts when multiple processes set PRAGMAs simultaneously -const setPragma = (pragma: string) => { - try { - db.run(pragma); - } catch (e) { - // Ignore PRAGMA errors - they're optimizations, not critical - } -}; - -setPragma('PRAGMA cache_size = -1048576'); // 1GB cache per connection -setPragma('PRAGMA temp_store = MEMORY'); -setPragma('PRAGMA mmap_size = 1073741824'); // 1GB memory-mapped I/O -setPragma('PRAGMA journal_mode = WAL'); // WAL mode for better concurrent read performance -setPragma('PRAGMA query_only = ON'); // Read-only mode -setPragma('PRAGMA page_size = 8192'); // Optimal page size -setPragma('PRAGMA synchronous = OFF'); // Disable synchronous writes for better performance + // Disable synchronous writes for better performance const statements = { totalIcons: db.prepare('SELECT total_count FROM overview WHERE id = 1'), From 33fed5f463de24a6f09bae84df2fb6536470f362 Mon Sep 17 00:00:00 2001 From: lovestaco Date: Sun, 7 Dec 2025 23:01:07 +0530 Subject: [PATCH 220/234] feat: add astro debugger --- frontend/.vscode/launch.json | 50 +++++++++++++++++++++++++++++++++--- 1 file changed, 47 insertions(+), 3 deletions(-) diff --git a/frontend/.vscode/launch.json b/frontend/.vscode/launch.json index d642209762..4cc0a05891 100644 --- a/frontend/.vscode/launch.json +++ b/frontend/.vscode/launch.json @@ -2,10 +2,54 @@ "version": "0.2.0", "configurations": [ { - "command": "./node_modules/.bin/astro dev", - "name": "Development server", + "type": "bun", "request": "launch", - "type": "node-terminal" + "name": "Debug Astro (Bun)", + + // Run the Astro dev server using Bun + "program": "./node_modules/astro/astro.js", + + // The arguments to pass to the program, if any. + "args": ["dev"], + + // The working directory of the program. + "cwd": "${workspaceFolder}", + + // The environment variables to pass to the program. + "env": {}, + + // If the environment variables should not be inherited from the parent process. + "strictEnv": false, + + // If the program should be run in watch mode. + // This is equivalent to passing `--watch` to the `bun` executable. + // You can also set this to "hot" to enable hot reloading using `--hot`. + "watchMode": false, + + // If the debugger should stop on the first line of the program. + "stopOnEntry": false, + + // If the debugger should be disabled. (for example, breakpoints will not be hit) + "noDebug": false, + + // The path to the `bun` executable, defaults to your `PATH` environment variable. + "runtime": "bun", + + // The arguments to pass to the `bun` executable, if any. + // Unlike `args`, these are passed to the executable itself, not the program. + "runtimeArgs": ["--inspect=ws://localhost:6499/fdt"] + }, + { + "type": "bun", + "request": "attach", + "name": "Attach to Bun", + + // The URL of the WebSocket inspector to attach to. + // This value can be retrieved by using `bun --inspect`. + "url": "ws://localhost:6499/fdt", + // Optional path mapping for remote debugging + "localRoot": "${workspaceFolder}", + "remoteRoot": "/app" } ] } From 54fa49fd7dff194e8e09e0ff542ecc952bc04924 Mon Sep 17 00:00:00 2001 From: Ganesh Kumar Date: Mon, 8 Dec 2025 18:39:47 +0530 Subject: [PATCH 221/234] feat: Introduce `cluster_hash` for TLDR pages, optimizing queries and refactoring sitemap generation. --- frontend/db/tldrs/README.md | 8 ++++---- frontend/scripts/tldr/README.md | 1 + 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/frontend/db/tldrs/README.md b/frontend/db/tldrs/README.md index fff9c6616f..b0735026d1 100644 --- a/frontend/db/tldrs/README.md +++ b/frontend/db/tldrs/README.md @@ -42,11 +42,11 @@ The queries are optimized for specific frontend use cases: * **Query**: ```sql SELECT url, metadata FROM pages - WHERE url LIKE ? + WHERE cluster_hash = ? ORDER BY url LIMIT ? OFFSET ? ``` -* **Logic**: Uses SQL `LIMIT` and `OFFSET` to fetch only the requested chunk of records, avoiding loading thousands of commands into memory. +* **Logic**: Uses `cluster_hash` for efficient filtering (instead of `LIKE`) and SQL `LIMIT`/`OFFSET` to fetch only the requested chunk of records. ### 3. Fetching a Single Page (`getTldrPage`) * **Goal**: Get the content for a specific command. @@ -54,8 +54,8 @@ The queries are optimized for specific frontend use cases: * **Returns**: Rendered HTML content and full metadata. ### 4. Sitemap Generation -* **`getTldrSitemapCount`**: Counts total URLs for the sitemap index. -* **`getTldrSitemapUrls`**: Fetches a chunk of 5000 URLs for individual sitemap files using `LIMIT` and `OFFSET`. +* **`getTldrSitemapCount`**: Counts total URLs for the sitemap index (Root + Clusters + Pages). +* **`getTldrSitemapUrls`**: Fetches a chunk of URLs for individual sitemap files using a logic-based approach in the worker (iterating through Root -> Clusters -> Pages) with `LIMIT` and `OFFSET`. ## Data Flow diff --git a/frontend/scripts/tldr/README.md b/frontend/scripts/tldr/README.md index 2d6941b7f6..8f1f2a5a04 100644 --- a/frontend/scripts/tldr/README.md +++ b/frontend/scripts/tldr/README.md @@ -59,6 +59,7 @@ The `tldr_to_db.go` script performs the following steps to transform raw markdow - **`pages`**: Stores individual command pages. - `url_hash` (PK): Integer hash of the URL. - `url`: The full path (e.g., `/freedevtools/tldr/common/tar/`). + - `cluster_hash`: Integer hash of the cluster name (FK). - `html_content`: Rendered HTML from markdown. - `metadata`: JSON string containing title, description, keywords, etc. - **`cluster`**: Stores metadata for command groups (platforms like `common`, `linux`). From e49ce53ca3ddfb00f9c40704f8e6d12d829695ed Mon Sep 17 00:00:00 2001 From: Ganesh Kumar Date: Mon, 8 Dec 2025 20:17:02 +0530 Subject: [PATCH 222/234] feat: Optimize TLDR sitemap and cluster count retrieval via updated overview schema and add SSR utility tests. --- frontend/db/tldrs/tldr-worker.ts | 4 +- frontend/scripts/tldr/test_sitemap.ts | 31 ------ frontend/scripts/tldr/tldr_to_db.go | 13 ++- frontend/test/ssr/tldr/tldr-utils.test.ts | 109 ++++++++++++++++++++++ 4 files changed, 121 insertions(+), 36 deletions(-) delete mode 100644 frontend/scripts/tldr/test_sitemap.ts create mode 100644 frontend/test/ssr/tldr/tldr-utils.test.ts diff --git a/frontend/db/tldrs/tldr-worker.ts b/frontend/db/tldrs/tldr-worker.ts index bee175f9d1..0a5e8a012a 100644 --- a/frontend/db/tldrs/tldr-worker.ts +++ b/frontend/db/tldrs/tldr-worker.ts @@ -66,7 +66,7 @@ const statements = { ), // Optimized sitemap queries - getClusterCount: db.prepare('SELECT COUNT(*) as count FROM cluster'), + getClusterCount: db.prepare('SELECT total_clusters as count FROM overview WHERE id = 1'), getClustersPaginated: db.prepare( 'SELECT name FROM cluster ORDER BY name LIMIT ? OFFSET ?' @@ -78,7 +78,7 @@ const statements = { // Count total URLs for sitemap index (1 for root + clusters + pages) getSitemapCount: db.prepare( - `SELECT (SELECT COUNT(*) FROM cluster) + (SELECT COUNT(*) FROM pages) + 1 as count` + `SELECT total_clusters + total_pages + 1 as count FROM overview WHERE id = 1` ), }; diff --git a/frontend/scripts/tldr/test_sitemap.ts b/frontend/scripts/tldr/test_sitemap.ts deleted file mode 100644 index 2bb5771c4a..0000000000 --- a/frontend/scripts/tldr/test_sitemap.ts +++ /dev/null @@ -1,31 +0,0 @@ - -import path from 'path'; -import { Worker } from 'worker_threads'; - -const workerPath = path.resolve('../../db/tldrs/tldr-worker.ts'); -const dbPath = path.resolve('../../db/all_dbs/tldr-db-v1.db'); - -const worker = new Worker(workerPath, { - workerData: { - dbPath, - workerId: 1, - }, - execArgv: ['--loader', 'ts-node/esm'], // Use ts-node loader if needed, or just run with bun -}); - -worker.on('message', (message) => { - if (message.ready) { - console.log('Worker ready'); - worker.postMessage({ - id: '1', - type: 'getSitemapUrls', - params: { limit: 10, offset: 0 }, - }); - } else if (message.id === '1') { - console.log('Sitemap URLs:', message.result); - worker.terminate(); - } else if (message.error) { - console.error('Error:', message.error); - worker.terminate(); - } -}); diff --git a/frontend/scripts/tldr/tldr_to_db.go b/frontend/scripts/tldr/tldr_to_db.go index db52b2b9d9..a60e491b77 100644 --- a/frontend/scripts/tldr/tldr_to_db.go +++ b/frontend/scripts/tldr/tldr_to_db.go @@ -210,10 +210,15 @@ func ensureSchema(db *sql.DB) error { _, err = db.Exec(` CREATE TABLE IF NOT EXISTS overview ( id INTEGER PRIMARY KEY CHECK(id = 1), - total_count INTEGER NOT NULL + total_count INTEGER NOT NULL, + total_clusters INTEGER NOT NULL DEFAULT 0, + total_pages INTEGER NOT NULL DEFAULT 0 ); `) - return err + if err != nil { + return err + } + return nil } func main() { @@ -340,7 +345,9 @@ func main() { tx.Commit() // 4. Overview - if _, err := db.Exec("INSERT INTO overview (id, total_count) VALUES (1, ?)", len(allPages)); err != nil { + totalClusters := len(pagesByCluster) + totalPages := len(allPages) + if _, err := db.Exec("INSERT INTO overview (id, total_count, total_clusters, total_pages) VALUES (1, ?, ?, ?)", totalPages, totalClusters, totalPages); err != nil { log.Fatal(err) } diff --git a/frontend/test/ssr/tldr/tldr-utils.test.ts b/frontend/test/ssr/tldr/tldr-utils.test.ts new file mode 100644 index 0000000000..8c3ad7d361 --- /dev/null +++ b/frontend/test/ssr/tldr/tldr-utils.test.ts @@ -0,0 +1,109 @@ + +import { afterAll, describe, expect, it } from 'bun:test'; +import { + getAllTldrClusters, + getTldrCluster, + getTldrCommandsByClusterPaginated, + getTldrOverview, + getTldrPage, + getTldrSitemapCount, + getTldrSitemapUrls, +} from '../../../db/tldrs/tldr-utils'; +import { cleanupWorkers } from '../../../db/tldrs/tldr-worker-pool'; + +// Helper to measure execution time +async function measureTime(fn: () => Promise): Promise<{ result: T; duration: number }> { + const start = performance.now(); + const result = await fn(); + const end = performance.now(); + return { result, duration: end - start }; +} + +describe('TLDR SSR Utils', () => { + // Ensure workers are cleaned up after tests + afterAll(async () => { + await cleanupWorkers(); + }); + + it('getTldrOverview should return total count > 0', async () => { + const { result, duration } = await measureTime(() => getTldrOverview()); + console.log(`getTldrOverview took ${duration.toFixed(2)}ms`); + expect(result).toBeGreaterThan(0); + expect(typeof result).toBe('number'); + }); + + it('getAllTldrClusters should return list of clusters', async () => { + const { result, duration } = await measureTime(() => getAllTldrClusters()); + console.log(`getAllTldrClusters took ${duration.toFixed(2)}ms`); + expect(result.length).toBeGreaterThan(0); + + // Check for known clusters + const common = result.find(c => c.name === 'common'); + const linux = result.find(c => c.name === 'linux'); + expect(common).toBeDefined(); + expect(linux).toBeDefined(); + expect(common?.count).toBeGreaterThan(0); + }); + + it('getTldrCluster should return correct metadata for "common"', async () => { + const { result, duration } = await measureTime(() => getTldrCluster('common')); + console.log(`getTldrCluster('common') took ${duration.toFixed(2)}ms`); + expect(result).toBeDefined(); + expect(result?.name).toBe('common'); + expect(result?.count).toBeGreaterThan(0); + expect(result?.preview_commands.length).toBeGreaterThan(0); + }); + + it('getTldrCluster should return null for non-existent cluster', async () => { + const { result, duration } = await measureTime(() => getTldrCluster('non-existent-cluster-123')); + console.log(`getTldrCluster('non-existent') took ${duration.toFixed(2)}ms`); + expect(result).toBeNull(); + }); + + it('getTldrCommandsByClusterPaginated should return commands for "common"', async () => { + const { result, duration } = await measureTime(() => getTldrCommandsByClusterPaginated('common', 1, 10)); + console.log(`getTldrCommandsByClusterPaginated('common', 1, 10) took ${duration.toFixed(2)}ms`); + expect(result.length).toBeGreaterThan(0); + expect(result.length).toBeLessThanOrEqual(10); + + // Check structure + const cmd = result[0]; + expect(cmd.name).toBeDefined(); + expect(cmd.url).toBeDefined(); + expect(cmd.description).toBeDefined(); + }); + + it('getTldrCommandsByClusterPaginated should return empty array for out of range page', async () => { + const { result, duration } = await measureTime(() => getTldrCommandsByClusterPaginated('common', 9999, 10)); + console.log(`getTldrCommandsByClusterPaginated('common', 9999, 10) took ${duration.toFixed(2)}ms`); + expect(result).toEqual([]); + }); + + it('getTldrPage should return content for "common/tar"', async () => { + const { result, duration } = await measureTime(() => getTldrPage('common', 'tar')); + console.log(`getTldrPage('common', 'tar') took ${duration.toFixed(2)}ms`); + expect(result).toBeDefined(); + expect(result?.html_content).toContain('tar'); + expect(result?.metadata).toBeDefined(); + expect(result?.metadata.title).toBeDefined(); + }); + + it('getTldrPage should return null for non-existent page', async () => { + const { result, duration } = await measureTime(() => getTldrPage('common', 'non-existent-command-123')); + console.log(`getTldrPage('common', 'non-existent') took ${duration.toFixed(2)}ms`); + expect(result).toBeNull(); + }); + + it('getTldrSitemapCount should return total count', async () => { + const { result, duration } = await measureTime(() => getTldrSitemapCount()); + console.log(`getTldrSitemapCount took ${duration.toFixed(2)}ms`); + expect(result).toBeGreaterThan(0); + }); + + it('getTldrSitemapUrls should return URLs', async () => { + const { result, duration } = await measureTime(() => getTldrSitemapUrls(10, 0)); + console.log(`getTldrSitemapUrls(10, 0) took ${duration.toFixed(2)}ms`); + expect(result.length).toBe(10); + expect(result[0]).toBe('/freedevtools/tldr/'); // Root should be first + }); +}); From 529c0a285b05217473564cb0e5fb216da1e7fb6e Mon Sep 17 00:00:00 2001 From: Ganesh Kumar Date: Mon, 8 Dec 2025 20:18:20 +0530 Subject: [PATCH 223/234] feat: Document proposed TLDR schema changes and add a `test` script to `package.json`. --- frontend/tldr-schema-change.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 frontend/tldr-schema-change.md diff --git a/frontend/tldr-schema-change.md b/frontend/tldr-schema-change.md new file mode 100644 index 0000000000..02fa0bc38d --- /dev/null +++ b/frontend/tldr-schema-change.md @@ -0,0 +1,18 @@ +# Overivew + +Currenlty i have implemented tldr schema in the db/tldrs/tldr-schema.ts file. +and tldr_to_db.go file to insert tldr data into the db. + +But for any small change in migration we might waste time in future + +folow rules of scheam of db/emojis +this also folow the same rules of sitemap generation. present in src/pages/emojis + +First explain How db is created scripts/emojis + +and explain how pagination and other things are implemented in src/pages/emojis. + +no need to change tldr end pages logic +i.e., freedevtools/tldr/adb/adb this will be same. + +Only fix main_page table and remove sitemap table let it be same as emojis sitemap generation From c035c65091e6d3a89167fbbe8b7341c28170e3f8 Mon Sep 17 00:00:00 2001 From: lovestaco Date: Mon, 8 Dec 2025 20:21:47 +0530 Subject: [PATCH 224/234] fix: working chrome profiler --- frontend/.vscode/launch.json | 42 +++++++++++++++++++++++++----------- 1 file changed, 30 insertions(+), 12 deletions(-) diff --git a/frontend/.vscode/launch.json b/frontend/.vscode/launch.json index 4cc0a05891..2bcdbdcf90 100644 --- a/frontend/.vscode/launch.json +++ b/frontend/.vscode/launch.json @@ -27,29 +27,47 @@ "watchMode": false, // If the debugger should stop on the first line of the program. - "stopOnEntry": false, + "stopOnEntry": true, // If the debugger should be disabled. (for example, breakpoints will not be hit) - "noDebug": false, + // "noDebug": false, // The path to the `bun` executable, defaults to your `PATH` environment variable. "runtime": "bun", // The arguments to pass to the `bun` executable, if any. // Unlike `args`, these are passed to the executable itself, not the program. - "runtimeArgs": ["--inspect=ws://localhost:6499/fdt"] + // "runtimeArgs": ["--inspect=ws://localhost:6499/fdt"] + "runtimeArgs": ["--inspect"] }, { - "type": "bun", + "type": "chrome", + "request": "launch", + "name": "Debug Chrome (Client-side)", + "url": "http://localhost:4321/freedevtools/svg_icons/", + "webRoot": "${workspaceFolder}", + "sourceMaps": true, + "sourceMapPathOverrides": { + "webpack:///./*": "${webRoot}/*", + "webpack:///src/*": "${webRoot}/src/*", + "webpack:///*": "*", + "webpack:///./~/*": "${webRoot}/node_modules/*" + } + }, + { + "type": "chrome", "request": "attach", - "name": "Attach to Bun", - - // The URL of the WebSocket inspector to attach to. - // This value can be retrieved by using `bun --inspect`. - "url": "ws://localhost:6499/fdt", - // Optional path mapping for remote debugging - "localRoot": "${workspaceFolder}", - "remoteRoot": "/app" + "name": "Attach to Chrome", + "port": 9222, + "webRoot": "${workspaceFolder}", + "sourceMaps": true + } + ], + "compounds": [ + { + "name": "Debug Full Stack (Bun + Chrome)", + "configurations": ["Debug Astro (Bun)", "Debug Chrome (Client-side)"], + "stopAll": true } ] } From c301605cff2996898848e7ac752785044aaa7da3 Mon Sep 17 00:00:00 2001 From: lovestaco Date: Mon, 8 Dec 2025 20:35:27 +0530 Subject: [PATCH 225/234] fix: fdt --- frontend/.vscode/launch.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/frontend/.vscode/launch.json b/frontend/.vscode/launch.json index 2bcdbdcf90..0df22bfb83 100644 --- a/frontend/.vscode/launch.json +++ b/frontend/.vscode/launch.json @@ -38,7 +38,7 @@ // The arguments to pass to the `bun` executable, if any. // Unlike `args`, these are passed to the executable itself, not the program. // "runtimeArgs": ["--inspect=ws://localhost:6499/fdt"] - "runtimeArgs": ["--inspect"] + "runtimeArgs": ["--inspect=ws://localhost:6499/fdt"] }, { "type": "chrome", From c7b656e50fd22ff39acf519ae0823a4089a7a313 Mon Sep 17 00:00:00 2001 From: Ganesh Kumar Date: Mon, 8 Dec 2025 20:49:51 +0530 Subject: [PATCH 226/234] refactor: move tldr page title and description from metadata to direct database columns and data structures --- frontend/db/tldrs/tldr-schema.ts | 6 ++- frontend/db/tldrs/tldr-worker.ts | 12 +++-- frontend/package.json | 3 +- frontend/scripts/tldr/tldr_to_db.go | 50 ++++++++++++------- .../src/pages/tldr/[platform]/[slug].astro | 4 +- frontend/test/ssr/tldr/tldr-utils.test.ts | 3 +- 6 files changed, 49 insertions(+), 29 deletions(-) diff --git a/frontend/db/tldrs/tldr-schema.ts b/frontend/db/tldrs/tldr-schema.ts index 746e941cdb..ea73451365 100644 --- a/frontend/db/tldrs/tldr-schema.ts +++ b/frontend/db/tldrs/tldr-schema.ts @@ -1,11 +1,11 @@ export interface PageMetadata { - title: string; - description: string; keywords: string[]; features: string[]; } export interface Page { + title: string; + description: string; html_content: string; metadata: PageMetadata; } @@ -36,6 +36,8 @@ export interface RawClusterRow { export interface RawPageRow { url_hash: number; url: string; + title: string; + description: string; html_content: string; metadata: string; } \ No newline at end of file diff --git a/frontend/db/tldrs/tldr-worker.ts b/frontend/db/tldrs/tldr-worker.ts index 0a5e8a012a..f09849658b 100644 --- a/frontend/db/tldrs/tldr-worker.ts +++ b/frontend/db/tldrs/tldr-worker.ts @@ -49,12 +49,12 @@ const statements = { ), getPage: db.prepare( - `SELECT html_content, metadata FROM pages WHERE url_hash = ?` + `SELECT title, description, html_content, metadata FROM pages WHERE url_hash = ?` ), // New query for paginated commands in a cluster getCommandsByClusterPaginated: db.prepare( - `SELECT url, metadata FROM pages + `SELECT url, description, metadata FROM pages WHERE cluster_hash = ? ORDER BY url LIMIT ? OFFSET ?` @@ -130,9 +130,11 @@ parentPort?.on('message', (message: QueryMessage) => { const { platform, slug } = params; const hashKey = `${platform}/${slug}`; const hash = hashNameToKey(hashKey); - const row = statements.getPage.get(hash) as { html_content: string; metadata: string } | undefined; + const row = statements.getPage.get(hash) as { title: string; description: string; html_content: string; metadata: string } | undefined; if (row) { result = { + title: row.title, + description: row.description, html_content: row.html_content, metadata: JSON.parse(row.metadata) }; @@ -149,7 +151,7 @@ parentPort?.on('message', (message: QueryMessage) => { clusterHash, limit, offset - ) as { url: string; metadata: string }[]; + ) as { url: string; description: string; metadata: string }[]; result = rows.map(row => { const meta = JSON.parse(row.metadata); @@ -158,7 +160,7 @@ parentPort?.on('message', (message: QueryMessage) => { return { name, url: row.url, - description: meta.description, + description: row.description, // Use column directly features: meta.features }; }); diff --git a/frontend/package.json b/frontend/package.json index 2adfbb0f67..c191a47c73 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -24,7 +24,8 @@ "pagespeed": "bun run scripts/pageSpeed.cjs", "pagespeed:all": "bun run scripts/pageSpeed.cjs --all", "pagespeed:minimal": "bun run scripts/pageSpeed.cjs --minimal", - "pagespeed:all:minimal": "bun run scripts/pageSpeed.cjs --all --minimal" + "pagespeed:all:minimal": "bun run scripts/pageSpeed.cjs --all --minimal", + "test": "bun test" }, "dependencies": { "@astrojs/node": "^9.5.0", diff --git a/frontend/scripts/tldr/tldr_to_db.go b/frontend/scripts/tldr/tldr_to_db.go index a60e491b77..46090d1d8b 100644 --- a/frontend/scripts/tldr/tldr_to_db.go +++ b/frontend/scripts/tldr/tldr_to_db.go @@ -33,8 +33,10 @@ const ( type Page struct { UrlHash int64 Url string // Kept for reference + Title string + Description string HtmlContent string - Metadata string // JSON + Metadata string // JSON (keywords, features) } type MainPage struct { @@ -60,10 +62,8 @@ type ProcessedPage struct { } type PageMetadata struct { - Title string `json:"title"` - Description string `json:"description"` - Keywords []string `json:"keywords"` - Features []string `json:"features"` + Keywords []string `json:"keywords"` + Features []string `json:"features"` } type Frontmatter struct { @@ -179,23 +179,38 @@ func parseTldrFile(path string) (*ProcessedPage, error) { // --- Database --- func ensureSchema(db *sql.DB) error { + // Drop tables if they exist to force schema update + if _, err := db.Exec("DROP TABLE IF EXISTS pages"); err != nil { + return err + } + if _, err := db.Exec("DROP TABLE IF EXISTS cluster"); err != nil { + return err + } + if _, err := db.Exec("DROP TABLE IF EXISTS overview"); err != nil { + return err + } + // Pages table - Simplified + fmt.Println("Creating pages table...") _, err := db.Exec(` - CREATE TABLE IF NOT EXISTS pages ( + CREATE TABLE pages ( url_hash INTEGER PRIMARY KEY, url TEXT NOT NULL, -- Kept for reference cluster_hash INTEGER NOT NULL, + title TEXT DEFAULT '', + description TEXT DEFAULT '', html_content TEXT DEFAULT '', - metadata TEXT DEFAULT '{}' -- JSON + metadata TEXT DEFAULT '{}' -- JSON (keywords, features) ) WITHOUT ROWID; `) if err != nil { - return err + return fmt.Errorf("failed to create pages table: %w", err) } // Cluster table - Cluster Metadata (similar to emojis category) + fmt.Println("Creating cluster table...") _, err = db.Exec(` - CREATE TABLE IF NOT EXISTS cluster ( + CREATE TABLE cluster ( hash INTEGER PRIMARY KEY, name TEXT NOT NULL, count INTEGER NOT NULL, @@ -203,12 +218,13 @@ func ensureSchema(db *sql.DB) error { ) WITHOUT ROWID; `) if err != nil { - return err + return fmt.Errorf("failed to create cluster table: %w", err) } // Overview table + fmt.Println("Creating overview table...") _, err = db.Exec(` - CREATE TABLE IF NOT EXISTS overview ( + CREATE TABLE overview ( id INTEGER PRIMARY KEY CHECK(id = 1), total_count INTEGER NOT NULL, total_clusters INTEGER NOT NULL DEFAULT 0, @@ -216,7 +232,7 @@ func ensureSchema(db *sql.DB) error { ); `) if err != nil { - return err + return fmt.Errorf("failed to create overview table: %w", err) } return nil } @@ -270,7 +286,7 @@ func main() { if err != nil { log.Fatal(err) } - stmt, err := tx.Prepare("INSERT INTO pages (url_hash, url, cluster_hash, html_content, metadata) VALUES (?, ?, ?, ?, ?)") + stmt, err := tx.Prepare("INSERT INTO pages (url_hash, url, cluster_hash, title, description, html_content, metadata) VALUES (?, ?, ?, ?, ?, ?, ?)") if err != nil { log.Fatal(err) } @@ -278,17 +294,15 @@ func main() { for _, p := range allPages { meta := PageMetadata{ - Title: p.Title, - Description: p.Description, - Keywords: p.Keywords, - Features: p.Features, + Keywords: p.Keywords, + Features: p.Features, } metaJson, _ := json.Marshal(meta) // Calculate cluster hash clusterHash := get8Bytes(hashString(p.Cluster)) - _, err = stmt.Exec(p.UrlHash, p.Url, clusterHash, p.HtmlContent, string(metaJson)) + _, err = stmt.Exec(p.UrlHash, p.Url, clusterHash, p.Title, p.Description, p.HtmlContent, string(metaJson)) if err != nil { log.Printf("Error inserting page %s: %v", p.Name, err) } diff --git a/frontend/src/pages/tldr/[platform]/[slug].astro b/frontend/src/pages/tldr/[platform]/[slug].astro index 9e4dbc681c..c846719187 100644 --- a/frontend/src/pages/tldr/[platform]/[slug].astro +++ b/frontend/src/pages/tldr/[platform]/[slug].astro @@ -110,8 +110,8 @@ if (pageNumber !== null) { htmlContent = page.html_content || ''; - const title = page.metadata.title || command; - const description = page.metadata.description || `Documentation for ${command} command`; + const title = page.title || command; + const description = page.description || `Documentation for ${command} command`; const breadcrumbItems = [ { label: 'Free DevTools', href: '/freedevtools/' }, diff --git a/frontend/test/ssr/tldr/tldr-utils.test.ts b/frontend/test/ssr/tldr/tldr-utils.test.ts index 8c3ad7d361..1c033f7bb2 100644 --- a/frontend/test/ssr/tldr/tldr-utils.test.ts +++ b/frontend/test/ssr/tldr/tldr-utils.test.ts @@ -84,8 +84,9 @@ describe('TLDR SSR Utils', () => { console.log(`getTldrPage('common', 'tar') took ${duration.toFixed(2)}ms`); expect(result).toBeDefined(); expect(result?.html_content).toContain('tar'); + expect(result?.title).toBeDefined(); + expect(result?.description).toBeDefined(); expect(result?.metadata).toBeDefined(); - expect(result?.metadata.title).toBeDefined(); }); it('getTldrPage should return null for non-existent page', async () => { From 7aa1e147a3133b14663a55cffe2f529288a4d745 Mon Sep 17 00:00:00 2001 From: Ganesh Kumar Date: Mon, 8 Dec 2025 20:58:43 +0530 Subject: [PATCH 227/234] feat: Refactor tldr page schema to use dedicated title and description columns, update related queries, and enhance sitemap count with new overview statistics. --- frontend/db/tldrs/README.md | 6 +++--- frontend/db/tldrs/tldr-worker.ts | 2 +- frontend/scripts/tldr/README.md | 6 +++++- 3 files changed, 9 insertions(+), 5 deletions(-) diff --git a/frontend/db/tldrs/README.md b/frontend/db/tldrs/README.md index b0735026d1..fe52c96527 100644 --- a/frontend/db/tldrs/README.md +++ b/frontend/db/tldrs/README.md @@ -41,7 +41,7 @@ The queries are optimized for specific frontend use cases: * **Goal**: Get a specific page of commands for a platform. * **Query**: ```sql - SELECT url, metadata FROM pages + SELECT url, description, metadata FROM pages WHERE cluster_hash = ? ORDER BY url LIMIT ? OFFSET ? @@ -51,10 +51,10 @@ The queries are optimized for specific frontend use cases: ### 3. Fetching a Single Page (`getTldrPage`) * **Goal**: Get the content for a specific command. * **Query**: Selects from `pages` by `url_hash`. -* **Returns**: Rendered HTML content and full metadata. +* **Returns**: Rendered HTML content, title, description, and metadata. ### 4. Sitemap Generation -* **`getTldrSitemapCount`**: Counts total URLs for the sitemap index (Root + Clusters + Pages). +* **`getTldrSitemapCount`**: Counts total URLs for the sitemap index (Root + Clusters + Pages) using pre-calculated counts from the `overview` table. * **`getTldrSitemapUrls`**: Fetches a chunk of URLs for individual sitemap files using a logic-based approach in the worker (iterating through Root -> Clusters -> Pages) with `LIMIT` and `OFFSET`. ## Data Flow diff --git a/frontend/db/tldrs/tldr-worker.ts b/frontend/db/tldrs/tldr-worker.ts index f09849658b..7c9950ac2f 100644 --- a/frontend/db/tldrs/tldr-worker.ts +++ b/frontend/db/tldrs/tldr-worker.ts @@ -54,7 +54,7 @@ const statements = { // New query for paginated commands in a cluster getCommandsByClusterPaginated: db.prepare( - `SELECT url, description, metadata FROM pages + `SELECT url, description FROM pages WHERE cluster_hash = ? ORDER BY url LIMIT ? OFFSET ?` diff --git a/frontend/scripts/tldr/README.md b/frontend/scripts/tldr/README.md index 8f1f2a5a04..d1a40e657a 100644 --- a/frontend/scripts/tldr/README.md +++ b/frontend/scripts/tldr/README.md @@ -60,8 +60,10 @@ The `tldr_to_db.go` script performs the following steps to transform raw markdow - `url_hash` (PK): Integer hash of the URL. - `url`: The full path (e.g., `/freedevtools/tldr/common/tar/`). - `cluster_hash`: Integer hash of the cluster name (FK). + - `title`: Title of the page. + - `description`: Description of the page. - `html_content`: Rendered HTML from markdown. - - `metadata`: JSON string containing title, description, keywords, etc. + - `metadata`: JSON string containing keywords and features. - **`cluster`**: Stores metadata for command groups (platforms like `common`, `linux`). - `hash` (PK): Integer hash of the cluster name. - `name`: Cluster name (e.g., `common`). @@ -69,6 +71,8 @@ The `tldr_to_db.go` script performs the following steps to transform raw markdow - `preview_commands_json`: JSON array of the first 5 commands for preview. - **`overview`**: Stores global statistics. - `total_count`: Total number of commands across all platforms. + - `total_clusters`: Total number of clusters. + - `total_pages`: Total number of pages. ### 2. File Parsing From 3c2dfca22cc2d84c874bdb881d89dfa28c2e404a Mon Sep 17 00:00:00 2001 From: Ganesh Kumar Date: Mon, 8 Dec 2025 21:28:37 +0530 Subject: [PATCH 228/234] feat: select metadata column in paginated commands by cluster query --- frontend/db/tldrs/tldr-worker.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/frontend/db/tldrs/tldr-worker.ts b/frontend/db/tldrs/tldr-worker.ts index 7c9950ac2f..f09849658b 100644 --- a/frontend/db/tldrs/tldr-worker.ts +++ b/frontend/db/tldrs/tldr-worker.ts @@ -54,7 +54,7 @@ const statements = { // New query for paginated commands in a cluster getCommandsByClusterPaginated: db.prepare( - `SELECT url, description FROM pages + `SELECT url, description, metadata FROM pages WHERE cluster_hash = ? ORDER BY url LIMIT ? OFFSET ?` From 5ffd6a2f28e7279a33b8be168c4715c2c655c5df Mon Sep 17 00:00:00 2001 From: Lince Mathew Date: Mon, 8 Dec 2025 21:41:09 +0530 Subject: [PATCH 229/234] fix: mapage path --- .../src/pages/man-pages/[category]/[subcategory]/_Page.astro | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/frontend/src/pages/man-pages/[category]/[subcategory]/_Page.astro b/frontend/src/pages/man-pages/[category]/[subcategory]/_Page.astro index 40b06c3a06..a9c5eb87b3 100644 --- a/frontend/src/pages/man-pages/[category]/[subcategory]/_Page.astro +++ b/frontend/src/pages/man-pages/[category]/[subcategory]/_Page.astro @@ -39,8 +39,7 @@ const breadcrumbItems = [ href: `/freedevtools/man-pages/${category}/${subcategory}/`, }, { - label: manPage.slug, - href: `/freedevtools/man-pages/${manPage.main_category}/${manPage.sub_category}/${manPage.slug}/`, + label: manPage.slug }, ]; From 0a35fbfcf4d6731ae9712b0127cbd3cf02e8081e Mon Sep 17 00:00:00 2001 From: lovestaco Date: Tue, 9 Dec 2025 02:21:20 +0530 Subject: [PATCH 230/234] fix: bun to node --- frontend/md/bun_to_node.md | 128 +++++++++++++++++++++++++++++++++++++ 1 file changed, 128 insertions(+) create mode 100644 frontend/md/bun_to_node.md diff --git a/frontend/md/bun_to_node.md b/frontend/md/bun_to_node.md new file mode 100644 index 0000000000..3c76eb68d0 --- /dev/null +++ b/frontend/md/bun_to_node.md @@ -0,0 +1,128 @@ +# Migrate from Bun to Node.js + +## Overview + +Replace Bun runtime with Node.js by: + +1. Replacing `bun:sqlite` with `better-sqlite3` in database code +2. Updating all package.json scripts to use `node`/`npm` instead of `bun` +3. Cleaning build artifacts and reinstalling dependencies + +## Files to Modify + +### Database Code Changes + +**`db/svg_icons/svg-worker.ts`** + +- Replace `import { Database } from 'bun:sqlite'` with `import Database from 'better-sqlite3'` +- Update database initialization: `new Database(dbPath)` (note: `{ readonly: true }` option is NOT supported in better-sqlite3) +- Use `db.exec('PRAGMA ...')` instead of `db.pragma()` for setting PRAGMA values +- Update comment on line 2 to reflect better-sqlite3 +- Add error handling around database initialization + +**`db/banner/banner-utils.ts`** + +- Replace `import { Database } from 'bun:sqlite'` with `import Database from 'better-sqlite3'` +- Update database initialization: `new Database(dbPath)` (note: `{ readonly: true }` option is NOT supported) +- Change PRAGMA calls from `db.run('PRAGMA ...')` to `db.exec('PRAGMA ...')` +- The `db.run()` method works the same way for regular SQL statements + +**`db/svg_icons/svg-worker-pool.ts`** + +- Update comment on line 2 to reflect better-sqlite3 instead of bun:sqlite + +### Package Configuration + +**`package.json`** + +- Add `better-sqlite3` to dependencies +- Add `@types/better-sqlite3` to devDependencies +- Update scripts: +- `dev`: Change from `bun --max-old-space-size=16384` to `node --max-old-space-size=16384` +- `dev:light`: Change from `bun --max-old-space-size=4096` to `node --max-old-space-size=4096` +- `build`: Change from `bun run` to `node` +- `build:mcp`, `build:tldr`, `build:icons`, `build:emojis`, `build:man-pages`, `build:index`: Change from `bun` to `node` +- `serve-ssr`: Change from `bun run` to `node` +- `banner:generate`, `pagespeed*`: Change from `bun run` to `node` +- Remove `@types/bun` from devDependencies (if present) + +### No Changes Needed + +- `integrations/copy-worker.mjs` - No bun-specific code +- `integrations/critical-css-inlining.mjs` - No bun-specific code +- `integrations/wrap-astro.mjs` - No bun-specific code +- `src/middleware.ts` - No bun-specific code +- `astro.config.mjs` - No bun-specific code + +## Execution Steps + +1. Remove build artifacts: `node_modules`, `dist`, `.astro` +2. Update all files listed above +3. Run `npm install` to install dependencies including better-sqlite3 + - Note: If you encounter CUDA-related errors with `onnxruntime-node`, use: `ONNXRUNTIME_NODE_INSTALL_CUDA=skip npm install` +4. **Compile worker files for development** (required for Node.js): + ```bash + npx esbuild db/svg_icons/svg-worker.ts --outfile=db/svg_icons/svg-worker.js --format=esm --target=node18 --bundle=false --platform=node + ``` + + - This creates a `.js` file that the worker pool can load in development mode + - The worker pool looks for `.js` files first, then falls back to `.ts` files +5. Start server: `npm run dev` +6. Wait 30 seconds, then test: `curl localhost:4321/freedevtools/svg_icons/` +7. If test fails, check console logs and fix issues iteratively + +## Important Differences from bun:sqlite + +### Database Initialization + +- **bun:sqlite**: `new Database(dbPath, { readonly: true })` - supports readonly option +- **better-sqlite3**: `new Database(dbPath)` - readonly option NOT supported, use `PRAGMA query_only = ON` instead + +### PRAGMA Usage + +- **bun:sqlite**: Can use `db.run('PRAGMA ...')` or `db.pragma('key', 'value')` +- **better-sqlite3**: Must use `db.exec('PRAGMA key = value')` - the `pragma()` method is for reading values, not setting them + +### API Compatibility + +- `stmt.get()`, `stmt.all()`, and `db.run()` methods are compatible +- `db.prepare()` works the same way +- `better-sqlite3` API is synchronous and compatible with the current code structure + +## Worker File Compilation + +### Current State + +- Worker files (`.ts`) must be compiled to JavaScript (`.js`) for Node.js to execute them +- The `copy-worker.mjs` integration only compiles workers during build (`astro:build:done` hook) +- In development, you must manually compile worker files before starting the dev server + +### Manual Compilation + +```bash +npx esbuild db/svg_icons/svg-worker.ts --outfile=db/svg_icons/svg-worker.js --format=esm --target=node18 --bundle=false --platform=node +``` + +### Future Automation (Recommended) + +The `copy-worker.mjs` integration could be enhanced to also compile workers in development mode by adding an `astro:server:setup` or `astro:config:setup` hook. This would: + +- Automatically compile worker files when the dev server starts +- Watch for changes and recompile on file modifications +- Eliminate the need for manual compilation + +Example enhancement to `copy-worker.mjs`: + +```javascript +hooks: { + 'astro:server:setup': async ({ server }) => { + // Compile workers for development + await compileWorkers(); + }, + 'astro:build:done': async ({ dir }) => { + // Existing build-time compilation + } +} +``` + +This would ensure worker files are always available in both development and production environments. From 92b5c84bbe5c04e70fab49f3e73ba79d240a9dff Mon Sep 17 00:00:00 2001 From: lovestaco Date: Tue, 9 Dec 2025 18:37:40 +0530 Subject: [PATCH 231/234] fix: working cpu profiling on dev server --- frontend/.vscode/launch.json | 71 ++++++++---------------------------- 1 file changed, 15 insertions(+), 56 deletions(-) diff --git a/frontend/.vscode/launch.json b/frontend/.vscode/launch.json index 0df22bfb83..d84db808dd 100644 --- a/frontend/.vscode/launch.json +++ b/frontend/.vscode/launch.json @@ -2,72 +2,31 @@ "version": "0.2.0", "configurations": [ { - "type": "bun", + "command": "./node_modules/.bin/astro dev", + "name": "Development server", "request": "launch", - "name": "Debug Astro (Bun)", - - // Run the Astro dev server using Bun - "program": "./node_modules/astro/astro.js", - - // The arguments to pass to the program, if any. - "args": ["dev"], - - // The working directory of the program. - "cwd": "${workspaceFolder}", - - // The environment variables to pass to the program. - "env": {}, - - // If the environment variables should not be inherited from the parent process. - "strictEnv": false, - - // If the program should be run in watch mode. - // This is equivalent to passing `--watch` to the `bun` executable. - // You can also set this to "hot" to enable hot reloading using `--hot`. - "watchMode": false, - - // If the debugger should stop on the first line of the program. - "stopOnEntry": true, - - // If the debugger should be disabled. (for example, breakpoints will not be hit) - // "noDebug": false, - - // The path to the `bun` executable, defaults to your `PATH` environment variable. - "runtime": "bun", - - // The arguments to pass to the `bun` executable, if any. - // Unlike `args`, these are passed to the executable itself, not the program. - // "runtimeArgs": ["--inspect=ws://localhost:6499/fdt"] - "runtimeArgs": ["--inspect=ws://localhost:6499/fdt"] + "type": "node-terminal", + "skipFiles": ["**/node_modules/**"] }, { "type": "chrome", "request": "launch", - "name": "Debug Chrome (Client-side)", - "url": "http://localhost:4321/freedevtools/svg_icons/", - "webRoot": "${workspaceFolder}", - "sourceMaps": true, + "name": "Launch Chrome against localhost", + "url": "http://localhost:4321", + "webRoot": "${workspaceFolder}/src", "sourceMapPathOverrides": { - "webpack:///./*": "${webRoot}/*", - "webpack:///src/*": "${webRoot}/src/*", - "webpack:///*": "*", - "webpack:///./~/*": "${webRoot}/node_modules/*" - } - }, - { - "type": "chrome", - "request": "attach", - "name": "Attach to Chrome", - "port": 9222, - "webRoot": "${workspaceFolder}", - "sourceMaps": true + "vite:///src/*": "${webRoot}/*" + }, + "skipFiles": ["**/node_modules/**"] } ], "compounds": [ { - "name": "Debug Full Stack (Bun + Chrome)", - "configurations": ["Debug Astro (Bun)", "Debug Chrome (Client-side)"], - "stopAll": true + "name": "Launch Dev Server and Chrome", + "configurations": [ + "Development server", + "Launch Chrome against localhost" + ] } ] } From dad1292f4a75d29de39662872520d1e9473249ff Mon Sep 17 00:00:00 2001 From: lovestaco Date: Tue, 9 Dec 2025 19:34:39 +0530 Subject: [PATCH 232/234] feat: working cpu profiling on prod server --- frontend/.vscode/launch.json | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/frontend/.vscode/launch.json b/frontend/.vscode/launch.json index d84db808dd..fed74dff36 100644 --- a/frontend/.vscode/launch.json +++ b/frontend/.vscode/launch.json @@ -18,6 +18,38 @@ "vite:///src/*": "${webRoot}/*" }, "skipFiles": ["**/node_modules/**"] + }, + { + "type": "node", + "request": "launch", + "name": "Profile Production Server", + "runtimeExecutable": "/home/lovestaco/.nvm/versions/node/v22.17.0/bin/node", + "runtimeArgs": [ + "--cpu-prof", + "--cpu-prof-dir", + "${workspaceFolder}", + "--cpu-prof-name", + "production-profile-${date:yyyy-MM-dd-HH-mm-ss}.cpuprofile" + ], + "program": "${workspaceFolder}/dist/server/entry.mjs", + "cwd": "${workspaceFolder}", + "skipFiles": ["**/node_modules/**"], + "sourceMaps": true, + "outFiles": ["${workspaceFolder}/dist/**/*.js"] + }, + { + "type": "node", + "request": "launch", + "name": "Profile Production Server (Inspector)", + "runtimeExecutable": "/home/lovestaco/.nvm/versions/node/v22.17.0/bin/node", + "runtimeArgs": ["--inspect"], + "program": "${workspaceFolder}/dist/server/entry.mjs", + "cwd": "${workspaceFolder}", + "skipFiles": ["**/node_modules/**"], + "sourceMaps": true, + "outFiles": ["${workspaceFolder}/dist/**/*.js"], + "console": "integratedTerminal", + "internalConsoleOptions": "neverOpen" } ], "compounds": [ From 5236b0225ce09f07e0b7b78af1c7f3fdfe6dd703 Mon Sep 17 00:00:00 2001 From: Ganesh Kumar Date: Wed, 17 Dec 2025 19:09:10 +0530 Subject: [PATCH 233/234] test: Add performance test for tldr worker and update Makefile. --- frontend/db/tldrs/tldr-worker.ts | 7 +++---- frontend/serve/Makefile | 12 ++++++++++-- 2 files changed, 13 insertions(+), 6 deletions(-) diff --git a/frontend/db/tldrs/tldr-worker.ts b/frontend/db/tldrs/tldr-worker.ts index f09849658b..16ad311938 100644 --- a/frontend/db/tldrs/tldr-worker.ts +++ b/frontend/db/tldrs/tldr-worker.ts @@ -54,7 +54,7 @@ const statements = { // New query for paginated commands in a cluster getCommandsByClusterPaginated: db.prepare( - `SELECT url, description, metadata FROM pages + `SELECT url, description FROM pages WHERE cluster_hash = ? ORDER BY url LIMIT ? OFFSET ?` @@ -151,17 +151,16 @@ parentPort?.on('message', (message: QueryMessage) => { clusterHash, limit, offset - ) as { url: string; description: string; metadata: string }[]; + ) as { url: string; description: string }[]; result = rows.map(row => { - const meta = JSON.parse(row.metadata); // Extract name from URL or metadata const name = row.url.split('/').filter(Boolean).pop() || ''; return { name, url: row.url, description: row.description, // Use column directly - features: meta.features + features: [] // No metadata available }; }); break; diff --git a/frontend/serve/Makefile b/frontend/serve/Makefile index b53492e343..34a37e675b 100644 --- a/frontend/serve/Makefile +++ b/frontend/serve/Makefile @@ -113,11 +113,11 @@ test-tldr-req-pmd: @echo "⬇️ Server up -----------" @sleep 2 @echo "⬇️ Warming up server logs will be flushed for the below requests, ignore the below curls -----------" - hey -z 10s -host hexmos-local.com http://127.0.0.1/freedevtools/tldr/adb/adb + hey -z 10s -host hexmos-local.com http://127.0.0.1/freedevtools/tldr/npm/npm-fund/ @python3 pmd_logs.py --peek @sleep 2 @echo "⬇️ Requesting tldr 10sec" - hey -z 10s -host hexmos-local.com http://127.0.0.1/freedevtools/tldr/adb + hey -z 10s -host hexmos-local.com http://127.0.0.1/freedevtools/tldr/npm/ @python3 pmd_logs.py --peek @sleep 2 @echo "10s Requesting tldr main" @@ -128,6 +128,14 @@ test-tldr-req-pmd: hey -z 1m -host hexmos-local.com http://127.0.0.1/freedevtools/tldr/npm/npm-fund/ @python3 pmd_logs.py --peek @sleep 2 + @echo "1min Requesting tldr main" + hey -z 1m -host hexmos-local.com http://127.0.0.1/freedevtools/tldr/npm + @python3 pmd_logs.py --peek + @sleep 2 + @echo "1min Requesting tldr main" + hey -z 1m -host hexmos-local.com http://127.0.0.1/freedevtools/tldr + @python3 pmd_logs.py --peek + @sleep 2 @echo "5min Requesting tldr 5min" hey -z 5m -host hexmos-local.com http://127.0.0.1/freedevtools/tldr/npm/npm-fund/ @python3 pmd_logs.py --peek From 24ac9a90214cabaaf53e5eb781857b810548cf94 Mon Sep 17 00:00:00 2001 From: Ganesh Kumar Date: Wed, 17 Dec 2025 19:37:12 +0530 Subject: [PATCH 234/234] Remove unwanted documentation file. --- frontend/tldr-schema-change.md | 18 ------------------ 1 file changed, 18 deletions(-) delete mode 100644 frontend/tldr-schema-change.md diff --git a/frontend/tldr-schema-change.md b/frontend/tldr-schema-change.md deleted file mode 100644 index 02fa0bc38d..0000000000 --- a/frontend/tldr-schema-change.md +++ /dev/null @@ -1,18 +0,0 @@ -# Overivew - -Currenlty i have implemented tldr schema in the db/tldrs/tldr-schema.ts file. -and tldr_to_db.go file to insert tldr data into the db. - -But for any small change in migration we might waste time in future - -folow rules of scheam of db/emojis -this also folow the same rules of sitemap generation. present in src/pages/emojis - -First explain How db is created scripts/emojis - -and explain how pagination and other things are implemented in src/pages/emojis. - -no need to change tldr end pages logic -i.e., freedevtools/tldr/adb/adb this will be same. - -Only fix main_page table and remove sitemap table let it be same as emojis sitemap generation