Add R backend-controlled format options to data explorer #987

kv9898 · 2025-12-11T14:11:28Z

Addresses posit-dev/positron#4818, in tandem with posit-dev/positron#11051.

The combined end result looks like (using the code example in the original issue):

This pull request introduces frontend formatting options for the data explorer backend and ensures that these options are provided in backend state replies. The changes also include the logic to calculate formatting options based on the current R session settings.

Frontend formatting options integration:

Added a new format_options field of type Option<FormatOptions> to the BackendState struct in crates/amalthea/src/comm/data_explorer_comm.rs to support frontend display customization.
When replying with backend state in RDataExplorer::get_state, the code now includes current formatting options by calling a new helper method.

Calculation of formatting options:

Introduced the current_format_options method in RDataExplorer to compute formatting options based on the R session's scipen option, determining integral digit limits and other display parameters.
Imported get_option from harp to retrieve R session options necessary for formatting calculations.

whereayan · 2025-12-15T07:41:08Z

That's great — nice work! 👍I’ll try out your way, but still hope the official team jumps in to fix it.

kv9898 · 2025-12-15T07:49:56Z

That's great — nice work! 👍I’ll try out your way, but still hope the official team jumps in to fix it.

Cool! You will probably need to build both Ark and Positron for it to take effect. I'll have my fingers crossed that you don't run into compilation errors!

juliasilge · 2025-12-15T19:49:58Z

Thank you for this PR, @kv9898! This looks like a good approach to me, and I have asked @dfalbel to do a detailed review.

dfalbel

Thank you @kv9898 !

We should add a test in https://github.com/posit-dev/ark/blob/f0a0e689892e8a45c2b3b2ef9978d8c805e8c804/crates/ark/tests/data_explorer.rs checking if the state is changed after changing that option.

You can use eg: https://github.com/kv9898/ark/blob/9c14a03e3e517e9b3534a2a9a6fcc428a7b707ab/crates/ark/tests/data_explorer.rs#L826

crates/ark/src/data_explorer/r_data_explorer.rs

…igits

kv9898 · 2025-12-16T02:44:27Z

@dfalbel It is actually impossible to 100% replicate the R behaviour for (at least) 2 reasons:

For vectors/data.frames R tends to display the whole variable in the scientific notation. I therefore only try to follow R's scalar formatting as closely as possible
R's calculation is based on text length of the number, which is different from Positron's calculation based on integral+decimal digits.

I have rewritten the logic to match with the scalar printing behaviour in R using the following examples

# Set the scipen option to a specific value
options("scipen" = 0, "digits" = 7) # default to 0 and 7

test <- data.frame(num = c(0.000145, 0.0000145, 10000, 100000, 12345.123456))
print(0.000145) # fixed notation
print(0.0000145) # scientific notation in R, 6 d.p. in Positron
print(10000) # fixed notation
print(100000) # scientific notation
print(12345.123456) # 2 d.p.

options("scipen" = 1, "digits" = 7) # scipen to 1
print(0.000145) # all fixed
print(0.0000145)
print(10000) 
print(100000)
print(12345.123456) # 2.d.p.

options("scipen" = 0, "digits" = 8) # digits to 8
print(0.000145) # fixed notation
print(0.0000145) # scientific notation, 6 d.p. in Positron
print(10000) # fixed notation
print(100000) # scientific notation
print(12345.123456) # 3.d.p.

) * Initial plan * Add test for format_options state change after R option modification Co-authored-by: kv9898 <105025148+kv9898@users.noreply.github.com> * Improve error handling in test_format_options_state_change Co-authored-by: kv9898 <105025148+kv9898@users.noreply.github.com> * Remove unnecessary move keyword in closure Co-authored-by: kv9898 <105025148+kv9898@users.noreply.github.com>

github-actions · 2025-12-16T04:33:44Z

Thank you all for this PR! We ask that you sign our Contributor License Agreement before we accept your contribution. You can sign the CLA by posting a comment on this PR saying:

I have read the CLA Document and I hereby sign the CLA

2 out of 3 committers have signed the CLA.
✅ (kv9898)[https://github.com/kv9898]
✅ (juliasilge)[https://github.com/juliasilge]
❌ @copilot
_{You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot.}

kv9898 · 2025-12-16T04:35:02Z

Thank you @kv9898 !

We should add a test in https://github.com/posit-dev/ark/blob/f0a0e689892e8a45c2b3b2ef9978d8c805e8c804/crates/ark/tests/data_explorer.rs checking if the state is changed after changing that option.

You can use eg: https://github.com/kv9898/ark/blob/9c14a03e3e517e9b3534a2a9a6fcc428a7b707ab/crates/ark/tests/data_explorer.rs#L826

The test has been added in 9569779!

juliasilge · 2025-12-16T17:37:34Z

recheck

dfalbel · 2025-12-16T18:09:02Z

crates/ark/tests/data_explorer.rs

+        // max_integral_digits = (10 + 5).max(1) = 15
+        // large_num_digits = (10 - 5).max(0) = 5
+        // small_num_digits = (10 + 6).max(1) = 16
+        assert_eq!(format_options.max_integral_digits, 15);


Looks like tests are failing with: thread 'test_format_options_state_change' (32147) panicked at crates/ark/tests/data_explorer.rs:3160:9: assertion `left == right` failed left: 5 right: 15

Any idea @kv9898 ?

It only fails for R 4.2 🤔

IDK, this might be related: wch/r-source@7f20c19

There's something special about scipen. Because digits are being correctly evaluated. Perhaps we can't use get_option for this, and instead we bay need to evaluate getOption from the R interpreter.

It only fails for R 4.2 🤔

Thx for checking that out. Because the test worked for me with R 4.5.2:

E:\ark [scipen ≡] > cargo test -p ark --test data_explorer Compiling ark v0.1.220 (E:\ark\crates\ark) warning: unused import: `std::time::Duration` --> crates\ark\src\fixtures\dummy_frontend.rs:7:5 | 7 | use std::time::Duration; | ^^^^^^^^^^^^^^^^^^^ | = note: `#[warn(unused_imports)]` (part of `#[warn(unused)]`) on by default warning: `ark` (lib) generated 1 warning (run `cargo fix --lib -p ark` to apply 1 suggestion) Finished `test` profile [unoptimized + debuginfo] target(s) in 14.20s Running tests\data_explorer.rs (target\debug\deps\data_explorer-27782ce78c282663.exe) running 40 tests test test_export_with_sort_order ... ok test test_basic_mtcars ... ok test test_column_labels_edge_cases ... ok test test_column_labels ... ok test test_column_labels_haven_compatibility ... ok test test_column_labels_missing ... ok test test_data_explorer_special_values ... ok test test_data_table_support ... ok test test_data_update_num_rows ... ok test test_empty_data_frame_column_profiles ... ok test test_empty_data_frame_data_values ... ok test test_empty_data_frame_schema ... ok test test_export_data ... ok test test_format_options_state_change ... ok test test_empty_data_frame_state ... ok test test_frequency_table ... ok test test_get_data_values_by_indices ... ok test test_histogram ... ok test test_histogram_single_bin_same_values ... ok test test_invalid_filters_preserved ... ok test test_live_updates ... ok test test_matrix_support ... ok test test_null_counts ... ok test test_row_names_matrix ... ok test test_schema_identification ... ok test test_search_filters ... ok test test_search_schema_combined_filters ... ok test test_search_schema_data_type_filters ... ok test test_search_schema_edge_cases ... ok test test_search_schema_no_matches ... ok test test_search_schema_sort_orders ... ok test test_search_schema_text_filters ... ok test test_search_schema_text_with_sort_orders ... ok test test_search_schema_type_sort_orders ... ok test test_set_membership_filter ... ok test test_single_row_data_frame_column_profiles ... ok test test_summary_stats ... ok test test_tibble_support ... ok test test_update_data_filters_reapplied ... ok test test_women_dataset ... ok test result: ok. 40 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.83s

~~Currently, I don't have much of a clue.~~ Feel free to edit my code if you find a solution! Wrapping the call in as.integer() seems to work.

IDK, this might be related: wch/r-source@7f20c19

There's something special about scipen. Because digits are being correctly evaluated. Perhaps we can't use get_option for this, and instead we bay need to evaluate getOption from the R interpreter.

Tried that but sadly it didn't work.

kv9898 · 2025-12-17T09:01:14Z

Now the tests are passed!

kv9898 · 2025-12-17T15:27:12Z

recheck

@juliasilge Ark seems use its own RequireCLA script and does not follow your change in posit-dev/cla-assistant@b7814b8. Note Copilot is not in:

ark/.github/workflows/cla.yml

Line 52 in 910bb6d

    
                     allowlist: DavisVaughan, dependabot[bot], dfalbel, isabelizimm, jonvanausdeln, lionel-, nstrayer, petetronic, positron-bot[bot], seeM, sharon-wang, softwarenerd, timtmok, wesm

This reverts commit b0508b6.

This reverts commit bc4f44b.

This reverts commit aa29e72.

dfalbel

Looks good to me!

@juliasilge I'm still not fully convinced that R's defaults are better compared to Positron Data explorer defaults.

There's no way though to identify if users have explicitly set scipen and digits, ie, if they are really overriding the defaults.

But this probably something that can be handled on the UI side, ie, I'm assuming that in the front-end side PR we'll allow users to decide if they want data explorer to match R's settings or not.

crates/ark/src/data_explorer/r_data_explorer.rs

Co-authored-by: Daniel Falbel <dfalbel@gmail.com>

juliasilge · 2025-12-17T22:33:42Z

recheck

kv9898 · 2025-12-19T17:31:57Z

Can we merge this now?

juliasilge · 2025-12-19T22:31:48Z

I'm going to work on this soon, together with the PR to /positron. Thanks!

kv9898 added 2 commits December 11, 2025 21:21

Add format_options as part of BackendState

f58e61b

fetch scipen option

9c14a03

This was referenced Dec 11, 2025

Add backend-controlled format options to data explorer posit-dev/positron#11051

Open

Data Explorer - scientific notation - does not adhere to options(scipen = xxx) posit-dev/positron#4818

Open

juliasilge requested a review from dfalbel December 15, 2025 19:49

dfalbel requested changes Dec 15, 2025

View reviewed changes

crates/ark/src/data_explorer/r_data_explorer.rs Outdated Show resolved Hide resolved

dfalbel reviewed Dec 15, 2025

View reviewed changes

crates/ark/src/data_explorer/r_data_explorer.rs Outdated Show resolved Hide resolved

kv9898 added 6 commits December 16, 2025 09:46

Also use digits

c10d632

Remove clamping for maximum values

a8cae77

Add comments for scipen and digits defaults

03924fb

2 d.p. by default

7e3086f

max integral digits and small num digits only depend on scipen, not d…

74afed2

…igits

Add function comment

35f1c6a

juliasilge requested a review from dfalbel December 16, 2025 17:44

dfalbel requested changes Dec 16, 2025

View reviewed changes

This was referenced Dec 17, 2025

[WIP] Investigate test failure specific to R version 4.2.2 kv9898/stock-management#69

Closed

Fix scipen option not updating in R 4.2 via Rf_GetOption1 kv9898/ark#2

Merged

kv9898 force-pushed the scipen branch from c596754 to 9569779 Compare December 17, 2025 03:55

use as.integer() this time (#3)

910bb6d

kv9898 requested a review from dfalbel December 17, 2025 15:17

juliasilge added 3 commits December 17, 2025 09:51

Copilot does not need to sign the CLA

aa29e72

Maybe Copilot's name is copilot

bc4f44b

Actually, it's name is copilot[bot]

b0508b6

juliasilge mentioned this pull request Dec 17, 2025

Copilot does not have to sign the CLA #996

Merged

juliasilge added 3 commits December 17, 2025 10:01

Revert "Actually, it's name is copilot[bot]"

bdfe49e

This reverts commit b0508b6.

Revert "Maybe Copilot's name is copilot"

0ba2914

This reverts commit bc4f44b.

Revert "Copilot does not need to sign the CLA"

98b3fd2

This reverts commit aa29e72.

dfalbel approved these changes Dec 17, 2025

View reviewed changes

crates/ark/src/data_explorer/r_data_explorer.rs Outdated Show resolved Hide resolved

Streamline code

afa46f6

Co-authored-by: Daniel Falbel <dfalbel@gmail.com>

Add R backend-controlled format options to data explorer #987

Are you sure you want to change the base?

Add R backend-controlled format options to data explorer #987

Uh oh!

Conversation

kv9898 commented Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

whereayan commented Dec 15, 2025

Uh oh!

kv9898 commented Dec 15, 2025

Uh oh!

juliasilge commented Dec 15, 2025

Uh oh!

dfalbel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kv9898 commented Dec 16, 2025

Uh oh!

github-actions bot commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kv9898 commented Dec 16, 2025

Uh oh!

juliasilge commented Dec 16, 2025

Uh oh!

dfalbel Dec 16, 2025

Choose a reason for hiding this comment

Uh oh!

dfalbel Dec 16, 2025

Choose a reason for hiding this comment

Uh oh!

dfalbel Dec 16, 2025

Choose a reason for hiding this comment

Uh oh!

kv9898 Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kv9898 Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kv9898 Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

kv9898 commented Dec 17, 2025

Uh oh!

kv9898 commented Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dfalbel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

juliasilge commented Dec 17, 2025

Uh oh!

kv9898 commented Dec 19, 2025

Uh oh!

juliasilge commented Dec 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kv9898 commented Dec 11, 2025 •

edited

Loading

github-actions bot commented Dec 16, 2025 •

edited

Loading

kv9898 Dec 17, 2025 •

edited

Loading

kv9898 Dec 17, 2025 •

edited

Loading

kv9898 commented Dec 17, 2025 •

edited

Loading