Skip to content

Track strings to calculate codeSurvivalRate #2365

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
meigaoms wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Track strings to calculate codeSurvivalRate #2365

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
dd4fab8
Track strings to calculate codeSurvivalRate
Dec 3, 2025
2fde3c1
Update test for the new telemetry adding
Dec 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 21 additions & 3 deletions src/extension/prompts/node/codeMapper/codeMapperService.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -305,10 +305,22 @@ function reportEditSurvivalEvent(res: EditSurvivalResult, { requestId, speculati
"survivalRateFourGram": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "isMeasurement": true, "comment": "The rate between 0 and 1 of how much of the AI edit is still present in the document." },
"survivalRateNoRevert": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "isMeasurement": true, "comment": "The rate between 0 and 1 of how much of the ranges the AI touched ended up being reverted." },
"didBranchChange": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "isMeasurement": true, "comment": "Indicates if the branch changed in the meantime. If the branch changed (value is 1), this event should probably be ignored." },
"timeDelayMs": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "isMeasurement": true, "comment": "The time delay between the user accepting the edit and measuring the survival rate." }
"timeDelayMs": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "isMeasurement": true, "comment": "The time delay between the user accepting the edit and measuring the survival rate." },
"textBeforeAiEdits": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "Array of text strings before AI edits were applied for each edit region." },
"textAfterAiEdits": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "Array of text strings after AI edits were applied for each edit region." },
"textAfterUserEdits": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "Array of text strings after user modifications for each edit region." }
Comment on lines +309 to +311
Copy link

Copilot AI Dec 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security concern: The GDPR metadata for textBeforeAiEdits, textAfterAiEdits, and textAfterUserEdits is missing proper data classification. These fields contain user code which may include sensitive information and should have appropriate "classification" metadata specified.

Suggested change
"textBeforeAiEdits": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "Array of text strings before AI edits were applied for each edit region." },
"textAfterAiEdits": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "Array of text strings after AI edits were applied for each edit region." },
"textAfterUserEdits": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "Array of text strings after user modifications for each edit region." }
"textBeforeAiEdits": { "classification": "CustomerContent", "purpose": "FeatureInsight", "comment": "Array of text strings before AI edits were applied for each edit region. Contains user code and may include sensitive information." },
"textAfterAiEdits": { "classification": "CustomerContent", "purpose": "FeatureInsight", "comment": "Array of text strings after AI edits were applied for each edit region. Contains user code and may include sensitive information." },
"textAfterUserEdits": { "classification": "CustomerContent", "purpose": "FeatureInsight", "comment": "Array of text strings after user modifications for each edit region. Contains user code and may include sensitive information." }

Copilot uses AI. Check for mistakes.
}
*/
res.telemetryService.sendMSFTTelemetryEvent('codeMapper.trackEditSurvival', { requestId, speculationRequestId, requestSource, chatRequestModel, mapper }, {
res.telemetryService.sendMSFTTelemetryEvent('codeMapper.trackEditSurvival', {
requestId,
speculationRequestId,
requestSource,
chatRequestModel,
mapper,
textBeforeAiEdits: res.textBeforeAiEdits ? JSON.stringify(res.textBeforeAiEdits) : undefined,
textAfterAiEdits: res.textAfterAiEdits ? JSON.stringify(res.textAfterAiEdits) : undefined,
textAfterUserEdits: res.textAfterUserEdits ? JSON.stringify(res.textAfterUserEdits) : undefined,
Comment on lines +320 to +322
Copy link

Copilot AI Dec 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security concern: User code content is being sent through regular telemetry (sendMSFTTelemetryEvent) on lines 314-323. While currentFileContent is appropriately sent only through sendInternalMSFTTelemetryEvent (line 329) and sendEnhancedGHTelemetryEvent (line 345), the new text fields are included in regular telemetry which may not be appropriate for code content.

Suggested change
textBeforeAiEdits: res.textBeforeAiEdits ? JSON.stringify(res.textBeforeAiEdits) : undefined,
textAfterAiEdits: res.textAfterAiEdits ? JSON.stringify(res.textAfterAiEdits) : undefined,
textAfterUserEdits: res.textAfterUserEdits ? JSON.stringify(res.textAfterUserEdits) : undefined,
// Removed textBeforeAiEdits, textAfterAiEdits, textAfterUserEdits from regular telemetry event

Copilot uses AI. Check for mistakes.
}, {
survivalRateFourGram: res.fourGram,
survivalRateNoRevert: res.noRevert,
timeDelayMs: res.timeDelayMs,
Expand All @@ -321,6 +333,9 @@ function reportEditSurvivalEvent(res: EditSurvivalResult, { requestId, speculati
chatRequestModel,
mapper,
currentFileContent: res.currentFileContent,
textBeforeAiEdits: res.textBeforeAiEdits ? JSON.stringify(res.textBeforeAiEdits) : undefined,
textAfterAiEdits: res.textAfterAiEdits ? JSON.stringify(res.textAfterAiEdits) : undefined,
textAfterUserEdits: res.textAfterUserEdits ? JSON.stringify(res.textAfterUserEdits) : undefined,
}, {
survivalRateFourGram: res.fourGram,
survivalRateNoRevert: res.noRevert,
Expand All @@ -332,7 +347,10 @@ function reportEditSurvivalEvent(res: EditSurvivalResult, { requestId, speculati
headerRequestId: speculationRequestId,
completionTextJson: res.currentFileContent,
chatRequestModel,
requestSource
requestSource,
textBeforeAiEdits: res.textBeforeAiEdits ? JSON.stringify(res.textBeforeAiEdits) : undefined,
textAfterAiEdits: res.textAfterAiEdits ? JSON.stringify(res.textAfterAiEdits) : undefined,
textAfterUserEdits: res.textAfterUserEdits ? JSON.stringify(res.textAfterUserEdits) : undefined,
}, {
timeDelayMs: res.timeDelayMs,
survivalRateFourGram: res.fourGram,
Expand Down
9 changes: 9 additions & 0 deletions src/extension/test/node/utils.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -294,6 +294,15 @@ suite('EditSurvivalTracker', () => {
{
"fourGram": 0.75,
"noRevert": 1,
textAfterAiEdits: [
'\r\n\r\n\t\tif (!args) {\r\n\t\t\tthrow new Error("invalid json document!");\r\n\t\t}'
],
textAfterUserEdits: [
'\r\n\r\n\t\tif (!args) {\r\n\t\t\tthrow new Error("");\r\n\t\t}'
],
textBeforeAiEdits: [
'\r\n'
]
}
);
});
Expand Down
19 changes: 16 additions & 3 deletions src/extension/tools/node/abstractReplaceStringTool.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -307,10 +307,20 @@ export abstract class AbstractReplaceStringTool<T extends { explanation: string
"survivalRateFourGram": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "isMeasurement": true, "comment": "The rate between 0 and 1 of how much of the AI edit is still present in the document." },
"survivalRateNoRevert": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "isMeasurement": true, "comment": "The rate between 0 and 1 of how much of the ranges the AI touched ended up being reverted." },
"didBranchChange": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "isMeasurement": true, "comment": "Indicates if the branch changed in the meantime. If the branch changed (value is 1), this event should probably be ignored." },
"timeDelayMs": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "isMeasurement": true, "comment": "The time delay between the user accepting the edit and measuring the survival rate." }
"timeDelayMs": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "isMeasurement": true, "comment": "The time delay between the user accepting the edit and measuring the survival rate." },
"textBeforeAiEdits": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "Array of text strings before AI edits were applied for each edit region." },
"textAfterAiEdits": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "Array of text strings after AI edits were applied for each edit region." },
"textAfterUserEdits": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "Array of text strings after user modifications for each edit region." }
Comment on lines +311 to +313
Copy link

Copilot AI Dec 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security concern: The telemetry fields textBeforeAiEdits, textAfterAiEdits, and textAfterUserEdits contain user code but lack proper data classification in the GDPR metadata. These fields need either:

  1. Appropriate "classification" metadata (e.g., "CustomerContent") in the GDPR annotations
  2. Or should only be sent through internal telemetry channels

Additionally, these fields are sent through regular telemetry events which may not be appropriate for code content that could contain sensitive information.

Suggested change
"textBeforeAiEdits": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "Array of text strings before AI edits were applied for each edit region." },
"textAfterAiEdits": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "Array of text strings after AI edits were applied for each edit region." },
"textAfterUserEdits": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "Array of text strings after user modifications for each edit region." }
"textBeforeAiEdits": { "classification": "CustomerContent", "purpose": "FeatureInsight", "comment": "Array of text strings before AI edits were applied for each edit region." },
"textAfterAiEdits": { "classification": "CustomerContent", "purpose": "FeatureInsight", "comment": "Array of text strings after AI edits were applied for each edit region." },
"textAfterUserEdits": { "classification": "CustomerContent", "purpose": "FeatureInsight", "comment": "Array of text strings after user modifications for each edit region." }

Copilot uses AI. Check for mistakes.
}
*/
res.telemetryService.sendMSFTTelemetryEvent('codeMapper.trackEditSurvival', { requestId: this._promptContext?.requestId, requestSource: 'agent', mapper: 'stringReplaceTool' }, {
res.telemetryService.sendMSFTTelemetryEvent('codeMapper.trackEditSurvival', {
requestId: this._promptContext?.requestId,
requestSource: 'agent',
mapper: 'stringReplaceTool',
textBeforeAiEdits: res.textBeforeAiEdits ? JSON.stringify(res.textBeforeAiEdits) : undefined,
textAfterAiEdits: res.textAfterAiEdits ? JSON.stringify(res.textAfterAiEdits) : undefined,
textAfterUserEdits: res.textAfterUserEdits ? JSON.stringify(res.textAfterUserEdits) : undefined,
}, {
survivalRateFourGram: res.fourGram,
survivalRateNoRevert: res.noRevert,
timeDelayMs: res.timeDelayMs,
Expand All @@ -319,7 +329,10 @@ export abstract class AbstractReplaceStringTool<T extends { explanation: string
res.telemetryService.sendGHTelemetryEvent('replaceString/trackEditSurvival', {
headerRequestId: this._promptContext?.requestId,
requestSource: 'agent',
mapper: 'stringReplaceTool'
mapper: 'stringReplaceTool',
textBeforeAiEdits: res.textBeforeAiEdits ? JSON.stringify(res.textBeforeAiEdits) : undefined,
textAfterAiEdits: res.textAfterAiEdits ? JSON.stringify(res.textAfterAiEdits) : undefined,
textAfterUserEdits: res.textAfterUserEdits ? JSON.stringify(res.textAfterUserEdits) : undefined,
}, {
survivalRateFourGram: res.fourGram,
survivalRateNoRevert: res.noRevert,
Expand Down
19 changes: 16 additions & 3 deletions src/extension/tools/node/applyPatchTool.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -390,10 +390,20 @@ export class ApplyPatchTool implements ICopilotTool<IApplyPatchToolParams> {
"survivalRateFourGram": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "isMeasurement": true, "comment": "The rate between 0 and 1 of how much of the AI edit is still present in the document." },
"survivalRateNoRevert": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "isMeasurement": true, "comment": "The rate between 0 and 1 of how much of the ranges the AI touched ended up being reverted." },
"didBranchChange": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "isMeasurement": true, "comment": "Indicates if the branch changed in the meantime. If the branch changed (value is 1), this event should probably be ignored." },
"timeDelayMs": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "isMeasurement": true, "comment": "The time delay between the user accepting the edit and measuring the survival rate." }
"timeDelayMs": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "isMeasurement": true, "comment": "The time delay between the user accepting the edit and measuring the survival rate.",
"textBeforeAiEdits": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "Array of text strings before AI edits were applied for each edit region." },
"textAfterAiEdits": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "Array of text strings after AI edits were applied for each edit region." },
"textAfterUserEdits": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "Array of text strings after user modifications for each edit region." }}
Comment on lines +394 to +396
Copy link

Copilot AI Dec 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security concern: The new telemetry fields textBeforeAiEdits, textAfterAiEdits, and textAfterUserEdits contain user code content but lack proper data classification in the GDPR metadata. Code content may contain sensitive information (API keys, credentials, proprietary logic) and should be classified appropriately (e.g., "classification": "CustomerContent" or similar) or only sent through internal/enhanced telemetry channels.

Currently these fields are:

  1. Sent through regular telemetry (sendMSFTTelemetryEvent, sendGHTelemetryEvent) without proper classification
  2. Missing the "classification" key in the GDPR metadata annotations

Compare with how currentFileContent is handled - it's only sent through sendInternalMSFTTelemetryEvent and sendEnhancedGHTelemetryEvent, not through regular telemetry events.

Suggested change
"textBeforeAiEdits": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "Array of text strings before AI edits were applied for each edit region." },
"textAfterAiEdits": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "Array of text strings after AI edits were applied for each edit region." },
"textAfterUserEdits": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "Array of text strings after user modifications for each edit region." }}
"textBeforeAiEdits": { "classification": "CustomerContent", "purpose": "FeatureInsight", "comment": "Array of text strings before AI edits were applied for each edit region." },
"textAfterAiEdits": { "classification": "CustomerContent", "purpose": "FeatureInsight", "comment": "Array of text strings after AI edits were applied for each edit region." },
"textAfterUserEdits": { "classification": "CustomerContent", "purpose": "FeatureInsight", "comment": "Array of text strings after user modifications for each edit region." }}

Copilot uses AI. Check for mistakes.
}
*/
res.telemetryService.sendMSFTTelemetryEvent('applyPatch.trackEditSurvival', { requestId: this._promptContext?.requestId, requestSource: 'agent', mapper: 'applyPatchTool' }, {
res.telemetryService.sendMSFTTelemetryEvent('applyPatch.trackEditSurvival', {
requestId: this._promptContext?.requestId,
requestSource: 'agent',
mapper: 'applyPatchTool',
textBeforeAiEdits: res.textBeforeAiEdits ? JSON.stringify(res.textBeforeAiEdits) : undefined,
textAfterAiEdits: res.textAfterAiEdits ? JSON.stringify(res.textAfterAiEdits) : undefined,
textAfterUserEdits: res.textAfterUserEdits ? JSON.stringify(res.textAfterUserEdits) : undefined,
}, {
Comment on lines +403 to +406
Copy link

Copilot AI Dec 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Performance concern: JSON.stringify() is called on potentially large arrays of text strings for every telemetry event. If users make many edits across large code regions, this could create performance overhead. Consider:

  1. Limiting the size or number of text entries tracked
  2. Truncating long text strings before serialization
  3. Caching the stringified result if the same data is sent to multiple telemetry events

Copilot uses AI. Check for mistakes.
survivalRateFourGram: res.fourGram,
survivalRateNoRevert: res.noRevert,
timeDelayMs: res.timeDelayMs,
Expand All @@ -402,7 +412,10 @@ export class ApplyPatchTool implements ICopilotTool<IApplyPatchToolParams> {
res.telemetryService.sendGHTelemetryEvent('applyPatch/trackEditSurvival', {
headerRequestId: this._promptContext?.requestId,
requestSource: 'agent',
mapper: 'applyPatchTool'
mapper: 'applyPatchTool',
textBeforeAiEdits: res.textBeforeAiEdits ? JSON.stringify(res.textBeforeAiEdits) : undefined,
textAfterAiEdits: res.textAfterAiEdits ? JSON.stringify(res.textAfterAiEdits) : undefined,
textAfterUserEdits: res.textAfterUserEdits ? JSON.stringify(res.textAfterUserEdits) : undefined,
}, {
survivalRateFourGram: res.fourGram,
survivalRateNoRevert: res.noRevert,
Expand Down
11 changes: 10 additions & 1 deletion src/platform/editSurvivalTracking/common/editSurvivalReporter.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,13 @@ export interface EditSurvivalResult {
* See ArcTracker.
*/
readonly arc?: number;

/**
* Text states for each edit region
*/
readonly textBeforeAiEdits?: string[];
readonly textAfterAiEdits?: string[];
readonly textAfterUserEdits?: string[];
}

export class EditSurvivalReporter {
Expand Down Expand Up @@ -104,7 +111,6 @@ export class EditSurvivalReporter {

const currentBranch = this._getCurrentBranchName();
const didBranchChange = currentBranch !== this._initialBranchName;

this._sendTelemetryEvent({
telemetryService: this._telemetryService,
fourGram: survivalRate.fourGram,
Expand All @@ -113,6 +119,9 @@ export class EditSurvivalReporter {
didBranchChange,
currentFileContent: this._document.getText(),
arc: this._arcTracker?.getAcceptedRestrainedCharactersCount(),
textBeforeAiEdits: survivalRate.textBeforeAiEdits,
textAfterAiEdits: survivalRate.textAfterAiEdits,
textAfterUserEdits: survivalRate.textAfterUserEdits,
});
}

Expand Down
13 changes: 12 additions & 1 deletion src/platform/editSurvivalTracking/common/editSurvivalTracker.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,13 +36,17 @@ export class EditSurvivalTracker {
* fourGram: Number between 0 (no edits survived) and 1 (all edits survived).
* noRevert: Number between 0 (the text after user edits equals the text before the AI edits) and 1 (the text after user edits does not revert any text to the initial state)
*/
computeTrackedEditsSurvivalScore(): { fourGram: number; noRevert: number } {
computeTrackedEditsSurvivalScore(): { fourGram: number; noRevert: number; textBeforeAiEdits: string[]; textAfterAiEdits: string[]; textAfterUserEdits: string[] } {
let similarityScoreSumFourGram = 0;
let similarityScoreSumMax = 0;

let noRevertSum = 0;
let noRevertSumMax = 0;

const allTextBefore: string[] = [];
const allTextAfter: string[] = [];
const allTextCurrent: string[] = [];

const ranges = this._originalEdits.getNewRanges();
const updatedRanges = applyEditsToRanges(ranges, this._combinedEditsSinceStart);

Expand All @@ -54,6 +58,10 @@ export class EditSurvivalTracker {
const newRange = updatedRanges[i];
const textAfterUserEdits = this._text.substring(newRange.start, newRange.endExclusive);

allTextBefore.push(textBeforeAiEdits);
allTextAfter.push(textAfterAiEdits);
allTextCurrent.push(textAfterUserEdits);

const similarity = compute4GramTextSimilarity(textAfterUserEdits, textAfterAiEdits);

const aiEditSimilarity = compute4GramTextSimilarity(textAfterAiEdits, textBeforeAiEdits);
Expand All @@ -75,6 +83,9 @@ export class EditSurvivalTracker {
return {
fourGram: similarityScoreSumMax === 0 ? 1 : (similarityScoreSumFourGram / similarityScoreSumMax),
noRevert: noRevertSumMax === 0 ? 1 : (noRevertSum / noRevertSumMax),
textBeforeAiEdits: allTextBefore,
textAfterAiEdits: allTextAfter,
textAfterUserEdits: allTextCurrent,
};
}
}
Expand Down