Question about support for full case-replace support with dabbrev

[yuki-tsubaki]

In issue #116 there was a suggestion/patch provided by soficshift (in an edit) that added cape-dabbrev specific support for aggressive case-replace (i.e. also supporting capital expansion to lowercase abbreviation replace). Would you be open to adding this or something like it?

I know you mentioned that dabbrev is not a very good back-end and you don't want to work around all its quirks.

At the same time (unless you have issues with that solution and/or see something I don't), I think that

1. The more aggressive case replace strategy is more appropriate for dabbrev because of the buffer sourced completions (as opposed to a dictionary which all completions are assumed to be canonical forms).
2. That behavior would match the behavior of dabbrev.
3. At least in my usage, the function with that modification hasn't been perceptibly slower.

For me, the cape-dabbrev is much more useful with the more aggressive case replace because I often use it in org-mode files in English and it can be disruptive when I am trying to complete some long technical word and it changes to a capital because another use was at the beginning of a sentence.

Of course I respect if you want to keep it how it is in which case I'll just use the modified version, but I just wanted to share my experience/opinion and clarify your position on adding this.

[minad]

How is the modified behavior supposed to work? If case-replace=t then uppercased words will be lowercased automatically? I see your point about words at the beginning of sentences but there are also names and languages with uppercase words. It seems to me that case-replace is too restricted. One needs at least three configuration settings: always replace, never replace and capitalize if input is capitalized.

[yuki-tsubaki]

Yeah, you are right that optimally it would be even more configurable, but at least for me in this case the value of having suggestions from capitalized words in my buffers is high and the utility of fixing my capitalization when I type something that must be uppercase (like proper nouns) as lowercase is nonexistent (because I generally already know to capitalize them so I capitalize the abbreviation).

The only issue I really see with they provided is is that it doesn't handle pascal/camel case well (because the whole word is downcased). Honestly I never really have to use that case pattern, but for the sake of robustness and generalizability, I fixed it by swapping the (downcase w) from their suggestion to (concat (substring input 0 1) (substring w 1)).

With that switch it handles pascal/camel case properly by only replacing the case of the first letter (thus preserving intraword capitalization).

Of course if you wanted to make this more customizable you could wrap all this in conditionals with some configuration variable, but at least for dabbrev, I think this manages it basically exactly how I would expect with minimal extra code and configuration complexity.

---

Edit: noticed you asked about the modified behavior

The original suggestion downcases the entire completion when the completion starts with a non-lower case letter.

My modification replaces the first character of the completion with the first character of the input but leaves the rest of the word as is (lower).

Also, I didn't notice earlier but apparently their version doesn't check dabbrev-case-replace before doing that so that should be fixed.

I need to go now, but I'll fix that later and send it. (Sorry, I'm a bit busy with other stuff now, but I'll do it as soon as I have some time.)

[yuki-tsubaki]

@minad got some time again so here is what I have now.

Basically if (1) case folding and case replace are enabled (according to the value rules specified in the dabbrev-... variable documentation) and (2) the first character is lowercase, the first character of the abbreviation is used in place of the first character of the completion (effectively doing the case replace while preserving intraword capitalization

(defun cape--dabbrev-list (input)
    "Find all Dabbrev expansions for INPUT."
    (cape--silent
      (let ((dabbrev-check-other-buffers (not (null cape-dabbrev-check-other-buffers)))
	    (dabbrev-check-all-buffers (eq cape-dabbrev-check-other-buffers t)))
	(dabbrev--reset-global-variables))
      (cons
       (apply-partially #'string-prefix-p input)
       (cl-loop with min-len = (+ cape-dabbrev-min-length (length input))
		with ic = (cape--case-fold-p dabbrev-case-fold-search)
		;; Could be refactored into a `cape--case-replace-p' function
		with cr = (if (eq dabbrev-case-replace 'case-replace)
				       case-replace
				     dabbrev-case-replace)
		for w in (dabbrev--find-all-expansions input ic)
		if (>= (length w) min-len) collect
		(let ((dw (if (and cr
				   ic 
				   (not (let (case-fold-search) 
					  (string-match-p "[[:lower:]]" w))))
			      
			      w (concat (substring input 0 1) (substring w 1)))))
		  (cape--case-replace (and ic dabbrev-case-replace) input dw))))))

I did some basic benchmarking (not super good probably should have made a macro that gets stdev and the mean or repeats each time so I can do actual statistics but oh well).

;; Current version
;; All have 743 completion suggestions
;; My bench macro gives mean time as the result. I ran with 15 iterations
;; The numbers are in seconds
(bench 15
  (cape--dabbrev-list "t"))
6.809360726674398

(bench 15
  (cape--dabbrev-list "T"))
6.821476332346598

;; My version
(bench 15
  (cape--dabbrev-list-cr "t"))
6.812762673695882

(bench 15
  (cape--dabbrev-list-cr "T"))
6.77250452041626
6.798343912760417
;; I got this once before on my non-recorded one for another one so I
;; assume it is an outlier caused by some external factor
7.147141234079997

In general, it seems not to have a significant effect on performance (at least within cases that are noticeably long and thus would matter). In real usage (i.e. not completing one letter abbreviations with a 7.1M text buffer open) I have had no issues with the modified version.

Seeing multi-second completion times does show what you meant about dabbrev having very poor performance. If I get annoyed enough and have time sometime I might look into making a more performant replacement.

---

Edit:
Well, good thing I searched. Turns out there may already be someone who implemented exactly the things I would have tried to make dabbrev faster (i.e. idle time word finding and putting the completions into a good data-structure for fast access).

phillord/pabbrev: Predictive Abbreviation Expansion for Emacs

Looks promising, though I haven't actually tried it yet.

[minad]

Yes, I've also discussed with Dmitry Gutov about a Dabbrev replacement a few years back. It would be ideal to have a good reusable library based on a tree-based data structure. The tree would be populated in the background and could be used as Capf backend. That's the reason why I think we should not proceed to tweak Dabbrev further. Dabbrev is a dead end.