Refactor course-definition.yml token definitions and documentation

Author: rohitpaulk

course-definition.yml

@@ -177,9 +177,10 @@ stages:
         - Examples: `VAR`, `IDENTIFIER`, `STRING`, `EOF` etc.
       - `<lexeme>`: The actual sequence of characters that formed the token.
         - Examples: `var`, `breakfast`, `"bagels"` etc.
-        - This is an empty string for the `EOF` token.
+        - For an `EOF` token, the lexeme is an empty string.
       - `<literal>`: The literal value of the token.
-        - For most tokens this is `null`, it only holds a value for `STRING` / `NUMBER` tokens.
+        - For most tokens this is `null`.
+        - For `STRING`/`NUMBER` tokens, it holds the value of the string/number.
 
       The `EOF` token is a special token that represents the end of the file. All calls to `tokenize` will include an `EOF` token at the end.