diff --git a/docs/datagrid_audit_setup.md b/docs/datagrid_audit_setup.md index 935472e3..78c30467 100644 --- a/docs/datagrid_audit_setup.md +++ b/docs/datagrid_audit_setup.md @@ -9,8 +9,8 @@ After installing the required Elastic components for Data Grid Audit, the integr > Please review the following important information before proceeding: -> * **For Existing Data Grid Audit Customers:** You must be on Elasticsearch 7.17 when initially running this setup. After the setup is complete, you can upgrade to Elasticsearch 8.17.3. -> * Before upgrading to Elasticsearch 8.17.3, the `ESIndexCreationSetting` may need to be updated. For details, refer to the [Instance setting Details](https://help.relativity.com/Server2024/Content/System_Guides/Instance_Setting_Guide/Instance_setting_descriptions.htm#ESIndexCreationSettings). +> * **For Existing Data Grid Audit Customers:** You must be on Elasticsearch 7.17 when initially running this setup. After the setup is complete, you can upgrade to Elasticsearch 8.x or 9.x. +> * Before upgrading to Elasticsearch 8.x or 9.x, the `ESIndexCreationSetting` may need to be updated. For details, refer to the [Instance setting Details](https://help.relativity.com/Server2024/Content/System_Guides/Instance_Setting_Guide/Instance_setting_descriptions.htm#ESIndexCreationSettings). > * Always verify the minimum required Elasticsearch version in your specific release bundle, as it may differ from the versions mentioned here. ### Prerequisites diff --git a/docs/elasticsearch-configurations/manual-elasticsearch-configuration.md b/docs/elasticsearch-configurations/manual-elasticsearch-configuration.md new file mode 100644 index 00000000..a223215d --- /dev/null +++ b/docs/elasticsearch-configurations/manual-elasticsearch-configuration.md @@ -0,0 +1,151 @@ +# Manual Elasticsearch Configuration: Index Field Limit + +## Overview + +This guide provides the required steps to update and optimize Elasticsearch settings. These adjustments help ensure that Elasticsearch is configured to handle the expected data volume, prevent indexing issues, and maintain reliable system performance. + +## Update the Index Field Limit + +Elasticsearch enforces a limit on the total number of fields that can exist within an index. This is controlled by the `index.mapping.total_fields.limit` setting, which helps protect the cluster from mapping explosions and excessive memory usage. + +When documents contain more fields than the value configured in `index.mapping.total_fields.limit`, Elasticsearch may reject the document or ignore additional dynamic fields depending on index settings, which can result in missing data or ingestion failures. This can lead to: + +- Missing or incomplete data +- Inconsistent query results +- Difficulty performing accurate analysis and aggregations +- Document ingestion failures + +To ensure all fields are properly indexed and stored, the `index.mapping.total_fields.limit` value must be increased to accommodate the actual number of fields present in the incoming documents. + +This documentation provides step-by-step instructions for manually updating Elasticsearch index settings to adjust the field limit and prevent data loss caused by exceeding the configured threshold. + +### Steps to Update the Index Field Limit + +### Prerequisites + +1. Administrator privileges in Kibana/Elasticsearch. + +#### Step 1: Update the Field Limit + +1. Log in to Kibana as a user with administrative privileges. +2. In Kibana, go to **Stack Management** → **Index Management** → **Component Templates**. +3. Use the search box to find the **apm@settings** component template. + ![Screenshot: apm@settings in Kibana](../../resources/post-install-verification-images/elasticsearch-index-settings/apm-settings.png) +4. Click the **Edit** button in the **Actions** column. +5. In the edit view, navigate to the **Index settings** section. + ![Screenshot: edit apm@settings in Kibana](../../resources/post-install-verification-images/elasticsearch-index-settings/index-settings.png) +6. Add or update the field limit setting: + - Key: `index.mapping.total_fields.limit` + - Value: the required limit (for example, `2500`) + + ![Screenshot: update limit in Kibana](../../resources/post-install-verification-images/elasticsearch-index-settings/apm-settings-edit-limit.png) + + > [!NOTE] + > - The default value of `index.mapping.total_fields.limit` is `1000`. The `limit` field is not included by default; for a first-time update, add the `limit` field and set it to the required value. + > - Increasing this limit excessively may increase memory usage and impact cluster stability, so it should be set slightly above the expected maximum number of fields. + +7. After updating the field limit, navigate to the **Review** tab and click **Save component template**. +8. Verify that the **apm@settings** component template shows the updated field limit. + ![Screenshot: verify limit in Kibana](../../resources/post-install-verification-images/elasticsearch-index-settings/apm-settings-updated-limit-verify.png) + + +#### Step 2: Perform a Rollover to Apply the New Settings + +> [!NOTE] +> A rollover is only required when APM data streams already contain existing backing indices. +In the case of a fresh installation, no data has been ingested yet, and the APM data streams have not created any backing indices. +Because of this, Elasticsearch will automatically create the first index using the updated component template settings, and no rollover action is needed. + +APM data streams consist of multiple backing indices (for example: metrics-apm-default-000123). +Updating the component template does not retroactively apply changes to these existing backing indices. + +Because the data stream already has active backing indices, the updated field limit will only apply to new indices created after the change. + +To apply the new settings, the following steps are required: +- Identify the correct APM data stream. +- Trigger a rollover to create a new backing index. +- Verify that the newly created index includes the updated limit. + + +1. Identify the APM Data Stream + 1. Go to **Discover**. + 2. Select the **Data view** associated with the logs/metrics/traces you are working with. + 3. Open one of the latest documents to expand its details. + 4. Locate the `_index` field. + + In the document details, find the value of the `_index` field. + Example: `.ds-logs-apm.app.relsvr_logging-default-2025.11.26-000013` + ![Screenshot: data stream in Kibana](../../resources/post-install-verification-images/elasticsearch-index-settings/latest-document.png) + + 5. Derive the data stream name + + Remove the `.ds-` prefix and the `date/suffix` portion from the index name. + + Using the example above: + `logs-apm.app.relsvr_logging-default` + + This is the data stream name to use with the rollover API. + +2. Perform a Rollover + + 1. In Kibana, navigate to **Management** → **Dev Tools** → **Console**. + 2. Run the rollover command in the console window. Paste the following API request (replace `` with the actual data stream name). + ``` + POST //_rollover + ``` + 3. Click the **Run** button (the arrow icon on the right side of the console input panel) to execute the request. In the response panel, ensure that the output contains: + + - `"rolled_over": true`, and + - The `"new_index"` field is present with the new backing index name. + +
+ Sample JSON response + + ``` + { + "acknowledged": true, + "shards_acknowledged": true, + "old_index": ".ds-logs-apm.app.relsvr_logging-default-2025.12.17-000002", + "new_index": ".ds-logs-apm.app.relsvr_logging-default-2025.12.17-000003", + "rolled_over": true, + "dry_run": false, + "lazy": false, + "conditions": {} + } + ``` +
+ + > [!NOTE] + > Ensure the data stream name is valid; otherwise, the rollover request fails. + +3. Verify the Updated Field Limit + + 1. In Kibana, navigate to **Management** → **Dev Tools** → **Console**. + 2. Run the GET command in the console window. Paste the following API request (replace `` with the `"new_index"` value returned by the rollover response). + ``` + GET //_settings + ``` + 3. Click the **Run** button (the arrow icon on the right side of the console input panel) to execute the request. In the response panel, ensure that the output contains: + + - `"index.mapping.total_fields.limit": `. + +
+ Sample JSON response + + ``` + { + ".ds-logs-apm.app.relsvr_logging-default-2025.12.17-000003": { + "settings": { + "index": { + "mapping": { + "total_fields": { + "limit": "2500", + "ignore_dynamic_beyond_limit": "true" + } + }, + } + } + } + } + ``` +
\ No newline at end of file diff --git a/docs/elasticsearch_pre_installation_overview.md b/docs/elasticsearch_pre_installation_overview.md index 8d5ca380..6f1bf3da 100644 --- a/docs/elasticsearch_pre_installation_overview.md +++ b/docs/elasticsearch_pre_installation_overview.md @@ -282,7 +282,7 @@ If you have used Elasticsearch for the optional Data Grid Audit feature on Relat https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry#enable-long-paths-in-windows-10-version-1607-and-later -3. **Verify the minimum supported version of Elastic** - All Relativity products have been certified with Elastic Stack 8.17.x, though Data Grid Audit alone is supported on 7.17.x. If you intend to use a single cluster for both, you must install the same version of Elasticsearch on all nodes in the cluster, and that version must be the higher of the minimum versions for Environment Watch and Data Grid Audit, if different. +3. **Verify the minimum supported version of Elastic** - All Relativity products have been certified with Elastic Stack 8.x or 9.x, though Data Grid Audit alone is supported on 7.17.x. If you intend to use a single cluster for both, you must install the same version of Elasticsearch on all nodes in the cluster, and that version must be the higher of the minimum versions for Environment Watch and Data Grid Audit, if different. 4. **At least the minimum Relativity major version and patch** specified in the Environment Watch bundle you intend to deploy is installed on all servers in the environment. See the [System Requirements](../README.md) for the minimum version required. diff --git a/docs/elasticsearch_setup_development.md b/docs/elasticsearch_setup_development.md index ce77490e..8e921749 100644 --- a/docs/elasticsearch_setup_development.md +++ b/docs/elasticsearch_setup_development.md @@ -14,16 +14,16 @@ If you download a .zip or other file from the internet, Windows may block the fi ![Unblock file screenshot](../resources/troubleshooting-images/unblocked.png) -### Step 1: Download and Install Elasticsearch 8.17.3 on one server -1. Download Elasticsearch 8.17.3 +### Step 1: Download and Install the Windows .zip package for Elasticsearch version 8.x or 9.x on one server +1. Download Elasticsearch 8.x or 9.x 1. Visit [Elastic’s official download page](https://www.elastic.co/downloads/elasticsearch). - 2. Download the 8.17.3 Windows .zip version. + 2. Download the 8.x or 9.x Windows .zip version. 3. Before extracting, see [How to Unblock Downloaded Files](#how-to-unblock-downloaded-files). 4. Extract the files to `C:\elastic` -2. Install and Configure Elasticsearch 8.17.3 +2. Install and Configure Elasticsearch 8.x or 9.x 1. Open an elevated PowerShell and run the following command to start Elasticsearch and perform the auto installation steps: ``` - C:\elastic\elasticsearch-8.17.3\bin\elasticsearch.bat + C:\elastic\elasticsearch-{version}\bin\elasticsearch.bat ``` @@ -42,7 +42,7 @@ If you download a .zip or other file from the internet, Windows may block the fi 3. Open an elevated PowerShell and run the following command to install Elasticsearch as a Windows service: ``` - C:\elastic\elasticsearch-8.17.3\bin\elasticsearch-service.bat install + C:\elastic\elasticsearch-{version}\bin\elasticsearch-service.bat install ``` The output will look similar to: ``` @@ -53,7 +53,7 @@ If you download a .zip or other file from the internet, Windows may block the fi 3. Run Elasticsearch as a Windows Service 1. Open an elevated PowerShell and run the following command to start the Elasticsearch service: ``` - C:\elastic\elasticsearch-8.17.3\bin\elasticsearch-service.bat start + C:\elastic\elasticsearch-{version}\bin\elasticsearch-service.bat start ``` The output will look similar to: ``` @@ -61,7 +61,7 @@ If you download a .zip or other file from the internet, Windows may block the fi The service 'elasticsearch-service-x64' has been started. ``` 4. Enable Stack Monitoring - 1. Navigate to the Elasticsearch configuration folder (e.g., `C:\elastic\elasticsearch-8.17.3\config`) and open the **elasticsearch.yml** file. + 1. Navigate to the Elasticsearch configuration folder (e.g., `C:\elastic\elasticsearch-{version}\config`) and open the **elasticsearch.yml** file. 2. Add the following line to enable Stack Monitoring: ``` xpack.monitoring.collection.enabled: true @@ -72,7 +72,7 @@ If you download a .zip or other file from the internet, Windows may block the fi ``` 5. Reset the Elastic (Admin) User Password 1. The following command resets the password for the `elastic` user, which is the default superuser (admin) account in Elasticsearch. This account is required for logging in to Kibana and for performing administrative tasks such as managing users, roles, and system settings. - 2. Navigate to ElasticSearch's bin folder(`C:\elastic\elasticsearch-8.17.3\bin`) + 2. Navigate to ElasticSearch's bin folder(`C:\elastic\elasticsearch-{version}\bin`) 3. Open an elevated PowerShell and run the following command: ``` .\elasticsearch-reset-password -u elastic @@ -88,11 +88,11 @@ If you download a .zip or other file from the internet, Windows may block the fi 6. Install the 'mapper-size' plugin 1. Open an elevated PowerShell and run the following command to install the 'mapper-size' plugin: ``` - C:\elastic\elasticsearch-8.17.3\bin\elasticsearch-plugin install mapper-size + C:\elastic\elasticsearch-{version}\bin\elasticsearch-plugin install mapper-size ``` 2. To verify the 'mapper-size' plugin is installed, run: ``` - C:\elastic\elasticsearch-8.17.3\bin\elasticsearch-plugin list + C:\elastic\elasticsearch-{version}\bin\elasticsearch-plugin list ``` 3. Restart the Elasticsearch Service. To restart the Elasticsearch service, run the following in an elevated PowerShell session: ``` @@ -100,7 +100,7 @@ If you download a .zip or other file from the internet, Windows may block the fi ``` The output will look similar to: ``` - WARNING: Waiting for service 'Elasticsearch 8.17.3 (elasticsearch-service-x64) (elasticsearch-service-x64)' to stop... + WARNING: Waiting for service 'Elasticsearch 8.x or 9.x (elasticsearch-service-x64) (elasticsearch-service-x64)' to stop... ``` 7. Verify Elasticsearch Server @@ -137,15 +137,15 @@ If you download a .zip or other file from the internet, Windows may block the fi ### Step 2: Install and Configure Kibana -1. Download Kibana 8.17.3 - 1. Download and extract the 8.17.3 Windows .zip version of Kibana from [Elastic’s official Kibana download page](https://www.elastic.co/downloads/kibana). +1. Download Kibana 8.x or 9.x + 1. Download and extract the 8.x or 9.x Windows .zip version of Kibana from [Elastic’s official Kibana download page](https://www.elastic.co/downloads/kibana). 2. Before extracting, see [How to Unblock Downloaded Files](#how-to-unblock-downloaded-files). 2. Start Kibana from the command line 1. Navigate to Kibana's `bin` folder (e.g., `C:\elastic\kibana\bin`). 2. Open an elevated PowerShell and run the following command: ``` - C:\Kibana\kibana-8.17.3\bin\kibana.bat + C:\Kibana\kibana-{version}\bin\kibana.bat ``` 3. If successful, you should see output indicating that the Kibana server has started and is listening on port 5601. Look for lines similar to: ``` @@ -158,9 +158,9 @@ If you download a .zip or other file from the internet, Windows may block the fi 1. In your terminal, click the generated link to open Kibana in your browser. 2. In your browser, paste the enrollment token that was generated in the terminal when you started Elasticsearch, then click the Configure Elastic button to connect your Kibana instance with Elasticsearch. [See where the enrollment token is generated.](#enrollment-token-generation) - 3. If the token has expired, generate a new one by running the following command in the Elasticsearch's bin folder (e.g., `C:\elastic\elasticsearch-8.17.3\bin`). + 3. If the token has expired, generate a new one by running the following command in the Elasticsearch's bin folder (e.g., `C:\elastic\elasticsearch-{version}\bin`). ``` - C:\elastic\elasticsearch-8.17.3\bin\elasticsearch-create-enrollment-token --scope kibana + C:\elastic\elasticsearch-{version}\bin\elasticsearch-create-enrollment-token --scope kibana ```
Sample output @@ -177,7 +177,7 @@ If you download a .zip or other file from the internet, Windows may block the fi 1. Open an elevated PowerShell and run the following command: ``` - C:\Kibana\kibana-8.17.3\bin\kibana-encryption-keys generate + C:\Kibana\kibana-{version}\bin\kibana-encryption-keys generate ``` 2. If successful, you will see output showing the generated encryption keys. For example: @@ -193,7 +193,7 @@ If you download a .zip or other file from the internet, Windows may block the fi
- 3. Copy the generated encryption keys and paste them at the end of your `kibana.yml` file (e.g., `C:\Kibana\kibana-8.17.3\config\kibana.yml`). The configuration should look similar to: + 3. Copy the generated encryption keys and paste them at the end of your `kibana.yml` file (e.g., `C:\Kibana\kibana-{version}\config\kibana.yml`). The configuration should look similar to:
Sample kibana.yml configuration @@ -208,7 +208,7 @@ If you download a .zip or other file from the internet, Windows may block the fi 4. Restart the Kibana service, by opening an elevated PowerShell and run the following command: ``` - C:\Kibana\kibana-8.17.3\bin\kibana.bat + C:\Kibana\kibana-{version}\bin\kibana.bat ``` 5. To verify success, check the terminal output for lines indicating that Kibana has started successfully. You can also refer to the screenshots below: @@ -229,7 +229,7 @@ If you download a .zip or other file from the internet, Windows may block the fi > Only use NSSM if you want Kibana to start automatically as a service on Windows. If you do not wish to use NSSM, simply run `kibana.bat` manually. > > ``` - > C:\Kibana\kibana-8.17.3\bin\kibana.bat + > C:\Kibana\kibana-{version}\bin\kibana.bat > ``` 1. Download the latest NSSM executable from https://nssm.cc/download and place it in the C drive (e.g., `C:\nssm-2.24`). @@ -253,7 +253,7 @@ If you download a .zip or other file from the internet, Windows may block the fi > If you accidentally install the Kibana service before completing your configuration (for example, by pressing Return too early in the NSSM dialog), you can easily edit the service properties afterward. This allows you to update the application path, log file settings, or other options without reinstalling the service. > To edit the service properties, open an elevated PowerShell and run the following command: `C:\nssm-2.24\win64\nssm.exe edit kibana` - 4. In the I/O tab, enter the full path of a log file where the service logs will be stored. For example, create a folder in the Kibana directory (e.g., `C:\Kibana\kibana-8.17.3\service_logs`) and a blank log file (e.g., `C:\Kibana\kibana-8.17.3\service_logs\kibana_service.log`). + 4. In the I/O tab, enter the full path of a log file where the service logs will be stored. For example, create a folder in the Kibana directory (e.g., `C:\Kibana\kibana-{version}\service_logs`) and a blank log file (e.g., `C:\Kibana\kibana-{version}\service_logs\kibana_service.log`). 5. Copy the full log file path into the stdout and stderr sections: @@ -285,13 +285,13 @@ If you download a .zip or other file from the internet, Windows may block the fi 1. Prerequisites to setup APM Server - Elastic and Kibana should be configured and services should be up and running. -2. Download APM Server 8.17.3 +2. Download APM Server 8.x or 9.x 1. Visit [Elastic’s APM Server page](https://www.elastic.co/downloads/apm). - 2. Download and extract the 8.17.3 Windows .zip file. + 2. Download and extract the 8.x or 9.x Windows .zip file. 3. Before extracting, see [How to Unblock Downloaded Files](#how-to-unblock-downloaded-files). 4. Extract the files to `C:\`. -3. Configure APM Server (`C:\apm-server-8.17.3-windows-x86_64\apm-server.yml`) +3. Configure APM Server (`C:\apm-server-{version}-windows-x86_64\apm-server.yml`) 1. An API key is required for configuring both APM and Beats. To create an API key: 1. Log in to Kibana (`http://:5601`) using the `elastic` credential. @@ -306,7 +306,7 @@ If you download a .zip or other file from the internet, Windows may block the fi > Copy and save `id` and `api_key` values immediately and store them securely according to your organization’s credential management and security policies. - 2. Navigate to the apm-server folder (e.g., `C:\apm-server-8.17.3-windows-x86_64`) and open the `apm-server.yml` file using a text editor. + 2. Navigate to the apm-server folder (e.g., `C:\apm-server-{version}-windows-x86_64`) and open the `apm-server.yml` file using a text editor. 3. Update the `apm-server.yml` file to match the following sample configuration. Replace the placeholder values (``, ``, ``, ``) as needed for your environment. Below is a sample configuration: @@ -333,7 +333,7 @@ If you download a .zip or other file from the internet, Windows may block the fi 2. Run the following command to install the APM Server as a Windows service: ``` - PowerShell.exe -ExecutionPolicy UnRestricted -File C:\apm-server-8.17.3-windows-x86_64\install-service.ps1 + PowerShell.exe -ExecutionPolicy UnRestricted -File C:\apm-server-{version}-windows-x86_64\install-service.ps1 ``` The output will look similar to: diff --git a/docs/environment-watch/post-install-verification/elasticsearch-cluster-health.md b/docs/environment-watch/post-install-verification/elasticsearch-cluster-health.md index 4324ab7c..ff3657bb 100644 --- a/docs/environment-watch/post-install-verification/elasticsearch-cluster-health.md +++ b/docs/environment-watch/post-install-verification/elasticsearch-cluster-health.md @@ -33,12 +33,14 @@ Confirm cluster health status, total nodes, shards, indices, and document count - Data size. **Expected Result:** -- Cluster health is **Green (Healthy)**. +- Under any Data Store Dashboard, click on Cluster Overview link. +- Verify cluster health is **Green (Healthy)**. - Nodes, shards, indices, and documents display current, non-zero values. - No unassigned shards. - Data size is displayed accurately. **Screenshot:** +![Screenshot: Cluster Overview](../../../resources/post-install-verification-images/elasticsearch-cluster-health/cluster-overview.png) ![Screenshot: Cluster Health Summary](../../../resources/post-install-verification-images/elasticsearch-cluster-health/cluster-health-summary.png) diff --git a/docs/environment_watch_install_other_integrations.md b/docs/environment_watch_install_other_integrations.md index 2182d301..dad18e0f 100644 --- a/docs/environment_watch_install_other_integrations.md +++ b/docs/environment_watch_install_other_integrations.md @@ -21,6 +21,12 @@ The RabbitMQ integration enables monitoring of RabbitMQ queues, exchanges, and n - [Setting Up RabbitMQ Integration](rabbitmq/rabbitmq_integration.md) +### Manual Elasticsearch Configuration + +The Manual Elasticsearch Configuration helps to update and optimize essential Elasticsearch settings. + +- [Setting Up Elasticsearch Configuration](elasticsearch-configurations/manual-elasticsearch-configuration.md) + ## Next Step [Click here for the next step](./environment-watch/post-install-verification.md) \ No newline at end of file diff --git a/docs/iis_log_configuration.md b/docs/iis_log_configuration.md new file mode 100644 index 00000000..a4609c36 --- /dev/null +++ b/docs/iis_log_configuration.md @@ -0,0 +1,374 @@ +# IIS Log Field Configuration for Environment Watch + +## Overview + +This document provides step-by-step instructions for configuring IIS log fields required for proper log ingestion by Environment Watch. Incorrect or incomplete IIS log field configuration can result in parsing errors, incomplete data, or ingestion failures. + +The Environment Watch monitoring agent uses OpenTelemetry (OTEL) collectors to parse and ingest IIS logs. The configuration requires specific fields to be enabled in IIS logging to ensure successful data collection and analysis. + +## Prerequisites + +- IIS installed on the Server +- Administrative access to the IIS Manager +- Environment Watch monitoring agent installed (see [Install Environment Watch Monitoring Agents](install_environment_watch_monitoring_agents.md)) + +## Terminology and Abbreviations + +The following abbreviations are used throughout this document: + +| Abbreviation | Description | +|-------------|-------------| +| IIS | Internet Information Services | +| OTEL | OpenTelemetry | +| HTTP | Hypertext Transfer Protocol | +| URI | Uniform Resource Identifier | +| IP | Internet Protocol | +| W3C | World Wide Web Consortium | +| Win32 | Windows API status code | + +## Required IIS Log Fields + +The following fields **must** be enabled in your IIS log configuration for successful ingestion. These fields are configured in IIS Manager under W3C Logging Fields. + +| Field Name | IIS Log Field | Description | Required | +|------------|---------------|-------------|----------| +| `date` | date | Date when the log entry was created | Yes | +| `time` | time | Time when the log entry was created | Yes | +| `s-ip` | s-ip | Server IP address | Yes | +| `cs-method` | cs-method | HTTP request method (GET, POST, etc.) | Yes | +| `cs-uri-stem` | cs-uri-stem | URI stem (path) of the request | Yes | +| `cs-uri-query` | cs-uri-query | Query string parameters | Yes | +| `s-port` | s-port | Server port number | Yes | +| `cs-username` | cs-username | Authenticated username | Yes | +| `c-ip` | c-ip | Client IP address | Yes | +| `cs(User-Agent)` | cs(User-Agent) | User agent string from the client | Yes | +| `cs(Referer)` | cs(Referer) | Referer URL | Yes | +| `sc-status` | sc-status | HTTP status code | Yes | +| `sc-substatus` | sc-substatus | HTTP substatus code | Yes | +| `sc-win32-status` | sc-win32-status | Windows status code | Yes | +| `time-taken` | time-taken | Time taken to process the request (milliseconds) | Yes | + +## Configuration Steps + +### Step 1: Open IIS Manager + +1. Press **Win + R** to open the Run dialog +2. Type `inetmgr` and press **Enter** +3. The Internet Information Services (IIS) Manager will open + +### Step 2: Navigate to Logging Configuration + +1. In the left-hand **Connections** pane, expand your server node +2. Select the **Sites** folder or a specific website you want to configure +3. In the center **Features View** pane, double-click **Logging** + + > [!NOTE] + > You can configure logging at the server level (applies to all sites) or at the individual site level. For Environment Watch, configure at the server level to ensure consistent logging across all Relativity web applications. + +### Step 3: Configure Log File Format + +1. In the **Logging** configuration page, ensure the **Format** dropdown is set to **W3C** +2. Click the **Select Fields...** button next to the Format dropdown + +### Step 4: Select Required Fields + +1. In the **W3C Logging Fields** dialog, ensure the following fields are checked: + + **Standard Fields:** + - ☑ **date** (Date) + - ☑ **time** (Time) + - ☑ **s-ip** (Server IP Address) + - ☑ **cs-method** (Method) + - ☑ **cs-uri-stem** (URI Stem) + - ☑ **cs-uri-query** (URI Query) + - ☑ **s-port** (Server Port) + - ☑ **cs-username** (User Name) + - ☑ **c-ip** (Client IP Address) + - ☑ **sc-status** (Protocol Status) + - ☑ **sc-substatus** (Protocol Substatus) + - ☑ **sc-win32-status** (Win32 Status) + - ☑ **time-taken** (Time Taken) + + **Custom Fields (expand the sections as needed):** + - ☑ **cs(User-Agent)** - Located under "Request Headers" + - ☑ **cs(Referer)** - Located under "Request Headers" + +2. Verify that all required fields are selected +3. Click **OK** to close the dialog + +### Step 5: Configure Log File Location (Optional) + +1. Note the **Directory** path where IIS logs are stored (default: `%SystemDrive%\inetpub\logs\LogFiles`) +2. This path must match the `OTEL_IIS_FILE_LOG_PATH` environment variable configured during monitoring agent installation +3. If you need to change the log directory: + - Update the **Directory** field in IIS Manager + - Update the `OTEL_IIS_FILE_LOG_PATH` environment variable to match + - Restart the Relativity Environment Watch service + + > [!IMPORTANT] + > The default log path pattern should be: `C:\inetpub\logs\LogFiles\W3SVC*\*.log` + +### Step 6: Apply Configuration + +1. Click **Apply** in the **Actions** pane on the right +2. The changes will take effect immediately for new log entries +3. Existing log files will retain their previous format + + > [!NOTE] + > You do not need to restart IIS for logging configuration changes to take effect. However, the new format will only apply to new log entries. + +### Step 7: Verify Log Format + +1. Navigate to your IIS log directory (e.g., `C:\inetpub\logs\LogFiles\W3SVC1\`) +2. Open the most recent log file with a text editor (e.g., Notepad) +3. Verify the `#Fields:` header line matches the required format: + + ``` + #Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken + ``` + +5. Verify that log entries contain all the required fields (example): + + ``` + 2025-06-10 22:14:03 ::1 POST /Relativity/Identity/connect/token - 443 - ::1 - - 200 0 0 25951 + ``` + +## OpenTelemetry Configuration Overview + +The Environment Watch monitoring agent uses the following OTEL configuration to parse IIS logs. This configuration is automatically deployed during installation. + +**Configuration File Location:** +``` +C:\ProgramData\Relativity\EnvironmentWatch\Services\InfraWatchAgent\Templates\otelcol-config-rel-web.yaml +``` + +> [!NOTE] +> This is a template file used by the monitoring agent. Do not modify these files directly as they are managed by the Environment Watch service. + +### File Log Receiver Configuration + +```yaml +receivers: + filelog/iis: + include: "${env:OTEL_IIS_FILE_LOG_PATH}" + include_file_name: true + start_at: beginning + operators: + # Filter out comment lines starting with # + - type: filter + expr: 'body matches "^#"' + drop: true + # Parse log entries using regex + - type: regex_parser + regex: ^(?P\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})\s+(?P\S+)\s+(?P\S+)\s+(?P\S+)\s+(?P\S*)\s+(?P\d+)\s+(?P\S*)\s+(?P\S+)\s+(?P.*?)\s+(?P.*?)\s+(?P\d+)\s+(?P\d+)\s+(?P\d+)\s+(?P\d+) +``` + +### Understanding the Regex Pattern + +The regex pattern above is designed to parse IIS log entries in W3C format. Here's a breakdown of each component: + +#### Regex Components Explained + +| Regex Component | Description | Matches | Example | +|----------------|-------------|---------|---------| +| `^` | Start of line anchor | Beginning of the log entry | - | +| `(?Ppattern)` | Named capture group | Creates a field with the specified name | - | +| `\d{4}-\d{2}-\d{2}` | Date format | YYYY-MM-DD | `2025-06-10` | +| `\d{2}:\d{2}:\d{2}` | Time format | HH:MM:SS | `22:14:03` | +| `\s+` | One or more whitespace | Separates fields | Space or tab | +| `\S+` | One or more non-whitespace | Required field value | `POST`, `::1` | +| `\S*` | Zero or more non-whitespace | Optional field value (can be `-`) | `-` or `param=value` | +| `.*?` | Non-greedy match any character | User-Agent or Referer (may contain spaces) | `Mozilla/5.0...` | +| `\d+` | One or more digits | Numeric values | `443`, `200`, `25951` | + +#### Field-by-Field Breakdown + +**Example log entry:** +``` +2025-06-10 22:14:03 ::1 POST /Relativity/Identity/connect/token - 443 - ::1 - - 200 0 0 25951 +``` + +**Regex pattern matched to fields:** + +1. **Date and Time** - `(?P\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})` + - Pattern: `\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}` + - Matches: `2025-06-10 22:14:03` + - Explanation: Four digits (year), dash, two digits (month), dash, two digits (day), space, time in HH:MM:SS format + +2. **Server IP** - `(?P\S+)` + - Pattern: `\S+` (one or more non-whitespace) + - Matches: `::1` (IPv6 localhost) + - Explanation: Captures any non-whitespace characters (IP address or hostname) + +3. **HTTP Method** - `(?P\S+)` + - Pattern: `\S+` + - Matches: `POST` + - Explanation: HTTP verb (GET, POST, PUT, DELETE, etc.) + +4. **URI Stem** - `(?P\S+)` + - Pattern: `\S+` + - Matches: `/Relativity/Identity/connect/token` + - Explanation: The requested URL path without query parameters + +5. **Query String** - `(?P\S*)` + - Pattern: `\S*` (zero or more non-whitespace) + - Matches: `-` (indicates no query string) + - Explanation: Query parameters after `?` in URL, or `-` if none + +6. **Server Port** - `(?P\d+)` + - Pattern: `\d+` (one or more digits) + - Matches: `443` + - Explanation: Port number the server received the request on + +7. **Username** - `(?P\S*)` + - Pattern: `\S*` (zero or more non-whitespace) + - Matches: `-` (indicates anonymous/no authentication) + - Explanation: Authenticated username, or `-` if not authenticated + +8. **Client IP** - `(?P\S+)` + - Pattern: `\S+` + - Matches: `::1` (IPv6 localhost) + - Explanation: IP address of the client making the request + +9. **User-Agent** - `(?P.*?)` + - Pattern: `.*?` (non-greedy match any character) + - Matches: `-` (or full User-Agent string like `Mozilla/5.0...`) + - Explanation: Browser/client identification string; `.*?` allows spaces, stops at next whitespace boundary + +10. **Referer** - `(?P.*?)` + - Pattern: `.*?` (non-greedy match any character) + - Matches: `-` (or full Referer URL) + - Explanation: The page that linked to this request; `.*?` allows spaces + +11. **HTTP Status** - `(?P\d+)` + - Pattern: `\d+` + - Matches: `200` + - Explanation: HTTP response status code (200, 404, 500, etc.) + +12. **HTTP Substatus** - `(?P\d+)` + - Pattern: `\d+` + - Matches: `0` + - Explanation: IIS-specific substatus code providing additional detail + +13. **Win32 Status** - `(?P\d+)` + - Pattern: `\d+` + - Matches: `0` + - Explanation: Windows system error code (0 = success) + +14. **Time Taken** - `(?P\d+)` + - Pattern: `\d+` + - Matches: `25951` + - Explanation: Request processing time in milliseconds + + +> [!IMPORTANT] +> The regex pattern **must match the exact field order** in the IIS log `#Fields:` header. If fields are reordered or added/removed in IIS configuration, the regex will fail to parse correctly. + +### Field Mapping + +The OTEL collector maps IIS log fields to the following internal field names: + +| IIS Field | OTEL Field Name | +|-----------|-----------------| +| date time | `iis_log_date_time` | +| s-ip | `iis_log_hostname` | +| cs-method | `iis_log_http_request_method` | +| cs-uri-stem | `iis_log_http_target` | +| cs-uri-query | `iis_log_http_query` | +| s-port | `iis_log_http_server_port` | +| cs-username | `iis_log_end_username` | +| c-ip | `iis_log_http_client_address` | +| cs(User-Agent) | `iis_log_cs_useragent` | +| cs(Referer) | `iis_log_cs_referer` | +| sc-status | `iis_log_http_server_status` | +| sc-substatus | `iis_log_http_server_substatus` | +| sc-win32-status | `iis_log_http_sc_win32_status` | +| time-taken | `iis_log_http_server_duration` | + +#### Visual Parsing Example + +Here's how the regex pattern maps to an actual IIS log entry: + +``` +Log Entry: +2025-06-10 22:14:03 ::1 POST /Relativity/Identity/connect/token - 443 - ::1 - - 200 0 0 25951 + +Field Mapping: +├─ 2025-06-10 22:14:03 → iis_log_date_time +├─ ::1 → iis_log_hostname +├─ POST → iis_log_http_request_method +├─ /Relativity/Identity/... → iis_log_http_target +├─ - → iis_log_http_query (empty) +├─ 443 → iis_log_http_server_port +├─ - → iis_log_end_username (anonymous) +├─ ::1 → iis_log_http_client_address +├─ - → iis_log_cs_useragent (empty) +├─ - → iis_log_cs_referer (empty) +├─ 200 → iis_log_http_server_status +├─ 0 → iis_log_http_server_substatus +├─ 0 → iis_log_http_sc_win32_status +└─ 25951 → iis_log_http_server_duration (25.951 seconds) +``` + +### Processors + +The configuration includes a processor to convert the `time-taken` field to an integer type: + +```yaml +processors: + attributes/iis_log_http_server_duration: + actions: + - key: iis_log_http_server_duration + action: convert + converted_type: int +``` + +## Verification in Kibana + +After configuring IIS logging and ensuring the Environment Watch monitoring agent is running, verify that logs are being ingested into Elasticsearch: + +### Step 1: Log into Kibana + +1. Open your web browser and navigate to Kibana (default: `https://:5601`) +2. Log in with your Elasticsearch credentials + +### Step 2: Navigate to Discover + +1. Click on the **Discover** menu item in the left navigation panel +2. Select data view logs-* + +### Step 3: Set the Time Range + +1. In the top right corner, click the time filter +2. Select **Last 15 minutes** or **Last 1 hour** to view recent logs +3. Click **Apply** + +### Step 4: Search for IIS Logs + +In the search bar, enter the following query to filter for IIS logs: + +``` +labels.iis_log_http_request_method:* +``` + +Or search for a specific field: + +``` +labels.iis_log_http_target:"/Relativity*" +``` + +### Step 5: Verify Fields are Populated + +1. Expand one of the log entries by clicking the arrow icon + + ![IIS Log Details Arrow](../resources/iis_log_details_arrow.png) + +2. Verify that all IIS log fields are present and populated: + - `labels.iis_log_date_time` + - `labels.iis_log_hostname` + - `labels.iis_log_http_request_method` + - `labels.iis_log_http_target` + - `labels.iis_log_http_server_status` + - `labels.iis_log_http_server_duration` and all other mapped fields + + ![IIS Log Field Mapping](../resources/iis__log_field_mapping.png) diff --git a/docs/troubleshooting/elasticsearch.md b/docs/troubleshooting/elasticsearch.md index 26f2295e..ba34a5df 100644 --- a/docs/troubleshooting/elasticsearch.md +++ b/docs/troubleshooting/elasticsearch.md @@ -73,7 +73,7 @@ This document provides troubleshooting guidance for common Elasticsearch issues ``` 3. Check Elasticsearch Logs: - 1. Navigate to the log directory (default: `C:\elastic\elasticsearch-8.17.3\logs\`). + 1. Navigate to the log directory (default: `C:\elastic\elasticsearch-{version}\logs\`). 2. Review the Elasticsearch log file (`elasticsearch.log`) for error messages. 3. Check the slow logs and garbage collection logs if present. 4. For every error in the Elasticsearch log, provide troubleshooting for that specific error. diff --git a/docs/troubleshooting/relativity-server-cli.md b/docs/troubleshooting/relativity-server-cli.md index 8e46546e..823e9e55 100644 --- a/docs/troubleshooting/relativity-server-cli.md +++ b/docs/troubleshooting/relativity-server-cli.md @@ -151,4 +151,17 @@ This section covers common errors encountered during the Environment Watch and D relsvr.exe setup ``` +### Incorrect Invariant Server Name + +**Symptoms:** +- The CLI reports an error during the toggle stage due to an incorrect Invariant server name in the instance settings. + + ![Invariant Server Instance Settings Error](../../resources/ew-cli-toggle-stage-error.png) + +**Troubleshooting Steps:** +1. **Verify Invariant Server Name Instance Settings in Relativity:** + 1. Sign in to the Relativity web UI as an administrator. + 2. Navigate to **Instance Settings**, and update the `InvariantServerName` setting to match the Invariant Server. + + For full setup instructions, see [Relativity_Server_CLI Setup](../relativity_server_cli_setup.md). \ No newline at end of file diff --git a/resources/Installer_hostmetric.png b/resources/Installer_hostmetric.png index dd13eb5e..41ba7d16 100644 Binary files a/resources/Installer_hostmetric.png and b/resources/Installer_hostmetric.png differ diff --git a/resources/environment_watch_product_overview_008.png b/resources/environment_watch_product_overview_008.png index 57780553..41ba7d16 100644 Binary files a/resources/environment_watch_product_overview_008.png and b/resources/environment_watch_product_overview_008.png differ diff --git a/resources/ew-cli-toggle-stage-error.png b/resources/ew-cli-toggle-stage-error.png new file mode 100644 index 00000000..c2a90687 Binary files /dev/null and b/resources/ew-cli-toggle-stage-error.png differ diff --git a/resources/iis__log_field_mapping.png b/resources/iis__log_field_mapping.png new file mode 100644 index 00000000..54346f96 Binary files /dev/null and b/resources/iis__log_field_mapping.png differ diff --git a/resources/iis_log_details_arrow.png b/resources/iis_log_details_arrow.png new file mode 100644 index 00000000..2ad8b98d Binary files /dev/null and b/resources/iis_log_details_arrow.png differ diff --git a/resources/post-install-verification-images/elasticsearch-cluster-health/cluster-overview.png b/resources/post-install-verification-images/elasticsearch-cluster-health/cluster-overview.png new file mode 100644 index 00000000..b7d148ce Binary files /dev/null and b/resources/post-install-verification-images/elasticsearch-cluster-health/cluster-overview.png differ diff --git a/resources/post-install-verification-images/elasticsearch-cluster-health/monitoring-agent.png b/resources/post-install-verification-images/elasticsearch-cluster-health/monitoring-agent.png new file mode 100644 index 00000000..7e97c440 Binary files /dev/null and b/resources/post-install-verification-images/elasticsearch-cluster-health/monitoring-agent.png differ diff --git a/resources/post-install-verification-images/elasticsearch-index-settings/apm-settings-edit-limit.png b/resources/post-install-verification-images/elasticsearch-index-settings/apm-settings-edit-limit.png new file mode 100644 index 00000000..b6e87a3a Binary files /dev/null and b/resources/post-install-verification-images/elasticsearch-index-settings/apm-settings-edit-limit.png differ diff --git a/resources/post-install-verification-images/elasticsearch-index-settings/apm-settings-updated-limit-verify.png b/resources/post-install-verification-images/elasticsearch-index-settings/apm-settings-updated-limit-verify.png new file mode 100644 index 00000000..a2f39edc Binary files /dev/null and b/resources/post-install-verification-images/elasticsearch-index-settings/apm-settings-updated-limit-verify.png differ diff --git a/resources/post-install-verification-images/elasticsearch-index-settings/apm-settings.png b/resources/post-install-verification-images/elasticsearch-index-settings/apm-settings.png new file mode 100644 index 00000000..84d6c364 Binary files /dev/null and b/resources/post-install-verification-images/elasticsearch-index-settings/apm-settings.png differ diff --git a/resources/post-install-verification-images/elasticsearch-index-settings/index-settings.png b/resources/post-install-verification-images/elasticsearch-index-settings/index-settings.png new file mode 100644 index 00000000..77acd606 Binary files /dev/null and b/resources/post-install-verification-images/elasticsearch-index-settings/index-settings.png differ diff --git a/resources/post-install-verification-images/elasticsearch-index-settings/latest-document.png b/resources/post-install-verification-images/elasticsearch-index-settings/latest-document.png new file mode 100644 index 00000000..50144971 Binary files /dev/null and b/resources/post-install-verification-images/elasticsearch-index-settings/latest-document.png differ diff --git a/resources/post-install-verification-images/monitoring-agents/data-populated.png b/resources/post-install-verification-images/monitoring-agents/data-populated.png index c8ee8388..68dd17c9 100644 Binary files a/resources/post-install-verification-images/monitoring-agents/data-populated.png and b/resources/post-install-verification-images/monitoring-agents/data-populated.png differ diff --git a/resources/post-install-verification-images/monitoring-agents/explore-discover-view.png b/resources/post-install-verification-images/monitoring-agents/explore-discover-view.png index 7d412cd0..0fed8778 100644 Binary files a/resources/post-install-verification-images/monitoring-agents/explore-discover-view.png and b/resources/post-install-verification-images/monitoring-agents/explore-discover-view.png differ diff --git a/resources/post-install-verification-images/monitoring-agents/filter-dropdown-populated.png b/resources/post-install-verification-images/monitoring-agents/filter-dropdown-populated.png index 685f19bc..6f51c036 100644 Binary files a/resources/post-install-verification-images/monitoring-agents/filter-dropdown-populated.png and b/resources/post-install-verification-images/monitoring-agents/filter-dropdown-populated.png differ diff --git a/resources/post-install-verification-images/monitoring-agents/hosts-agent-versions.png b/resources/post-install-verification-images/monitoring-agents/hosts-agent-versions.png index f4f1a2bf..0097728d 100644 Binary files a/resources/post-install-verification-images/monitoring-agents/hosts-agent-versions.png and b/resources/post-install-verification-images/monitoring-agents/hosts-agent-versions.png differ diff --git a/resources/relativity_alerts_verification_003_003.png b/resources/relativity_alerts_verification_003_003.png index 29c99f53..fe10d747 100644 Binary files a/resources/relativity_alerts_verification_003_003.png and b/resources/relativity_alerts_verification_003_003.png differ diff --git a/resources/troubleshooting-images/unblocked.png b/resources/troubleshooting-images/unblocked.png index f566ae72..59cb2542 100644 Binary files a/resources/troubleshooting-images/unblocked.png and b/resources/troubleshooting-images/unblocked.png differ