You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: content/operate/rc/databases/rdi/_index.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -75,7 +75,7 @@ Before you can create a data pipeline, you must have:
75
75
Please be aware of the following limitations:
76
76
77
77
- The target database must be a Redis Cloud Pro database hosted on Amazon Web Services (AWS). Redis Cloud Essentials databases and databases hosted on Google Cloud do not support Data Integration.
78
-
- The target database must use multi-zone [high availability]({{< relref "/operate/rc/databases/configuration/high-availability" >}}).
78
+
- The target database must use [high availability]({{< relref "/operate/rc/databases/configuration/high-availability" >}}). It can use either single-zone or multi-zone high availability.
79
79
- The target database can use TLS, but can not use mutual TLS.
80
80
- The target database cannot be in the same subscription as another database that has a data pipeline.
Copy file name to clipboardExpand all lines: content/operate/rc/databases/rdi/define.md
+12-6Lines changed: 12 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -36,6 +36,9 @@ After you have [prepared your source database]({{<relref "/operate/rc/databases/
36
36
1. If your database requires mTLS, select **Use mTLS**. Enter the ARN of your [Client certificate secret]({{< relref "/operate/rc/databases/rdi/setup#create-database-credentials-secrets" >}}) in the **Client Certificate Secret ARN** field and the ARN of your [Client key secret]({{< relref "/operate/rc/databases/rdi/setup#create-database-credentials-secrets" >}}) in the **Client Key Secret ARN** field.
37
37
{{<imagefilename="images/rc/rdi/rdi-define-mtls.png"alt="The Source database connectivity section, with Use TLS selected and the Client Certificate Secret ARN and Client Key Secret ARN fields." >}}
38
38
1. If your database requires mTLS with a client key passphrase, enter the ARN of your [Client key passphrase secret]({{< relref "/operate/rc/databases/rdi/setup#create-database-credentials-secrets" >}}) in the **Please add a secret ARN for the password to use with the secret store** field.
39
+
1. Select **Advanced properties** to configure additional optional properties for your pipeline.
You can add any [Debezium source property](https://debezium.io/documentation/reference/stable/connectors/) for your source database type in the **Collector source properties** section and any [Redis server Debezium sink property](https://debezium.io/documentation/reference/stable/operations/debezium-server.html#_redis_stream) in the **Collector sink properties** section.
1. Redis Cloud will attempt to connect to PrivateLink. If your PrivateLink does not allow automatic acceptance of incoming connections, accept the incoming connection on AWS PrivateLink to proceed. See [Accept or Reject PrivateLink connection requests](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html#accept-reject-connection-requests).
@@ -73,10 +76,6 @@ After your pipeline is provisioned, you will be able to define your pipeline. Yo
73
76
74
77
If any tables are missing a unique constraint, a warning will appear in the **Data modeling** section. Select **Manage columns** to select the columns that define a unique constraint for those tables.
1. Select the Redis data type to write keys to the target. You can choose **Hash key** or **JSON key** if the target database supports JSON.
96
+
{{<imagefilename="images/rc/rdi/rdi-configure-new-pipeline.png"alt="The data modeling section, with Hash Key selected."width=75% >}}
98
97
99
98
You can also supply one or more [transformation job files]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples" >}}) that specify how you want to transform the captured data before writing it to the target. Select **Upload jobs** to upload your job files.
100
99
@@ -105,6 +104,13 @@ After your pipeline is provisioned, you will be able to define your pipeline. Yo
1. In the **Advanced properties** section, you can add any processor properties to control how the data is processed. See the [RDI configuration file reference]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#processors" >}}) for all available processor properties.
108
+
109
+
{{<imagefilename="images/rc/rdi/rdi-processor-properties.png"alt="The advanced properties section with processor properties." >}}
1. Review the tables you selected in the **Review and deploy** section. If everything looks correct, select **Confirm & Deploy** to start ingesting data from your source database.
Copy file name to clipboardExpand all lines: content/operate/rc/databases/rdi/view-edit.md
+11-8Lines changed: 11 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -20,10 +20,10 @@ The **Status** table shows statistics for the whole data pipeline:
20
20
-**Status**: The status of the data pipeline. Possible statuses include:
21
21
| Status | Description |
22
22
|--------|-------------|
23
-
|{{<imagefilename="images/rc/rdi/pipeline-status-initial-sync.png#no-click"height=30pxalt="Initial Sync"class="inline" >}}| The data pipeline is ingesting all records from the source database into the target database. |
24
-
|{{<imagefilename="images/rc/rdi/pipeline-status-streaming.png#no-click"height=30pxalt="Streaming"class="inline" >}}| The data pipeline is capturing new changes from the source database as they happen. Changes in the source database are added to the target database within a few seconds. |
25
-
|{{<imagefilename="images/rc/rdi/pipeline-status-stopped.png#no-click"height=30pxalt="Stopped"class="inline" >}}| The data pipeline has been [stopped](#stop-and-restart-data-pipeline). |
26
-
|{{<imagefilename="images/rc/rdi/pipeline-status-error.png#no-click"height=30pxalt="Error"class="inline" >}}| There is an error in the data pipeline. [Reset the pipeline](#reset-data-pipeline) and contact support if the issue persists. |
23
+
|**Initial Sync**| The data pipeline is ingesting all records from the source database into the target database. |
24
+
|**Streaming**| The data pipeline is capturing new changes from the source database as they happen. Changes in the source database are added to the target database within a few seconds. |
25
+
|**Stopped**| The data pipeline has been [stopped](#stop-and-restart-data-pipeline). |
26
+
|**Error**| There is an error in the data pipeline. [Reset the pipeline](#reset-data-pipeline) and contact support if the issue persists. |
27
27
-**Total ingested**: Total number of records ingested from the source database.
28
28
-**Total inserted**: Total number of records inserted into the target database.
29
29
-**Total filtered**: Total number of records filtered from being inserted into the target database.
@@ -62,10 +62,6 @@ To change the data you want to ingest from the data pipeline:
62
62
63
63
If any tables are missing a unique constraint, a warning will appear in the **Data modeling** section. Select **Manage columns** to select the columns that define a unique constraint for those tables.
1. In the **Advanced properties** section, you can add any processor properties to control how the data is processed. See the [RDI configuration file reference]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#processors" >}}) for all available processor properties.
94
+
95
+
{{<imagefilename="images/rc/rdi/rdi-processor-properties.png"alt="The advanced properties section with processor properties." >}}
0 commit comments