From b2ec400e91d094487f41f76ba4ffd8c0163128fa Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jan=20Van=C3=AD=C4=8Dek?= Date: Tue, 14 Oct 2025 13:32:39 +0200 Subject: [PATCH] docs: clarify AWS ECR as the only supported Docker registry for Keboola components --- extend/component/deployment/index.md | 3 +- extend/component/docker-tutorial/index.md | 7 +- extend/component/docker-tutorial/registry.md | 9 +- extend/component/docker-tutorial/setup.md | 122 +++++++++---------- extend/component/tutorial/index.md | 4 +- extend/generic-extractor/publish.md | 4 +- extend/generic-extractor/running.md | 4 +- overview/index.md | 2 +- 8 files changed, 73 insertions(+), 82 deletions(-) diff --git a/extend/component/deployment/index.md b/extend/component/deployment/index.md index 8caf1cdc..238827b3 100644 --- a/extend/component/deployment/index.md +++ b/extend/component/deployment/index.md @@ -299,8 +299,7 @@ allows you to obtain the repository for a component and push credentials to that [running components](/extend/component/running/#running-a-component), for example, how to obtain the AWS registry credentials. If you want to get even more low level, you can use the [Developer Portal API](https://kebooladeveloperportal.docs.apiary.io/#) directly. It also allows you to [generate credentials for a service account](https://kebooladeveloperportal.docs.apiary.io/#reference/0/vendor/create-service-account) -programmatically. Apart from our AWS ECR registry, we also support running images stored in [Quay.io](https://quay.io/repository/) -and [Docker Hub](https://hub.docker.com/) registries. +programmatically. We use our AWS ECR registry for hosting all component images. ## Test Live Configurations Testing your component can be simply added as part of the script in `.travis.yml` file. See an example in diff --git a/extend/component/docker-tutorial/index.md b/extend/component/docker-tutorial/index.md index 004212a8..fdefbeba 100644 --- a/extend/component/docker-tutorial/index.md +++ b/extend/component/docker-tutorial/index.md @@ -34,10 +34,9 @@ Docker Images are created by executing the instructions written in a **Dockerfil file consisting mostly of shell commands which must be executed to prepare the application for running. Docker images can be based on other images. So if you need minor modification to a system, you do not have to build the whole thing from scratch. If you want Images to be -reused, *push* your Dockerfile to a **Docker registry**. The registries ([Dockerhub](https://hub.docker.com/), -[Quay](https://quay.io/)) will build the image; anyone interested in using it can download it. -[AWS ECR](https://aws.amazon.com/ecr/) is a private repository and has no build triggers. You need to push the images manually or -using a [deploy script](/extend/component/deployment/) in your CI pipeline. +reused, *push* your Dockerfile to a **Docker registry**. For Keboola components, we use +[AWS ECR](https://aws.amazon.com/ecr/). You need to push the images using a +[deploy script](/extend/component/deployment/) in your CI pipeline. Docker image names are based on the following scheme: `registry-name/account-name/image-name:tag` where _registry-name_ and _account-name_ can sometimes be omitted. For example, you can refer to a Docker _hello-world_ image as: `hello-world` diff --git a/extend/component/docker-tutorial/registry.md b/extend/component/docker-tutorial/registry.md index 2d65f3ac..18f70556 100644 --- a/extend/component/docker-tutorial/registry.md +++ b/extend/component/docker-tutorial/registry.md @@ -11,11 +11,8 @@ redirect_from: An important part of the Docker ecosystem is a **Docker registry**. It acts as a folder of images, taking care of their storing and building. -[Docker Hub](https://hub.docker.com/) is the official Docker registry. - -For reliability reasons, we strongly recommend to use the [Amazon AWS ECR](https://aws.amazon.com/ecr/) -[provisioned by the **Keboola Developer Portal**](/extend/component/deployment/). -We also support Docker Hub and [Quay](https://quay.io/), both public and private repositories. +For reliability reasons, we use the [Amazon AWS ECR](https://aws.amazon.com/ecr/) +[provisioned by the **Keboola Developer Portal**](/extend/component/deployment/). This is the only supported Docker registry for Keboola components. ## Working with Registry In order to run an image, **pull** (`docker pull`) the image to your machine. The `docker run` @@ -88,4 +85,4 @@ trigger a new build of the Docker image. Also note that the image automatically or branch name. So, when you push a commit to the `master` branch, you will get an image with a tag `master` (which will move away from any older image builds). When creating a `1.0.0` tag, you will get an image with a `1.0.0` tag. -When using images in Keboola, we **highly recommend to use our [ECR repository](/extend/component/deployment/)**. +When using images in Keboola, you must use our [ECR repository](/extend/component/deployment/). diff --git a/extend/component/docker-tutorial/setup.md b/extend/component/docker-tutorial/setup.md index d3bc810c..7b86bf9e 100644 --- a/extend/component/docker-tutorial/setup.md +++ b/extend/component/docker-tutorial/setup.md @@ -1,61 +1,61 @@ ---- -title: Installation and Running -permalink: /extend/component/docker-tutorial/setup/ -redirect_from: - - /extend/docker/tutorial/setup/ ---- - -* TOC -{:toc} - -To work with Docker, you need to have it installed. If you do not have a computer with a Docker machine, you can obtain a -cheap [hosted server](https://marketplace.digitalocean.com/apps/docker). You can also -run everything locally. To install a Docker machine on Win/Mac, -use [Docker Community Edition](https://hub.docker.com/search?type=edition&offering=community). For -other systems, see the [documentation](https://docs.docker.com/install/). - -## Getting Started -To test that everything is running correctly, start with an example -from the Docker [documentation](https://docs.docker.com/get-started/). -Run the following commands on the command line: - - docker run hello-world - - -or - - docker run docker.io/library/hello-world:latest - -If this works, use any image published in any Docker -registry, e.g. [Docker Hub](https://hub.docker.com/), [Quay](https://quay.io/) or [AWS ECR](https://aws.amazon.com/ecr/). -Note that in some configurations, you may need to use `sudo` to run `docker`. If you run into problems, consult the -official troubleshooting ([Windows](https://docs.docker.com/docker-for-windows/troubleshoot/), [Mac](https://docs.docker.com/docker-for-mac/troubleshoot/)). - -## Generally Useful Commands -- [`docker run`](https://docs.docker.com/engine/reference/run/): Run an -image (create a container and run the command in `ENTRYPOINT` or `CMD` section of **Dockerfile**). -- [`docker build`](https://docs.docker.com/engine/reference/commandline/build/): Build -an image (execute instructions in **Dockerfile** and create a runnable image). -- [`docker pull`](https://docs.docker.com/engine/reference/commandline/pull/): Pull -a newer version of an image (force update of the cached copy). - -## Sharing Files -Sharing files between the host OS and the container is done using the `--volume` parameter of the `docker run` command: - - docker run --volume=/hostPath/:/containerPath/ imageName - -Do not use any spaces around `:`. The details of file sharing are somewhat -[dependent on the host OS](https://docs.docker.com/storage/volumes/) you are using. -There is a very -useful [guide for Rocker image](https://github.com/rocker-org/rocker/wiki/Sharing-files-with-host-machine), which -describes all the sharing options in great detail. - -On Linux systems, where Docker runs natively, there is really not much to think about. The only things that can bite -you are file and directory permissions. Keep in mind that files created within the running Docker container on -a mounted host volume will be created with the permissions of users of that container (not the host OS). - -On Mac OS, follow the [official example](https://docs.docker.com/docker-for-mac/osxfs/). -The limitation is that your host path must be in a shared directory (e.g., `/Users/`). - -On Windows, follow the [official example](https://docs.docker.com/docker-for-windows/#shared-drives). -The limitation is that you must use absolute paths with `docker run`. +--- +title: Installation and Running +permalink: /extend/component/docker-tutorial/setup/ +redirect_from: + - /extend/docker/tutorial/setup/ +--- + +* TOC +{:toc} + +To work with Docker, you need to have it installed. If you do not have a computer with a Docker machine, you can obtain a +cheap [hosted server](https://marketplace.digitalocean.com/apps/docker). You can also +run everything locally. To install a Docker machine on Win/Mac, +use [Docker Community Edition](https://hub.docker.com/search?type=edition&offering=community). For +other systems, see the [documentation](https://docs.docker.com/install/). + +## Getting Started +To test that everything is running correctly, start with an example +from the Docker [documentation](https://docs.docker.com/get-started/). +Run the following commands on the command line: + + docker run hello-world + + +or + + docker run docker.io/library/hello-world:latest + +If this works, use any image published in any Docker +registry, e.g. [Docker Hub](https://hub.docker.com/), [Quay](https://quay.io/) or [AWS ECR](https://aws.amazon.com/ecr/). +Note that in some configurations, you may need to use `sudo` to run `docker`. If you run into problems, consult the +official troubleshooting ([Windows](https://docs.docker.com/docker-for-windows/troubleshoot/), [Mac](https://docs.docker.com/docker-for-mac/troubleshoot/)). + +## Generally Useful Commands +- [`docker run`](https://docs.docker.com/engine/reference/run/): Run an +image (create a container and run the command in `ENTRYPOINT` or `CMD` section of **Dockerfile**). +- [`docker build`](https://docs.docker.com/engine/reference/commandline/build/): Build +an image (execute instructions in **Dockerfile** and create a runnable image). +- [`docker pull`](https://docs.docker.com/engine/reference/commandline/pull/): Pull +a newer version of an image (force update of the cached copy). + +## Sharing Files +Sharing files between the host OS and the container is done using the `--volume` parameter of the `docker run` command: + + docker run --volume=/hostPath/:/containerPath/ imageName + +Do not use any spaces around `:`. The details of file sharing are somewhat +[dependent on the host OS](https://docs.docker.com/storage/volumes/) you are using. +There is a very +useful [guide for Rocker image](https://github.com/rocker-org/rocker/wiki/Sharing-files-with-host-machine), which +describes all the sharing options in great detail. + +On Linux systems, where Docker runs natively, there is really not much to think about. The only things that can bite +you are file and directory permissions. Keep in mind that files created within the running Docker container on +a mounted host volume will be created with the permissions of users of that container (not the host OS). + +On Mac OS, follow the [official example](https://docs.docker.com/docker-for-mac/osxfs/). +The limitation is that your host path must be in a shared directory (e.g., `/Users/`). + +On Windows, follow the [official example](https://docs.docker.com/docker-for-windows/#shared-drives). +The limitation is that you must use absolute paths with `docker run`. diff --git a/extend/component/tutorial/index.md b/extend/component/tutorial/index.md index 5dcbde05..803cb594 100644 --- a/extend/component/tutorial/index.md +++ b/extend/component/tutorial/index.md @@ -236,9 +236,7 @@ We offer free hosting of your Docker images in the **[Amazon Container Registry All repositories in AWS ECR are private. When you create your component using the method shown above, we have just provisioned you with the Docker image hosting and you do not need to worry about it any more. -We also support the DockerHub and Quay.io registries, both public and private. However, as they are more prone to -outages and beyond our control, we recommend using our reliable AWS ECR. Use DockerHub or Quay.io only if you, -for instance, want the image to be public. +AWS ECR is the **only supported registry** for Keboola components, providing reliable hosting for all Docker images. ## Summary You have just created your own Keboola component. Although it does not do much, it shows the easiest path diff --git a/extend/generic-extractor/publish.md b/extend/generic-extractor/publish.md index 88d3dc58..bc3c4547 100644 --- a/extend/generic-extractor/publish.md +++ b/extend/generic-extractor/publish.md @@ -36,9 +36,7 @@ have created the component, edit it, and fill in the following details: - **Region** -- leave empty - **UI options** --- set to `genericTemplatesUI` -For a list of available tags, see the [Generic Extractor GitHub repository](https://github.com/keboola/generic-extractor/) or -[Generic Extractor Quay repository](https://quay.io/repository/keboola/generic-extractor/), both of which contain the same tags -as the above AWS ECR repository. It is also possible to use the `latest` tag, which points to the highest available tag. However, +For a list of available tags, see the [Generic Extractor GitHub repository](https://github.com/keboola/generic-extractor/). It is also possible to use the `latest` tag, which points to the highest available tag. However, we recommend that you configure your component with a specific tag and update it manually to avoid problems with breaking changes in future Generic Extractor releases. diff --git a/extend/generic-extractor/running.md b/extend/generic-extractor/running.md index 45c02e9d..375ee86f 100644 --- a/extend/generic-extractor/running.md +++ b/extend/generic-extractor/running.md @@ -75,11 +75,11 @@ configuration you want to execute. For example: Then run Generic Extractor in the current directory by executing the following command on *nix systems: - docker run -v ($pwd):/data quay.io/keboola/generic-extractor:latest + docker run -v ($pwd):/data 147946154733.dkr.ecr.us-east-1.amazonaws.com/developer-portal-v2/ex-generic-v2:latest or on Windows: - docker run -v %cd%:/data quay.io/keboola/generic-extractor:latest + docker run -v %cd%:/data 147946154733.dkr.ecr.us-east-1.amazonaws.com/developer-portal-v2/ex-generic-v2:latest You should see: diff --git a/overview/index.md b/overview/index.md index 6cf4538c..254cabd5 100644 --- a/overview/index.md +++ b/overview/index.md @@ -21,7 +21,7 @@ The following chart shows how Keboola is structured. All Keboola parts are brief Everything you can do in the Keboola UI can be done programatically using the API of the corresponding component. All of our components have API documentation on [Apiary](https://keboola.docs.apiary.io/#) and most of them have a public [Github repository](https://github.com/keboola/). -Our Docker components are built either on [DockerHub](https://github.com/keboola/), [Quay](https://quay.io/organization/keboola) or privately on [AWS ECR](https://aws.amazon.com/ecr/). +Our Docker components are built on [AWS ECR](https://aws.amazon.com/ecr/). This means that there are virtually **endless possibilities of what can be done with Keboola programmatically**.