Skip to content

Commit b25ab5b

Browse files
mhjacksmhjacks
authored
Merge pull request #403 from mhjacks/secrets_blog
Secrets blog
2 parents eb6023e + e542dfc commit b25ab5b

File tree

1 file changed

+312
-0
lines changed

1 file changed

+312
-0
lines changed

‎content/blog/2024-01-26-more-secrets-options.md‎

Lines changed: 312 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,312 @@
1+
---
2+
date: 2024-01-26
3+
title: More Secrets Options Now Available with Validated Patterns
4+
summary: Validated Patterns now supports alternatives to both HashiCorp Vault and the External Secrets Operator
5+
author: Martin Jackson
6+
blog_tags:
7+
- patterns
8+
- secrets
9+
---
10+
11+
# More Secrets Options Now Available with Validated Patterns
12+
13+
## Overview
14+
15+
One of the things about the kubernetes application management experience that
16+
we wanted to explore and improve as part of the Validated Patterns initiative
17+
was the secrets handling in GitOps. So we worked out a scheme that stored
18+
secret material in a dedicated secrets store, using the External Secrets Operator
19+
to project that secret material into the applications using ESO's powerful and
20+
convenient abstraction and templating features. At that time, HahsiCorp Vault
21+
was well supported and popular in the community, as was using ESO to retrieve
22+
secrets from it. All of the major hyperscaler keystores are supported (AWS,
23+
Azure, GCP), but multi- or hybrid- cloud solutions that can be "self-contained"
24+
are either less well supported or the solutions themselves lean towards SaaS
25+
offerings.
26+
27+
Almost two years later, the secrets landscape has shifted somewhat. As a Red
28+
Hat project, Validated Patterns initiative gravitates towards Red Hat-supported
29+
solutions, and neither HashiCorp Vault nor ESO are currently Red Hat supported.
30+
Meanwhile, the only backend we provided a code path to drive with ESO was Vault.
31+
32+
This nullifies one of the major reasons we wanted to use ESO in the first place -
33+
namely, the ability to easily swap out secrets backends in case Vault was not
34+
usable for some reason. Earlier in 2023, one of our engineers did a proof of
35+
concept of ESO support using the AWS Secrets Manager backend - demonstrating that
36+
ESO delivered on its promise of multiple secret store support. Adapting ESO was
37+
the easy part - the hard part is building more abstraction into the VP secrets
38+
handling code that runs on pattern install.
39+
40+
To this end, we decided we would expand secrets support by introducing at least
41+
one new backend - and we chose the kubernetes backend, because it is self-contained
42+
(that is, it can be run on-prem and requires no new products or projects to be
43+
installed), and is a useful vehicle for introducing an abstraction layer for
44+
validated patterns secret parsing. In addition, dealing with kubernetes secrets
45+
objects directly has the side effect of enabling us to provide a mechanism for
46+
users to inject their secrets directly into their patterns, bypassing the need
47+
to use any secrets manager or ESO at all. This also provides benefits to installations
48+
where only commercially supported solutions can be installed, since ESO is
49+
currently not commercially supported by any entity.
50+
51+
In a nutshell, the new features depend on an abstraction of secret file parsing, so
52+
that the secrets are held in memory in a datastructure that is then processed and
53+
loaded by the appropriate backend code.
54+
55+
Users of the pattern framework will be able to change secrets backends as straightforwardly
56+
as we can make possible. The only other change the user will need to make (to use another
57+
ESO backend) is to use the backend's mechanism to refer to keys. (For example: in Vault,
58+
keys have have names like `secret/data/global/config-demo`; in the Kubernetes backend
59+
it would just be the secret object name that's being used to store the secret material,
60+
such as `config-demo`).
61+
62+
## Chart changes
63+
64+
The clusterSecretStore related chart elements have moved to
65+
values-global.yaml, specifically `global.secretStore.backend`. For example, from
66+
multicloud-gitops:
67+
68+
```yaml
69+
global:
70+
pattern: multicloud-gitops
71+
options:
72+
useCSV: false
73+
syncPolicy: Automatic
74+
installPlanApproval: Automatic
75+
secretStore:
76+
backend: kubernetes
77+
```
78+
79+
Previously, the `secretStore` setting was done per-chart; but ordinarily this
80+
setting will hold for the entire cluster, which may include many charts. Because
81+
of this, we will move these settings in our charts. (Older configs will not
82+
break if they still use vault; and it is still an option to override each
83+
chart if you want to do that.)
84+
85+
Individual secret keys used for ESO lookups will need to be overridden or changed
86+
to use the kubernetes backend.
87+
88+
This values-global setting is also used by the Ansible workflow to decide which
89+
backend to inject secrets into.
90+
91+
## The "vault" Backend - Unchanged Interface, New plumbing
92+
93+
The vault support now has a new code path to follow, but supports exactly the same
94+
features in the same ways it always has. Vault is still the default backend for
95+
patterns, and all existing patterns should be able to adopt the new code path without
96+
making any changes. Any other experience is a bug we will fix.
97+
98+
All of the defaults for the new code path are designed to work with existing
99+
vault-based configurations; the new features are entirely optional and we do not
100+
expect breakage or regressions to existing vault-based configurations.
101+
102+
## The "kubernetes" Backend *new*
103+
104+
New features have been introduced to the secrets structure to support the use of the
105+
Kubernetes ESO backend. See the below details for the new options and how they are processed
106+
by the `kubernetes` parsing strategy.
107+
108+
## The "none" Backend *new*
109+
110+
New features have been introduced to the secrets structure to support not using an ESO
111+
ESO backend, but rather injecting secrets objects directly into the pattern. This violates
112+
the spirit of GitOps by not recording the details of the deployment in a versioned way.
113+
However users might want or need to make this tradeoff for different reasons. See the below
114+
details for the new options and how they are processed by the `none` parsing strategy.
115+
116+
At present, any external secret objects will need to be deleted from the repository to use
117+
the `none` backend - since the ArgoCD application will not sync when a non-existing CRD is
118+
referenced.
119+
120+
## How to Use a non-default Backend
121+
122+
We have provided Makefile targets to switch between backends. These targets edit the pattern
123+
configuration files in-place; in keeping with the GitOps philosophy they change the files in
124+
git that control how the application is deployed.
125+
126+
These Makefile targets are:
127+
128+
```text
129+
secrets-backend-vault Edits values files to use default Vault+ESO secrets config
130+
secrets-backend-kubernetes Edits values file to use Kubernetes+ESO secrets config
131+
secrets-backend-none Edits values files to remove secrets manager + ESO
132+
```
133+
134+
Run the makefile target in your repo to effect the necessary changes. If the command makes changes,
135+
it will display them in `git diff` format, and it will be up to you to commit and push the result
136+
to effect the change. Nothing will change on your cluster until you commit and push.
137+
138+
## Using the old system - The `legacy-load-secrets` Makefile target
139+
140+
The existing vault-utils codepath is available via the `legacy-load-secrets`
141+
Makefile target. If secrets loading fails, or you just want to use the other
142+
system, you can run `make legacy-load-secrets` after `make install` and it will
143+
run those scripts and the Ansible playbooks and roles associated with them.
144+
145+
## Deprecation of v1.0 Secrets
146+
147+
The v1.0 secrets format has not been used in the Validated Patterns framework
148+
for over a year now. The v2.0 framework is a strict superset of the v1.0
149+
framework. Support for the v1.0 framework is still available via the
150+
`legacy-load-secrets` code path, but this may be removed in the future.
151+
152+
## Updates to the Secrets v2.0 Schema
153+
154+
New features have been added at both the file level and the per-secret level
155+
to support the new backends:
156+
157+
### Top-level Additions
158+
159+
#### `secretStoreNamespace`
160+
161+
example:
162+
```yaml
163+
---
164+
version: "2.0"
165+
secretStoreNamespace: 'validated-patterns-secrets'
166+
167+
secrets:
168+
```
169+
170+
A new top-level key has been introduced in the secrets file:
171+
`secretStoreNamespace`. This defaults to `validated-patterns-secrets`. This is
172+
the namespace that ESO uses as its special secret store, which serves the same
173+
architectural purpose as vault does in the default installation. (Secrets are
174+
installed into this namespace as Kubernetes secrets objects, and ESO allows for
175+
them to be copied or templated out using ESO mechanisms).
176+
177+
#### `defaultAnnotations`
178+
179+
example:
180+
181+
```yaml
182+
defaultAnnotations:
183+
validatedpatterns.io/pattern: 'myPattern'
184+
```
185+
186+
This data structure is a hash, or dictionary. These labels will be applied to all
187+
secrets objects unless they have per-secret annotations set. Labels are only added to
188+
kubernetes based secrets objects (using the `kubernetes` or `none`) backends. The
189+
vault loader ignores these settings.
190+
191+
#### `defaultLabels`
192+
193+
example:
194+
195+
```yaml
196+
defaultLabels:
197+
patternType: 'edge'
198+
patternEnvironment: 'production'
199+
```
200+
201+
This data structure is a hash, or dictionary. These labels will be applied to all
202+
secrets objects unless they have per-secret labels set. Labels are only added to
203+
kubernetes based secrets objects (using the `kubernetes` or `none`) backends. The
204+
vault loader ignores these settings.
205+
206+
### Per-secret Additions
207+
208+
#### `targetNamespaces`
209+
210+
example:
211+
212+
```yaml
213+
secrets:
214+
- name: config-demo
215+
targetNamespaces:
216+
- config-demo
217+
fields:
218+
```
219+
220+
This option is ignored by `vault` and `kubernetes` backends, and only used by the `none` backend.
221+
Normally, you will only need to add your secret to one namespace at a time. However, if you do need
222+
to copy a secret that is identical except for the namespace it goes into, you can add multiple
223+
`targetNamespaces` each namespace specified will get a copy of the secret.
224+
225+
There is not a default target namespace for the none backend, so omitting this field from a config
226+
parsed for the `none` backend is an error.
227+
228+
#### `labels`
229+
230+
example:
231+
```yaml
232+
secrets:
233+
- name: config-demo
234+
labels:
235+
patternType: 'edge'
236+
patternEnvironment: 'production'
237+
fields:
238+
```
239+
240+
In this case, these labels will only be applied to any `config-demo` secret objects created
241+
by the framework. This option is only used by the `none` and `kubernetes` backends and
242+
ignored by the `vault` backend. If `defaultLabels` are specified at the top level of the
243+
file, per-secrets labels will override them.
244+
245+
#### `annotations`
246+
247+
example:
248+
249+
```yaml
250+
secrets:
251+
- name: config-demo
252+
annotations:
253+
validatedpatterns.io/pattern: 'myPattern'
254+
fields:
255+
```
256+
257+
In this case, this annotation will only be applied to any `config-demo` secret objects created
258+
by the framework. This option is only used by the `none` and `kubernetes` backends and
259+
ignored by the `vault` backend. If `defaultAnnotations` are specified at the top level of the
260+
file, per-secrets annotations will override them.
261+
262+
## Under the Hood - Python and Ansible Code
263+
264+
The main changes here were to factor out the code that did the file parsing and actual secret
265+
loading into different modules. The `parse_secrets_info` module now reads all of the file contents
266+
and renders all of the secrets it can before turning the process over to an appropriate secrets
267+
loader.
268+
269+
### The process_secrets playbook
270+
271+
The `process_secrets` understands the backend configured for the different backends from values-global,
272+
and follows the appropriate strategy.
273+
274+
### parse_secrets_info Ansible Module
275+
276+
`parse_secrets_info` understands the different backends, and parses the secrets file into an in-memory
277+
structure that can then be handed over to a loader specific to the backend. There is an additional
278+
script, `common/scripts/process_secrets/display-secrets-info.sh <secrets_file> <backend_type>` that
279+
can be used to view how the secrets are parsed. This will display secrets on the terminal, so use
280+
with caution. It creates a `parsed_secrets` structure that should be generally useful, as well as
281+
`vault_policies` (Specifically for Vault support). Additionally, it creates a `kubernetes_secret_objects`
282+
structure suitable to hand over to the Ansible k8s.core collection directly.
283+
284+
### vault_load_parsed_secrets Ansible Module
285+
286+
`vault_load_parsed_secrets` is responsible for setting up the commands to load the secrets into vault,
287+
and running them.
288+
289+
### The k8s_secret_utils Ansible Role
290+
291+
`k8s_secret_utils` is used for loading both the `kubernetes` and `none` backends. It
292+
293+
### Changes to to vault_utils Ansible Role
294+
295+
Some code has been factored out of `vault_utils` and now lives in roles called `cluster_pre_check` and
296+
`find_vp_secrets` roles. A new task file has been added, `push_parsed_secrets.yaml` that knows how to use
297+
the `parsed_secrets` structure generated by `parse_secrets_info`. The existing code in the other task files
298+
remains.
299+
300+
## Developing a new backend
301+
302+
To provide support for an additional backend, the framework will need changes to:
303+
304+
- The `golang-external-secrets` chart (to support the new provider)
305+
- The shell and ansible framework for loading (understanding the new backend name and
306+
developing behaviors for it).
307+
308+
## Conclusion
309+
310+
The Validated Patterns framework strives to offer solutions to real-world problems, and we hope you will find
311+
these new features useful. If you run into problems, we will do our best to
312+
[help](https://github.com/validatedpatterns/common/issues)!

0 commit comments

Comments
 (0)