Parameterized tests #615

chlab · 2025-03-25T07:29:53Z

chlab
Mar 25, 2025

To be able to define more complex tests, it would be useful to be able to define parameterized tests, as seen in other testing frameworks or technologies, e.g.

Here's how this could work from the point of view of writing the tests. Let's say we have a Deployment definition like:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: 'foo-deployment'
  selector:
    matchLabels:
      app: 'foo'
spec:
  replicas: {{ .Values.deployment.replicaCount }}

and we want to ensure that there's a PodDisruptionBudget for each Deployment:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: 'foo-pdb'
spec:
  minAvailable: {{ sub .Values.deployment.replicaCount 1 }}
  selector:
    matchLabels:
      app: 'foo'

We could write a test like this:

tests:
  - it: should have a PodDisruptionBudget for each Deployment
    parameters:
      # each key defined here adds a "test dimension" to the execution, as well as a named variable containing the current value of the list
      # can be a constant or a jsonpath selector
      deployment: $[?(@.kind == 'Deployment')]
    # should fail if skipEmptyTemplates is false
    documentSelector:
      path: $[?(@.kind == 'PodDisruptionBudget')].selector.matchLabels.app
      value: $deployment.metadata.selector.matchLabels.app
    asserts:
      - greaterOrEqual:
          path: spec.minAvailable
          value: $deployment.spec.replicas - 1 # bonus: this could enable dynamic assertions

There would be a test result per Deployment found. The result set (combination of parameters + documentSelector) should be allowed to be empty.

Parametrized tests could also work will for simpler use cases, such as my question described here. In this case, we could just define a constant with a list of RoleBindings we want to test for:

tests:
  - it: 'Should have required RoleBindings'
    parameters:
      roleBindingName: ['managed-admin-rb', 'managed-view-rb']
    documentSelector:
      path: kind
      value: RoleBinding
      matchMany: true
    asserts:
      - equal:
          path: metadata.name
          value: $roleBindingName

I would love to hear your thoughts on this @quintush. We increasingly rely on helm unittest at work and would be happy to take a stab at contributing this functionality if we can agree that this makes sense.

This would solve a few issues brought up in the past:

chlab · 2025-03-25T07:47:25Z

chlab
Mar 25, 2025
Author

Ooh I just found #222 and will check this out first

4 replies

chlab Mar 25, 2025
Author

This kind of fulfills the example with a constant/yaml list as a parameter but doesn't allow the same flexibility as the solution proposed above. Also, I don't particularly like having templated test files as documented in #222. A native support for parametrization would be more elegant imho. What do you think @quintush? Like I said, we would be willing to work on this if you think it makes sense as it's proposed.

chlab Apr 2, 2025
Author

@quintush Is this something that would be worth working on on our side or does it not fit the idea of helm unittest?

quintush Apr 3, 2025
Maintainer

Hello @chlab,

First of all thanks for your suggestions.

I do see the benefit of having parameterized tests, to keep the tests more reusable.
Implementation #222 , was a good starting point to learn the impact.
As you can see it requires some pre-processing (which is currently done, by wrapping the tests in a helmchart).

Currently I'm primarily focussed on implementing some kind of pre-processing.
This could be a starting point for parameterized tests, but also conditional assertions (#203 ) and even library tests (#171 ).
I can imagine that pre-processing can also help in filling the parameters.

So to conclude, I do see a fit for the future.
If you want to start on it, feel free to create a PR, please make sure to allow the maintainer to edit the PR, so we can collaborate on the PR and finally I can give you full credits to the implementation.

Greetings,
@quintush

chlab Apr 3, 2025
Author

Thanks for getting back to me @quintush!

We could probably start on this in a week or so. Is there anything we should know in regards to how you would implement this or what your thoughts are in regards to pre-processing? Otherwise we'll do a code review of the plugin first and open a PR for discussion with a suggestion on how we would implement it.

Cheers

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parameterized tests #615

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 1 comment 4 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Parameterized tests #615

Uh oh!

Uh oh!

chlab Mar 25, 2025

Replies: 1 comment · 4 replies

Uh oh!

chlab Mar 25, 2025 Author

Uh oh!

Uh oh!

chlab Mar 25, 2025 Author

Uh oh!

chlab Apr 2, 2025 Author

Uh oh!

quintush Apr 3, 2025 Maintainer

Uh oh!

chlab Apr 3, 2025 Author

chlab
Mar 25, 2025

Replies: 1 comment 4 replies

chlab
Mar 25, 2025
Author

chlab Mar 25, 2025
Author

chlab Apr 2, 2025
Author

quintush Apr 3, 2025
Maintainer

chlab Apr 3, 2025
Author