From idea to implementation
Let’s say you need to arrange people at tables for a party, you will naturally try to put colleagues close to each other. But why? Over the years, you have learnt from experience, and you may not realise this immediately when you are solving a mathematical problem. You just place people who like to interact closely together; nothing special.
Press enter or click to view image in full size
I will rephrase above — you put the least conflicting characters close to each other, you are making groups of people for each table, or even tables. You found a way to minimise conflicts across all tables! That is what MaxSAT is all about — to form groups of things, as few as possible, so that there is as little conflict between them as possible.
In nature, problems pop up randomly; terms like “minimal” and “maximal” appear spontaneously.
So, in this spontaneous way, I came across the “table character allocation” problem manifestation in another form, this time at work in IT.
I saw that my colleague’s team faces this issue: they have a limited number of validation environments and no easy way to scale up or spin up additional environments. They create software, e.g. micro-services, and when it comes to integration testing, people start overwriting the application state, throwing away other people’s test results.
It may be just a messy development process, or they may have assumed that fixing this is impossible or not worth the effort.
The basic conflict carrier is — change, they change the state of application based on so-called Feature Toggles. This simple object tells the developer or Product Owner that they can switch on or switch off part of the functionality of their application, for e.g. A/B testing.
Feature Toggles should be short-lived because, over time, when they are enabled and in Production and of value to customers, their underlying functionality should become a solid part of the core functionality.
But experience shows that a single feature toggle lifetime may extend up to several months, in worst cases, to years.
When we write unit tests for a microservice, we can use mocks.
When we write integration tests, it can be a bit harder; in our case, solutions like Togglz (a software product) were not an option (don’t ask me why).
Then they just limited the number of tested states to a minimum.
What about test coverage? Still quite high, but some remaining tests have filters not to fire when a given Feature Toggle is in a certain state, and they are just toggled manually when test suites are executed.
So, not a real problem at all? They applied the 80/20 rule, and everything is fine, right?
You can stop reading here if you see no problem.I want to show you that we can address this.
And then you can decide if it was worth it.
Rather than 95% automated with a few % requiring manual action, I want to close the gap to achieve 100% automation.
I will show you how to optimise execution using JUnit 6 and Java 21, and utilise most of the JUnit library’s extension points.
Java , conflict, and order
Imagine these two Test classes (Java):
package org.example; import com.junit.toggleaware.annotation.FeatureToggle; import com.junit.toggleaware.annotation.ToggleAwareTest; import com.junit.toggleaware.annotation.ToggleValue; import com.junit.toggleaware.jupiter.TestRunContext; import org.junit.jupiter.api.MethodOrderer; import org.junit.jupiter.api.Order; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.TestMethodOrder; import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertTrue; @ToggleAwareTest @TestMethodOrder(MethodOrderer.OrderAnnotation.class) public class TestOne { @Test @FeatureToggle(name="ft1", value = ToggleValue.ON) @Order(2) public void firstTest() { assertTrue(TestRunContext.currentToggleState().get("ft1")); System.out.println("FirstTest"); } @Test @FeatureToggle(name="ft1", value = ToggleValue.OFF) @Order(1) public void secondTest() { assertFalse(TestRunContext.currentToggleState().get("ft1")); System.out.println("SecondTest"); } }
package org.example; import com.junit.toggleaware.annotation.FeatureToggle; import com.junit.toggleaware.annotation.ToggleAwareTest; import com.junit.toggleaware.annotation.ToggleValue; import com.junit.toggleaware.jupiter.TestRunContext; import org.junit.jupiter.api.MethodOrderer; import org.junit.jupiter.api.Order; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.TestMethodOrder; import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertTrue; @ToggleAwareTest @TestMethodOrder(MethodOrderer.OrderAnnotation.class) public class TestTwo { @Test @Order(1) @FeatureToggle(name="ft1", value = ToggleValue.ON) public void firstTest() { assertTrue(TestRunContext.currentToggleState().get("ft1")); System.out.println("FirstTest Two"); } @Test @Order(2) @FeatureToggle(name="ft1", value = ToggleValue.OFF) public void secondTest() { assertFalse(TestRunContext.currentToggleState().get("ft1")); System.out.println("SecondTest Two"); } }
You have tests annotated with information about the exact Feature Toggle state they are supposed to test the application with.
You want this desired state of the application enforced before the test is executed and restored to the original state after the test or test suite finishes.
But you are limited to running them all against a single environment at a time.
Now look at this execution log of both test classes:
A
Applied state {ft1=false}
SecondTest
Applied state {ft1=true}
FirstTest
Applied state {ft1=true}
FirstTest Two
Applied state {ft1=false}
SecondTest Two
B
[ToggleAware] applyToggleState {ft1=false}
[ToggleAware] stateChanges [ft1: <unset> -> false]
SecondTest
SecondTest Two
[ToggleAware] applyToggleState {ft1=true}
[ToggleAware] stateChanges [ft1: false -> true]
FirstTest Two
FirstTest
Please notice how many Feature Toggle state changes you see on A) versus B)
The first block shows 4 state changes; the second only 2, thanks to the solution to “table character allocation” you applied to your party tables (if you really cared). We want the minimum number of groups/batches with the maximum number of tests in each batch that do not conflict.
Why does this difference matter? When you have 4% tests of 2000 tests, you have much to control, and you should care. Remember, before each batch in this configuration, you want to change the state of the feature toggle for the same single environment you are testing. This takes time, introduces latency, and creates a dependency on another microservice during test execution.
Combinatorial explosion — we need to address this; our goal is not to test all combinations of feature toggle states. Our goal is to test a subset of combinations we decide are worth considering. And to do it without causing hassle and manual re-work.
Compatibility with existing JUnit 6 infrastructure — we do not want an entire rewrite of JUnit; it is a stable tool, and we want only to slightly expand its use cases, mainly by annotations. So new annotations should integrate flawlessly into existing infrastructure and not change how tests are executed by default, only when we decide to. The Order annotation should behave exactly the same way as it does in the latest JUnit implementation.
Operational models for our extension
Run it with an IDE — we can’t enforce intelligent batching as of writing this article due to the limitations of JUnit 6 API [1]. But we can make it work with existing and newly annotated tests in a regular way that does not impose Feature Toggle state-derived batch order. Efficiently, we will just do more toggle switching (a small library will do it for us) when we run this way, or we can disable the FTs by default (this part may require extra work beyond this article).
Through mvn test — similar to above
Through a custom test suite aggregate — here we can impose batching like this:
EnvironmentController envController = new MyEnvironmentController(); ToggleAwareLauncher launcher = new ToggleAwareLauncher(envController); ToggleAwareExecutionResult result = launcher.executeClasses(List.of( CheckoutV2SmokeIT.class, CheckoutV2RefundIT.class, LegacyCheckoutIT.class )); System.out.println("Batches: " + result.plan().size()); System.out.println("Failures: " + result.totalFailures()); result.writeHtmlReport(Path.of("target", "toggle-aware-report.html"));
Or like this by package name:
ToggleAwareExecutionResult result =
launcher.executePackage("com.mycompany.checkout");
Or even limit ourselves to Feature Toggle-related tests only:
ToggleAwareExecutionResult result =
launcher.executeFeatureToggleTestsInPackage("com.mycompany.checkout");
But do we need another report apart from the JUnit default one? Maybe not, but to present some possibilities, I decided to added it (see next section screens).
Grouping toggles into categories
You dislike this convention applied to every test, repeated, duplicated:
@FeatureToggle(name="loyalty.points", value = ToggleValue.ON)
We can move it to one central place — a catalogue class container enum and expose to the IDE with meta annotation like this so you get the completion while typing at the test class or test method level:
package com.junit.toggleaware.example; import com.junit.toggleaware.annotation.FeatureToggle; import com.junit.toggleaware.annotation.ToggleValue; import java.lang.annotation.ElementType; import java.lang.annotation.Retention; import java.lang.annotation.RetentionPolicy; import java.lang.annotation.Target; @Target({ElementType.TYPE, ElementType.METHOD}) @Retention(RetentionPolicy.RUNTIME) @FeatureToggle(catalog = ExampleFeatureToggles.class, entry = "LOYALTY_POINTS", value = ToggleValue.OFF) public @interface LoyaltyPointsOff { }
And under the hood, we can elaborate on Feature Toggle properties by maintaining FT properties like this:
package com.junit.toggleaware.example; import com.junit.toggleaware.annotation.FeatureToggleDefinition; enum ExampleFeatureToggles implements FeatureToggleDefinition { CHECKOUT_V2("checkout.v2", "2023-01-15", "2027-12-31"), LOYALTY_POINTS("loyalty.points", "2025-09-01", "2026-03-01"), FT1("ft1", "", "2026-03-01"); private final String toggleName; private final String introducedDate; private final String endOfLifeDate; ExampleFeatureToggles(String toggleName, String introducedDate, String endOfLifeDate) { this.toggleName = toggleName; this.introducedDate = introducedDate; this.endOfLifeDate = endOfLifeDate; } @Override public String toggleName() { return toggleName; } @Override public String introducedDate() { return introducedDate; } @Override public String endOfLifeDate() { return endOfLifeDate; } }
What could a final report with FT states and filters look like for each batch of tests executed?
Or even better, as you can pre-filter and mark in each executed batch of tests which feature toggles are marked in code as expired:
To help with a proper clean-up of old FTs.
How this all works
Architecturally we group extension functionality into packages:
1. annotations
2. discovery
3. exec
4. jupiter
5. plan
6. report
annotations
This contains classes/interfaces that are most often used when writing tests.
How users should use it:
– Annotate test classes or methods with @ FeatureToggle(…) to say which toggle state a test requires.
– Annotate a test class with ‘@ ToggleAwareTest(…) to activate the JUnit 5/6 extensions.
– Optionally use ‘@ ToggleSetup(…) for named setup steps and to influence batching order.
discovery
Used usually indirectly through ToggleAwareLauncher.
– Advanced users may use it if they want to inspect test requirements without executing tests.
exec
This is the main programmatic integration package.
– Users who want batch-aware execution instantiate ToggleAwareLauncher.
– Users must provide an EnvironmentController so the library can actually apply toggle state in their system.
jupiter
Used indirectly by placing @ ToggleAwareTest on a test class.
– Advanced users may need to provide an EnvironmentControllerProvider so direct runs can apply toggle state.
plan
Used mostly through ToggleAwareLauncher.plan (…) methods.
– Advanced users can plug in a custom BatchPlanner (Greedy or MaxSAT/SMT-based).
report
Used indirectly through ToggleAwareLauncher or ToggleAwareExecutionResult.
– Advanced users can render or write reports themselves.
Where is this MaxSAT solver located? It is in the plan package.
How to configure the default EnvironmentController responsible for switching toggle states on external services?
by adding MEDA-INF.services resource file -> com.junit.toggleaware.exec.EnvironmentControllerProvider
Inside of it, place the name of the implementation class that delivers an instance of EnvironmentController for your environment.
[1] — Launcher-managed mode: ToggleAwareLauncher does discovery, builds TestRequirements, computes PlannedBatch values with MaxSatPlanner, applies environment state batch by batch, and then executes only the tests assigned to each batch.
but
Plain JUnit mode: IDE run actions and standard mvn test invoke the JUnit Platform directly. In that path, Jupiter extensions can influence individual test behaviour, but they do not take over global discovery and rescheduling of the full test plan.
