Secret Manager is a fully managed and scalable service for storing, manipulating, auditing, and accessing confidential information used in Google Cloud services, including GKE and Compute Engine. A key part of any secret management strategy is managing the deletion and destruction of secrets. To provide further advanced capability, Google Cloud has announced that the delayed destruction of secret versions of Secret Manager is now fully available. This new feature helps ensure that confidential information is not mistakenly deleted by an accidental or deliberate malicious attack.
Table of Contents
Table of Contents
How It Works?
Secret Manager has two primary components: secrets and secret versions. A secret is a global object within a project that holds metadata and multiple versions of secrets. The secret version contains the actual secret data, including credentials, API keys, or certificates.
Customers usually faced challenges while managing the secret material lifecycle in Secret Manager:
- Destruction of a secret version is irreversible, which means once your secret material is destroyed, there is no way to recover.
- There is a lack of actionable alerts when there’s an attempt to delete any critical secrets, which diminishes the likelihood of prompt intervention by an administrator.
To tackle these issues, we have implemented two features related to the delayed destruction of secrets: a customizable delay duration to stop the immediate deletion of secret versions, and a new Pub/Sub event notification that informs you when a destruction attempt is made.
Before delayed destruction was introduced, a secret version could transition between enabled, disabled, and destroyed phases. The disabled state is reversible and restricts access to the secret version, while the destroyed state is, by default, immediate and irreversible.
Now, when a destruction action occurs, a soft-delete is executed on the specific version by changing its status to disabled. The secrets are retained in an archival period for a configurable duration of N days, set by administrators using the TTL_DURATION field. During this time, an administrator can opt to restore the secret version by re-enabling it and switching it back to an enabled state. Once the delay period ends, the secret version is permanently deleted.
Observability and alerting via Pub/Sub notification
Observability is a key element of a well-defined security posture. To inform organizational admins and SecOps specialists about any attempts to destroy a secret version, we have added a new optional Pub/Sub notification named SECRET_VERSION_DESTROY_SCHEDULED.Once enabled, any scheduled destruction will notify the appropriate Pub/Sub topic, allowing the on-call personnel to analyze the change and if necessary restore the secret version instead of allowing destruction to proceed.
Complement delayed destruction with least privilege access
Activating the delayed destruction feature alone does not fully protect against the deletion of secret material. It’s crucial to implement proper access controls for your secrets. Even with this feature enabled, an administrator can still decide to delete the entire secret if they choose. Therefore, it’s essential to limit admin access to a small, highly trusted group of users. For routine lifecycle management of secrets, users should be assigned restricted roles that provide only the minimum privileges necessary to perform their tasks:
- Secret Manager Admin：Assigned to individuals with complete control over secrets, their versions, and lifecycle management. This role is usually limited to a security admin group and not widely shared.
- Secret Version Manager / Secret Version Adder：Assigned to DevOps engineers or service accounts involved in managing the lifecycle of secret versions (such as adding, enabling, disabling, or destroying) or handling rotation workflows. These roles do not permit any changes to secret properties.
Fulfilling Customer requirements
Input from customers who worked with us on the development of delayed destruction has enhanced our resilience to cyber attacks and improved backup processes.
“The ‘delay destruction of secret versions’ capability in Secret Manager helps us defend against disruptive attacks like ransomware or accidental deletion of important secrets in a simple and effective manner. As a financial services company, the security and resilience of our IT platforms is at the heart of what we do,” said Dominik Raub, chief information security officer, Crypto Finance AG, a Deutsche Börse Group company.
“We store credentials and key materials in Secret Manager to ensure secure access for our applications. Because these secrets are vital for our applications and infrastructure, they must be safeguarded against both malicious and accidental deletion,” he explained. “Previously, this was a significant challenge across all the cloud platforms we use. We reached out to our partners at Google regarding this issue and were impressed by their collaborative approach and the quick delivery of a solution. Now, we can protect our secrets from deletion with just a few clicks, eliminating the need for complicated backup procedures, all while maintaining the security provided by Secret Manager.”
If your organization is not currently using Google Cloud or Google Workspace, you can contact Google Cloud Elite Partner Microfusion to provide exclusive pre-sales consultation and professional technical support after purchasing Google Cloud and Google Workspace to solve your data relocation troubles and various difficulties in use. More importantly, Microfusion continues to provide customers with the latest news of Google Cloud, Google Workspace, technology news newsletters and themed webinars/physical workshops.
This article is translated and adapted from the official Google Cloud blog.
