SOPS, short for Secrets Operations, is a unique open-source text file editor that seamlessly combines text editing and file encryption/decryption into a single, user-friendly workflow.
Traditionally, when you want to secure a text file, you typically follow these steps:
Use your preferred text editor for creating, editing, and managing the text content, and save it as a file.
Employ a separate encryption/decryption tool to encrypt the entire file.
When you need to access the encrypted file: initially, you must decrypt the file using an encryption/decryption tool.Subsequently, open the decrypted file (now in plain text format) using your preferred text editor.
The drawback of this conventional approach is quite apparent: it necessitates the use of two distinct tools (a text editor and an encryption/decryption tool) to accomplish a single task.
SOPS, on the other hand, streamlines this process by integrating both text editing and encryption/decryption capabilities within a single application.
This eliminates the need for switching between multiple tools and simplifies the workflow, making it more efficient and user-friendly.
How does SOPS work?
In essence, SOPS offers a seamless combination of encryption/decryption and text editing in a single, fully automated package. Its standout feature lies in its automated workflow:
1. File creation
When you create a file using SOPS, it automatically employs the encryption method of your choice to encrypt the file before saving it to your storage.
This entire process occurs without any manual intervention. Consequently, if you attempt to open the file using a different text editor, you won’t be able to access its content due to the encryption.
2. File retrieval
When you open an encrypted file with SOPS, it takes care of both decrypting the file and presenting its contents to you.
While this may seem like a two-step operation, SOPS seamlessly handles it, making it appear as a single, user-friendly step for the end users.
In summary, SOPS streamlines the encryption and decryption of text files within an automated workflow, simplifying the user experience and enhancing security.
What are the benefits/highlights of SOPS?
1. Flexibility
SOPS offers remarkable flexibility by supporting a wide range of encryption methods, including:
GPG (for those unfamiliar, read on)
AWS KMS
GCP KMS
HashiCorp Vault
This versatility simplifies the management and editing of sensitive files. Whether you prefer to edit a text file using a local GPG key or need to work with an ENV file encrypted by HashiCorp Vault, SOPS provides a solution.
There’s no need to stick to a single encryption method for all your files because SOPS offers high customization, allowing you to tailor encryption to your specific needs and preferences.
2. SOP with PGP keys
In brief, GPG, or GnuPG, is a command-line tool that implements the PGP (Pretty Good Privacy) encryption program.
PGP, which stands for Pretty Good Privacy, is a robust encryption program designed to provide cryptographic privacy and authentication for various forms of data communication.
It serves multiple purposes, including signing, encrypting, and decrypting texts, emails, files, directories, and even whole disk partitions, all aimed at enhancing the security of communication.
On the other hand, GPG, or GnuPG, is a free implementation of the OpenPGP standard, as defined in RFC4880. GPG is a versatile command-line tool that enables users to encrypt and sign their data and communications. It also offers an extensive key management system and access modules for various public key directories.
Now that we’ve clarified this potentially confusing nomenclature, we can proceed to create a PGP key using the GPG command-line tool. Subsequently, we’ll explore how SOPS functions in conjunction with this key.
Creating PGP Key with GPG
SOPS Configuration
SOPS with Text Files
SOPS with structured Data
3. SOPS with HashiCorp Vault
Let’s provide a concise introduction to HashiCorp Vault. At its core, Vault serves as a secrets manager but offers a broader range of capabilities, including encryption management, key management, and more.
It’s an open-source project that you can deploy within your own infrastructure, and HashiCorp also offers a cloud-based service.
Now, for our SOPS tutorial, we need to establish a local Vault service for testing purposes. The most efficient way to achieve this is by utilizing Docker.
$ docker run -d -p8200:8200 vault:1.2.0 server -dev -dev-root-token-id=toor
Once you have set up Vault locally within a Docker container, you can verify its functionality by executing the following commands:
After successfully launching Vault and ensuring it’s operational, let’s proceed to generate a key that SOPS can utilize:
4. SOPS with AWS KMS
Creating a cryptographic key in AWS KMS (Key Management Service) using the AWS CLI is a straightforward process. If you’ve already installed the AWS CLI and configured it with the necessary permissions, you’re ready to go. Here’s how you can create an AWS KMS key:
The need for SOPS: Simplifying secrets management
Let me share a story from my past experience that highlights the importance of a tool like SOPS.
Years ago, I was deeply involved in a significant fin-tech project that revolved around a complex OpenShift cluster, essentially a customized Kubernetes environment.
Within this cluster, we used namespaces to segregate different environments, and at the heart of our infrastructure lay OpenShift as our primary secrets management solution.
Here’s how it worked: whenever we needed to modify the value of a secret, we’d navigate to the OpenShift console and make the necessary changes there.
If a secret was required in a Jenkins pipeline, we’d simply add it as an OpenShift secret, and it would automatically synchronize with Jenkins. Everything appeared to be running smoothly.
However, things took a turn when we needed to share a secret with a team member or migrate a secret to another cluster or environment. To achieve this, we had to export secrets as Kubernetes secret YAML files and then share those files. This process introduced two major challenges.
Firstly, it posed a security risk. What if one of these YAML files ended up in the wrong hands? It could potentially expose sensitive information to unauthorized individuals.
Secondly, it hampered productivity.
Since our YAML files couldn’t automatically sync with the OpenShift cluster, we ran the risk of working with outdated secrets. Keeping these files up to date was a cumbersome and error-prone task.
In general, relying solely on Kubernetes native secrets as our single source of truth wasn’t an ideal practice. Yet, consuming secrets from external sources within a Kubernetes environment wasn’t straightforward either.
Even if the secrets were meant for internal cluster use only, we had to create hardcoded YAML copies. The questions that arose were: Where should we store these YAML files, and how could we securely share them?
Enter SOPS. This tool was a game-changer for us. It allowed us to encrypt sensitive files with minimal overhead.
With SOPS, storing and sharing secret files became hassle-free, and we could rest assured that our secrets were well-protected.
What are the limitations of SOPS?
Let’s be crystal clear. SOPS is not a secrets manager. While it can serve as a quick and cost-effective solution for storing sensitive information in encrypted files, its utility has its boundaries.
As your use cases expand, you’re likely to encounter limitations. Specifically, relying solely on SOPS for secrets management may not be the best choice, and here’s why:
1. Limited to file format
SOPS primarily deals with secrets in file format. While this is suitable for certain scenarios, you’ll inevitably reach a point where you require API access to your secrets.
SOPS isn’t designed for this purpose, and attempting to build a custom solution on top of it would likely result in a clunky and impractical setup.
2. Lacks comprehensive secrets management features
Secrets managers are purpose-built tools designed to offer flexibility in handling secrets. They support various formats, policies, and access methods for your secrets.
This includes storing secrets as files, Kubernetes native YAML configurations, access from applications, by authorized users, and more.
Modern secrets managers come equipped with numerous integrations, making them versatile solutions for secrets management.
3. Automation and integration
Secrets managers excel in providing automation and integration capabilities.
They seamlessly integrate with command-line interfaces (CLIs), continuous integration/continuous deployment (CI/CD) systems, orchestration platforms, and other tools.
This ensures that all your secrets-related needs can be met without the need to manually encrypt and manage files.
4. Discovering an alternative: Doppler
Recently, I came across a promising newcomer: Doppler. This innovative tool can automatically synchronize secrets with a Kubernetes (K8s) cluster, eliminating the necessity to store K8s secrets in YAML files.
Even more impressively, Doppler can initiate the restart of a pod when a dependent secret undergoes an update.
This not only streamlines secrets management but also enhances the security and efficiency of your Kubernetes environment.
In summary, while SOPS serves a valuable purpose in certain scenarios, it’s crucial to recognize its limitations.
When your secrets management requirements evolve beyond file-based solutions, it’s wise to explore dedicated secrets managers like Doppler and others. These tools are equipped to handle a wide array of use cases and offer the automation and integration capabilities needed to keep your secrets secure and accessible.
