Applications have become the backbone of our daily activities, from complex stock trading platforms to simple hotel or medical bookings. Yet, beneath the surface, the lifeline of these digital ecosystems relies on secrets – confidential information like database passwords, API keys, and access tokens.
The responsible and secure management of these secrets is non-negotiable, as any mishandling could result in severe breaches, jeopardizing data integrity and user trust in the digital realm.
Effective secrets management is paramount for all organizations. Secrets can inadvertently leak through various means, such as version control systems (avoid hardcoding secrets!), private messages, emails, and other channels.
Leaked secrets can erode trust and credibility, and even result in financial consequences and legal repercussions. Hence, a robust secrets management strategy is essential.
This blog will take you through different ways of managing secrets in Python.
Things to remember before starting
To work with this task, you will need the following:
1. Python and pip installed
Ensure that Python and pip are installed on your local machine.
2. Familiarity with python and command line interface (CLI)
Have a basic understanding of Python programming and be comfortable with using the Command Line Interface for executing commands and scripts.
3. Python Integrated Development Environment (IDE)
It is always recommended to use a Python Integrated Development Environment (IDE) like PyCharm or Visual Studio Code to facilitate coding and project management.
4. Fundamental cloud knowledge
Possess a foundational understanding of cloud computing concepts and services, as this task may involve interactions with cloud resources.
These prerequisites will enable you to effectively engage with the task at hand.
How can you manage secrets in python?
Managing secrets securely in Python is crucial for protecting sensitive information like API keys, passwords, and tokens. Here are four different ways to manage secrets in Python:
1. Environment variables
Environment variables, established by the operating system or users, serve as program settings.
They provide a means to securely store and access secrets within your code. To create an environment variable on macOS or a Linux system, employ the following syntax:
$ export variable_name=value
$ export API_KEY=testkeyTo access the operating system environment variable, you can utilize the “os” package. The sample code is as follows:
import os
# Replace 'VARIABLE NAME' with the name of the environment variable you want to access
variable_value = os.environ.get('VARIABLE NAME')
if variable_value is not None:
print(f"The value of VARIABLE_NAME is: {variable_value}")
else:
print("VARIABLE_NAME is not set.")
2. Configuration files
env File An .env file serves as a container for environment variables in Python. These variables, configured externally, shape the behavior of Python code.
Typically, .env files store sensitive information like secret keys and passwords.
For accessing the content of the .env file, we’ll employ the python-dotenv package. To commence, begin by installing the package via the following command:
pip install python-dotenv
For testing purposes, generate a .env file and insert the following confidential information:
# .env file
API_KEY=your_api_key_here
DATABASE _URL=y_our_database_url_here
SECRET_KEY=your_secret_key_here
DEBUG=True
After completing the previous steps, proceed to create a main.py file and incorporate the following code snippet. In this code, the load_dotenv() function is employed to retrieve data from the .env file.
from dotenv import load_dotenv
import os
# Load environment variables from the .env file
load_dotenv()
# Access environment variables
api_key = os.getenv("API_KEY")
database_url = os.getenv("DATABASE_URL")
secret_key = os.getenv("SECRET_KEY")
debug = os.getenv("DEBUG")
# Example usage of the loaded environment variables
print(f"API_KEY: {api_key}")
print(f"DATABASE_URL: {database_url}")
print(f"SECRET_KEY: {secret_key}")
print(f"DEBUG: {debug}")
Another approach is utilizing the dotenv_values() function, which transforms the secrets into a dictionary. You can access these secrets using the following code snippet:
from dotenv import dotenv_values
# Load environment variables as a dictionary
env_vars = dotenv_values(".env")
# Access environment variables from the dictionary
api_key = env_vars.get("API_KEY")
database_url = env_vars.get("DATABASE_URL")
secret_key = env_vars.get("SECRET_KEY")
debug = env_vars.get("DEBUG")
# Example usage of the loaded environment variables
print(f"API_KEY: {api_key}")
print(f"DATABASE_URL: {database_url}")
print(f"SECRET_KEY: {secret_key}")
print(f"DEBUG: {debug}")
In larger projects, managing multiple .env files becomes necessary. You might have a .env file for your local development environment and a env.dev file for your cloud development production environment.
To handle such scenarios effectively, consider the following code snippet:
from dotenv import dotenv_values
secrets = dotenv_values(".env")
local_secrets = dotenv_values(".env.dev")
def main():
api_key = secrets.get("API_KEY")
api_secret = local_secrets.get("SECRET_KEY")
print(f"API_KEY: {api_key}")
print(f"API_SECRET: {api_secret}")
if __name__ == "__main__":
main()
3. JSON file
For better organization of secrets, you can opt for a JSON file. Let’s create a secrets.json file and insert the following confidential information into it:
{
"API_KEY": "your_api_key_here",
"API_SECRET": "your_api_secret_here",
"DATABASE_URL": "your_database_url_here",
"SECRET_KEY": "your_secret_key_here",
"DEBUG": true
}
With the JSON file prepared, let’s create a function to retrieve secrets from it:
import json
def get_secrets():
with open('secrets.json') as secrets_file:
secrets = json.load(secrets_file)
return secrets
def main():
secrets = get_secrets()
api_key = secrets.get("API_KEY")
api_secret = secrets.get("API_SECRET")
database_url = secrets.get("DATABASE_URL")
secret_key = secrets.get("SECRET_KEY")
debug = secrets.get("DEBUG")
# Example usage of the loaded secrets
print(f"API_KEY: {api_key}")
print(f"API_SECRET: {api_secret}")
print(f"DATABASE_URL: {database_url}")
print(f"SECRET_KEY: {secret_key}")
print(f"DEBUG: {debug}")
if __name__ == "__main__":
main()
4. Clouds secret manager
Major cloud service providers offer integrated secrets management solutions to safeguard sensitive information. Notable options include:
AWS Secrets Manager:
Amazon Web Services’ solution for managing, rotating, and retrieving secrets and credentials.
Secret Manager (by Google Cloud):
Google Cloud’s offering for secure storage and retrieval of sensitive data.
Azure Key Vault:
Microsoft Azure’s service for managing keys, secrets, and certificates.
AWS Secrets Manager enjoys widespread adoption. To interact with it programmatically using Python and Boto3, let’s write a function for creating and accessing secrets.
import boto3
def fetch_secret_from_aws(secret_name):
try:
session = boto3.session.Session()
client = session.client(service_name='secretsmanager', region_name='us-east-1')
get_secret_value_response = client.get_secret_value(SecretId=secret_name)
return get_secret_value_response['SecretString']
except Exception as e:
print(e)
return None
def create_secret_in_aws(secret_name, secret_value):
try:
session = boto3.session.Session()
client = session.client(service_name='secretsmanager', region_name='us-east-1')
client.create_secret(Name=secret_name, SecretString=secret_value)
return True
except Exception as e:
print(e)
return False
5. KMS
A Key Management System (KMS) plays a pivotal role in securely managing cryptographic keys, ensuring their integrity, and controlling their access.
It’s a cornerstone for organizations to generate, store, distribute, revoke, and monitor keys.
KMS simplifies centralized key management, strengthening security and safeguarding keys for authorized users.
HashiCorp Vault stands out as a leading open-source KMS solution. It offers a plethora of features and benefits, including versatile secret and key management across various environments, robust security controls, and impressive scalability.
Let’s proceed to create a function for the purpose of reading and writing secrets to a specific path within HashiCorp Vault:
import hvac
def read_secret_from_vault(path, token):
try:
client = hvac.Client(url='http://your-vault-url:8200')
client.token = token
response = client.read(path)
if response and 'data' in response:
return response['data']
else:
print("Secret not found at the specified path.")
return None
except Exception as e:
print(f"Error reading secret from Vault: {e}")
return None
def write_secret_to_vault(path, data, token):
try:
client = hvac.Client(url='http://your-vault-url:8200')
client.token = token
response = client.write(path, **data)
if response and response['request_id']:
print(f"Secret written to Vault at path: {path}")
return True
else:
print("Failed to write secret to Vault.")
return False
except Exception as e:
print(f"Error writing secret to Vault: {e}")
return False
Summing up
Managing secrets is a critical aspect of application development. When developers embed secrets in plaintext directly into their applications, it introduces a significant security risk.
If these secrets are exposed or discovered, malicious actors can exploit them to gain unauthorized access to sensitive data.
Alternatively, one can adopt another approach, which involves storing secrets in the source code repository but in an encrypted form, and then securely sharing them with the development team.
This method offers flexibility and security if you leverage a tool such as Mozilla SOPS.
By utilizing Mozilla SOPS, you can encrypt sensitive information within your source code repository.
This approach enhances security because even if the repository is accessed by unauthorized individuals, the secrets remain protected as long as they are properly encrypted.
Furthermore, team members with the appropriate decryption permissions can access and use these secrets as needed, ensuring that the sensitive data is kept secure throughout the development process.
In short, Mozilla SOPS provides a robust solution for securely managing secrets within a development team, minimizing the risk of security breaches while maintaining flexibility and collaboration.
