A system for distributing and managing secrets


Keywhiz is a system for managing and distributing secrets. It can fit well with a service oriented architecture (SOA). Here is an overview in presentation format

Every organization has services or systems that require secrets. Secrets like:

  • TLS certificates/keys
  • GPG keys
  • API tokens
  • database credentials

Common practices include putting secrets in config files next to code or copying files to servers out-of-band. The former is likely to be leaked and the latter difficult to track.

Keywhiz makes managing secrets easier and more secure. Keywhiz servers in a cluster centrally store secrets encrypted in a database. Clients use mutually authenticated TLS (mTLS) to retrieve secrets they have access to. Authenticated users administer Keywhiz via CLI. To enable workflows, Keywhiz has automation APIs over mTLS.

Keywhiz is reliable and used in production, however some upcoming changes may break API backward compatibility. See our roadmap.

System components

Keywhiz Server

Keywhiz Server provides JSON APIs for accessing and managing secrets. It is written in Java and based on Dropwizard.


KeywhizFs is a FUSE-based file system, providing secrets as if they are files in a directory. Transparently, secrets are retrieved from a Keywhiz Server using mTLS with a client certificate.

Presenting secrets as files makes Keywhiz compatible with nearly all software. Outside of Keywhiz administration, consumers of secrets only have to know how to read a file.

KeywhizFs stores all secrets in memory only and never persisted to disk. If KeywhizFs is unmounted or the server loses power, all secrets will be safely removed from that server.

To mitigate a Keywhiz Server outage, KeywhizFs maintains a local cache of previously accessed secrets. Unless the server is rebooted or KeywhizFs unmounted, applications can happily continue accessing secrets previously accessed.

Keywhiz CLI

Keywhiz CLI is a Java program for Keywhiz administration. Clients, secrets, and groups can be queried, added, removed, or associated with each other. Users can authenticate and use the CLI.

Public Key Infrastructure

Keywhiz makes heavy use of mTLS and X509 certificates. It can even help distribute and rotate them for other services! There is the assumption of a PKI system though. If one does not exist or a PKI is wanted for development consider certstrap for a simple, initial PKI.

Data model

The data model is composed of clients, secrets, groups, and users.

Clients are identified by a client certificate. A database entry can enable/disable access or escalate automation privileges.

Secrets have a globally unique name and can contain any binary content. For example, helloworldsrv-database.yml. Secrets are immutable, but multiple ordered versions can exist at once. Arbitrary key-value metadata can be added for extensibility.

Groups are the glue binding clients and secrets. Clients are assigned membership into groups (e.g. for the service-on-host, service, etc.). Secrets are also assigned to groups. If a client and secret share any group, the client will have access to the secret. Groups are not hierarchical and cannot be in other groups.

Users are authenticated Keywhiz administrators. Authentication is customizable with LDAP and bcrypt-hashed passwords currently supported. After initial authentication subsequent requests require an encrypted cookie from Keywhiz Server.


Source code for the Server and CLI is available in square/keywhiz. KeywhizFs source is in square/keywhiz-fs. To checkout the Keywhiz source code:

$ git clone && cd keywhiz

Starting the Server

Keywhiz can store data in PostgreSQL, MySQL or H2. H2 is the simplest database for development purpose and all the data is stored in /tmp/h2_data/keywhizdb_development. For production systems you should use PostgreSQL or MySQL.

From the base of the keywhiz repository, build the server:

$ mvn package -am -pl server -P h2

Run any migrations:

$ java -jar server/target/keywhiz-server-*-SNAPSHOT-shaded.jar preview-migrate server/src/main/resources/keywhiz-development.yaml
$ java -jar server/target/keywhiz-server-*-SNAPSHOT-shaded.jar migrate server/src/main/resources/keywhiz-development.yaml

Add an administrative user:

$ java -jar server/target/keywhiz-server-*-SNAPSHOT-shaded.jar add-user server/src/main/resources/keywhiz-development.yaml

Start the server:

$ java -jar server/target/keywhiz-server-*-SNAPSHOT-shaded.jar server server/src/main/resources/keywhiz-development.yaml

Building the CLI

From the base of the keywhiz repository:

$ mvn package -am -pl cli

Run the CLI and get a usage statement:

$ ./cli/target/keywhiz-cli-*-SNAPSHOT-shaded.jar

In development, you can use --devTrustStore, e.g.:

$ ./cli/target/keywhiz-cli-*-SNAPSHOT-shaded.jar --devTrustStore --user keywhizAdmin list groups

You may want to alias this command for convenience:

$ alias keywhiz.cli="/path/to/keywhiz-cli-*-SNAPSHOT-shaded.jar"

Mounting KeywhizFs

Refer to the KeywhizFs README.


More examples are available in the Wiki, but here are a few.

In development, you can use --devTrustStore and these examples assume that you want to use the default keywhizAdmin user.

In production, you will want to create your own users and make sure that your certificates are properly signed and trusted.

Adding a Secret

Using Keywhiz CLI
$ keywhiz.cli --devTrustStore --user keywhizAdmin login
$ keywhiz.cli add secret --name mySecretName < mySecretFile
Using Keywhiz automation API

The automation API requires a client certificate and automationAllowed=true in the clients DB table. For development purpose, you can use the pre-generated client.p12 keystore:

$ cat request.json
  "description":"example kerberos keytab",
$ curl --cert ./server/src/test/resources/clients/client.p12:ponies -H "Content-Type:application/json" -d @request.json https://localhost:4444/automation/secrets/

Assigning a Secret to a Group

Using Keywhiz CLI
$ keywhiz.cli --devTrustStore --user keywhizAdmin assign secret --name example.keytab --group kerberos
Using Keywhiz automation API
$ curl --cert ./server/src/test/resources/clients/client.p12:ponies -X PUT "https://localhost:4444/automation/secrets/$SECRET_ID/groups/$GROUP_ID"


Latest JARs

Source code for Keywhiz and this website is available on GitHub.


Upcoming features

  • Replace KeywhizFS to remove FUSE reliance
  • Redesigned, more coherent APIs
  • Database integrity checks


If you would like to contribute code to Keywhiz you can do so through GitHub by forking the repository and sending a pull request.

When submitting code, please make every effort to follow existing conventions and style in order to keep the code as readable as possible. Please also make sure your code compiles and passes tests by running mvn clean verify.

Before your code can be accepted into the project you must also sign the Individual Contributor License Agreement (CLA).


Copyright 2015 Square, Inc.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
See the License for the specific language governing permissions and
limitations under the License.