How Binance's Secure, “Break-Glass” Host Access System Works
Main Takeaways
Accessing the host can be a security concern, but is necessary in an emergency.
Binance has developed a system that allows for “break-glass” host access for emergencies while maintaining security. Additional permissions can also be requested when needed
Host access and specific cmd permissions are granted via an internal "ChatOps" with both high efficiency and security.
All stakeholders can efficiently discuss the case’s background in a dynamically created chat group . After a single click approval or rejection, all stuff is logged and can be subsequently audited for security reasons.
Learn more about how we balance security and troubleshooting with a Break Glass Access System.
Why Did We Create this Solution?
An experienced system engineer has likely been in a data center in the dead of night at least once. Even with expertly designed and implemented services, products, and systems, there is always an unknown element.
As Werner Vogels, Chief Technology Officer of Amazon, said, “Everything fails, all the time.” Most cases don’t require intervention as the service self-heals. However, occasionally, it becomes necessary to dive into the issue’s epicenter: the host itself.
Granting access to a host is a double-edged sword. While it could help to identify the root cause of the problem, it raises security concerns, including
How can we know who is doing what?
How can we provide on-demand access?
Who approves this access?
How can we limit privilege escalation within the host?
To address these concerns at Binance, we run privilege escalation in a tiered structure while troubleshooting within the instance.
We’ve also implemented ChatOps to streamline approval operations, making everything accessible, auditable, tidy, and efficient. As an event-driven approach like ChatOps provides speed, we can create ephemeral, tiny war rooms to isolate the context and discussion.
Pieces of the Puzzle
In Binance, our engineering team created generic APIs to interact with chat tools for many purposes, including our Break Glass Access system.
The Break Glass System comprises:
An ephemeral Linux user environment with a restricted set of shell commands.
A tiered privilege system integrated with a chat tool for requesting additional permission.
A provisioner for running privilege escalation as a backend.
Ephemeral restricted shell
When users connect to the host, their user IDs and home directories are created and deleted once all sessions are logged out. This dynamic environment brings some advantages to us, including:
Easy user management among thousands of instances in a dynamic cloud environment.
An immutable host environment after troubleshooting.
Restricted Bash, also known as rbash, is a limited version of the standard Bash shell. It was designed to provide a more controlled environment for users with restricted access.
In our Break Glass Access scenario, we confine users to a predefined set of commands and disable certain features. These features include setting environment variables, executing commands, using redirections, etc., rbash effectively prevents unauthorized actions and safeguards sensitive information.
We’ve also improved how this system works and introduced restrictions to command parameters. For example, in some situations, a restricted user can use docker ps or docker log but can not use any other docker command unless elevated to a suitable Tier level.
Tiered privilege escalation system
We have implemented a tiered privilege system to cater to different troubleshooting scenarios and provide more fine-grained access control. Users can request access to Tier 2 or Tier 3 through the request CLI developed by Go.
Tier 2 provides a predefined set of commands (that we define) executable within the rbash. This tier is suitable for situations where users need to perform specific tasks that don’t require full administrative access.
On the other hand, Tier 3 grants sudo access, allowing users to perform more advanced troubleshooting with different available command sets. Even with this highest privileged access, they can use sudo but are unable to switch users.
The tiered system offers several advantages ;
Flexibility: Users can request the appropriate level of access based on their troubleshooting needs.
Security: Limiting access to only the necessary commands or privileges minimizes the risk of unauthorized actions.
Auditability: Each tier request is logged, providing a clear audit trail of who requested access, when, and for what purpose.
Efficiency: The tiered system ensures that users have the right level of access to resolve issues quickly without compromising security.
Users can use request CLI to list their requests and check the status of their approvals. The CLI provides transparency, making it easy for users to track their requests and communicate with approvers if required.
From the reviewer's perspective, they see a chat group with required information about the host, project, tier, and user. The requestor is also a member of this chat group, so there is no friction between the reviewer and the requestor. Reviewers can ask any questions and understand why this has been requested.
Provision pipeline
Upon approval, a provision pipeline will make the command in the approved Tier available to the user in a specific host. The provision pipeline will symbolically link enabled commands on the user’s $HOME/bin directory, where the user can run those commands without any restriction.
In summary;
All remote shell access activity is logged and monitored.
Every engineer with remote access to an instance is limited by a restricted shell by default.
Every remote shell access user has a right to request a privilege escalation.
Every privilege escalation request starts a chat group between the user and the approvers.
Remote shell access users can have immediate effect once their request is approved.
Overall, our Break Glass Access solution balances necessary access for critical tasks and maximum security by implementing a tiered privilege escalation system and leveraging Restricted Bash.
Closing Thoughts
When developing our secure “Break-Glass” Access system, it was essential to prioritize security principles such as security by design, least privilege, and separation of duties.
Having a tiered privilege escalation system allows for granular control over access levels. Restricted shell and ephemeral user environments also help maintain security when necessary access is allowed. Finally, integrating ChatOps ensures engineers and reviewers an easy and streamlined user experience.
We believe organizations may find the balance between providing necessary access to engineers and maintaining robust security aligned with a well-known principle; “trust, but verify.”