Into the Cloud Conclusion – Securing Applications

Introduction

Applications need to be developed primarily with security in mind.  A lot of things are involved but I will focus on two things in this blog post:

  1. We need to know which actors are using our system.
  2. We need to expose features only to users that need them, and have privilege to access them.

The first one deals with authentication – granting a user some sort of trusted credential (like an ID card). The second one deals with authorization – granting that credential a set of entitlements.

Architecture

In the diagram below, we will be authenticating users through Amazon Cognito. Cognito will be configured to authenticate through three methods:

  1. Using local amazon user pools (users signing up though our app)
  2. Using facebook (users signing in through their facebook login)
  3. Through google (users accessing the app with their google account).

Security architecture showing integration with Amazon Cognito using facebook and google identity providers

Standards

Like I stated in my earlier posts, it is important to design a security architecture that leverages standards so that it can be interoperable with other systems. We used the OpenID protocol. This made it easier to integrate using Angular Auth OIDC Client for the user interface and Spring Security Oauth2 library for the backend.

Authentication

How can we trust that the token passed to the backend service comes from the user instead of a malicious client? How do we identify the user from the token? How do we ensure that the right token is delivered to the client for use?  How do we ensure that users have a good user experience when logging in to the application? These are the questions you need to ask yourself when designing an authentication system.

Gaining access to the System and Single Sign On (SSO)

We implemented authorization code grant flow using (PKCE) on the angular user interface. The authorization code grant type flow redirects login to the authorization provider (Amazon Cognito in this case). The login and signup forms are served by Amazon Cognito. This means that the user’s sign in and signup experience can be customized by changing the Cognito configuration. A few things can be achieved:

  1. Sign-in though external identity providers like Facebook, Google and other OpenID providers
  2. Sign-In through SAML providers.
  3. Enabling multi factor authentication according to the security requirements of your application.
  4. Configuring the session properties like access token validity period, session time, validity of refresh tokens and so on.
  5. Determining what user info is needed
  6. Styling the sign up and login pages.

Signing in through external identity providers help you to login to multiple applications without signing in. For example, once I have logged in to my google account, by clicking the google login, I do not need to enter my user name and password again. This approach provides a seamless way for users to access applications.

One of the major security vulnerabilities of the authorization code grant flow is that the code returned by the authorization server can be intercepted and a malicious actor can use that code to obtain an access token on behalf of the user. This is because public clients are normally used with authorization grant type and no client secrets are used. A common way to mitigate this is using Proof of Key for Code Exchange (PKCE). Most OpenID clients (including angular-auth-oidc-client) and authorization providers (in this case Cognito) support this. This method prevents interception because:

  1. Before the redirect is made to the authorization provider, the client generates two cryptographically related tokens: the code challenge and the code verifier.
  2. Only the client knows these token pairs and their relationship can be validated using cryptographic algorithms.
  3. The client adds the code challenge to the authorization request and the authorization server keeps track of the code challenge by associating it with the request along with the generated authorization code.
  4. The client is required to pass the authorization code and the code verifier in order to obtain the access token. Since only the client knows the verifier, no malicious user can obtain the token without the verifier.
  5. The authorization server validates the relationship between the challenge and the verifier using the selected cryptographic algorithms.

Benefits of using OpenID

In this application, the front end and backend strictly integrates with Amazon Cognito using the OpenID protocol. Because of this:

  1. I can reuse established open id libraries without re-inventing the wheel
  2. I have a solid foundation on the security flow since the protocol is well documented
  3. I can switch authentication providers and use others like Keycloak or Okta without implementation code changes.

Proof of Identity

Once the client makes a token request with a valid authorization code and code verifier, Amazon Cognito issues three tokens:

  1. An access token
  2. An identity token
  3. A refresh token.

These tokens are cryptographically signed by Cognito using public-key cryptography as per the OpenID standard using the JWT specification. In this specification:

  1. The authorization server maintains a key pair. A private key (kept secret by the authorization server) that it uses to sign generated tokens and a public key that it exposes for resource servers to retrieve.
  2. The client attaches the access token to a request.
  3. The resource server validates the request by validating the signature of the token using the public key.

In short, proof of identity is established based on trust. If the resource server can confirm that the token comes from the authorization server, then it trusts that information. The token is a JWT token that contains claims that identify the client. One of those claims is the username claim that contains the user id of the user.

Authorization

Once the identity of the actor has been established, the system needs to know the capabilities of the actor. What is this user allowed to do on this system? This is implemented in different ways in different systems. There are two aspects of authorization:

  1. Obtaining the user entitlements
  2. Enforcing the user entitlements

Both aspects can be taken care of by the same, or different authorization systems. Once practice I find useful is to de-couple authentication from authorization because you will, most likely, not use the same solution for both and you do not want to be tied down to a solution because you cannot isolate them in your logic.

In my case, I am only using Cognito for authentication. Cognito has a native integration with Amazon Verified Permissions which handles the above two aspects through authorization policy configuration and enforcement. Because I isolated both in my design, I am free to start with a much simpler authorization system using database based role-based-access control. In future, if I want to use something more elaborate like the Amazon Verified Permissions, I can easily integrate.

Like I said in the begining of the Authorization section, authorization behaviours fall into two categories:

  1. Those that handle 1 and 2 together, all you have to do is simply ask the authorization server the question “does the user have access to this resource?”
  2. Those that handle 1 and expect that you handle 2. They provide you with the list of the user’s entitlements and it is up to you to enforce that.

I am currently using the second approach, retrieving the user’s entitlement from the database.

Backend Authorization

Being a spring boot service, authentication and authorization are handled using spring security. I implemented a custom granted authority resolver. This is an implementation of spring’s Converter interface that converts a JWT token to a collection of granted authorities.

public class AppJwtGrantedAuthorityResolver implements Converter< Jwt , Collection<GrantedAuthority> > {

@Override
publicCollection<GrantedAuthority>convert(Jwtsource){
…
 }
}

}
 

Spring security already has APIs that help you to enforce access based on what entitlements the user has. So I can configure things like:

  1. Users can only have access to the GET ‘/hymns’ endpoint if they have the ‘hymns.view’ permission.
  2. Users can only have access to the POST ‘/hymns’ endpoint if they have the ‘hymns.create’ permission.

Front-end Authorization

Front-end authorization follows the same approach as backend but in this case, instead of restricting apis, we restrict the following:

  1. Access to pages on the app
  2. Visibility of components
  3. Functionality (enabling or disabling) of certain features or components.

These can be achieved in angular by:

  1. Implementing route guards
  2. Using entitlement based directives
  3. Using conditional statements applied to templates.

NOTE: It is advisable to implement both server and client size authorization.

Back to Amazon Verified Permissions

Amazon Verified Permissions is a user authorization service newly introduced by amazon that enables you to define flexible authorization rules that specify when and how much access a user has. These rules are defined using the open source Cedar language for specifying access control.

Like I said earlier in this post, as a resource server, you ask Amazon Verified Permissions if a user has access to the intended request. This service takes that authorization request, passes it though a set of rules you defined using the cedar language, and responds with an ALLOW or REJECT response.

This approach is extremely flexible and allows you to specify access rules based on any user and request criteria. Application teams can configure their access requirements in a manner that helps them efficiently manage access controls.

One drawback is that calls has to be made to the authorization API for every request and this can be costly and inefficient. This can be mitigated by:

  1. Caching the authorization decisions
  2. Batching authorization requests for multiple functionalities using the batch authorization API.

Secret Management

I cannot conclude without talking briefly about how to manage secrets. Database and private key credentials need to be stored securely to avoid malicious users gaining access to your system. The AWS Systems Manager provides a parameter store that enables you to store different parameters including secrets. This stores secrets as SecureStrings that are encrypted using the default AWS managed encryption key or a customer managed encryption key that you can manage. Encryption keys can be created using the Amazon Key Management Service.

Using the default AWS managed key is free, but you can only access the secret from within the account, using AWS APIs. If you need more flexibility and you want to share secrets with multiple AWS accounts, you will need to create a customer managed key, and of course, this is not free.

The EC2 Journey Unfolds – Routing Traffic

Introduction

In my earlier article I discussed my experiences with the Amazon EC2 instance. I also talked about the VPNs and security groups that are needed to set up the network at which the EC2 virtual machine resides. Today, I will detail my approaches to getting network traffic to the virtual machine.

Routing Architecture

Routing Architecture

In my simple scenario, I am using the same EC2 instance to host my Nginx router (which serves my Angular resources for the front-end) and the Spring Boot service which runs the Hymns API. The Nginx server listens to port X and the spring boot service listens to port Y. Both service use the HTTP protocol.

Target Groups

Target groups are what AWS uses to know where to route traffic to. After AWS determines the IP address, it need to know which port and protocol to use to send the data packets. This is what a target group provides.

The Big Picture

I plan to build many applications in future, as well as APIs that other developers can use to collaborate with. I will be using mariarch.com to serve web applications and pages, and mariach.io to serve collaboration APIs. Although the routing solution I will start with will be simple, I will have to bear this in mind so that my system can evolve appropriately.

The routing big picture. This shows how user interface and backend traffic will be routed.

Flow for API Traffic

  1. Client makes an API call
  2. The request reaches an API gateway
  3. The API gateway routes the requests based on the sub-domain to the required target group
  4. The request reaches the target group.

Flow for User Interface Traffic

  1. Client loads a page on the browser
  2. Requests from the browser are received by a load balancer
  3. The load balancer routes the request based on domain to the required target group
  4. Target groups define the port and protocol that an Nginx server listens to
  5. An Nginx server can be configured to host multiple applications and will contain rules to serve the correct asset based on the subdomain used in the request
  6. The Nginx server returns the requested access.

Api Gateways with Amazon API Gateway

This is by far the most feature backed approach to take and also, not the simplest to configure. AWS provides two categories of HTTP traffic routing that you can use: The HTTP api and the REST api. I was initially confused as to the difference between these two offerings and, on studying the documentation of these features I got to understand their peculiarities. I will briefly summarize them below.

HTTP APIs only provide basic routing and you can create APIs based on region. You can configure these apis to use Mutual TLS authentication (HTTPS), and authorize using IAM, amazon cognito, custom authorization using AWS lambda functions and JWT tokens.  You can also use custom domains on these APIS.

Rest APIs build on all the features of HTTP APIs and they can be edge-optimized and private. You can set up these apis to use certificate authentication or authenticate using AWS’s custom WAF (Web Application Firewall). You can also configure resource polices to achieve fine grained authorization. Also, you can create API keys that your API clients will authenticate with and enforce rate limiting and throttling per client. Please see the developer guide for more details about these two offerings.

Typical AWS Api Gateway Configuration

Typical AWS API Gateway configuration showing how the API gateway routes traffic to a target group through a network load balancer.

The typical way to configure AWS API Gateway is through a network load balancer. AWS provides a detailed guide on how to do this and I happened to follow the same with no issues.

Simple API Gateway Using Application Load Balancers

Using the AWS API gateway will surely give you the most flexibility and will enable you to achieve much more without writing code. But this requires an extra component which will eventually add to your overall costs. If you don’t require the features that the API gateway provides (like me at this point who only wants a simple routing by sub domain name), you can simply use an application load balancer. It enables you to specify basic routing rules. In my case, I am using the application load balancer to route traffic by domain name.

Routing traffic using application load balancers.

Network versus Application Load Balancers

AWS provides different categories of Load Balancers of which application and network load balancers are the recommended and most popular. Network load balancers route traffic at the network level (i.e it uses protocols like TCP and UDP). This means that the rules you specify here are network related. Application load balancers, on the other hand, route traffic on an application level (using protocols like HTTP, HTTPS, e.t.c). This enables you to route based on HTTP related criteria like path, headers e.t.c.

Transport Layer Security (TLS) and Certificates

Users will not trust you when you do not use https (I mean, I get skeptical when I navigate to a site without https). Even worse are HTTPS sites that are flagged with the dreaded “certificate invalid” or  “site not trusted” messages. These messages are related to the certificate used to establish the secure connections.

AWS API Gateways and load balancers allow you to listen to secure endpoints by associating a certificate with them. AWS gives you an easy way to create and manage public certificates. These certificates are signed by AWS certificate authority and will show up as a legitimately signed certificate (check hymns.mariarch.com which was signed by a public AWS certificate). Certificates provided by AWS are managed using AWS certificate manager. This services allows you to create public certificates for free and will not charge you for using public certificates. You can decide to associate multiple domain names to a certificate or use wildcards. Please note that when creating certificates with AWS certificate manager, you will be required to validate the sub-domains associated with those certificates. Again, AWS provides a detailed, step-by-step, process on how to do this which involves associating a few domain records to your domain. If your domain is managed by AWS using Route53, it’s much more easier.

One thing to note using AWS public certificates is that:

  1. They are signed by AWS Certificate Authority
  2. You can only access the certificate details
  3. You can not access the private key of the certificate.

This effectively means that you can not use these certificate to secure any component apart from those that are natively integrated to AWS certificate management (i.e API gateways and load balancers). I learned this the hard way when trying to route traffic directly to my EC2 instance (you don’t have access to private keys of these certificates to configure any deployed service like nginx for http).

AWS Private certificates give you much more control and freedom to use any Certificate Authority of your choice. It also gives you access to the private keys as well. And, as you guessed it, it is not free. Please see the AWS documentation for certificate management for more information.

Routing Traffic Directly to EC2 Instances

Like you know, this is the simplest approach because it does not require any extra component from AWS. We just need to:

  1. Add network rules to the security group to permit traffic from the IP addresses you want ( you can permit all, but this is discouraged)
  2. Run a service that listens to a particular port. The default port for http is 80, that for https is 443 (If you want your urls to be without a port, you should use these defaults).
  3. To use TLS, you will need to use an AWS private certificate or obtain a certificate from an external provider.

This option, on its own, is discouraged because that the network interfaces of the EC2 instances are exposed directly. However, this can be a viable option of you already have routing providers outside of AWS. In this case, your network rules should only allow traffic coming from these routing providers.

So interesting right? Now that I have looked at routing traffic, I will circle back to securing APIs using Amazon Cognito in the next post.

First Deployment Journey – Starting a Project with AWS

Background

For the past two weeks, I had two goals – to explore AWS and to launch my software development business. These two goals are similar in the sense that they involve delving into a deep abyss of unknowns with a lot of experiments. It has been an awesome journey so far and of course, I will keep you updated.

Overview

My first goal this past month was to kick of an online presence of my new business – Mariarch. This will involve launching a proof-of-concept introduction of an ambitious project, to curate all the catholic hymns in the world and give Catholics an easy way to discover and learn hymns as we travel around the world. (If this is something you are interested in, by the way, please drop me an email). The objective was to deploy an initial, simple version of this project with bare-minimum features. (You can check the application here)

Architecture

To tie this to my second goal, Mariarch Hymns (the name of the project), needed to be designed using AWS infrastructure.

Initial architecture

The simplest way to have implemented this was a single server architecture with a server side UI framework like freemarker or to serve single page architecture UI clients from within the same server. This will consume the least resources. But since I have the vision of creating mobile applications and exposing these APIs for collaboration with other teams, it made sense to isolate the frontend from the backend implementation using Rest APIs.

Cheap as Possible is The Goal

AWS offers a reasonable free tier offering that I aim to leverage in this exploration and ensure that I spend as little as possible in this initial stage. While cost reduction is a goal, I will be designing in such a way that my application can evolve whenever its needed.

Avoiding Provider Lock-in

One of my core principles of design in this application is to avoid adopting solutions that will make change difficult. Because I am still in the exploratory phase, I need to be able to easily migrate away from AWS to another cloud provider like Azure or GCP if I ever decided that AWS does not meet my needs. In order to be free to do this, I had to ensure the following:

  1. Use open source frameworks and standards and tools where possible and avoid proprietary protocols. A good example is integrating my UI client with Amazon Cognito using the standard OpenID protocol as opposed to relying heavily on AWS Amplify.
  2. Integrating with secret management systems like AWS parameter store and AWS secret manager using a façade pattern to ensure that other secret providers can be easily integrated to the application system.

Software Stack

  1. Angular 18
  2. Java 21
  3. Spring boot 3.3.2
  4. Nginx
  5. AWS
  6. Mongo DB

Mongo V Dynamo DB

I’m pretty sure you are wondering why I chose Mongo DB as opposed to Dynamo DB since I am exploring AWS infrastructure. Dynamo DB is not suited for applications with complex queries to the database. It is a highly efficient key value store. Hopefully I will get into more detail if I have a need for the Dynamo. Fortunately, AWS partners with Mongo Atlas to provide managed Mongo Atlas databases in the same AWS region.

EC2 – The Core

At the heart of AWS offering are its Elastic Cloud Computing systems it calls EC2 (I’m thinking C2 means cloud computing). These are simply virtual machines that you can provision to run your services. AWS provides a host of configurations that allows you to pick the operating system, resource and graphics requirements that you need. (I will not get in to that since AWS has an extensive documentation on this). AWS also provides the serverless version, ECS Fargate, that enables you to focus on our application code rather than the virtual machine. On a level above this,  containers can be deployed through Docker, Amazon Elastic Kubernetes Service, or Amazon’s provision of the Openshift Red Hat Version. For this purpose, I used the simplest, the EC2 Amazon Linux virtual machine (the free tier configuration).

The Amazon Linux virtual machine is basically a Linux instance tailored for AWS cloud infrastructure. It provides tools to seamlessly integrate with the AWS ecosystem. You could do this on any other image version by installing the corresponding AWS tools.

EC2 instances generally run in their own Virtual Private Cloud. This means that you cannot communicate with EC2 instances unless you add rules to grant access to specific network interfaces like protocols and ports. One interesting thing to note is that while AWS provides a free tier offering for EC2, it doesn’t for VPCs and you need a VPC for every EC2 instance. This means that you will spend some money running EC2 instances since they need to run within a VPC even if you are on the free tier. I have attached my initial cost breakdown for the first 2 weeks of use of AWS. We see that while I’m on the free tier and all services I use are still free, I still need to pay for the VPC.

Cost breakdown showing VPC expenses.

Securing EC2 Instances

A VPC is defined by its ID and a subnet mask. Subnet masks are what is used to decide what IP addresses are used for that network. EC2 allows you to configure security groups that define who gets to access what interface of the virtual system. Security groups are reusable. They allow you to define multiple network access rules and apply these rules to different AWS components like other EC2 instances or load balancers. A network access rule comprises of the following:

  1. A type (e.g. HTTP, SSH, e.t.c)
  2. A network protocol (TCP, UDP)
  3. A port
  4. Source IPs (These can be defined by selecting VPC ids, or entering subnet masks or IP addresses).

As a rule of thumb, always deny by default, and grant access to components that need that access. Avoid wildcard rules in your security group.

Another aspect of securing EC2 instances is granting the instance permission to access other AWS services. In my application, I needed to retrieve deployment artefacts from AWS S3 in order to install them on the EC2 instance. To achieve this, the EC2 instance needs to be able to pull objects from S3. As discussed in the previous post, AWS provides a robust security architecture that ensures that all components are authenticated and authorized before they can access any AWS service. This includes EC2 instances.

IAM Roles enable AWS components to access other AWS services.  EC2 instances can be assigned AWS roles. This ensures that the credentials corresponding to the assigned roles are available for processes to use in that virtual environment. In IAM, you can configure a set of policies to be associated with an IAM role. Please check out the IAM documentation for more details.

Routing Traffic to EC2 Instances

Because EC2 instances run in their own sandbox, they cannot be accessed externally without certain measures. In the next post will go through three basic approaches:

  1. Simplest – Exposing the EC2 interface to the outside world
  2. Using an application load balancer
  3. Using an API gateway and a network load balancer.

Into The AWS Cloud – Part 1: Workforce Security, Organizations and the Identity Center

Introduction

We spent our first weekend in July in Center Parcs Longford, Ireland; a truly exciting and fun filled resort (fun for the kids, I was just playing aggressive catchup). We did a lot of swimming and playing with water and so did a million other people. When we came home, we started feeling the effects of being exposed to different environmental conditions. I fell sick, my son fell sick… and I think it has only just begun. Perhaps if I had been a bit more vigilant and prepared for these conditions (if that was possible).

This is how the concept of security is. Most people think security is about “preventing bad actors from damaging our system”. Yes, that’s one aspect. But it also entails being vigilant and watchful. It involves guarding yourself from – surprisingly – yourself. The last point is crazy right? But think about it. How many of us has left their child alone and the child opens the window (imagine if the window doesn’t have a child lock). How many of us has accidentally deleted a resource or configured something that we have no business configuring? If we simply didn’t have that access, that would not have happened.

AWS was built with security in mind and that is one of the reasons it excels so much and can be used for mission critical and sensitive applications. In this blog, and the other blogs in this series, I will not be going through what AWS already has in its documentation – It has extensive and well-formed documentation and guides for all its services. Rather, I will be going through what I call the “spirit, essence and use cases” for these aspects. This will enable you to have a high level understanding of these concepts before perusing the documentation.

AWS Approach to Security

My approach is simple, since we are not perfect and we make errors (some mistakes, some malignant), we need to be vigilant and guard ourselves, keep track of these errors we make so we know how to fix and prevent them. AWS seems to think in the same vein and has developed lots of services tailored to all aspects of security. Some of their guiding principles are:
1. Audit – all actions any one or machine makes are logged and can be tracked.
2. Least privilege access – Customers, employees and services can be given access to only what they need, at the time they need it and nothing more than that.
3. Universal Identity and hierarchies – Identities are uniform and can be grouped in such a way that policies are applied and enforced uniformly.
4. Separation of customers from workforce – Authentication and authorization flows, while they are conceptually the same, are different for customers and employees. This enables security flows tailored to improving customer experience for customers.
5. Zero trust architecture – This ensures that the authorization of customers, employees and services are checked immediately before a resource is accessed.
6. Relationship between identity and billing – All users are directly tied to AWS accounts which are used by AWS for billing. Because of this relationship, it is easy to track what resources are being used by which AWS account.

Authorization Approach

AWS authorizes all requests before they are processed to achieve a zero trust architecture. In a typical flow:
1. The client logs in and is authenticated.
2. The client makes are request to access a certain resource
3. AWS obtains the security context of this request. This context contains everything about:
a. The user
b. The AWS account behind that user and its associated organization hierarchy
c. What the user is trying to access
d. The user’s privileges
4. AWS evaluates the security context against a set of policies. These are rules that specify whether to allow or deny a specific request. A policy contains the following:
a. The resource pattern (to match the resource uri of the request)
b. The action e.g S3:ObjectDelete ( to delete an object on S3)
c. Set of conditions (expressions that specify when this rule should be used)
d. The effect property (Whether to allow or deny a request)
5. In a list of polices to be executed, the first matched policy is used and the request is either “Allow”ed or “Reject”ed based on the value of the Effect property.

The policy syntax allows us to achieve the following kinds of restrictions:
1. Allow all S3 actions for resources starting with “project-a/” if the user’s project is “project-a”
2. Allow all S3 actions for all resources if the user’s role is SystemAdmin
3. Deny all S3 actions for all resources if the role is Accouting

Account and Billing

If you are thinking of using AWS, you likely belong to an organization; a startup; or you are an individual looking explore or develop something. The simplest use case is the individual working on a project. In this case the individual is the sole contributor and manager of that project. This user can simply create an AWS account and use this account to create and manage resources. However, in large organizations, it becomes tricky. Many organizations and businesses have different departments or sections each with their own budget and financial accounting requirements . Also these different departments may or may not use the same AWS services. Even if they use the same aws service i.e AWS S3, they will likely use different resources (i.e buckets in this case). In special cases, departments may even be required to share resources. AWS needs to provide a flexible, secure and transparent way of organizing these kinds of access requirements.

The unit of billing and resource ownership is the AWS account. IAM Users and Resources are created under an AWS Account. This is how AWS knows “who” to bill for certain resources. IAM users are created and associated to accounts. But these accounts can be grouped logically in a nested manner so that the organization can have a top-down overview of the expenditure of the organization. This is where AWS Organizations come in.

AWS Organizations enable us to use and manage multiple AWS accounts in the same organization. Multiple accounts can be grouped together so that polices can be administered uniformly on accounts in these groups.

When you enable AWS Organizations, one organization is created for you called the “root”. You can create new AWS accounts or invite existing AWS accounts to this organization. One or more organization units can be created to group one or more accounts. An organization unit can comprise of multiple accounts and organization units.

Sample AWS Account Organization Hierarchy

In the example above, we created 3 organization units and 6 accounts.

  1. The organization unit OU3 has OU1 as parent
  2. The accounts are distributed to each organization unit (shown in blue)
  3. There is always going to be one account ACC that is directly associated with the root organization and this is the management account. The management account is ultimately where all bills go to but AWS categorizes the bills according to the individual account use.

In addition to billing, AWS organizations helps to ensure that each accounts can only use a specific AWS features. By defining service control policies (SCPs) and attaching these profiles to an account or organization unit, we are able to limit what AWS feature a particular account can use.  AWS has a lot of features and I am sure that organizations do not want all departments to be able to utilize all the features that AWS has to offer. What they typically would do is specify a list of features that a particular department can use. Once an account is granted access to these features then users can be created under these accounts and granted permissions to use those features.

For example, supposing ACC4 and ACC5 are only allowed to use AWS S3 ( I don’t know why I keep using this, but let’s just play along), we can define an SCP that grants access to only S3 service and attach this SCP to the OU3 organization unit. This way, both ACC4 and ACC5 accounts are restricted accordingly. If IAM users are created with ACC4, they will only be able to grant them S3 specific permissions. In detail:

  1. ACC4 can create an IAM User User 1 and grant permission to read from S3
  2. ACC5 can create an IAM User user 2 and grant permission to write to S3.
  3. ACC4 cannot grant User 1 a permission to access Amazon SNS because the SCP only allows using S3 permissions
  4. ACC4 cannot grant User 2 permission to access Amazon SQS because the SCP only allows using S3 permissions.

In addition to Service Control Polices (SCPs), other policies can be created and attached to organization units and accounts:

  1. AI services opt-out policies – control whether AWS AI services can store or use your content
  2. Backup policies – Enforce consistent backup plans
  3. Tag policies – Enforce a certain standard of tagging for all resources. You can define keys and their allowed values.

Identity and Access

There is a distinction between an AWS account and a User. Accounts are associated with resources and are used to create Users. In AWS, there are two kinds of users: root users and IAM users. Root Users are users automatically created and associated with the AWS account.

IAM Users are created in IAM or IAM Identity Center and these are associated to an AWS account. One or more IAM users can be created for an account. IAM Users are typically humans (organization staff) or machines that access any of the resources created in your account.

It is worth noting that AWS discourages the use of the root user for things that can be done with an IAM user with a correct privilege. They advise that the root user should be used for actions that only a root user can perform. The credentials for the aws account should be securely managed.

The IAM Identity Center enables administrators to manage IAM users and groups across the organization. These users have to be assigned the following:

  1. An associated AWS account
  2. One or more permission sets (specifying what the user can perform in that account).

Each permission set the user is assigned can be seen as a designated role. The Identity center provides a link to an access portal to enable users access the management console with any designated role. Assuming the user has the “developer” and “director” permission sets assigned, the user takes the following steps:

  1. Clicks on the portal link
  2. Enters the username and password
  3. The user is redirected to a page listing all the aws accounts the user has access to and the permission sets. The user can click on either “developer” or “director”. By clicking one permission set, the user is redirected to the management console and can use the management console as a “developer” or a “director”
  4. On the management console, the user can see which user and designated role is being used.

AWS also has a guide for this.

Conclusion and Thoughts

AWS has robust mechanisms for handling security and ensuring that you design your systems with security in mind. By using Amazon CloudTrail, you can monitor AWS account activity as well. AWS Organizations enable you to structure your accounts and billing to tailor that of your organization.

AWS has the IAM and the IAM Identity Center for managing access (i.e creating users, groups and assigning permissions). They are similar in use case and AWS advices to use the IAM Identity Center as much as possible because it facilitates configuring access at an organization level. This ensures that you have a uniform and transparent access enforcement system. With this, we can efficiently enforce access for our workforce.

To manage customer access, AWS provides AWS Cognito and AWS Verified Permissions. These two systems are so broad and powerful that they deserve their own section.

Into The AWS Cloud

Innovation is a constant phenomenon in or world today. In order to keep up with the speed of innovation businesses must be able to adapt quickly. One of the factors restricting businesses from evolving is IT infrastructure.

IT infrastructure comprises of everthing a business needs to survive in this digital age like email providers, servers, media storage, processing units, softwares, routers and so on. Businesses that manage their own IT infrastructure have to dedicate a percentage of their workforce to manage and evolve this infrastructure as the business grows. This task is especially cumbersome for lean businesses without the capacity to dedicate staff for this purpose.

With the advent of “the cloud”, more and more businessess choose to “outsource” all or part of their IT infrastructure so that it is managed by a separate company. This company ensures that computing and network resources are available and shared efficiently. These “cloud” providers now provide exact billing strategies that ensure that you only pay for what you use (as opposed to investing in a 16GB ram server and only use it for emails). These days, we find that it is more cost effective especially for lean businesses to move to cloud architecture and focuse on their core competencies.

In the following Sections, I will delve into the AWS cloud, exploring and reviewing the services they provide and the practical use cases for small businesses.

Do you want me to review or check out any aspect of AWS for you? Feel free to comment and let me know or contact me.

The Holy Architect

Introduction

The title sounds like a far-fetched ideal. (Just like the caption of this blog post generated from Microsoft’s Copilot, powered by DALL.E 3) For some, it is inconceivable to associate two words from seemingly different domains of life. Most times, people associate holiness to saints and people devoted to their religion. In our world today, and for some people, it is inconceivable, or at least impractical, to achieve the zenith of our career and still be devoted to our religion. For others, because we are at the zenith of our career and live in comfort, we see no need to embark on such a journey.

Misconceptions about Holiness

What thought crosses your mind when you hear that someone is holy? The person spends an inordinate amount of time in a religious institution? The person prays a lot? The person always talks about spiritual things and reads a lot of spiritual books? Or perhaps the person is frequently mentioned in religious circles. Some people go as far as to think that because these people spend so much time devoted to their religion, they possible cannot have a thriving career.

It is easy to think and postulate about matters we have little or no experience with and it is often easy to judge people by external appearances. But by careful consideration of the concept of holiness, one would realize that the topic is much more profound and vital to everyday living.

What is Holiness?

The oxford dictionary give us something that is not entirely helpful – “The state of being Holy”. With “Holy” meaning “dedicated or consecrated to God or a religious purpose; sacred”. In the simplest term, it is state of realizing that you are meant for something and you give your whole self for that purpose.  One who recognizes that his life has meaning does well to find and fulfil that meaning to which he found. And like any other endeavor in life which requires skills, there are some skills that are required for you to faithfully achieve your life’s goal. I’m not talking about career skills this time, because our lives are much greater than our careers. I am talking about fundamental skills – virtues.

The Pursuit of Virtue can be applied to every aspect of our lives

I recently came across a book titled “The Art of Living” by Dr Edward Sri. It made me aware that basic virtues are needed in all aspects of our lives, even in our careers. After that, I attended some project management courses, and to my amazement, these same concepts (albeit talked about limitedly in different lexicon) were discussed as fundamental to managing teams, individuals and one’s self. I will try to explain how virtues are vital with a story.

I have so much experience in software development and people usually encourage me to write about what I know and I actually know that it will help others if I do so. With the virtue of prudence, I am able to evaluate the benefits of documenting my knowledge. This virtue also makes it possible to listen to my peers and take this advice. Also, looking back at my career, I have benefitted from the mentorship and guidance of  so many great individuals, and used numerous resources (I cannot imagine my career without google and stackoverflow). The virtue of justice makes me understand that because I have received so much, I ought to give back the same way I received.

Knowing that I need to write to help others is not enough. This knowledge does not actually help anyone. I’ve known this, but because I lacked two other fundamental virtues, I have not been able to successfully and continuously document my experiences in a manner that will truly inspire and help others. With the virtue of fortitude, I would be able to persevere in my writings even when the topic I am writing on seems difficult; able to be patient and continue my blogs even when my kids distract me, or I am heartbroken, or grieving the death of a loved one; with fortitude, I can be free to dedicate my effort to writing great pieces that would inspire, inform and educate my readers and not settle for mediocre blog posts. With the virtue of temperance, I will be able to overcome distractions that get in the way of my completing this blog posts like food, amusement, sex and so on ( which of their own, are not bad). With this virtue, I will also avoid excusing myself from writing my blog post because I have another (still legitimate) responsibility.

When I am able to write blog posts easily, consistently, promptly and with joy, then I have attained a level of perfection what will ensure that my efforts truly make a difference to people’s lives. Note that this perfection does not concern itself with the work in its nature but to the manner at which the work is done. Attaining this level of perfection, through the study of these virtues and the practice of attaining these virtues, in every aspect of our lives including our careers is the basis of holiness.

The Virtuous Architect

There can never be a perfect software architect. But that shouldn’t stop us from studying and practicing to be. If we set our aim to the cosmos and fall short of it by landing on the sky, then we have done something exceptional. However, if we tell ourselves: “we cannot attain the cosmos” and settle for a “more realistic” goal as the sky, we might find ourselves falling smack on the ground. And like all life’s endeavors, the cardinal virtues – prudence, justice, temperance and fortitude – are the basic skills every software architect needs to design useful systems, collaborate with clients and stakeholders to own their vision and work with the team to make this vision a reality.

Prudent Architecture

Prudence generally means applying the right reason to things. Prudence helps the architect determine the right course of action based on the given context. This context can be gotten from the careful consideration of the data and metrics surrounding that endeavor – for example telemetry data for performance enhancement , business capability use cases and statistics for feature elucidation, or bug report data and tests for bug fixing. Experiences should also be sought from best practices. This can be in the form of external global standards, or internal company adopted standards. Standards not only ensure inter-operability, but encourage architects to rely and build upon the knowledge and wisdom of their predecessors and counterparts.

Listening and seeking advice from mentors and other professionals who have had similar experiences is also key to making the right decision. No matter how knowledgeable you think you are, you have a lot more to learn. This realization is a major catalyst for your growth, not only as a software architect, but as a person. Common ways to seek advice is to participate in software development communities like stackoverflow; searching for answers in search engines; talking to someone you know has more experience; or even talking to your peers. You might be surprised that by just talking, you find the solution to your problem.

A prudent architect carefully collects data – being mindful of, and guarding against, biases in data collection. It is so easy for us humans to be biased and not even know about it. Last year, when  I wanted to get a car, I began to notice cars similar to the one I was about to purchase. Weird right?

Once we have carefully considered the options and data involved, we need to make a decision. We can be prudent architects and avoid making decisions based on impulse and feelings and take time to consider all information. But by delaying decisions or being indecisive, we end up undoing what we have done and causing harm. A prudent architect makes a decision in a timely manner after weighing the context surrounding it. And after making that decision, the architect sticks to it because he knows that the decision was carefully made considering the available context. This does not mean that the architect is obstinate or does not update his position based on new information or data. Because we are all imperfect beings, we should always be open to correction.

There are common reasons that make us indecisive. The most common one is anxiety. We are worried about the future and whether this solution will stand the test of time. To this I answer that no temporal thing in this world is worth being anxious about. What’s the worst that could happen? Critically think about it. Secondly, as architects, especially in agile development, we don’t expect to deliver the perfect solution in the first iteration. As long as we are honest with ourselves; listen to our stakeholders and our team; and keep track of improvements and feedback, we will surely get the job done.

Responsibilities of an Architect

The virtue of justice makes us aware of our responsibilities and what we owe others. We owe our client transparency and honesty. We need to be able to listen to and provide solutions to the concerns of stakeholders. We need to be accountable to our managers and ensure that we do everything in our power to execute quality deliverables in a timely manner. We owe our team mates guidance and leadership and we need to be a rock for our team in times of crisis and uncertainty. This virtue motivates us and drives us to be excellent individuals. We often feel disillusioned or discouraged because we have lost sight of the true reason we do things. The virtue of justice pins the true essence of our actions on our foreheads that we may not lose sight of the true reason for our endeavors.

Navigating through Troubled Waters

Now we have the developed the best architecture. We are able to answer all the questions our stakeholders throw at us. We present our architecture to the governance team and they applaud. Awesome! Then what next? What happens when it’s time to implement and we meet a few hiccups? We see that things don’t go according to plan and stakeholders become worried and start questioning our abilities. What happens when the realization that the implementation is taking longer than expected and we receive immense pressure from management? In my circle we call these “fires”.

It’s in these common situations that we need perseverance which is an aspect of fortitude. With perseverance, we can wade the muddy waters of difficulty calmly, think clearly and come up with solutions, under pressure, that will get you and your team to the finish line. With perseverance we can objectively work at the solution to a problem and seek to solving the root cause instead of taking the short cut or the easy way out.

Constancy is the third aspect of fortitude that enables us to continue working to solve a problem and not abandon it for a lesser or more trivial task. We have all faced that situation where we have a critical bug to fix or some part of the architecture to figure it out, and because it is too difficult at the time we tell ourselves “oh let me go and do PR reviews”.

Sometimes, the work is not difficult and the challenges we face are internal. We feel disappointed because we were just passed off for a promotion; we feel heart broken when a loved one leaves us; or grief when someone we love dies. We can also feel anxious about being a husband, wife or father and so on. A lot of factors, apart from the nature of the task itself, can contribute, and make the execution of our tasks difficult and the virtue of patience helps us carry these burdens effectively and achieve great things despite them.

And when I mention great things, I do mean them. This is because magnanimity, the fourth aspect of fortitude helps us aspire for greatness in everything we do. A magnanimous architect is not just content with implementing something that “works”, but designing a system that is truly beautiful, magnificent, scalable and efficient given the resources at his disposal.

Avoiding Distractions

Anything that will prevent us from achieving our task as software architects are distractions and should be avoided. It doesn’t matter if they are good or evil. The virtue of temperance helps us regulate our passions in order to focus on what is truly important.

One of such common passions as software architects is that of novelty – the desire to try new things and work with new tools. Seeking the latest technological advancements and the latest trends in the software development world. Seeking the best tool, within reason is the prudent approach as discussed, but seeking them for the sake of having them in our system, yields to unnecessary change and introduces unwanted behaviors in or system.

The Proud and Vain Architect

It is amazing what the vices of pride and vanity can do to a software architect. A proud architect believes that he is better than he actually is and above everyone else in his team. A humble architect knows what he truly is in relation to his team and stakeholders and strives for greatness given that knowledge. A vain architect does everything for his own glory.  A humble architect seeks progress of the project, and the team and the satisfaction of stakeholders. Because a proud architect thinks himself better than others, he refuses to listen, spends a lot of time talking in team meetings and makes everything about himself.  The proud architect does the team a great harm because he causes discord and strive in the team and goes through great length to impose his will on the team. More importantly the proud architect does himself a great harm by shutting himself from the team and depriving himself of the opportunity to grow and learn. Finally, because a vain architect seeks his own glory, he is afraid of making mistakes, he is always worried about his image and tries to make everyone happy instead of doing the right thing.

It is important to understand that a humble architect is not indecisive or timid or shy. In fact, because a software architect knows what he truly is – a human being with faults, a team member, an advisor to stakeholders – he knows that he can make mistakes and is not afraid to do so. He knows that he can say wrong things from time to time and so, he has confidence to talk because he is willing to accept corrections. He is not indecisive and anxious but full of great confidence because he knows that although he is a lowly human being, he places his trust in The One Who Knows.

My Experience as a Remote Software Developer

It is said that challenges are what makes life interesting, and overcoming these challenges is what makes life meaningful. Mid last year, I took a leap into an interesting new world of remote software development. As I got my first contract letter – I signed this electronically and if felt weird – I was filled with both excitement and uncertainty. How could I possibly contribute to a team without personal/physical contact with team members? How I assure them of my work output from miles away? Will they pay me on time? What happens if they do not pay me? How do I complain? All these questions, though some were silly, bothered me.

 

With all these uncertainties, I knew it was the right step. Working in Lagos, Nigeria has been a stressful experience where I got up by 4am, left my house by 5pm to beat traffic (I worked at Lekki before), and came back at 10pm. At some point, I felt extremely indifferent working because of the stress of commute. One day, I told myself “Enough is enough”, or as my Yoruba friends will say  “Otoge”. Also, I realized that I could accomplish far more than I currently was and, without limiting myself to location, I could grasp those opportunities. So with many job applications and interviews, I finally landed my first remote gig!

 

On my first day of remote work, I realized that I did not need to take a bath, or dress up, or step outside my room. That feeling was weird and oddly satisfying. I could put on a T-shirt and my underwear and I’m ready for work? Cool!. As the first few days of onboarding passed, I had to pick a work schedule that was comfortable for me. I worked from 8am to 5pm with an hour break at noon. This schedule works perfectly for me because I can wake up, say my prayer, exercise, go for mass and start my work by 8am, and by 5pm, I can rewind, relax and get personal work done afterward. Once I got used to this schedule, everything else went smoothly. The norm of working late and overtime in my previous work-place disappeared and I always found myself wrapping up my work by 5pm regardless of whether I accomplished the tasks for the day or not – face it, things can always be done the next day after a much-deserved rest.

 

Working remotely also has its challenges. My main challenge was to maintain the discipline to work at my specified time period. For me, there is this temptation to say “the work doesn’t matter”, or “it’s not worth it”, or ” I can make up for lost time”, and most times, giving in to this temptation ruins my whole week because my work either gets negatively impacted, or creeps into my personal life and affects it. Also, I noticed that due to a lack of physical activity, I started to gain weight. Since I did not spend hours trekking, or jumping buses to get to and from work, I wasn’t burning the calories that I should have been burning. Trying to counteract this effect with exercise has been difficult though, well, because I’m a lazy human being. But still, I try!

 

Arguably one of the most frustrating parts of working remotely is that you get to interact with your teammates in an asynchronous manner. This means that there is no guarantee that you will get a reply from your request at the moment you make it. I work with developers across different time-zones and great care has to be taken to plan communication so that it does not affect your work output. There was a time I needed help fixing a bug and no one, literally no one, responded to my requests. It was a hair-pulling situation, and with time I got to understand that eventually, they would reply and I had to be patient. The virtue of patience is key. So, these days, whenever I get stuck in communication, I just move on to some other thing and then switch back when I get the reply. Another thing I am still getting used to is the over-dependence on KPIs and metrics. Although I understand this quite perfectly – there needs to be a way the company knows you are really working – I still find measuring the output of work with Jira tickets, clocking out and clocking in, and daily status reports to be cumbersome.

 

It has been a crazy ride for me, and personally, I am loving working remotely. For those thinking of making this leap, I say, life is too short, just do it!

API Error Handling in Spring Framework: Using Exception Handlers

A TopTal post on Spring Boot rest API error handling here gives a foundation on error handling with Spring Boot. This article will focus on using error codes for characterizing errors; assigning exception handlers to groups of controllers; and the order of precedence in exception handling using exception handlers.

The practice of wrapping errors in an ApiError object, as described in the aforementioned TopTal blog post helps us to remove traces of implementation details (like exception names and stack traces) from our error responses and ensure that error responses always provide value to the consuming client.

 

Utilization of Error Codes

In addition to an ApiError object, an integer representing the specific error can be added as a property of the ApiError object. This enables the creation of a faster, more comprehensive, and extensible error handling implementation for the consuming client. If an error code is not used, the consuming client will be forced to process string representations of errors in addition to processing the HTTP status code. This is more CPU intensive than a simple integer comparison when error codes are used.

Error codes also provide a way of standardizing an API. A proper documentation of error codes returned by an API is sufficient for an API client to perform error handling when consuming the API.

By decoupling the HTTP status code from the error codes using error code status resolver implementations, we provide a way to change the HTTP status code of an error in an organized way, without modifying error handling or business logic.

Below is an example of an APIError class that can be used:


@Data
@Builder
public class ApiError {
private LocalDateTime timestamp;
private String error;
private int errorCode;
@Singular
private List<ErrorCause> causes;
private ApiError(LocalDateTime timestamp, String error, int errorCode, List<ErrorCause> causes) {
this.timestamp = timestamp == null? LocalDateTime.now(): timestamp ;
this.error = error == null? "": error;
this.errorCode = errorCode;
this.causes = causes == null? Collections.emptyList(): causes;
}
}

view raw

ApiError.java

hosted with ❤ by GitHub

Error Handling

Now that we have an ApiError class, how do we properly handle errors in API’s? Most errors that occur in API methods are cross-cutting concerns and it is good practice to separate the business logic from the error handling logic. This enables the same error handling logic to be used for multiple API methods without code duplication. The Spring MVC framework caters for this separation of concerns with the concept of Exception Handlers.

An @ExceptionHandler annotation labels a method in a controller or a controller advice as an exception handler. Exception handler methods are called when an exception (root or cause) thrown by the controller matches the exception handled by the handler.

To demonstrate this, we create HelloWorldController to return a SyntaxException when the name property contains a hyphen (apparently, we don’t like hyphenated names here lol).


@RestController
public class HelloController {
@PostMapping(path = "/sayHello")
public HelloResponse sayHello(@RequestBody HelloRequest request){
if(request.getName().contains("-")){
throw new SyntaxException();
}
return HelloResponse.builder()
.message(String.format("Hello %s!!", request.getName()))
.build();
}
}

Controller Exception Handlers

To Handle this exception with a controller exception handler, we declare a method in the controller annotated with @ExceptionHandler as shown:


@RestController
public class HelloController {
@PostMapping(path = "/sayHello")
public HelloResponse sayHello(@RequestBody HelloRequest request){
if(request.getName().contains("-")){
throw new SyntaxException();
}
return HelloResponse.builder()
.message(String.format("Hello %s!!", request.getName()))
.build();
}
@ExceptionHandler(SyntaxException.class)
public ApiError handleSyntaxException(){
return ApiError.builder()
.error("Syntax Error (from controller)")
.errorCode(991)
.build();
}
}

Once we make a post request with a hyphenated name, the syntax error occurs as shown below:


{
"timestamp": "2018-09-10T20:43:56.077",
"error": "Syntax Error (from controller)",
"errorCode": 991,
"causes": []
}

ControllerAdvice Exception Handlers

Exception handler methods can also be added to a ControllerAdvice. ControllerAdvices are component beans that contain methods that perform cross-cutting operations for multiple controllers. This is the most common way of writing ControllerAdvices because it decouples the exception handling logic from the business logic by placing them in separate files. To implement this, we create a ControllerAdvice and implement the desired method just as before.


@RestControllerAdvice
public class HelloControllerAdvice {
@ExceptionHandler(Exception.class)
public ApiError handleGeneralException(){
return ApiError.builder()
.error("General Error (from controller-advice)")
.errorCode(500)
.build();
}
@ExceptionHandler(SyntaxException.class)
public ApiError handleSyntaxException(){
return ApiError.builder()
.error("Syntax Error (from controller-advice)")
.errorCode(991)
.build();
}
}

In order for this implementation to work, the exception handler in the controller needs to be removed. This is because controller exception handlers have more priority than controller advice exception handlers.

Here, @RestControllerAdvice was used instead of @ControllerAdvice. The latter is a composite annotation of @ControllerAdvice and @ResponseBody

Applying ControllerAdvices to Selected Controllers

ControllerAdvices can be limited to “advice” specific controllers. This group can be defined by base package name, superclass, or annotation. Any of these fields can be set in the properties of the @ControllerAdvice.

Note that since exception handlers are applied at runtime, having selectors such as this reduces performance since it gives the application one extra condition to check in order to decide whether or not to apply the handler.

Matching Exceptions to Handlers

At runtime, the spring MVC framework selects one exception handler to handle an exception thrown by a controller. The selection algorithm follows these rules:

  1. Exception handlers declared inside the throwing controller are selected in preference to exceptions declared in ControllerAdvices.
  2. Among methods of an advice bean, a root exception match is preferred to a cause exception match.
  3. An exception match (whether root or cause) in an advise bean with a higher priority, specified by (@Order annotation or Ordered interface), is selected in preference with those of lower priorities.
  4. Two exception handler methods with the same target exception cannot be declared on the same ControllerAdvice. This will result in an IllegalStateException due to the ambiguous exception handler method mapping.
  5. Exception handlers belonging to the same advice are selected according to their specificity. An exact class match is preferred, then a sub-class match and so on.

Customizing HTTP Status Codes

The previous examples return an error response with the HTTP INTERNAL SERVER ERROR (500) by default. To change this, we can either return a ResponseEntity object or use the @ResponseStatus annotation.


@RestControllerAdvice
public class HelloControllerAdvice {
@ExceptionHandler(Exception.class)
public ResponseEntity<ApiError> handleGeneralException(){
return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR)
.body(
ApiError.builder()
.error("Syntax Error (from controller-advice)")
.errorCode(500)
.build()
);
}
@ExceptionHandler(SyntaxException.class)
@ResponseStatus(HttpStatus.BAD_REQUEST)
public ApiError handleSyntaxException(){
return ApiError.builder()
.error("Syntax Error (from controller-advice)")
.errorCode(991)
.build();
}

Summary

Exception handlers provide a convenient way to handle API errors. API Error Objects make error description more user-friendly and error codes help to efficiently characterize errors.

Codebase

The codebase for this tutorial can be found here.

Keep Koding Konstantly.

Adventures with Hyperledger Sawtooth Core

From the first paragraph of the introduction to hyperldeger sawtooth documentation: 

Hyperledger Sawtooth is an enterprise blockchain platform for building distributed ledger applications and networks. The design philosophy targets keeping ledgers distributed and making smart contracts safe, particularly for enterprise use.

It’s an ambitious open source distributed ledger framework that aims at providing a high rate of transactions per second (it claims a thousand per second); configurable consensus mechanisms; transaction processors (smart contracts) written in any language; and an overall modular architecture.

We took a keen interest on this platform because of its immense capabilities, so, in order to find out if this distributed ledger framework is really what it boasts to be on paper, I took the hyperledger sawtooth through a series of tests. And of course, I’m reporting my findings here.

 

About this Adventure

  • The released version 1.0 is used for all tests documented here.
  • The java sdk will be extensively used to make transaction processors.
  • All tests were done on a windows 10 system (docker toolbox)

 

Installation Requirements

Please proceed to the documentation for more information on the requirements for installation. From experience, you just need a system with enough ram and disk space to handle execution of docker containers. Docker is used for packaging the framework.

 

Installing Hyperledger Sawtooth

The installation process is simple, all that’s needed is to pull the git repository  and checkout to the version you want. In this case, we will be checking out to v1.0.0.

Cloning the Repository

git clone https://github.com/hyperledger/sawtooth-core

Checking out to v1.0.0

 git checkout tags/v1.0.0 -b sawtoothv1.0.0

Good! We now have our copy of v1.0.0 on the sawtoothv1.0.0 branch

 

Running Hyperledger sawtooth core

The code contains docker-compose files for different configurations that you could use as a starting point. Here, we will start with docker/compose/sawtooth-default.yaml.

cd /docker/compose
docker-compose -f sawtooth-default.yaml up

 

A brief introduction to the sawtooth architecture

An indepth exposee on the sawtooth architecture can be easily found in the documentation. We will just go through the basic components.

The hyperledger sawtooth system is made up of the following:

  1. Validator – This is the principal component of the sawtooth platform. It is responsible for block validation, block chaining and communicating with other validators on the network. The validator also validates transactions by delegating it to its transaction processor. It also ensures that transactions and batches are signed properly.
  2. Transaction Families – These are group transaction types that perform related actions. A transaction processor processes transactions belonging to a transaction family. Clients create transactions from a family and send it to the validator through the rest-api. The transaction processor is an independent service that communicates with the validator through TCP/UDP socket connections. A validator can be connected to any amount of transaction processors in order to process transactions belonging to different transaction families. The validator registers these transaction processors as soon as they are connected and deregisters them once they are disconnected. This way, a validator knows what transaction processor is attached to it.
  3. Rest Service – This exposes rest api endpoints for clients to integrate with the validator. A proper documentation of
  4. its endpoints be found on the sawtooth documentation. With this, we can send transaction in batches, query states, transactions and blocks, and listen for state changes through a websocket api endpoint.

 

On the sawtooth-default.yaml

This is a basic one-validator sawtooth system meant to show case the capabilities of the hyperledger sawtooth. It comprises of the following:

  1. Settings-tp: A transaction processor that processes transactions that can be used to configure the blockchain.
  2. Intkey-tp-python: An integer-key transaction processor, that provides transactions that can be used for testing.
  3.  Xo-tp-python: A tic-tac to transaction processor. Yes, you can play tic-tac to by sending transactions to the blockchain LOL.
  4. Validator – The validator (as the name implies)
  5. Rest api – The rest api (as the name implies)
  6. Shell – A command line utility for posting transactions of the various transaction families. It integrates with the rest api.

With a good internet connection, all modules are pulled quickly as docker-compose is run and you can see the colourful output on your command line.

 

Experimenting with the Integer Key

Run the following command in the docker terminal to enter into the shell environment

docker exec -it sawtooth-shell-default bash

We are going to create a batch and send to the int-key-transaction processor.

Intkey create_batch
Intkey load

 

More about IntKey

Intkey provides transactions that can be used to test deployed sawtooth distributed ledgers. These transactions are as follows:

intkey set: It sets a key to a specified value

intkey inc: this increases the value of a key by 1

intkey dec: this decreases the value

intkey show: this displays the value of a specific key

intkey list: this lists the values of all the keys.

intkey create_batch: creates a batch of intkey transactions

intkey load: Sends the created batch to the validator.

 

More about the intkey that the documentation won’t tell you

A close inspection of the intkey cli code written in python reveals how create_batch can be used. You can supply the following arguments:

-o –output:  Location of the output file (batches.inkey by default)

-c –count:  Number of batches modifying random keys (1 by default)

-B –max-batch-size:  Max transactions per batch (10 by default)

-K  –key-count:  Number of keys to set initially ( 1 by default)

 

create_batch generates a random word list (keys to set initially specified by -K or –key-count). It then generates “intkey set” transactions that initialize the values of those keys to a random number between 9000 and 100000, appending it to a batch. It generates one set transaction for each key. After this, it creates a random number of batches modifying the random keys. The number of batches created is specified by the -c / –count args.

Each batch created will have a random batch size of the range between 1 and the args -B / –max-batch-size. These transactions modify a random choice of keys and randomly increment or decrement the values.

Therefore, with the create_batch, the smallest number of transactions we can have is two batches, 1 transaction per batch. One batch for set, while another for increment or decrements.

Investigating Transaction Flow with Int Key

In order to properly investigate transaction flow, lets cover the theoretical background.
From the sawtooth documentation, the main component of the validator is the Journal, responsible for “maintaining and extending the BlockChain” for the validator. These responsibilities include the following:

  • Validating candidate blocks
  • Evaluating valid blocks to determine if they are the correct chain head
  • Generating new blocks to extend the chain.

 

A batch is a group of transactions that are atomically lumped together in a manner that when one fail, all fails. Batches can be made up of transactions from the same or different families.

The journal consumes blocks and batches that arrive at the validator via the interconnect, either through the gossip protocol or the rest api. The journal routes them internally.

The flow is as follows:

  1.  Blocks and batches arrive from the interconnect to the completer
  2. The completer ensures that all dependencies of the blocks and(or) batches have been satisfied and delivered downstream
  3. The completer delivers the blocks and batches to different pipelines.
  4. Completed Batches go to the BlockPublisher for validation and inclusion into a block
  5. Completed blocks are delivered to the chain controller for validation and fork resolution.
  6. Blocks are considered formally complete by the completer once all of their predecessors have been delivered to the ChainController and their batches field contains all the batches specified in the BlockHeader’s batch_ids list. The batches field is expected to be in the same order as the batch ids.
  7. Batches are considered complete once all of its independent transactions exist in the current chain or have been delivered to the BlockPublisher.

 

The Chain Controller

This is responsible for determining which chain the validator is currently on and coordinating any change-of-chain activities that need to happen.
The Chain Controller is designed to handle multiple block validation activities simultaneously. This means that once forks form, the Chain Controller can process these forks simultaneously. The current chain can be used while a deep fork is evaluated. Currently the thread pool is set to 1 (its hardcoded), so only one block is validated at a time.

 

Flow for updating blocks in the ChainController

  • Block enters the chain controller
  • Chain controller creates a block validator with the candidate block and current chain head and dispatches it to a thread pool for execution. Once the block validator has completed, it will callback the chain controller indicating whether the block should be used as the chain head. This call back occurs in three situations:
    • If the call back occurs after the chain head has been updated, a new block validator is created and dispatched to redo the fork resolution.
    • If the call back occurs when the chain head has not been updated, that new block becomes the chain head.
    • If the call back determines that the block should not become the chain head, then it is discarded. This occurs if there is an invalid block in the block’s chain, or is a member of a shorter or less desirable fork as determined by consensus.
  • The chain controller synchronizes chain head updates such that only one block validator result can be processed at a time.
  • The chain controller performs a chain head update using the block store, providing it with a list of commit blocks that are in the new fork and a list of decommit blocks that are in the block store that must be removed.
  • After the block store is updated, the block publisher is notified about the new chain head.

 

Block Validation Rules

In hyperledger sawtooth, block validation includes the following:

  • Transaction permissioning – Ensuring that the right entity submits transactions and batches.
  • On-chain block validation rules
  • Batch validation
  • Consensus verification
  • State hash check.

 

Block Publisher

The block publisher creates candidate blocks to extend the current chain. It does all the house-keeping work around creating a block but waits for the consensus-algorithm’s instruction on when to create a block and when to publish a block.
The flow in a block publisher is as follows:

  1.  Add new batches to scheduler
  2. Wait for consensus algorithm to create candidate block
  3. If candidate block is created, keep adding batches
  4. Wait for consensus algorithm to publish
  5. If time to publish, finalize candidate block and publish block
  6. When a block is created, an instance of Consensus.BlockPublisher is created that is responsible for guiding the creation of this candidate block. A transaction scheduler is created and all of the pending batches are submitted to it.

 

Results of intkey Testing

This will be posted soon enough, happy reading!

Life as a Software Developer

Are you new to writing code? Or have you mastered the arts and sciences of programming? Do you love coding in high level languages like python, java or C#, or you are a wizard in embeddable programming in C, or assembly language? Are you into abstract applications of programming like solving complex differential equations or do you fancy developing web applications? Regardless of what you use the skill of programming to do? We always go through the same things:

 

We have to cope with constant change

Well, to be fair, every human being (perhaps animals and plants thanks to climate) has to cope with change. But really, change in the software development world is different. We software developers have to cope with learning new processors; more exciting frameworks; more efficient programming languages; not to talk about upgrading our operating systems to support the kind of applications we want to install. As a spring-boot developer, change is common place. By the time the application you are developing goes live, the spring framework version you are using gets obsolete and the next thing that you must think of is how to upgrade. You will probably face coping with new software framework releases dealing with open source projects. It’s an awesome thing to know that frameworks are developing at a faster pace, but discipline is needed to cope with the change.

Be careful not to lose yourself

Have you ever walked along the road and someone angrily taps you on the head and yells, “Hey! Can’t you hear me greeting you.” Of course you cannot. You are busy thinking about whether or not to use reactive streams to handle event’s coming from state delta changes. But then again, you remember that the person is not in your head so you just have to apologize. On some days, the only thing I think about is solving the problem at hand. I guess the sage software developer portrayed wisdom by saying “One of the hallmarks of a good software developer is knowing when to walk away”. Indeed this is true. Many times that I resigned to the excruciating fate of not solving a problem on a particular day, after a much needed rest, I find the solution to that problem within the first hours of the next day. So once again devs… WALK AWAY.

We love all those HIGH’s

Oh, and those euphoric feelings you get when you pull of something complex and make it look simple? Who could get over that? I think it’s one of the perks plying a trade that has to do with “creating something out of nothing”. It makes you feel like you are working along-side God in doing his fantastic work of creation. One of the reasons I became endeared to programming is that with the right skill set and CPU, you could tell the computer to do anything. However, these feelings have caused a lot of developers to re-invent the wheel. Is it that you do not know that there is already a library that can handle conversion of a byte to a hexadecimal string? I mean why on earth would you want to re-create that? Although we all want to have that feeling that we created something, let’s not forget that what we create, we should be ready to manage. And if we are not prepared to manage such libraries, its best we find existing ones – ones that are managed by other organizations, there are lots of them I assure you!

 

Don’t forget those depressing moments

I recently hit a wall in development trying to figure out Hyperledger Sawtooth, and apply it to one of my projects. At that gigantic wall, faced with an abysmal feeling of despair, I had to re-evaluate my life’s choices. I also had this exact same feeling when I was trying to figure out how to implement the card personalization standard for Mastercard. With a bit of courage, I picked myself up, dusted myself and I am moving forward. Things are going to get though, like a wise entrepreneur always says, don’t let short term set-backs get to you. Think always of the long term. With a little perseverance, we will always see something really beautiful coming out from that supposedly gleam situation.

You want to know everything but then again… you cant.

I told myself one day that I wanted to explore Java 9. Out came spring framework 5, and oh, that one too! Let’s add it to my list. What about distributed ledgers? Machine learning? Big data? Artificial Neural Networks? Then the list of things to learn compiles. Don’t forget that we are on a tight deadline to deliver a feature and if you are in a fast growing organization like mine, you will be saddled with a lot of responsibilities here and there. With all these responsibilities, our learning list slowly becomes just a list and nothing more. Even worse, the list becomes a depressing agent for you because it constantly reminds you of all you failed to do. In these situations my dear, take life one step at a time. All that matters is your focus, not the amount of strides you take. Even if it takes you 10 years to finish your list (if you can even finish your list), the most important thing is that you learn something new every day. Make that a habit.

You find yourself googling or stack-overflow-ing the same thing over and over again

Well, what can I say, there’s no fix for this. Sorry.

Blog at WordPress.com.

Up ↑