Month: July 2017

In order to reduce operational risks, organizations put in CONTROLS, typically via Change Management processes. To minimize the frictions in your DevOps journey, and building on my previous Tip#4, let us look below for the Tip#5 for effective change management.

TIP #5 – Automate, Automate, and AGAIN Automate !

Change management typically includes a CAB (Change Advisory Board) meeting. This CAB meeting reviews the list of change sets, which the operations team filters and can either accept or reject,  to move the change-set into production.

The "Advisory" Board just became the "Gate Keeper", if you noticed!

Large enterprises will work with a traditional mindset. This traditional mindset assumes a Large Batch of changes, which may have been suited in the past for a BIG CAB meeting. But now with smaller change-sets (read as micro services) becoming the norm, imagine going through the rigor of a 2 hour CAB meeting. It will not be a very pleasant experience !!

Therefore we need to re-imagine the CAB meeting and ask the simple but difficult question - WHY DO I NEED A CAB ? 

Purpose of the CAB

Typically the purpose of the CAB meeting is to verify all the artifacts, as below-

• Have you tested your changes?
• Have you integrated the security practices?
• Have you tested the migration, rollback and can provide evidence?
• Is the change-set linked to the business need and do you have approvals?
• and the list can go on and on…..

Good news?

But there is good news now !  All these questions can be answered easily by automating your change management workflows. This is now supported by the convergence of the tooling across the application development life cycle and all the evidence required along with the artifacts can be easily built into your automated build and deployment pipelines and integrating with the change management workflows.

Thus the elimination of CAB is done by Automation of our workflows!

These automated workflow makes it possible for the Operations teams to trace the requirements as they become implemented, and improves the ability to see changes, the effects of changes, approvals, and gather the evidence in a self service model using telemetry.

Many teams start with an intermediate manual approval step, till they start to trust the teams, and their change sets. But this manual approval step also goes away, over a period of time depending on the maturity of your teams.

As the teams look to implement pre-approved change strategy, and look for future opportunities to keep on continuously improving and widening this definition, it is a win-win for both sides.

So if you still doing manual reviews in your change management teams, STOP ! Start to automate your workflows, start to automate your build process, start to add automated tests for security into your coding, start to automate your deployments to production, start to add Telemetry and automate the production reports, start to complete the feedback loop, and in the end increase the Transparency across the team.

So go ahead – Automate, Automate and again Automate !

Subscribe to my blog for more learning’s, and feel free to share your feedback here.

In order to reduce operational risks, organizations put in CONTROLS, typically via Change Management processes. To minimize the frictions in your DevOps journey, and building on my previous Tip#3, let us look below for the Tip#4 for effective change management.

TIP #4 – Use TELEMETRY to show evidence

Traditional thinking auditors look for evidence, and will typically ask for screenshots, configuration logs, settings etc. If you manage thousands of servers, this itself is a cumbersome activity, especially if you are launching and shutting down servers in the cloud all day.

Imagine the activities needed to manage the expectations and you would simply need an army to satisfy the audit needs!

But the auditors and compliance personnel cannot read code, and hence need all the help they can get, to satisfy the regulatory bodies. So you can help them with providing evidence using the following options.

 1. Create alternate data sources to present evidence 

Applications which are ‘operationally-aware’, will include telemetry data, including capturing error, warnings, events, trigger points, and logging this data to central\distributed stores. Typical telemetry systems (MS Insights/Kibana (logstash) ELK stack / Splunk etc.) can capture all this information and present in visual dashboards.  These dashboards can be customized, based on the needs of the users, and present the data at multiple levels of detail.

Auditors can slice and dice this data, and ‘self serve’ their audit needs !

2. Use Iterative approach to building CONTROLS evidence

As part of early engagement with the auditors, successful teams invite audit teams to their sprint planning and sprint reviews. This conversation can kick start rich discussions on how to build controls evidence in every sprint, instead of the end stage.

Teams can start to build controls right from the beginning! 

Sometimes the solutions to meeting the audit controls could be as simple as maintaining version control for all the artifacts. Other solutions could be simply linking all the artifacts across the complete application development life cycle. This allows traceability for each change set put in production.

To help you explore further, read up the fictitious narrative DevOps Audit Defense Toolkit. This provides some real life examples and links it all together.

So go ahead and start to build out your telemetry systems. Start doing early engagements with the change teams. Start to build out the controls iteratively, thereby building Trust and Transparency across the team.

Subscribe for more tips in my next post, and feel free to share your feedback here.

In order to reduce operational risks, organizations put in CONTROLS, typically via Change Management processes. To minimize the frictions in your DevOps journey, and building on my previous Tip#2, let us look below for the Tip#3 for effective change management.

TIP #3 – re-classify your change sets

Enterprises will have multiple change requests being pushed to production, of varying size, complexity and with different risk profiles. But existing change management processes today do not distinguish between these variations!

In reality, different change sets allow us to build different risk profiles.

So let us try to understand these variations in the Change sets, which can typically be classified into one of these 3 categories –

 1. STANDARD Change sets

These change sets are very low risk, and operations are familiar with these. These change sets have an established approval process in place. Examples – web style changes / data table updates etc.

2. NORMAL Change sets

These change sets are high risk, and operations are not familiar with these. These change sets typically use a CAB (Change Approval Board) process to approve/reject the changes. This process requires submitting change forms, with schedule, impacts, risks etc. Examples -New feature/product etc.

3. HIGH Urgency Change sets

These change sets are emergency changes, with potential high risk, and may need approvals from senior management. Examples -Security patch, Service fix patch etc.

Now with the above classification, we can aim to align with the operations teams and change management and ask for an agreement.

Agreement:  Can the STANDARD Change set be Pre-APPROVED?

As the standard changes sets are low risk => Operations teams do not need to approve. This agreement immediately give us the ability to define a pre- approval process. This allows us to deploy our change sets automatically (using  our automated deployment pipelines).

I am sure that this agreement itself will allow you to breathe more freely !

So go ahead and start working with your change management teams. Start to build these agreements, which allow you to auto-deploy to production, with complete Trust and Transparency across the team.

Subscribe for more tips in my next post, and feel free to share your feedback here.

In order to reduce operational risks, organizations put in CONTROLS, typically via Change Management processes. The outputs typically feed into the compliance/audit personnel needs, and satisfy them, but the legacy audit mindset CONFLICTS with the DevOps team mindset.

Therefore to minimize this friction, see my #1 tip in this post on how to work with the change management processes, and teams. I will be sharing more tips in my next posts.

TIP #1 – TALK to your Audit/Compliance team

1. ASK – Why does your audit team need the Change information? 

2. ASK – What will they do with the Change information? 

3. ASK – What level of granularity of data about the Change is required? 

4. ASK – Are there alternate sources of the same Change data?

5. ASK – When do they need this Change information?

Speaking the same language (audit-speak) and asking them questions, will give you as an IT team a better understanding of the Audit/Compliance process. You may be surprised by the technical nature of the various ACTS (Financial \ Healthcare etc.) and start to appreciate them even.

So just go ahead and START a conversation with your Audit/Compliance team members now, and you might be pleasantly surprised.

Subscribe for more tips in my next post, and feel free to share your feedback here.