10 Apr More Agile Than “Agile”

Joe Roushar – April 2018

Promises – Promises

Agile Information Technology (IT) promises to deliver automated capabilities to users more quickly, and, to a great extent, has achieved this. But to many, IT stands for Ivory Tower, and, other than the privileged few, most people in businesses are on their own unless their needs are shared by enough others that the potential benefits justify the cost of IT.

Is it possible to deliver new automated capabilities to users faster, more effectively meeting their needs? Is it possible to meet the diverse needs of users with many different perspectives and backgrounds without the long and painful “change request” and project cycles that have historically smudged IT’s reputation? Now that we are beginning to overcome the excessive failure rate of legacy IT project methods by using agile methods, what is next? The DevOps philosophy of full business stakeholder participation in IT projects has been very beneficial. But the business world is poised to surpass agile methodologies with new paradigms: Self-Service and the Citizen Developer.

Simply stated, there are cases in which IT resource limitations leave many business automation needs unmet.

How much more competitive would a company be if people anywhere in the organization, without technical training, could automate their own capabilities. The traditional approach of going to an IT department for all automation needs cannot handle everything so many projects are rejected. Business agility would take a leap forward if people throughout the organization could create their own automation to support business capabilities. This cannot be done without training and governance, but a small amount of governance evenly spread, can bring a tremendous amount of value. In this post, I will give specific examples of this, and describe how businesses can retool to make it happen.

Understanding Context Cross-Reference
Click on these Links to other posts and glossary/bibliography references

Section 8 #25

Table of Context

Agile SW Development

Automation development in businesses has traditionally involved “V” methodologies as I have referenced in earlier posts (MM) (SR). Software developed using a “Waterfall” process usually follows the “V” from the Requirements phase to Design, then to Implementation, Testing and Deployment once. That one cycle could take months or years. The long duration increases the risk that the resulting solution may not be as well suited to the users’ needs when it is finally delivered. It may not bring the value anticipated when it was first specified.

Agile attempts to divide and conquer by breaking the project up into bite-sized chunks, and delivering interim capabilities frequently. Disciplined acceptance testing of each deployed function can reveal early if it is aligned with the intended outcome. Using this strategy, it is also possible to redirect the effort if changes in users’ needs, the industry, technology or market indicate change is called for.

Agile methods are more focused than waterfall on continual user or stakeholder input. One long “V” often meant requirements tossed to IT and forgotten until the beast emerges from the other end of the tunnel not resembling the intent of the original business requestors. Short two- to four-week sprints, however, are much less likely to diverge. This risk of divergence is further reduced with with significant participation from non-IT stakeholders in brief daily alignment meetings called scrums. In this approach, user requirements are articulated as “stories” that are placed in a product backlog and prioritized for immediate, early, subsequent or deferred development. The backlog feeds a pipeline that can be adapted as each new sprint is planned, re-prioritizing efforts to adjust to the evolving business, market and user context.

Context: For years, businesses have used Scaled Agile Frameworks (SAFe) to help them adequately manage large projects. When setting Agile frameworks, you need to monitor each bit of code that is uploaded. Doing this is easy when using a program that allows you to tail log files with ease.  Combining agile methodology with other proven techniques, such as test-driven development or TDD, and good design patterns such as data-driven and lean programming dramatically improves the probability of good outcomes. Consequently, agile significantly improves the probability of success of a solution development effort, and may even incrementally increase the number of capabilities that can be delivered to an organization.

Another reliable way to improve automation project outcomes, whether agile or not, is to put enough emphasis on the role data plays in the success of the initiative. Information is fundamentally more agile than process because it is easier to recombine into different patterns to achieve business insights and transactions. Thus, concentrating on the key data elements and the processes and transformations needed to achieve the desired outcomes can improve the likelihood of project success. The importance of data focus applies to traditional databases, big data, streaming data and unstructured data. Agile strategies can increase project throughput: the rapidity with which new capabilities can be deployed and deliver business value. But, is incremental improvement enough?

DevOps methodology has an analysis strategy to identify potential answers to get beyond incremental progress. To achieve major improvement, find the most significant obstacle, and move the obstacle out of the way. The strategy is articulated in The Theory of Constraints. Consider a radical idea: What if the biggest constraint is that there is not enough IT to meet all business needs? If so, then why not minimize the role of IT in building new capabilities? In most companies, that is not possible today. This constraint is rapidly changing, and self-service is coming of age.

The self-service tools themselves are evolutions of familiar tools. But the positive impact can be revolutionary for organizations that can overcome the gravitational pull of tradition. The most advanced of these tools are beginning to integrate training into the actual development tool, guiding users step-by-step through the processes of  app development. While IT can typically only serve the privileged pack and the lucky lot that can get budget for their highest priority ideas, empowering citizen developers  with self-service tools, and governing the newly built apps using business and technology overseers can make businesses more lean and agile than scrums and sprints can offer.

There are two areas in which empowering users to automate their own capabilities are entering the modern enterprise: Drag-and-drop intelligent business process modeling (iBPM) and Self-Service Business Intelligence (SSBI).

SSBI

In a large organization, with many functional areas, each with many individuals focused on specific facets of the business function, the traditional processes of relying on a few data scientists, data modelers and BI app developers cannot possibly scale to satisfy the need for each knowledge worker to get better insights from data. When each report or cube or visualization could take many days or weeks to explain to IT people, design, build and refine, only the highest priority requests can be implemented by IT.

Most enterprise BI environments require IT delivering the specific reporting business users want only after long and painful IT project or resource request cycles. When there is no understanding of or access to business data except through IT, users among the “missing masses” may never receive the insights they need. probably too late anyway. So for one-off, quick data deliveries, traditional BI has been seen as pretty useless.

Emerging SSBI capabilities permit business users to discover needed data on their own, and use advanced tools to build, adapt and improve visualizations that deliver more meaningful insights on a broader range of topics without ever having to translate their requirements for IT people to implement. Gartner defines SSBI as “end users designing and deploying their own reports and analyses within an approved and supported architecture and tools portfolio” (Gartner Glossary). The folks at TechTarget further explain that Self-Service BI “enables business users to access and work with corporate data even though they do not have a background in statistical analysis, business intelligence (BI) or data mining” (TechTarget on SSBI). In order to do this, the business users need to have enough background in the business information systems to:

• Find the information they need
• Know how to combine it to derive credible insights
• Know how to avoid using KPIs incorrectly, to avoid conflicting reports

Southard Jones of BI firm, Birst says: “It’s not just putting a bunch of data in a big database, which is an outdated way to think about it. Instead, it’s about creating a data fabric in which all the routes between data are mapped at the application level” (Humphries Smackdown). Qlik, a leading BI tool vendor, provides the best underlying model for Self-Service in their Associative Model. “Qlik’s competitive advantage over other BI tools is that it manages associations in memory at the engine level and not at the application level. Every data point in every field of a table is associated with every other data point anywhere in the entire schema” (Hatzis 2017).

This all requires the platform to provide users with understandable representations of the parts of the Enterprise Information model that apply to their specific needs. Self-Service must provide context. TechTarget has several useful posts about Self-Service BI and have published a good comparison of the leading vendors in this space called: “Self-service BI software comparison: Tableau vs. PowerBI vs. Qlik Sense“. In this July 2017 guide, the growth of the market is described along with important trends. “And now the trend of smart data discovery — which involves using machine learning algorithms to detect meaningful insights in data, rather than forcing a person to find correlations — is playing a role in the ability of self-service platforms to deliver ROI” (SearchBusinessAnalytics).

Knowledge Value-Chain Instrumentation

What are the benefits of SSBI? “Allowing end users to make decisions based on their own queries and analyses frees up the organization’s business intelligence and information technology (IT) teams from creating the majority of reports and allows those teams to focus on other tasks that will help the organization reach its goals” (TechTarget on SSBI). Those other tasks may include creating the connectors to each of the business systems and data sources and managing the privileges to properly permit and restrict access to these highly valuable resources. As the number of potential sources expand to include social media and other external sources, this can become a full-time job.

In my next post I will decompose the anatomy of insights, and show how each step in the value chain can be automated.

iBPM and Self-Service Forms and Processes

Workload and workflow scripting systems combined with rules engines supporting forward and backward-chaining inference engines are rapidly entering the mainstream and displacing more time-intensive and laborious lower-level software development processes. I described this in a prior post about Rings of Power.

In many organizations, knowledge workers are able to document their processes using flowcharts. The example here is intentionally simplistic, at least to one who knows how to read the diagram. It could hide a massive amount of complexity in the implementation, yet a person who understands the intent of the process, for example, an app for adding newly acquired used cars to a dealership, may be able to understand and use it effectively. Underlying the “Prompt for Input” box may be a keyboard or barcode scanner input process. The “Process Information” box may automatically go to an external auto registry and download historical information about the car, add it to an inventory record, then post it to an “auto detailing” process or vendor who will prepare the vehicle for resale.

 

A fundamental assumption about enabling “citizen developers” to build or add capabilities using workflow tools is that the vendor and the company implementing the tool provide as many pre-configured drag-and-drop components as users may need to assemble processes that bring them value. Of course, the processes are almost always useless without data. In the process diagram above, almost all the steps require some exchange of information. The display, prompt and input capabilities may be implemented using a simple forms tool. Some of these tools are approaching the level of simplicity needed to empower citizen developers (eg. Capterra and Zapier). Even in the “Open the App” box, there could be an exchange of user information that tells the app enough about the user for the App to display the correct screen(s).

Besides logging in and entering data, these process tools can include notifications to other users and groups, process handoffs and escalations, and delivering search results. Tools such as Pega, which began it’s life as an interactive phone answering system, and Appian as well as a bunch of Open source tools such as Alfresco Activiti, Bonita and jBPM, have taken process modeling to a very high level. And while most of these are tools for IT professionals, they could be simplified and implemented to support the citizen developer.

The current standard for defining processes using a drag-and-drop method and recognizable symbols is called Business Process Modeling Notation 2.0 (BMPN 2.0). It is associated with another standard called Business Process Execution Language (BPEL) which powers engines that can read the notation and make it run on a computer. this combination of a human-readable process design technique and a powerful execution framework hides a massive amount of complexity, thereby protecting users from having to know how the bits and bytes flow between the user interface, memory, the coded services and the data stores. I am not convinced that vanilla BPMN2.0 is simple enough for most users, but it is a step in the right direction.

Low Code and No Code

Mendix, AppCloud/Lightening/Heroku from SFDC, Outsystems and other APaaS (Application Platform as a Service) vendors approach the bypassing of traditional code-intensive IT development differently, but with similar results. These platforms provide a layer of simple screens and guidance to enable people to build processes in a fraction of the time it would normally take them to design, code, test and deploy applications using Java, C#, Python, R or other programming languages. Notwithstanding their ease of use, most of these tools remain within the walls of the Ivory Tower. This is exacerbated by pricing that provides low-cost licenses or subscriptions for end-users, but charge hefty fees for the ability to build stuff, though there is some hopeful progress toward the citizenry.

Introducing Low Code tools that enable users to develop their own data entry forms and empowering them to change the way data is processed and moved about is evolving the role of IT and the accessibility of data in the enterprise. We are witnessing a democratization of information. The combination of agile or DevOps methods and highly customizable low-code platforms can be powerful transformation vehicles because they further shorten the time-to-value of automation initiatives. The next obvious step is to incorporate artificial intelligence (AI), machine learning (ML) and natural language understanding (NLU) into these tools so more of the complexity of automation is hidden from users, and they can use their special business knowledge to innovate without tripping over IT.

This may be excruciatingly obvious, as with most of the things I post in this blog, but users – citizen developers – will not be able to do self-service at all unless it is extremely easy (Burns 2017). The responsibility of making Self-Service BI, Processes and Forms usable falls to the tool vendors and the organizations that implement the tools. The vendors must innovate to hide the complexity from users and integrate the tutorials or training into the construction processes. The organizations that adopt and implement these tools need to redefine roles in business and IT to empower users to innovate their own solutions, and establish business and IT governance to ensure alignment and consistency. The validation processes shown in the Agile cycle illustration earlier in this post, and other established governance are needed to keep the information free from malicious and accidental inconsistency.

The benefits of moving in this direction to empower more users with more capabilities well-tailored to their specific needs are likely to make the businesses that do it much more competitive.

Click below to look in each Understanding Context section

Leave This Blank:Leave This Blank Too:Do Not Change This:

Your email: