Today, to stay competitive, every company must behave like a software company and delivering new solutions with high speed, quality and reliability. New features need to be introduced at a rapid pace, but with high quality expectations. At the same time, deployment releases and operations are becoming more complex, debugging and traceability in production are becoming increasingly difficult, security is more prominent, uptime and reliability needs are skyrocketing… the list goes on.
In short, traditional software engineering is changing dramatically. Some of the trends driving this disruption include:
- Cloud eat software. And that has not only changed the infrastructure where software is hosted, but also how applications are designed, developed, packaged and deployed. Cloud drives adoption of cloud-native technologies; serverless components and containerization have changed the way we build applications. The hard line between the development and infrastructure teams is blurring.
- Cybersecurity is now a concern. Rising cyber attacks and malware threats have left everyone vulnerable. Traditionally, software was released every few months, allowing plenty of time for security testing by specialists. Today, with the advent of very fast release cycles, security checks have to start early in the development process.
- AI/ML is the new electricity but creates shocks. AI/ML powered use cases are finding their way across platforms and across industries. Traditional software engineers are unfamiliar with AI/ML and ML developers are unfamiliar with technical disciplines and best practices, leaving a large gap in technical maturity.
- When one company raises the bar, others are expected to follow. Consumer expectations are constantly changing. They expect the same experience from Amazon, Uber and Google from every application. You are expected to provide a seamless, frictionless experience that your platform is available anytime, on any device, and that it is fast and secure.
- APIs drive innovation: Innovation at speed and scale isn’t easy, but APIs can enable just that by leveraging the collective power of the crowd. Today, any developer can use platform-provided APIs to build a new, innovative solution. An API-first mindset is emerging and the app store model (e.g. Apple App Store, Salesforce AppExchange, AWS Marketplace) has changed the way we think about innovation and creating new platform extensions.
- Low code/no code platforms evolve quickly, making it easy to develop small, situational applications without writing code. Citizen developers are on the rise.
- Modern architecture paradigms stimulate new ways of working. Modern platform architecture has evolved significantly in recent years from monolithic applications to SOA to microservices and API-based distributed platforms. Event-driven architectures have gained prominence. Streaming architecture has become increasingly popular to support massive scalability. This has a significant impact on the way engineering teams are organized and how they build, test, deploy and manage new platforms.
- Legacy platforms slow growth. Legacy applications built more than 20 years ago need to be modernized for today’s context. Modernization is a huge exercise depending on the age, technology and size of the application. But modernization is easier said than done. The legacy and the new worlds have to coexist for a while and need a bridging strategy, both from a technological and cultural perspective.
To meet these changing demands, engineering needs to become a lot more intelligent.
The advent of intelligent technology
We can bring intelligence to engineering by using more data to manage projects, using AI/ML analytics to improve engineering productivity and predictability, leveraging more automation, and applying the right process to the right problem.
Data-driven decision making in engineering
Engineering teams use numerous tools in the development environment for code repository, bug tracking, code scanning, build, test, deploy, etc. Popular tools include Azure DevOps, GitHub, Atlassian, and Jenkins. These tools generate a lot of detailed data. All these data silos can be aggregated and used to track interesting statistics around:
- Developer productivity: How much time is spent writing how much code versus other activities; story points delivered vs committed
- Software Quality: Number of bugs generated; speed at which bugs are resolved; number of bugs slipping into production
In addition, on top of this data, one can build an intelligence layer using machine learning to pull hidden insights, highlight co-relationships, and perform better root cause analysis. These insights can make the technical results more predictable. For example, whether or not developers can meet a release schedule, quality level, etc.
Data provides evidence of how your teams are performing. It helps remove potential roadblocks and enables better communication with all stakeholders. It can be used to better manage project performance and use it for objective decision making. Despite the availability of this data, tech leaders are still groping in the dark. Decisions are made based on intuition and gut feeling rather than hard data. It’s time to become more intelligent by using data to track project performance and make decisions.
Increase the productivity and predictability of technical services with AI
AI/ML is disrupting every industry and is now making its way into software engineering. Developer tools with AI/ML functionality are emerging that can be used throughout the software engineering lifecycle. These tools can not only save time for repetitive activities, but also improve software quality on large and complex tasks.
Tools with AI/ML functionality can improve a developer’s everyday life, from coding to testing. Today, AI-powered tools can scan and analyze code to provide intelligent code completion suggestions, flag any deviation from coding best practices (naming convention compliance, misuse of variables, etc.), perform peer review, code converting from one language to another, security vulnerabilities, etc. Dynamic Application Security Testing (DAST) solutions use AI to discover potential attack vectors in milliseconds, which would otherwise take a few days. AI/ML can help automatically generate HTML code from hand-drawn UI sketches. AI/ML can also help developers generate automatic unit tests of existing code and suggestions for improving tests.
In fact, testing is the largest category that can benefit from AI/ML-based intelligence, especially in large complex applications. QA teams that rely on manual testing cannot keep up with the speed at which code changes and releases are made today. AI/ML-based tools can help test teams in a number of specific ways.
- Change impact analysis: AI/ML can help analyze coverage and identify which tests to run based on what has changed in the application.
- Test Creation: AI/ML can help test teams create test cases based on clear English descriptions and can also learn how to improve tests over time; AI/M can also automatically heal broken tests. AI-powered tools can automatically convert manual UI tests into API tests.
- Visual testing: AI-powered image comparison technology can enable visual testing to analyze the UI screen differences detected during testing.
- Test analysis: AI can analyze test cases and defect statistics to increase test coverage and reduce the number of tests.
It’s time to get more intelligent by using AI-powered tools to grow tech teams, improve productivity, and bring predictability.
Intelligent automation: beyond development processes
Today, there are many automation options in the software development lifecycle. DevOps enables automation of building, testing, and deploying new versions of your software. You can also automate other activities such as test automation, code scanning, performance testing, etc. DevSecOps enables automation of security processes (eg code scanning for vulnerabilities) in the engineering lifecycle. AI/ML-powered tools can drive further automation improvements (for example, improve flow in your CI/CD pipelines). In general, automation enables higher team productivity and results in the fast or on-demand release cycles needed for business agility.
Automation is not only limited to development activities, it also applies to other parts of the engineering lifecycle, for example:
- Infrastructure-as-code (IaC) tools can automatically provision and configure cloud infrastructure environments (storage, network, etc.).
- Monitoring tools can automate the monitoring of availability, production load statistics and security vulnerabilities and generate alerts.
- Log management: The number of logs generated in the current ecosystem is huge and it is impractical to collect them manually. Log management tools can automatically collect and analyze information from logs.
By aggressively leveraging automation and using it throughout its engineering lifecycle, software engineering can become much more intelligent. You should automate low-value repetitive tasks and processes and use the time saved for higher-value activities.
Making Agile smarter
Several popular software engineering methodologies are available, such as Agile, Scaled Agile, the Spotify model, and more. Regardless of the methodology or process you use, intelligently applying the right methods and tools to the right problems can make them more effective, efficient and intelligent. For example, traditional Agile processes were originally designed in the production space and then adapted to develop mature software. But in today’s fast-paced market, development teams must quickly and continuously try out new ideas to demonstrate new business features or for technical validation of emerging technology. This is true not only in the early stages of new product development, but also on existing platforms where you need to keep up with changing market dynamics. Be ready to experiment.
We need to get smarter about selecting the right process and tailoring it to your needs rather than letting the process determine your outcome.
If software eats the world, complexity is eating software
While product engineering has been around for a long time, modern software engineering is a lot more complex than just writing code and following Agile processes. Intelligent technology practices bring solutions to market faster with better quality and reliability and gain more time for innovation to keep pace with today’s speed of business.